Adding custom metadata to Spark's dataframe - DataFriday 1x16
Sometimes, you want to keep track of metadata as you perform some transformations with your dataframe. Imagine adding the order of the field to your dataframe for example, or the ingestion date, or even more.
In the first lab, I will detail access to metadata and then add some metadata to the columns of the dataframe. I will first add the order, the source file, and the ingestion format. Then I will add more data, such as the processing date. Finally I will add a user metadata using a helper function.
In the second lab, I will join two dataframes and look at what happens with the extra metadata.
These labs are not linked from Spark in Action, 2nd edition (http://jgp.ai/sia). It's still a highly recommended read.
Repository: https://github.com/jgperrin/net.jgp.l...
Lab: #090, Sublab: #000
GitHub link: https://github.com/jgperrin/net.jgp.l...
Lab: #090, Sublab: #100
GitHub link: https://github.com/jgperrin/net.jgp.l...
This event is supported by La French Tech Raleigh-Research Triangle ( / frenchtechraleigh .
#DataFriday