28 - Create Spark RDD using External Datasets - Unit Test Code Demo 2
@backstreetbrogrammer
--------------------------------------------------------------------------------
Chapter 05 - Create Spark RDD using External Datasets - Unit Test Code Demo 2
--------------------------------------------------------------------------------
Spark can create distributed datasets from any storage source supported by Hadoop like:
Local file system
HDFS
MongoDB, Cassandra, HBase
Amazon S3, etc.
Spark supports:
Text files
SequenceFiles
Any Hadoop InputFormat
Text file RDDs can be created using SparkContext’s textFile() method. This method takes a URI for the file (either a local path on the machine, or a hdfs://, s3a://, etc. URI) and reads it as a collection of lines.
Example:
JavaRDD dataFile = sc.textFile("data.txt");
Once created, dataFile can be acted on by dataset operations like map or reduce.
Few important points to read files in Spark:
If using a path on the local filesystem, the file must also be accessible at the same path on worker nodes. Either copy the file to all workers or use a network-mounted shared file system.
All of Spark’s file-based input methods, including textFile, support running on directories, compressed files, and wildcards as well. For example, we can use:
sc.textFile("/my/directory")
sc.textFile("/my/directory/*.txt")
sc.textFile("/my/directory/*.gz")
The textFile() method also takes an optional second argument for controlling the number of partitions of the file. By default, Spark creates one partition for each block of the file (blocks being 128MB by default in HDFS) , but we can also ask for a higher number of partitions by passing a larger value. Note that we cannot have fewer partitions than blocks.
Github: https://github.com/backstreetbrogramm...
Apache Spark for Java Developers Playlist: • Apache Spark for Java Developers
Top Java Coding Interview Problems Playlist: • Top Java Coding Interview Problems
Java Serialization Playlist: • Java Serialization
Dynamic Programming Playlist: • Dynamic Programming
#java #javadevelopers #javaprogramming #apachespark #spark