LECTURE -60 - DATA ANALYTICS -DATA STRUCTURE OF FILES, , - BCA- S5.
Data Structure of Files
There are a couple of high-level containers that elaborate the specialized data structure in Hadoop to hold special types of data. For example, to maintain a binary log, the SequenceFile container provides the data structure to persist binary key-value pairs. We then can use the key, such as a timestamp represented by LongWritable and value by Writable, which refers to logged quantity.
There is another container, a sorted derivation of SequenceFile, called MapFile. It provides an index for convenient lookups by key.
These two containers are interoperable and can be converted to and from each other.
avro file based data structure in Hadoop
Apache Avro is a language-neutral data serialization system. It was developed by Doug Cutting, the father of Hadoop. Since Hadoop writable classes lack language portability, Avro becomes quite helpful, as it deals with data formats that can be processed by multiple languages. Avro is a preferred tool to serialize data in Hadoop.
Avro has a schema-based system. A language-independent schema is associated with its read and writes operations. Avro serializes the data which has a built-in schema. Avro serializes the data into a compact binary format, which can be deserialized by any application.
Avro uses JSON format to declare the data structures. Presently, it supports languages such as Java, C, C++, C#, Python, and Ruby.
Avro provides:
• Rich data structures.
• A compact, fast, binary data format.
• A container file, to store persistent data.
• Remote procedure calls (RPC).
• Simple integration with dynamic languages.
Creating Avro Schemas
An Avro schema is created using JSON format. JSON is short for JavaScript Object Notation, and it is a lightweight, text-based data interchange format that is intended to be easy for humans to read and write.
To describe an Avro schema, create a JSON record which identifies the schema, like this:
{
"type": "record",
"namespace": "com.example",
"name": "FullName",
"fields": [
{ "name": "first", "type": "string" },
{ "name": "last", "type": "string" }
]
}
The above example is a JSON record which describes schema that might be used by the value portion of a key-value pair in the store. It describes a schema for a person's full name.
Notice that for the record, there are four fields:
• type
Identifies the JSON field type. For Avro schemas, this must always be record when it is specified at the schema's top level. The type record means that there will be multiple fields defined.
• namespace
This identifies the namespace in which the object lives. Essentially, this is meant to be a URI that has meaning to you and your organization. • name
This is the schema name which, when combined with the namespace, uniquely identifies the schema within the store. In the above example, the fully qualified name for the schema is com.example.FullName.
• fields
This is the actual schema definition. It defines what fields are contained in the value, and the data type for each field. A field can be a simple data type, such as an integer or a string, or it can be complex data.