REST AP's data load into ADLS GEN2
REST API (Representational State Transfer API):
Definition:
REST (Representational State Transfer) is an architectural style for designing networked applications.
A REST API is a set of rules and conventions for building and interacting with web services that adhere to the principles of REST.
Key Characteristics:
Stateless:
Each request from a client to a server must contain all the information needed to understand and fulfill the request. The server should not store any client state between requests.
Client-Server Architecture:
Separation of concerns between the client and server components allows them to evolve independently.
Uniform Interface:
A uniform and consistent interface simplifies interactions and promotes scalability. It includes:
Resource Identification: Resources are identified by URIs (Uniform Resource Identifiers).
Resource Manipulation through Representations: Resources can have multiple representations (e.g., JSON, XML), and clients interact with these representations.
Stateless Communication:
Each request from a client to a server must contain all the information needed to understand and fulfill the request. The server should not store any client state between requests.
Cacheability:
Responses from the server can be explicitly marked as cacheable or non-cacheable.
Layered System:
The architecture can be composed of multiple layers, where each layer has a specific functionality and interacts only with adjacent layers.
Components of a REST API:
Resources:
Entities or services that are identified by URIs. Resources can be anything from a user profile to a product in an online store.
HTTP Methods (Verbs):
Commonly used HTTP methods include GET (retrieve), POST (create), PUT (update), DELETE (delete), etc.
Representation:
Data format (e.g., JSON or XML) used to represent the state of a resource during communication between the client and server.
URI (Uniform Resource Identifier):
Unique identifier for each resource in the system.
Azure Data Factory is a cloud-based data integration service provided by Microsoft Azure. It allows you to create, schedule, and manage data pipelines that can move data between various supported data stores. Azure Data Factory enables you to build complex data workflows for data preparation, transformation, and analytics. Here are key features and components of Azure Data Factory:
Key Concepts:
Pipeline:
A pipeline is a logical grouping of activities that together perform a task. It represents a workflow in which data flows from one or more sources to one or more destinations.
Activities:
Activities are the processing steps within a pipeline. They can perform various operations such as copying data, transforming data using HDInsight Hadoop, running Hive or Pig scripts, and more.
Data Flow:
Data Flow is a cloud-native ETL (Extract, Transform, Load) service that allows you to design and debug data transformation logic without writing code.
Linked Services:
Linked services define the connection information needed for Data Factory to connect to external resources. Examples include Azure Storage, Azure SQL Database, on-premises SQL Server, etc.
Data Sets:
Data sets represent the structure of the data that is being used by activities. They define the schema and location of the data.
Triggers:
Triggers define when a pipeline or activity should be executed. Triggers can be scheduled, event-based, or manual.
Integration Runtimes:
Integration runtimes define the compute infrastructure used by Data Factory to move data and perform activities. Azure Integration Runtime is the default runtime, but you can also use Self-hosted Integration Runtime for on-premises data.
Monitoring and Management:
Azure Data Factory provides monitoring dashboards and logs to track the performance of your pipelines and activities. Azure Monitor and Azure Log Analytics can also be integrated for more detailed monitoring.
Workflow:
Authoring:
Use the Azure Data Factory user interface, Visual Studio, or Azure DevOps to create and author pipelines, activities, and data flows.
Debugging:
Debug your pipelines and data flows to ensure they are working as expected before deployment.
Publish and Deploy:
Publish your data factory resources to the Azure Data Factory service, and then deploy your data factory to the appropriate environment (development, testing, production).
Monitoring and Management:
Monitor and manage your data factory using the Azure portal, Azure Monitor, and other tools.
Use Cases:
Data Migration:
Move data from on-premises systems to the cloud or between different cloud data stores.
Data Transformation:
Transform and clean data using data flows for analytics and reporting.
Data Orchestration:
Schedule and orchestrate complex workflows involving multiple data sources and destinations.