Coding from scratch a realtime Azure data and AI pipeline - the AI side
In this vid (part 2 of 2), I build a realtime ingesting DataBricks Delta Lake that refines the data ingested into Azure, supplements it with predictions from Microsoft Cognitive Services, as part of a Pipeline Model, and publishes the data to a realtime dashboard.
The video demonstrates:
Importing existing notebooks into Azure Databricks
Using the Event Hub connector to ingest realtime data into Databricks from Event Hubs
Reading and writing streams from the Delta Lake format
Using the MML Spark library to create Cognitive Services transformers for the data to add language detection, sentiment analysis, and image categorisation
Prepare the data for ingestion into power bi and use the power bi streaming capability in MML Spark
Create a realtime Databricks dashboard
Follow the how-to at https://github.com/lockedata/realtime...