Real Time Traffic Monitoring Pipeline with Apache NiFi, MySQL & Grafana
🎥 Video Description
This video demonstrates a Smart City Traffic Monitoring System implemented using Apache NiFi, MySQL, Python, and Grafana. The system simulates real-time traffic data, processes it through a data flow pipeline, detects congestion conditions, and visualizes results using an auto-refresh dashboard.
The solution is built using a data flow architecture, where Apache NiFi handles ingestion, validation, transformation, routing, and database integration.
🔧 What this video covers:
Data Generation
A Python script generates random traffic data in JSON format.
Data includes sensor ID, location, vehicle count, average speed, and road status.
Apache NiFi Data Flow
GetFile reads incoming JSON files.
EvaluateJsonPath extracts required attributes.
RouteOnAttribute validates data and detects congestion.
ReplaceText converts data into SQL queries.
PutSQL inserts processed data into MySQL.
PutFile handles invalid records.
Database Storage (MySQL)
Valid traffic data is stored in the traffic_data table.
Congestion alerts are stored in the traffic_alerts table.
Real-Time Visualization (Grafana)
Dashboard displays:
Average speed by location
Vehicle count trends
Traffic heatmap-style visualization
Congestion alerts
Auto-refresh enables near real-time monitoring.
🚦 Key Features
Continuous data ingestion using Apache NiFi
Rule-based congestion detection
Structured data storage in MySQL
Real-time dashboard with Grafana
Fault handling for invalid data
Modular and scalable data flow design
🎯 Outcome
This system demonstrates how data flow architecture can be applied to a real-world smart city problem. It enables automated traffic monitoring, efficient data processing, and real-time decision support through visualization.