How to Run Apache Airflow on MySQL Server | Integrate Apache Airflow with MySQL Server
The default database for airflow is SQLite Database which is a lightweight database and not recommended for running airflow in production environments. We already installed airflow on SQLite DB in last few videos. You can find it on our channel playlist section. We used windows sub system for linux to run apache airflow on windows without docker.
In this video we are going to Set up a Database Backend - Apache Airflow - MYSQL Server.
STEPS TO CONFIGURE APACHE AIRFLOW FOR MYSQL SERVER ON UBUNTU:
1) Update Package Manager:
sudo apt update
2) Install MySQL on Ubuntu:
sudo apt install mysql-server
3) Creating Database for Airflow:
CREATE DATABASE airflow_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'airflow_user' IDENTIFIED BY 'airflow_pass';
GRANT ALL PRIVILEGES ON airflow_db.* TO 'airflow_user';
4) Define Connection in airflow.cfg:
sql_alchemy_conn = mysql+mysqldb://airflow_user:[email protected]:3306/airflow_db
5) Installing MYSQL client on Ubuntu:
sudo apt install libmysqlclient-dev
6) Exporting Environment Variables:
export MYSQLCLIENT_CFLAGS="$(mysql_config --cflags)"
export MYSQLCLIENT_LDFLAGS="$(mysql_config --libs)"
7) Pip install MYSQL Client:
pip install mysqlclient
8) Creating an Airflow User:
airflow users create --username admin --password admin --role Admin --firstname admin --lastname admin --email [email protected]
9) Migrate Airflow:
airflow db migrate
Feel free to ask for any questions if you are stuck in any of the above steps.
Thank You.