pipeline을 따라 만들어보며 Airflow의 concept, object, usage를 습득하기. Table of Contents * Example Pipeline definition * It’s a DAG definition file * Importing Modules * Default Arguments * Instantiate a DAG * Schedule interval * Tasks * Templating with Jinja * Setting up Dependencies * Testing * Running the Script * command line Metadata Validation * Testing * Backfill Example Pipeline definition "

Airflow 설치하기 Table of Contents * Getting Airflow * Extra Packages Getting Airflow pip install airflow extra packages를 설치하고자 하는 경우 아래처럼 패키지명을 명시해준다. pip install "airflow[s3, postgres]" Extra Packages celery, jdbc, hive, mysql, s3 등등. 전체 목록은 아래 링크 참조. http://airflow.incubator.apache.org/installation.html

# airflow needs a home, ~/airflow is the default, # but you can lay foundation somewhere else if you prefer # (optional) export AIRFLOW_HOME=~/airflow # install from pypi using pip pip install airflow # initialize the database airflow initdb # start the web server, default port is 8080 airflow webserver -p 8080 * export AIRFLOW_