Airflow BashOperator Pass Arguments between Python Scripts

Question

In order to run Python scripts inside virtualenv when scheduled using Apache Airflow, the BashOperator is used.

For example,

task_a = BashOperator(
    task_id='task_a',
    bash_command='/path/to/virtualenv/bin/python /path/to/script_a.py'),
    dag=dag)

task_b = BashOperator(
    task_id='task_b',
    bash_command='/path/to/virtualenv/bin/python /path/to/script_b.py'),
    dag=dag)

In this case, how can we pass data between task_a and task_b, like how you could when using Xcom when the PythonOperator is used?

kaxil · Accepted Answer · 2020-05-09 21:08:55Z

2

Pass {{ ti.xcom_pull(task_ids='task_a') }} as an argument to your script in task_b. Remember only last line of the output of BashOperator is pushed to Xcom.

task_a = BashOperator(
    task_id='task_a',
    bash_command='/path/to/virtualenv/bin/python /path/to/script_a.py'),
    dag=dag)

task_b = BashOperator(
    task_id='task_b',
    bash_command="/path/to/virtualenv/bin/python /path/to/script_b.py {{ ti.xcom_pull(task_ids='task_a') }}"),
    dag=dag)

answered May 9, 2020 at 21:08

kaxil

19.1k3 gold badges67 silver badges87 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Airflow BashOperator Pass Arguments between Python Scripts

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related