How to use xCom in airflow dag file using python operator?

Question

I'm working on this airflow dag file to do some test with XCOM, but not sure how to use it between python operators. Can someone please help how to write the logic to pass a message between the python operators using XCOM push and pull functions. Below is the dag file that i'm working on...

The question is how to pass a message from each task to another task

@dag_factory
def create_dag():
    with DAG(
        dag_id="DSStest",
        default_args=default_args,
        schedule_interval=timedelta(1),
    ) as dag:
        # Define operators here, for example:

        output_file = path_in_workspace("testout")

        rscript_file = path_in_workspace("rtest2.R ")

        bcmd1 = "downloading some file here..."

        t0 = PythonOperator(
            task_id="start",
            python_callable=my_func2,
            provide_context=True,
            op_args=[output_file, 0],
        )

        t1 = PythonOperator(
            task_id="job1",
            python_callable=my_func1,
            provide_context=True,
            op_args=[output_file, 1],
        )

        t2 = PythonOperator(
            task_id="job2",
            python_callable=my_func1,
            provide_context=True,
            op_args=[output_file, 2],
        )

        t10 = PythonOperator(
            task_id="job10",
            python_callable=my_func2,
            provide_context=True,
            op_args=[output_file, 10],
        )

        t20 = BashOperator(
            task_id="job20",
            bash_command=bcmd1,
            queue={
                "worker_type": "png-bash-worker",
                "request_memory": "1G",
                "request_cpu": 1,
            },
        )

        # Define dependencies between operators here, for example:

        t0 >> t1
        t0 >> t2
        t1 >> t10
        t2 >> t10
        t10 >> t11
        t11 >> t20

        return dag  # Do not change this

I think it's not relevant to this example, but is the @dag_factory decorator something you've created? I haven't seen that before. — Taylor D. Edmiston
– Taylor D. Edmiston, Commented Aug 16, 2018 at 18:55
Possible duplicate of Python Airflow - Return result from PythonOperator — tobi6
– tobi6, Commented Aug 17, 2018 at 6:47
Please also see this answer on this question: stackoverflow.com/questions/50149085/… — tobi6
– tobi6, Commented Aug 17, 2018 at 6:47

Christopher Beck · Accepted Answer · 2018-08-16 20:37:04Z

3

I recommend to take a look at this example, it shows/explains everything regarding xcoms and PythonOperators:
example_xcom.py

The official airflow page also explains xcoms in a lot of detail:
official documentation

edited Aug 16, 2018 at 20:37

answered Aug 16, 2018 at 20:25

Christopher Beck

7651 gold badge8 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Brendan Over a year ago

That example is great, but let's not pretend the official page explains it in much detail at all, at least at the time of this comment. If you don't already know how xcoms work or have an example to go off of, the existing documentation for this is next to useless, and lacking in necessary context.

Collectives™ on Stack Overflow

How to use xCom in airflow dag file using python operator?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related