Airflow PythonOperator inside PythonOperator

Question

How can I run a PythonOperator inside another one PythonOperator?

The idea is: to call "main" function as a PythonOperator and then run a few other PythonOperators inside and scheduler them

The code is:

def printFunction(value):
    time.sleep(5)
    print(value)

def main():
    for i in range(10):
        task_2 = PythonOperator(
                        task_id='loop_task_2'+str(i),
                        python_callable = printFunction,
                        op_kwargs = {'value':i},
                        dag=dag,
                )
        task_3 = PythonOperator(
                        task_id='loop_task_3'+str(i),
                        python_callable = printFunction,
                        op_kwargs = {'value':i},
                        dag=dag,
                )
        task_2>>task_3

task = PythonOperator(
                        task_id='tor_task',
                        python_callable = main,
                        op_kwargs = {},
                        dag=dag,
                )

Elad Kalif · Accepted Answer · 2021-05-11 09:07:45Z

3

I think you are expecting that Airflow will create these operators/tasks in the UI but that is not going to happen. With that code you will have only tor_task in the UI. It's not very clear what you are trying to do but it seems that it would be better to just call printFunction directly in the loop.

In any case if you must use operator inside operator you should call the execute() method. See this answer for more information

answered May 11, 2021 at 9:07

Elad Kalif

16.4k3 gold badges29 silver badges63 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jjvzd Over a year ago

Sorry that my description is not clear. I want my PythonOperator call other PythonOperators in loop and them should be paralleled. That's what I need

Collectives™ on Stack Overflow

Airflow PythonOperator inside PythonOperator

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related