Python script execution using apache airflow

Question

I am new to airflow and I need to execute a python script using PythonOperator.

I am aware of defining a function in the DAG and then call it using python_callable. However, I have my code in a .py file and I want to run the whole script as it would be done from command line. The .py file currently accepts command line arguments, but I will modify it to get the values from the DAG op_kwargs.

If the only way to use the .py file is BashOperator, then do I need to change the way the script currently accepts command line arguments, as I guess the passed arguments will be treated as command line args?

Can you kindly help me with the DAG code for this?

irvifa · Accepted Answer · 2020-04-20 18:42:43Z

1

Let's say you save your code in a file named filename. Just write from filename import function, and then call the function using function(command, **context) from your DAG file. So in your DAG you'll have:

task = PythonOperator(
                task_id=task_id,
                python_callable=function,
                op_kwargs={'command': command},
                provide_context=True,
                dag=dag,
        )

edited Apr 20, 2020 at 18:42

answered Apr 20, 2020 at 18:35

irvifa

2,1132 gold badges19 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Santosh Satvik L Over a year ago

He wants to run the entire file as a script. Not only a particular function from that file. Can this be done with python operator? This can obviously be achieved with bash operator, but we lose out some convenience of the python operator, for example the requirements parameter.

irvifa Over a year ago

We can still create a wrapper function for all of the components located on the py file and then run it as a callable python function iirc

Santosh Satvik L Over a year ago

Its a script which is not segregated into functions. There isn't any function to call. I would have to modify that py file to keep all the code into a function like main, and then use it. Anyways, seems like I have to make do with bash operator.

Collectives™ on Stack Overflow

Python script execution using apache airflow

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related