Can I pass variable values to Python script from PythonOperator in Airflow using op_args or op_kwargs?

Question

From airflow DAG can I pass variable values as arguments to the python script using op_args or op_kwargs. In my airflow, Dag is importing my script as from scripts import my_script my python operator looks like this

PythonOperator(
    task_id='xxxxxx',
    python_callable=my_script.main,
    op_args=[bucket_name, prefix, source_blob_name, dest_bucket_name],
    dag=dag,
    trigger_rule='all_success'
)

I declared my variables in Airflow. I could be able to call my values bucket_name=Variable.get('bucket_name') here I want pass the value of bucket_name to my variable in Python script is that possible to do so?

Elad Kalif · Accepted Answer · 2022-04-21 19:52:14Z

1

In PythonOperator op_args, op_kwargs, templates_dict are templated fields.

So you can do :

PythonOperator(
    ...,
    op_args=['{{ var.value.bucket_name }}'],
    python_callable=my_script.main
)

Then your Python callable would be:

def main(*op_args):
    bucket_name = op_args[0]

You can also use op_kwargs / templates_dict:

PythonOperator(
    ...,
    templates_dict={'bucket_name', '{{ var.value.bucket_name }}'},
    python_callable=my_script.main
)

Then your Python callable would be:

def main(bucket_name, **context):
    ...

But there is no need to do either of them. There is no reason to pass argument that you can access directly with in your Python callable.

You can just do:

from airflow.models.variable import Variable
def main(**context):
    bucket_name = Variable.get('bucket_name')

This is perfectly safe since main is called only when PythonOperator is executed.

edited Apr 21, 2022 at 19:52

answered Apr 21, 2022 at 19:36

Elad Kalif

16.4k3 gold badges29 silver badges63 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Kriz Over a year ago

ops_args make sense to use , wondering how can i create placeholder for variables in my python script i have variables declared as bucket_name = 'XXX', prefix = 'XXX', source_blob_name = 'XXX', dest_bucket_name = 'XXX' should i declared them something like bucket_name = %s , prefix = %s

Elad Kalif Over a year ago

That is a matter of preference... You can choose the op_kwargs option.

Vinicius · Accepted Answer · 2022-04-21 19:36:19Z

0

You can use params inside your PythonOperator like this:

PythonOperator(
...
params={"bucket_name": bucket_name, "key": value}
...
)

And retrieve it:

def main(**kwargs):
 bucket_name = kwargs.get("bucket_name")
...

answered Apr 21, 2022 at 19:36

Vinicius

163 bronze badges

2 Comments

Kriz Over a year ago

my variable declarations are not inside main(), am believing I need to create a placeholder for my variables first in python script , can i do something like this bucket_name = %s ?

Vinicius Over a year ago

You could format it with f strings

Collectives™ on Stack Overflow

Can I pass variable values to Python script from PythonOperator in Airflow using op_args or op_kwargs?

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related