Airflow execute python script through bash operator

Question

I have a python script test2.py to connect to a remote server and execute the command. as below. This works on the command line.

Passing parameters as JSON and getting the response in JSON this works when executed as below in the command line.

python3.6 test2.py  '{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}'

I am trying to execute same through the airflow

from __future__ import print_function
from airflow.operators import BashOperator
from airflow.models import DAG
from datetime import datetime, timedelta


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2018, 9, 1),
    'email_on_failure': False,
    'email_on_retry': False,
    'schedule_interval': '@daily',
    'retries': 1,
    'retry_delay': timedelta(seconds=5),
}

dag = DAG(
    dag_id='DAG-3',
    default_args=default_args,
    dagrun_timeout=timedelta(minutes=10)
    )

cmd_command = "python3.6 /root/test2.py '{{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}}'"


t = BashOperator(
     task_id = 'some_id',
     bash_command = cmd_command,
     dag = dag)

I am seeing below error related to syntax.?

cmd_command = "python3.6 /root/test2.py '{{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}}'"
                                                       ^
SyntaxError: invalid syntax

Can you please help

Thank you

From security standpoint, why is test2.py taking the database credentials as argument? Those variables should be accessed as environment variables inside the test2.py. — alltej
– alltej, Commented Jul 7, 2022 at 15:03

SergiyKolesnikov · Accepted Answer · 2020-07-17 13:06:59Z

1

You use double quotes for JSON, but Python interprets them as start or end of a string. One way to resolve this is to escape double quotes inside JSON:

cmd_command = "python3.6 /root/test2.py '{\"hostname\": \"<server>\", \"username\":\"<test>\", \"password\":\"<test1>\", \"command1\":\"hostname\"}'"

edited Jul 17, 2020 at 13:06

answered Jul 17, 2020 at 12:29

SergiyKolesnikov

7,8254 gold badges33 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Eric Aya · Accepted Answer · 2023-03-12 09:45:42Z

0

I agree with Sergiy, you've got repeated " in your python line.

  cmd_args= r'{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}'
  cmd_command = f"python3.6 /root/test2.py '{cmd_args}'"

where

r'string' r is Python's RAW string, to switch off processing of spec chars, like \
f"string {python_var} " - new python3 explicit formatting string

another approach is to pass arguments by env variables:

bash_task = BashOperator(
    task_id="bash_task",
    bash_command="$PYTHON /root/test2.py '$my_params'",
    env={"my_params": r'{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}', 
         "PYTHON": 'python3.6'},
)

edited Mar 12, 2023 at 9:45

Eric Aya

70.2k36 gold badges190 silver badges266 bronze badges

answered Mar 12, 2023 at 8:46

Fedor

1167 bronze badges

Collectives™ on Stack Overflow

Airflow execute python script through bash operator

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related