Executing python and bash commands using DockerOperator

Question

Syntax to pass multiple commands (consisting of bash and python ) to the command parameter in DockerOperator For example if I have mix of command like -> python test.py , mkdir -p /test to be executed inside the docker container

Detail your question. What do you seek exactly? What have you tried? — Léopold Houdin
– Léopold Houdin, Commented Sep 22, 2018 at 16:55

Pablo Lozano · Accepted Answer · 2023-01-17 16:15:03Z

2

We achieved to run multiple commands with DockerOperator and retrieving the logs by filling the command argument like "bash -c 'command1 && command2 && command3'".

answered Jan 17, 2023 at 16:15

Pablo Lozano

498 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

akki · Accepted Answer · 2019-06-11 04:30:06Z

0

Pass multiple commands to the command parameter separated by &&, like this:

command='(command1) && (command2) && (command3)'

So the full code looks like:

t_docker = DockerOperator(
  task_id='docker_command',
  image='bde2020/spark-master:latest',
  api_version='auto',
  auto_remove=True,
  command='(python a.py) && (echo "done") && (mkdir /mydirectory) && (echo "directory created"))',
  network_mode='bridge'
)

answered Jun 11, 2019 at 4:30

akki

2,3422 gold badges26 silver badges39 bronze badges

4 Comments

akki Over a year ago

There seems to be a bug because of which it does not take list input, I just reported it here - issues.apache.org/jira/browse/…

andresg3 Over a year ago

This does not work for me. Is there another way to execute multiple commands

andresg3 Over a year ago

I tried: command='(echo "done") && (spark-submit --master local[*] /home/jovyan/pyspark_test01.py)' But got error: ERROR - docker container failed: {'Error': None, 'StatusCode': 127}

akki Over a year ago

Do you get the same error with command='(echo "done") && (echo "again")?

Collectives™ on Stack Overflow

Executing python and bash commands using DockerOperator

2 Answers 2

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related