I have airflow running on a docker container. I'm working with airflow version 2.0.2.
I know that I can actually create pools via the UI. But I'm looking for a way to create the pools programmatically on docker build via a pools.json file. Any way to help ?
You can do this programmatically via the REST API or CLI. The REST API allows for adding pools over HTTP one by one. The CLI has a command for importing pools from a JSON file:
airflow pools import [-h] [-v] FILEPATH
For example:
{
"pool_1": {"slots": 5, "description": ""},
"pool_2": {"slots": 5, "description": "test"}
}
The format is: "[pool name]": {"slots": [nr of slots], "description": "[description]"}
airflow pools import pools.json
The command works idempotent, so you can run it as many times as you like. Changes to existing pools result in an update, new pools will be created. Removing a pool from pools.json doesn't remove the pool in Airflow -- you'll have to do that manually.
Since it works idempotent, you can build a pools.json file into your Airflow image and run airflow pools import at the start of your container process, so that pools are checked at the start of every Airflow container.
Alternative way to create & use a pool in your DAG using python code (Rather than ui / cli)
# pools.py
from airflow.models import Pool
# USAGE: DONT FORGET .pool: @task(pool=s3_pool.**pool**)
s3_pool= Pool.create_or_update_pool(
"s3_pool",
slots=1,
description="Limit to 1 run of S3 ",
)
and your dag file:
# my_dag.py
# A generic operator
MyOperator(..., pool=s3_pool.pool)
# or using taskflow API
@task(pool=s3_pool.pool)
def my_task_name():
...
