bitnami/airflow repository overview

⁠Bitnami Secure Image for Apache Airflow

⁠What is Apache Airflow?

Overview of Apache Airflow⁠ Trademarks: This software listing is packaged by Bitnami. The respective trademarks mentioned in the offering are owned by the respective companies, and use of them does not imply any affiliation or endorsement.

⁠TL;DR

docker run --name airflow bitnami/airflow:latest

Warning: This quick setup is only intended for development environments. You are encouraged to change the insecure default credentials and check out the available configuration options in the Environment Variables⁠ section for a more secure d eployment.

⁠Why use Bitnami Secure Images?

Those are hardened, minimal CVE images built and maintained by Bitnami. Bitnami Secure Images are based on the cloud-optimized, security-hardened enterprise OS Photon Linux⁠. Why choose BSI images?

• Hardened secure images of popular open source software with Near-Zero Vulnerabilities
• Vulnerability Triage & Prioritization with VEX Statements, KEV and EPSS Scores
• Compliance focus with FIPS, STIG, and air-gap options, including secure bill of materials (SBOM)
• Software supply chain provenance attestation through in-toto
• First class support for the internet’s favorite Helm charts

Each image comes with valuable security metadata. You can view the metadata in our public catalog here⁠. Note: Some data is only available with commercial subscriptions to BSI⁠.

If you are looking for our previous generation of images based on Debian Linux, please see the Bitnami Legacy registry⁠.

⁠Supported tags and respective Dockerfile links

Learn more about the Bitnami tagging policy and the difference between rolling tags and immutable tags in our documentation page⁠.

You can see the equivalence between the different tags by taking a look at the tags-info.yaml file present in the branch folder, i.e bitnami/ASSET/BRANCH/DISTRO/tags-info.yaml.

Subscribe to project updates by watching the bitnami/containers GitHub repo⁠.

⁠Prerequisites

To run this application you need Docker Engine⁠ >= 1.10.0. Docker Compose⁠ is recommended with a version 1.6.0 or later.

⁠How to use this image

Airflow requires access to a PostgreSQL database to store information. We will use our very own PostgreSQL image⁠ for the database requirements. Additionally, if you pretend to use the CeleryExecutor, you will also need a Redis(R) server⁠.

⁠Using the Docker Command Line

1. Create a network

   docker network create airflow-tier
   

   Copy

2. Create a volume for PostgreSQL persistence and create a PostgreSQL container

   docker volume create --name postgresql_data
   docker run -d --name postgresql \
     -e POSTGRESQL_USERNAME=bn_airflow \
     -e POSTGRESQL_PASSWORD=bitnami1 \
     -e POSTGRESQL_DATABASE=bitnami_airflow \
     --net airflow-tier \
     --volume postgresql_data:/bitnami/postgresql \
     bitnami/postgresql:latest
   

   Copy

3. Create a volume for Redis(R) persistence and create a Redis(R) container

   docker volume create --name redis_data
   docker run -d --name redis \
     -e ALLOW_EMPTY_PASSWORD=yes \
     --net airflow-tier \
     --volume redis_data:/bitnami \
     bitnami/redis:latest
   

   Copy

4. Launch the Apache Airflow web container

   docker run -d --name airflow -p 8080:8080 \
     -e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
     -e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
     -e AIRFLOW_EXECUTOR=CeleryExecutor \
     -e AIRFLOW_DATABASE_NAME=bitnami_airflow \
     -e AIRFLOW_DATABASE_USERNAME=bn_airflow \
     -e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
     -e AIRFLOW_LOAD_EXAMPLES=yes \
     -e AIRFLOW_PASSWORD=bitnami123 \
     -e AIRFLOW_USERNAME=user \
     -e [email protected] \
     --net airflow-tier \
     bitnami/airflow:latest
   

   Copy

5. Launch the Apache Airflow scheduler container

   docker run -d --name airflow-scheduler \
     -e AIRFLOW_COMPONENT_TYPE=scheduler \
     -e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
     -e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
     -e AIRFLOW_EXECUTOR=CeleryExecutor \
     -e AIRFLOW_DATABASE_NAME=bitnami_airflow \
     -e AIRFLOW_DATABASE_USERNAME=bn_airflow \
     -e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
     -e AIRFLOW_LOAD_EXAMPLES=yes \
     -e AIRFLOW_WEBSERVER_HOST=airflow \
     --net airflow-tier \
     bitnami/airflow:latest
   

   Copy

6. Launch the Apache Airflow worker container

   docker run -d --name airflow-worker \
     -e AIRFLOW_COMPONENT_TYPE=worker \
     -e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
     -e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
     -e AIRFLOW_EXECUTOR=CeleryExecutor \
     -e AIRFLOW_DATABASE_NAME=bitnami_airflow \
     -e AIRFLOW_DATABASE_USERNAME=bn_airflow \
     -e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
     -e AIRFLOW_WEBSERVER_HOST=airflow \
     --net airflow-tier \
     bitnami/airflow:latest
   

   Copy

Access your application at http://your-ip:8080

⁠Using docker-compose.yaml

curl -LO https://raw.githubusercontent.com/bitnami/containers/main/bitnami/airflow/docker-compose.yml
docker-compose up

Please be aware this file has not undergone internal testing. Consequently, we advise its use exclusively for development or testing purposes. For production-ready deployments, we highly recommend utilizing its associated Bitnami Helm chart⁠.

If you detect any issue in the docker-compose.yaml file, feel free to report it or contribute with a fix by following our Contributing Guidelines⁠.

⁠Persisting your application

The Bitnami Airflow container relies on the PostgreSQL database & Redis to persist the data. This means that Airflow does not persist anything. To avoid loss of data, you should mount volumes for persistence of PostgreSQL data⁠ and Redis(R) data⁠

The above examples define docker volumes namely postgresql_data, and redis_data. The Airflow application state will persist as long as these volumes are not removed.

To avoid inadvertent removal of these volumes you can mount host directories as data volumes⁠. Alternatively you can make use of volume plugins to host the volume data.

⁠Mount host directories as data volumes with Docker Compose

The following docker-compose.yml template demonstrates the use of host directories as data volumes.

version: '2'
services:
  postgresql:
    image: bitnami/postgresql:latest
    environment:
      - POSTGRESQL_DATABASE=bitnami_airflow
      - POSTGRESQL_USERNAME=bn_airflow
      - POSTGRESQL_PASSWORD=bitnami1
    volumes:
      - /path/to/postgresql-persistence:/bitnami/postgresql
  redis:
    image: bitnami/redis:latest
    environment:
      - ALLOW_EMPTY_PASSWORD=yes
    volumes:
      - /path/to/redis-persistence:/bitnami
  airflow-worker:
    image: bitnami/airflow:latest
    environment:
      - AIRFLOW_COMPONENT_TYPE=worker
      - AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
      - AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
      - AIRFLOW_EXECUTOR=CeleryExecutor
      - AIRFLOW_DATABASE_NAME=bitnami_airflow
      - AIRFLOW_DATABASE_USERNAME=bn_airflow
      - AIRFLOW_DATABASE_PASSWORD=bitnami1
      - AIRFLOW_LOAD_EXAMPLES=yes
  airflow-scheduler:
    image: bitnami/airflow:latest
    environment:
      - AIRFLOW_COMPONENT_TYPE=scheduler
      - AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
      - AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
      - AIRFLOW_EXECUTOR=CeleryExecutor
      - AIRFLOW_DATABASE_NAME=bitnami_airflow
      - AIRFLOW_DATABASE_USERNAME=bn_airflow
      - AIRFLOW_DATABASE_PASSWORD=bitnami1
      - AIRFLOW_LOAD_EXAMPLES=yes
  airflow:
    image: bitnami/airflow:latest
    environment:
      - AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
      - AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
      - AIRFLOW_EXECUTOR=CeleryExecutor
      - AIRFLOW_DATABASE_NAME=bitnami_airflow
      - AIRFLOW_DATABASE_USERNAME=bn_airflow
      - AIRFLOW_DATABASE_PASSWORD=bitnami1
      - AIRFLOW_PASSWORD=bitnami123
      - AIRFLOW_USERNAME=user
      - [email protected]
    ports:
      - 8080:8080

⁠Mount host directories as data volumes using the Docker command line

1. Create a network (if it does not exist)

   docker network create airflow-tier
   

   Copy

2. Create the PostgreSQL container with host volumes

   docker run -d --name postgresql \
     -e POSTGRESQL_USERNAME=bn_airflow \
     -e POSTGRESQL_PASSWORD=bitnami1 \
     -e POSTGRESQL_DATABASE=bitnami_airflow \
     --net airflow-tier \
     --volume /path/to/postgresql-persistence:/bitnami \
     bitnami/postgresql:latest
   

   Copy

3. Create the Redis(R) container with host volumes

   docker run -d --name redis \
     -e ALLOW_EMPTY_PASSWORD=yes \
     --net airflow-tier \
     --volume /path/to/redis-persistence:/bitnami \
     bitnami/redis:latest
   

   Copy

4. Create the Airflow container

   docker run -d --name airflow -p 8080:8080 \
     -e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
     -e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
     -e AIRFLOW_EXECUTOR=CeleryExecutor \
     -e AIRFLOW_DATABASE_NAME=bitnami_airflow \
     -e AIRFLOW_DATABASE_USERNAME=bn_airflow \
     -e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
     -e AIRFLOW_LOAD_EXAMPLES=yes \
     -e AIRFLOW_PASSWORD=bitnami123 \
     -e AIRFLOW_USERNAME=user \
     -e [email protected] \
     --net airflow-tier \
     bitnami/airflow:latest
   

   Copy

5. Create the Airflow Scheduler container

   docker run -d --name airflow-scheduler \
     -e AIRFLOW_COMPONENT_TYPE=scheduler \
     -e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
     -e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
     -e AIRFLOW_EXECUTOR=CeleryExecutor \
     -e AIRFLOW_DATABASE_NAME=bitnami_airflow \
     -e AIRFLOW_DATABASE_USERNAME=bn_airflow \
     -e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
     -e AIRFLOW_LOAD_EXAMPLES=yes \
     -e AIRFLOW_WEBSERVER_HOST=airflow \
     --net airflow-tier \
     bitnami/airflow:latest
   

   Copy

6. Create the Airflow Worker container

   docker run -d --name airflow-worker \
     -e AIRFLOW_COMPONENT_TYPE=worker \
     -e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
     -e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
     -e AIRFLOW_EXECUTOR=CeleryExecutor \
     -e AIRFLOW_DATABASE_NAME=bitnami_airflow \
     -e AIRFLOW_DATABASE_USERNAME=bn_airflow \
     -e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
     -e AIRFLOW_WEBSERVER_HOST=airflow \
     --net airflow-tier \
     bitnami/airflow:latest
   

   Copy

⁠Configuration

⁠Load DAG files

Custom DAG files can be mounted to /opt/bitnami/airflow/dags.

⁠Installing additional python modules

This container supports the installation of additional python modules at start-up time. In order to do that, you can mount a requirements.txt file with your specific needs under the path /bitnami/python/requirements.txt.

⁠Environment variables

⁠Customizable environment variables

Note: the README for this container is longer than the DockerHub length limit of 25000, so it has been trimmed. The full README can be found at https://github.com/bitnami/containers/blob/main/bitnami/airflow/README.md⁠