SkaleData Docs
Airflow

Airflow overview

Managed Apache Airflow on your cloud: executors, worker queues, DAG development and deploys, and the base image.

SkaleData runs managed Apache Airflow on your own cloud. Each Airflow instance is an application on one of your clusters, deployed and configured from the console. These pages cover operating it:

  • Executors: Hybrid or Kubernetes. Task-start latency, isolation, idle cost, and node-pool capacity planning.
  • Worker queues: run several Celery worker pools with different machine shapes, node placement, and autoscaling, and route tasks between them.
  • Develop and deploy DAGs: run Airflow locally with skale airflow, deploy DAGs or a full image, and wire it into CI.
  • Custom images: what the maintained SkaleData base image includes and how to build your own image on top of it.
  • Image releases: the publish log for the base image.

To call an instance's REST API from a script or a CI job, see App APIs and credentials.

Destroying a cluster

What a cluster destroy deletes from your cloud account, what it keeps, and how to delete the cluster's data too.

Executors

Hybrid or Kubernetes executor: where tasks run, task-start latency, isolation, idle cost, and node-pool capacity planning.