Executors

Every SkaleData Airflow instance runs one of three executor modes, chosen on the app’s Execution tab: Celery, Kubernetes, or Hybrid. The executor decides where task processes run, and it’s the biggest lever you have over task-start latency, isolation, and idle cost.

Celery (default)

Persistent worker pods pull tasks from queues. Workers stay warm between tasks, so a task on a healthy queue starts in ~1–2 seconds. A KEDA autoscaler watches each queue’s backlog and scales workers between your Min/Max range — including down to zero for named queues.

• Best for: most workloads — short tasks, high task volume, fan-out DAGs, anything latency-sensitive.
• Resource model: tasks share their worker’s CPU/memory — sizing and node placement live on the Worker Queues tab. A task that needs more than its worker has will OOM the worker — route it to a bigger queue, or opt it into a Kubernetes pod (see Hybrid).
• Placement: workers follow their queue’s node pool; spot pools get the spot toleration automatically.

Kubernetes

No persistent workers — the scheduler launches one pod per task and deletes it when the task finishes. Pods for failed tasks are kept so you can kubectl describe them while debugging; delete them when you’re done.

Expect ~15–20 seconds of queue time per task: pod creation, image pull (cached after the first run on each node), and the DAG-sync init container. That’s the price of the benefits:

• Per-task isolation — every task gets its own CPU/memory (request = limit, Guaranteed QoS). Defaults come from the Kubernetes Task Defaults section on the Execution tab (500m / 1Gi / 1Gi ephemeral unless you change them); any task can override with executor_config={"pod_override": ...}.
• Scale to zero — nothing runs between tasks. Good for spiky, infrequent workloads.
• No noisy neighbors — a memory-hungry task can’t take out unrelated tasks, because there’s no shared worker to kill.

Redis (the Celery broker) isn’t deployed in Kubernetes-only mode.

Hybrid

Both executors at once: Celery handles every task by default, and DAG authors opt individual tasks into a Kubernetes pod:

heavy = PythonOperator(
    task_id="build_features",
    python_callable=build_features,
    executor="KubernetesExecutor",
    executor_config={  # optional — size/place the pod
        "pod_override": ...
    },
)

(Set executor in a DAG’s default_args to opt in a whole DAG.)

This is the recommended mode once you have even one outlier task: the fleet keeps Celery’s fast starts, and the 40 GiB-of-RAM monthly backfill gets its own right-sized pod without forcing you to run 40 GiB workers all month.

Worker-queue autoscalers automatically ignore Kubernetes-executed tasks in hybrid mode, so opted-in tasks never cause phantom Celery scale-ups.

Choosing

Switching modes is a config change: pick the executor on the Execution tab and Save & Apply. The apply rolls the affected components (see the capacity note below); queued and running tasks on the old executor drain normally.

Node pool capacity planning

Two sizing behaviors are worth planning for — both bit real rollouts:

Worker rollouts transiently need ~2× their steady-state CPU. Every Airflow pod runs with request = limit (Guaranteed QoS — it’s what makes the Analytics tab’s %-of-limit views and eviction protection work), so during a rolling update the old and new worker pods both hold their full reservation until the old ones finish draining. If a worker pool runs near its node-pool max nodes, a config change that rolls workers (executor switch, resize, image change) will leave new pods Pending until the old generation’s termination grace (10 minutes) frees the CPU. It self-resolves, but budget headroom if you want fast rollouts: keep the pool’s max at least one node above steady-state usage, or point big worker queues at their own pool.

Scale-to-zero queues and Kubernetes task pods rely on the cluster autoscaler. The first task after an idle period may also wait for a node to boot (~1–2 minutes on most clouds) before the pod can start. If that matters for a queue, set its Min Workers to 1 instead of 0.

Spot pools: Celery queue workers get the spot toleration automatically; Kubernetes-executor pods only land on spot if your pod_override adds both the node selector and the toleration — see the example on the Worker Queues page.