Executors

Every SkaleData Airflow instance runs one of two executor modes, chosen on the app's Execution tab: Hybrid (the default) or Kubernetes. The executor decides where task processes run — the biggest lever you have over task-start latency, isolation, and idle cost.

Hybrid (default)

The default, and the right choice for most workloads. Two executors run side by side: persistent Celery workers handle every task by default, and DAG authors opt individual tasks into their own Kubernetes pod.

Celery workers stay warm between tasks, so a task on a healthy queue starts in ~1–2 seconds. A KEDA autoscaler watches each queue's backlog and scales workers between your Min/Max range — including down to zero for named queues.

Resource model: by default, tasks share their worker's CPU/memory — sizing and node placement live on the Worker Queues tab. A task that needs more than its worker has will OOM the worker — route it to a bigger queue, or give it its own pod (below).
Placement: workers follow their queue's node pool; spot pools get the spot toleration automatically.

Opt a task into a Kubernetes pod when it's an outlier — too big, too isolated, or too rare to size a queue for:

heavy = PythonOperator(
    task_id="build_features",
    python_callable=build_features,
    executor="KubernetesExecutor",
    executor_config={  # optional — size/place the pod
        "pod_override": ...
    },
)

Set executor in a DAG's default_args to opt in a whole DAG. The fleet keeps Celery's fast starts, and the 40 GiB-of-RAM monthly backfill gets its own right-sized pod without forcing you to run 40 GiB workers all month. Worker-queue autoscalers automatically ignore these opted-in tasks, so they never cause phantom Celery scale-ups.

Kubernetes

No persistent workers — the scheduler launches one pod per task and deletes it when the task finishes. Pods for failed tasks are kept so you can kubectl describe them while debugging; delete them when you're done.

Expect ~15–20 seconds of queue time per task: pod creation, image pull (cached after the first run on each node), and the DAG-sync init container. That's the price of the benefits:

Per-task isolation — every task gets its own CPU/memory (request = limit, Guaranteed QoS). Defaults come from the Kubernetes Task Defaults section on the Execution tab (500m / 1Gi / 1Gi ephemeral unless you change them); any task can override with executor_config={"pod_override": ...}.
Scale to zero — nothing runs between tasks. Good for spiky, infrequent workloads.
No noisy neighbors — a memory-hungry task can't take out unrelated tasks, because there's no shared worker to kill.

Redis (the Celery broker) isn't deployed in Kubernetes-only mode.

Choosing

	Hybrid	Kubernetes
Task start latency	~1–2s by default (warm Celery); opt-in pods ~15–20s+	~15–20s per task
Isolation	Shared worker by default; opt-in pod per task	Pod per task
Idle cost	Celery minimum (0 for named queues)	Zero
Per-task sizing	Per queue, or per task via a pod	Per task
Good for	Most workloads — a fast fleet with heavy outliers	All-ephemeral, spiky, or untrusted tasks

Switching modes is a config change: pick the executor on the Execution tab and Save & Apply. The apply rolls the affected components (see the capacity note below); queued and running tasks on the old executor drain normally.

If you ran a Celery instance before this change, it now runs as Hybrid — same warm workers and queues for your existing tasks, plus the option to send one to a pod.

Right-sized task nodes (AWS)

On AWS, every Kubernetes-executor task pod gets its own node, provisioned on demand to match the pod's exact size. A 0.5-vCPU task lands on a small, cheap node; a memory-heavy task gets a memory-optimized one. Nothing is stranded, you don't manage a node pool for it, and when no tasks are running the task nodes scale to zero — so you pay only while work is in flight. (Under the hood this is Karpenter; you don't configure it.)

This covers both places Kubernetes task pods come from: the tasks you opt into pods in Hybrid mode, and every task under the Kubernetes executor.

Running large tasks

Because the node follows the task, you can request far more than any single fixed machine — up to roughly 94 vCPU / ~735 GiB per task (memory-heavy tasks land on a memory-optimized node). Set the size in Kubernetes Task Defaults on the Execution tab, or per task with a pod_override:

from kubernetes.client import models as k8s

big = PythonOperator(
    task_id="monthly_backfill",
    python_callable=run_backfill,
    executor="KubernetesExecutor",
    executor_config={
        "pod_override": k8s.V1Pod(
            spec=k8s.V1PodSpec(
                containers=[
                    k8s.V1Container(
                        name="base",
                        resources=k8s.V1ResourceRequirements(
                            requests={"cpu": "12", "memory": "48Gi"},
                            limits={"cpu": "12", "memory": "48Gi"},
                        ),
                    )
                ]
            )
        )
    },
)

A task larger than the ceiling stays Pending. The real limit is your AWS account's On-Demand vCPU quota — the total vCPUs across all your running nodes. If a large task won't schedule, raise it in the AWS console under Service Quotas → EC2 → Running On-Demand Standard instances.

What to expect

Warm start (~5–15s). When a task node is already up — a task ran recently, or a fan-out is packing several small tasks onto one node — the pod starts in seconds off the node's cached image.
Cold start (~40s–1 min). The first task after an idle stretch waits once for a fresh node to boot and pull the image. Everything after is warm until the nodes scale back to zero.
On-demand only. Task nodes never run on spot, so a task is never interrupted by a spot reclaim mid-run.

Hybrid keeps this economical: your routine tasks stay on warm Celery workers, and only the outliers you opt into pods pay the node lifecycle.

Right-sized task nodes are available on AWS clusters running a recent platform version. If your cluster predates it, upgrade it in place from the cluster's actions menu — no data migration, just a re-apply. Azure and GCP clusters have the same feature — see Right-sized task nodes (Azure) and Right-sized task nodes (GCP). On GCP it's automatic on every cluster.

Right-sized task nodes (Azure)

On Azure (AKS), every Kubernetes-executor task pod gets its own node, provisioned on demand to match the pod's exact size — the same right-sizing as on AWS. A 0.5-vCPU task lands on a small, cheap node; a memory-heavy task gets a larger one. Nothing is stranded, you don't manage a node pool for it, and the task nodes scale to zero when no tasks are running — so you pay only while work is in flight. (Under the hood this is AKS Node Auto-Provisioning, Azure's managed Karpenter; you don't configure it.)

As on AWS, this covers both places Kubernetes task pods come from: the tasks you opt into pods in Hybrid mode, and every task under the Kubernetes executor.

Running large tasks

Set the size in Kubernetes Task Defaults on the Execution tab, or per task with a pod_override — the example above works unchanged on Azure. Task nodes come from the general-purpose D-series, right-sized to each task up to a per-node ceiling of 64 vCPU by default (ask us if you need it raised).

A task larger than what's available stays Pending. The real limit is your Azure subscription's vCPU quota for the region and VM family. Raise it in the Azure portal under Subscriptions → Usage + quotas (or with az quota update) — bump both the Standard Dv-family counter and Total Regional vCPUs, since either can be the ceiling.

What to expect

Warm start. When a task node is already up — a task ran recently, or a fan-out is packing several small tasks onto one node — the pod starts in seconds off the node's cached image.
Cold start (~1 min). The first task after an idle stretch waits once for a fresh VM to boot and join the cluster. Everything after is warm until the nodes scale back to zero.
On-demand only. Task nodes never run on spot, so a task is never interrupted mid-run.

Hybrid keeps this economical the same way it does on AWS: your routine tasks stay on warm Celery workers, and only the outliers you opt into pods pay the node lifecycle.

On Azure, right-sized task nodes are a create-time cluster setting (they need Node Auto-Provisioning networking), so — unlike AWS — they can't be turned on with an in-place upgrade. They're available on new Azure clusters provisioned on a recent platform version; existing Azure clusters keep scheduling Kubernetes task pods onto their node pools. Reach out if you'd like a workload moved onto a NAP-enabled cluster.

Right-sized task nodes (GCP)

On GCP (GKE), every Kubernetes-executor task pod gets its own node, provisioned on demand to match the pod's exact size — the same right-sizing as on AWS and Azure. A 0.5-vCPU task lands on a small, cheap node; a memory-heavy task gets a larger one. Nothing is stranded, you don't manage a node pool for it, and the task nodes scale to zero when no tasks are running — so you pay only while work is in flight. (Under the hood this is GKE Node Auto-Provisioning, the managed GKE analog of Karpenter; you don't configure it.)

As on AWS and Azure, this covers both places Kubernetes task pods come from: the tasks you opt into pods in Hybrid mode, and every task under the Kubernetes executor.

Running large tasks

Set the size in Kubernetes Task Defaults on the Execution tab, or per task with a pod_override — the example above works unchanged on GCP. Task nodes come from the general-purpose N2 / E2 families, right-sized to each task.

A task larger than what's available stays Pending. The real limit is your Google Cloud project's vCPU quota — and on GCP there are two that apply: the regional CPU quota for the deploy region and the global CPUs (all regions) quota. Raise them in the Google Cloud console under IAM & Admin → Quotas (filter for CPUs) — bump both, since either can be the ceiling.

What to expect

Warm start (~5–15s). When a task node is already up — a task ran recently, or a fan-out is packing several small tasks onto one node — the pod starts in seconds off the node's cached image.
Cold start (~1 min). The first task after an idle stretch waits once for a fresh VM to boot and join the cluster. Everything after is warm until the nodes scale back to zero.
On-demand only. Task nodes never run on spot, so a task is never interrupted mid-run.

Hybrid keeps this economical the same way it does on AWS and Azure: your routine tasks stay on warm Celery workers, and only the outliers you opt into pods pay the node lifecycle.

On GCP, right-sized task nodes are on by default on every cluster — there's nothing to enable and no create-time or platform-version requirement (Node Auto-Provisioning is built into every GKE cluster we deploy). Unlike Azure, existing GCP clusters already have it.

Node pool capacity planning

Two sizing behaviors are worth planning for — both bit real rollouts:

Worker rollouts transiently need ~2× their steady-state CPU. Every Airflow pod runs with request = limit (Guaranteed QoS — it's what makes the Analytics tab's %-of-limit views and eviction protection work), so during a rolling update the old and new worker pods both hold their full reservation until the old ones finish draining. If a worker pool runs near its node-pool max nodes, a config change that rolls workers (executor switch, resize, image change) will leave new pods Pending until the old generation's termination grace (10 minutes) frees the CPU. It self-resolves, but budget headroom if you want fast rollouts: keep the pool's max at least one node above steady-state usage, or point big worker queues at their own pool.

Scale-to-zero queues and Kubernetes task pods rely on node autoscaling. The first task or worker after an idle period may wait for a node to boot before the pod can start — see Right-sized task nodes for the Kubernetes-executor timing on AWS (~40s–1 min cold), or ~1–2 minutes for a Celery queue's own pool. If cold starts matter for a queue, set its Min Workers to 1 instead of 0.

Spot pools: Celery queue workers get the spot toleration automatically; Kubernetes-executor pods only land on spot if your pod_override adds both the node selector and the toleration — see the example on the Worker Queues page.

On this page