Kubernetes binds GPU jobs only when every Pod fits

An eight-Pod GPU job that can run only as a complete group ties up expensive GPUs as soon as the scheduler places seven Pods individually on nodes. Kubernetes 1.37 brings this all-or-nothing scheduling into the core as a beta feature: the scheduler binds the group only when every member has room. Kueue and Volcano remain necessary for quota, fair share and cross-cluster queues.

Key takeaways

Related:DRA stability forces a rethink for GPU workloads  /  Kubernetes 1.34 and Knative: ready for AI workloads?

What is gang scheduling? All-or-nothing scheduling: the scheduler binds a group of Pods only when every member has a place. If one Pod cannot fit, the group waits.

PodGroups start only when every member has room

Kubernetes v1.37.0, codenamed Garhwal, has been a supported version since 26 August 2026 and includes 67 enhancements. A smaller subset matters for operations because it changes what happens to jobs that make sense only as a group.

Training runs, distributed inference and parallel computing jobs using the Message Passing Interface (MPI) model share the same failure mode. The scheduler places the first Pods on expensive GPU nodes. The last Pod waits for capacity. Instances already running hold memory and cards without completing a useful training epoch. Until now, the answer lay outside the core. Kueue, Volcano and coscheduling plugins kept these groups together. Kubernetes 1.37 incorporates basic group binding into the scheduler.

The Workload and PodGroup APIs are available as beta in the v1beta1 API version in this release, including gang scheduling. A PodGroup defines how many Pods must be schedulable together before the scheduler binds them. This gives GPU jobs the mechanism they need without requiring every team to run a separate queuing product as a mandatory layer.

The implementation goes beyond an API wrapper, as the Kubernetes project describes in its blog on workload-aware scheduling dated 8 September 2026. The native PodGroup queue queues only the top-level group rather than each Pod individually. The minCount threshold, the minimum number of Pods that must be schedulable together, can be changed later. An elastic gang can therefore grow or shrink without restarting Pods already scheduled. Workload-Aware Preemption and shared ResourceClaims for PodGroups are also beta. Dynamic Resource Allocation (DRA) with Extended Resources is even generally available (GA) in 1.37: a Pod can request a GPU using the traditional extended resource name and be assigned a device through DRA, without a separate device plugin running alongside it. This requires an installed DRA driver for the cards.

The scheduler thus handles group binding and device requests. Fair share across teams remains outside the core. That is precisely why Kueue remains necessary in production clusters.

Kueue stays while core features remain in beta

Kueue remains relevant even though Kubernetes now understands Pod groups. Kueue manages quota, fair share and cross-cluster queues, while Volcano provides batch policies that the scheduler does not replicate in 1.37. Core gang scheduling removes only the argument that GPU jobs start with incomplete groups without an add-on. The case for quota and fair share remains unchanged.

CompositePodGroup is alpha in 1.37. Hierarchical groups that place a training gang within a larger batch group are therefore planned rather than delivered; such hierarchies still require an add-on. Alpha testers upgrading from the previous release also face a breaking change: in 1.37, PodGroup v1alpha3 replaces v1alpha2. disruptionMode (the behavior when an incident affects the group) now accepts the values all (the entire group) and single (an individual member). Clusters already using the alpha APIs in production must adopt this renaming.

The beta API is broad enough for testing in a dedicated pool, using feature gates and expecting fields to change before general availability. It does not replace Kueue as the place for team quotas. The Job controller works with Workloads through the WorkloadWithJob feature gate on kube-apiserver and kube-controller-manager, so initial adoption through Jobs belongs in a test pool rather than on the production scheduler serving all namespaces.

The Node Lifecycle Conditions are also unfinished in 1.37. Kubernetes reserves five conditions with names such as DrainInProgress (draining underway) and MaintenancePlanned (maintenance scheduled). The feature gate is alpha, so no core controller reads these conditions yet. For now, a drain tool merely writes metadata for future releases.

Idle GPUs scale to zero only with the right metric

Alongside scheduling, scaling to zero determines GPU costs. The HorizontalPodAutoscaler (HPA) can scale workloads to zero replicas in 1.37. According to the Kubernetes blog dated 2 September 2026, the feature is beta and enabled by default in kube-apiserver and kube-controller-manager, so minReplicas 0 is no longer an alpha setting. The API server accepts the value. Once the metric falls below the threshold, the controller scales the workload down. When it rises again, the controller brings it back.

CPU and memory are insufficient for scaling to zero because the API server rejects an HPA containing only resource metrics. Object or external metrics are required, such as queue length, a custom metric from inference serving or an external load metric. Without this source, the last Pod stays running and the GPU remains occupied.

The default HPA downscale window for reaching zero is five minutes. GPUs therefore remain occupied when a batch queue is briefly empty. For inference spikes every few minutes, this window directly affects cost calculations. The ScaledToZero status condition distinguishes automatic scaling to zero from a manual pause. It is the difference between “the controller shut it down” and “someone set the replica count to zero”.

Native Histograms for Kubernetes metrics are also beta and enabled by default in 1.37. More detailed latency distributions change how autoscalers and dashboards see load, but they do not replace the queue metric HPA needs to scale to zero.

According to the project’s patch schedule, the next patch, 1.37.1, follows in the first few weeks after release. Until then, the test pool remains the place for gang scheduling and HPA scaling to zero.

Frequently asked questions

Image source: AI-generated (September 2026)

Translated from the German original using artificial intelligence. The German version is authoritative.