Ariff Nazhan

Article

KAI Scheduler vs HAMi

Credits to Scicom (MSC) Berhad and Scicom AI Enterprise for the resources.

KAI Scheduler and HAMi both decide which GPU a pod gets, and both can put several pods on one card. They solve different problems. KAI Scheduler decides who gets GPUs and when: queues, quotas, fair share and gang scheduling. HAMi decides which card and how much of it: it slices cards, caps memory per container, and lets a pod name the exact cards it runs on.

In short, they work at different levels. KAI is a Kubernetes scheduler: it works at the workload level and decides which pod or job runs, when, and on which node. HAMi works at the GPU level: it decides which card a pod gets and how much of it, and HAMi-core enforces that share inside the container. HAMi does not replace the Kubernetes scheduler, it extends it.

Where KAI Scheduler and HAMi sit. Workload level: KAI Scheduler, with queues, quotas, fair share, gang scheduling and preemption, decides who runs, when and on which node; without KAI this is the default kube-scheduler, which HAMi extends. Node level: a node with 8 GPUs. GPU level: HAMi, a device plugin and scheduler extender, decides which card and how much of it, pinning by UUID with a memory and compute share. Container level: HAMi-core caps memory and compute inside the container. Hardware: GPU 0 to GPU 7.
Where each one sits. Blue is KAI, green is HAMi.

We run several workloads on the same GPU nodes, and we had trouble putting each of them on specific GPUs. We tried both. This post compares them. The versions are HAMi 2.9.0 and KAI Scheduler v0.18.6.

What each one is

HAMi:

  1. It is the device plugin, plus a scheduler extender and a webhook. It hands out the cards itself.
  2. A pod can name the exact cards it runs on, by UUID.
  3. A pod asks for a share of each card's memory, and HAMi-core can enforce that share inside the container.
  4. It has no queues, quotas or gang scheduling.

KAI Scheduler:

  1. It is a scheduler only. The cards come from the NVIDIA device plugin or from DRA.
  2. A pod cannot name its cards. The device plugin picks them.
  3. Card sharing works through reservation pods that hold whole cards. Memory is not enforced by default.
  4. It has queues with quotas, fair share, preemption and gang scheduling.
  5. It is bigger: 7 controllers, 6 CRDs and 4 PriorityClasses, against HAMi's DaemonSet, Deployment and webhook.
How a pod that shares two cards gets them. HAMi: the pod names its cards, the webhook sends it to hami-scheduler, which takes the named cards if each has room, and the HAMi device plugin hands over exactly those cards. KAI Scheduler: the pod asks for a gpu-fraction in a queue, the scheduler checks the quota and picks GPU groups, the binder creates one reservation pod per card, and the NVIDIA device plugin gives each reservation pod any free card.
How a pod that shares two cards gets them. The shaded step is where the physical card is chosen. Drawn from HAMi's architecture and KAI's GPU sharing docs.

What each one is good at

KAI Scheduler architecture: workloads; the pod-grouper that groups pods into PodGroups; a tree of queues with quotas; the scheduling cycle (allocate, consolidate, reclaim, preempt, stale gang eviction); the binder, which also creates reservation pods for GPU sharing; and the device plugin or DRA that hands out the cards.
KAI Scheduler, after its scheduling deep dive. Blue is where KAI works.

KAI Scheduler (v0.18.6) is built around queues and gangs. A pod-grouper puts related pods into PodGroups that start together or not at all. The groups wait in a tree of queues, each with a guaranteed quota, a limit, an over-quota weight and a priority. Every scheduling cycle allocates first, then consolidates running pods to cut fragmentation, reclaims over-quota usage between queues, and preempts lower-priority work inside a queue, with fairness by DRF and optionally by usage over time. Priority and preemptibility are separate settings, and a minimum runtime protects a pod from early preemption. On top of that it places by binpack or spread, can be topology-aware, runs elastic jobs with a minimum and maximum size, shares GPUs with gpu-fraction or gpu-memory, schedules DRA ResourceClaims, works with cluster autoscalers, and records each decision as a Kubernetes Event.

HAMi architecture: workloads; a mutating webhook that routes GPU pods to the HAMi scheduler; the scheduler (kube-scheduler with the HAMi extender, or Volcano or Koordinator); device plugins for NVIDIA, Ascend, Cambricon, Hygon, Iluvatar and MetaX; inside the container HAMi-core for NVIDIA or a vendor runtime; and metrics of what each pod holds on each card.
HAMi, after its architecture diagram. Green is where HAMi works.

HAMi (v2.9.0) is built around the device. Its device plugin splits each card into slots, and a pod asks for N different cards and a share of each: memory in MiB or percent, and compute as a percent of the SMs. The pod can name its cards by UUID (nvidia.com/use-gpuuuid) or by model, or keep away from some. HAMi-core enforces those shares inside the container with no change to the application, and the container sees its cap as the card's memory; CUDA_DISABLE_CONTROL=true turns it off for one container. Placement is binpack or spread, for nodes and for cards, chosen per pod, and dynamic MIG is available per pod. It supports accelerators from several vendors, exports what each pod holds on each card, and needs no CRDs: a scheduler extender on kube-scheduler, a device plugin and a webhook.

Things to know before choosing

KAI cannot pin a pod to given cards. The device plugin picks them, so the only way to land on specific cards is to have them be the only free ones when the pod binds, and the placement can change after a restart. Its GPU sharing needs the nvidia.com/gpu.memory node label from GPU Feature Discovery; without it every bind of a gpu-fraction pod is rolled back. In v0.18.6 it also counts admin-access DRA claims as GPU usage, so with the GPU Operator's DRA driver a node looks fuller than it is. Uninstalling it through ArgoCD leaves its CRDs and the namespace of its reservation pods behind, and leftover reservation pods keep holding cards.

HAMi cannot run next to the NVIDIA device plugin, since both advertise nvidia.com/gpu. It counts slots, not cards: with deviceSplitCount: 10, an 8-GPU node shows nvidia.com/gpu: 80.

With either tool, engines that share a card should start one at a time. vLLM measures free GPU memory when it starts, and two engines measuring at once count each other's memory wrong.

Why we use HAMi

Our problem was placement: several workloads, each meant for specific cards, some sharing a card. HAMi covers that in one component.

  1. Pinning. A pod lists the card UUIDs it may use. The spec is the placement, and it survives restarts.
  2. Sharing by memory share. A pod asks for N cards and a share of each. HAMi will not place a pod past 100 % of a card.
  3. Enforcement per container. vLLM sizes its own memory, so vLLM pods run with HAMi-core off and the share is bookkeeping. Small services run with HAMi-core on, so they cannot grow into a neighbour's memory.
  4. Small footprint. A device plugin, a scheduler extender and a webhook, no CRDs.

A vLLM engine pinned to two cards that it shares with another engine:

metadata:
  annotations:
    nvidia.com/use-gpuuuid: "GPU-aaaa...,GPU-bbbb..."   # exactly these two cards
spec:
  runtimeClassName: nvidia
  containers:
    - name: vllm
      env:
        - name: CUDA_DISABLE_CONTROL      # HAMi-core off: vLLM manages its own memory
          value: "true"
      resources:
        limits:
          nvidia.com/gpu: "2"                   # two different cards
          nvidia.com/gpumem-percentage: "45"    # of each card

When KAI is the better choice

KAI is built for a shared training or batch cluster: many teams, jobs that come and go, quotas that must hold, gang scheduling for multi-node training, and preemption between priorities. If which card a pod lands on does not matter and the problem is who gets GPUs first, KAI solves that and HAMi does not.

If the problem is which card, and every card has a planned owner, HAMi does it directly. KAI gets there only through rollout order, with a reservation-pod layer and a GPU Feature Discovery dependency on top.