Running AI workloads on Kubernetes: GPU scheduling and inference cost

GPUs in a cluster

• GPUs are exposed to pods as a resource through a device plugin.
• A GPU is usually assigned whole to one pod unless sharing is set up.
• Nodes with GPUs should be tainted so that ordinary workloads do not land on them.

Inference specifics

• Models are large; loading time affects start-up, so plan scaling carefully.
• Batching requests raises GPU use but adds delay.
• Scale on a signal that reflects load, such as queue depth or request rate.

Controlling cost

• Use the smallest GPU that meets your latency target.
• Scale to zero where start-up time allows.
• Watch utilisation; an idle GPU is the costliest waste.

IICTP is a new independent credential and is not ISO/IEC 17024 accredited.