Spot, Reserved, or Serverless GPUs: Picking a Capacity Strategy

This is Part 4 of the GPU Economics for AI Teams series.

For platform engineers managing AI infrastructure, the primary constraint is rarely compute power—it is cost efficiency relative to fault tolerance. Choosing a GPU capacity strategy requires understanding the mechanism by which each model fails or succeeds. Spot instances trade stability for deep discounts via preemption. Reserved Instances (RIs) trade flexibility for price certainty. Serverless GPUs trade control for operational simplicity. The optimal strategy is almost always a hybrid that maps specific workload characteristics to the correct economic engine.

The Mechanics of Spot Preemption

Spot instances utilize unused cloud provider surplus capacity. Because this capacity is ephemeral, the provider can reclaim it with little notice. In AWS, this is called a Spot Interruption; in Google Cloud, it is Preemption. The mechanism is driven by market dynamics: if demand for GPU instances rises, the spot price increases, or existing spot instances are terminated to free resources for on-demand users.

The critical engineering challenge is not the price, but the preemption notice period. AWS provides a two-minute termination notice. This window is insufficient for most GPU workloads to complete a forward pass or save state to disk. Without explicit checkpointing mechanisms, a single preemption can lose hours of training progress.

Consider a PyTorch training loop running on cost-effective spot gpu capacity, such as A10G spot gpu instances. If the instance is preempted at step 50,000, the worker process dies. The orchestrator (e.g., Kubernetes with Karpenter or AWS Batch) must detect the pod termination, spin up a new instance, and resume training from the last saved checkpoint. This introduces overhead: network latency for downloading checkpoints and compute time wasted on redundant forward passes if the checkpoint is not granular enough.

Spot is viable only if the workload is idempotent or supports frequent checkpointing. For inference services, spot is risky unless combined with load balancing and rapid auto-scaling, but for training, it is the primary lever for cost reduction, often yielding up to 90% savings over on-demand pricing, though actual savings vary by instance type and region.

The Economics of Reserved Capacity

Reserved Instances (RIs) and Savings Plans represent a financial contract. You commit to a specific instance type, region, and tenancy for a term of one or three years. In exchange, you receive a significant discount compared to on-demand pricing, typically 30-70%. Unlike spot, reserved instances are not subject to preemption. They are guaranteed to be available as long as the capacity exists in the region, which is nearly always for popular instance types.

The mechanism here is risk mitigation. By paying upfront or committing to a monthly spend, you hedge against spot price volatility. However, RIs introduce a utilization penalty. If your workload is sporadic—such as a research team experimenting with new architectures—you may pay for reserved capacity that sits idle. This is the "capacity planning" trap: over-provisioning RIs leads to wasted capital, while under-provisioning forces you to use expensive on-demand instances for overflow.

Savings Plans, particularly the Compute Savings Plan, offer more flexibility than traditional RIs. They apply to any instance family in a region, allowing you to shift between GPU types (e.g., from P4d to P5) without losing the discount. This flexibility makes Savings Plans preferable for teams that iterate rapidly on hardware requirements.

For steady-state workloads, such as production inference endpoints with predictable traffic, RIs or Savings Plans are essential. They provide the stability required for Service Level Objectives (SLOs) and lock in costs for budgeting.

Serverless GPU Abstraction

Serverless GPU offerings, such as RunPod Serverless, Modal, Vast.ai, or AWS SageMaker Serverless Inference, abstract the underlying infrastructure entirely. You upload code, define resource requirements (e.g., 1x A100), and pay per second of execution time. There are no instances to manage, no patching, and no scaling policies to configure. The provider handles all provisioning and teardown.

The mechanism of serverless gpu involves cold starts. When a function is invoked after a period of inactivity, the provider must allocate a GPU, load the container image, and initialize the runtime. This can introduce latency ranging from seconds to minutes, depending on the image size and provider optimization. For interactive applications or real-time inference, this latency can degrade user experience.

Pricing for serverless GPUs is often higher per hour than on-demand instances when calculated over long durations. However, for bursty workloads with infrequent usage, the pay-per-second model can be cheaper because you avoid paying for idle time. Additionally, serverless models often include concurrency limits and throttling to prevent resource exhaustion, which can be a bottleneck for high-throughput inference.

Serverless is ideal for event-driven tasks, such as processing a batch of images upon upload, or for prototyping where infrastructure management is a distraction. It is less suitable for long-running training jobs or low-latency inference where cold starts are unacceptable.

Mapping Workloads to Strategies

The most effective capacity strategy combines all three models based on workload characteristics.

Long-Running Training Jobs: Use Spot instances with aggressive checkpointing. Implement a fallback mechanism: if spot instances are unavailable or too expensive, spin up on-demand instances or use a Savings Plan to cover the baseline. Tools like Ray Train or PyTorch Lightning simplify checkpointing and resumption. This maximizes cost efficiency while minimizing risk.

Production Inference: Use a hybrid approach. For steady-state traffic, use Reserved Instances or Savings Plans to cover the base load. For traffic spikes, use auto-scaling to on-demand or spot instances if latency allows. For highly variable, low-frequency traffic, consider serverless GPUs to avoid idle costs.

Interactive Notebooks and Development: Use Spot instances. These workloads are interactive and short-lived. Preemption is an annoyance but not a catastrophic failure. Configure the IDE or notebook environment to save state frequently.

Batch Processing: Use Spot instances with flexible fleet strategies. Submit jobs to a queue that can be satisfied by any available GPU type. This maximizes the chance of finding low-priced spot capacity.

Conclusion

There is no single best GPU capacity model. Spot instances offer the lowest cost but require engineering effort to handle preemption. Reserved Instances offer stability and predictable costs but require accurate utilization forecasting. Serverless GPUs offer operational simplicity but introduce latency and higher per-unit costs. Platform engineers must build systems that can dynamically switch between these models based on real-time cost and availability signals. The goal is not to pick one, but to orchestrate them.