Cold Starts in Serverless GPU Inference and How to Kill Them

Serverless GPU inference promises elasticity, but it introduces a specific failure mode that breaks the "pay-per-use" promise: the cold start. Unlike CPU-based serverless functions where cold starts are measured in milliseconds, GPU cold starts can take seconds or even minutes. This latency spike is not due to the GPU’s computational power; it is an I/O problem. The mechanism of failure is the transfer and deserialization of multi-gigabyte model weights from remote object storage into the GPU’s VRAM every time a new execution environment is provisioned.

To eliminate this latency, platform engineers must shift from a "load-on-demand" model to a "pre-warmed" or "streaming" architecture. This involves leveraging ephemeral storage snapshots and model streaming techniques to decouple weight loading from request processing.

The Anatomy of the Cold Start

When a serverless provider scales out a new GPU instance, the following sequence occurs:

1. Container Initialization: The hypervisor allocates resources, loads the OS kernel, and starts the container runtime. This takes ~100-500ms.
2. Driver Loading: The NVIDIA GPU driver initializes. This is relatively fast but requires kernel module loading.
3. Weight Download: The application code downloads the model weights (often 2GB–50GB+) from S3, GCS, or Azure Blob Storage. This is the primary bottleneck. Network egress/ingress limits and storage API latency make this slow.
4. Deserialization: The framework (PyTorch, TensorFlow, etc.) reads the weight files and converts them into GPU tensors. This step is CPU-bound and memory-intensive.
5. Ready State: The inference endpoint is now listening.

The critical insight is that steps 3 and 4 are deterministic and repeatable. They do not depend on the incoming request. Therefore, they should not block the first request.

Strategy 1: Snapshots (Ephemeral Storage Pre-warming)

The most effective way to kill cold starts is to avoid downloading weights entirely. However, relying on local ephemeral storage like /tmp on AWS Lambda is risky because that storage is wiped whenever a new execution environment is provisioned (i.e., on a cold start), not preserved across environments. To make persistence viable, you must either keep containers alive via provisioned concurrency or integrate with persistent file systems like Amazon EFS.

Mechanism

Instead of downloading weights on every cold start, you create a snapshot of the model. This snapshot is a compressed archive stored in a high-speed CDN or object storage. When a new container starts, a lightweight bootstrap script checks if the model exists in the persistent layer (e.g., EFS mount or a warm container's disk). If not, it downloads the snapshot and extracts it locally. Subsequent requests within that container’s lifecycle use the local copy.

Implementation

For platform engineers, this means restructuring the deployment pipeline:

1. Pre-build the Snapshot: During CI/CD, download the model weights and compress them. Upload this snapshot to a low-latency storage tier.
2. Container Entry Point: Modify the Dockerfile’s entry point to check for the snapshot.

# Simplified entrypoint logic
MODEL_PATH = "/tmp/model_weights"
SNAPSHOT_URL = "s3://bucket/model-snapshot.tar.gz"
 
if not os.path.exists(MODEL_PATH):
    # Download and extract snapshot
    subprocess.run(["aws", "s3", "cp", SNAPSHOT_URL, "/tmp/snapshot.tar.gz"])
    subprocess.run(["tar", "-xzf", "/tmp/snapshot.tar.gz", "-C", "/tmp"])
    os.remove("/tmp/snapshot.tar.gz")
 
# Load model from local disk (much faster than network)
model = load_model(MODEL_PATH)

Trade-offs

• Storage Cost: You pay for storing the snapshot. However, this is negligible compared to the cost of failed requests or poor user experience.
• Versioning: If the model changes, the snapshot URL must change, triggering a full re-download on the next cold start. This is acceptable because model updates are infrequent.

Strategy 2: Model Streaming

In some cases, even downloading a snapshot is too slow. Model streaming allows the inference engine to begin processing requests before all weights are fully loaded into VRAM. This is particularly effective for large language models (LLMs) with multiple layers.

Mechanism

Frameworks like vLLM or Triton Inference Server support loading models in chunks. The bootstrap script starts the inference server, which begins fetching weights over HTTP or gRPC while simultaneously accepting incoming requests for parts of the model that are already loaded.

How It Works

1. Bootstrapper Service: A lightweight service starts and initiates the model loading process.
2. Streaming Weights: Weights are streamed from a storage backend (e.g., S3) directly into the GPU memory.
3. Partial Readiness: The inference server exposes a health check endpoint. Initially, it returns "loading." Once enough layers are loaded to handle the expected workload, it returns "ready."
4. Request Routing: The load balancer routes traffic only after the health check passes. However, some advanced setups can route to the streaming instance once a minimum threshold of layers is loaded, reducing the perceived latency.

Implementation Example

Using vLLM:

from vllm import LLM
 
# Initialize with model from a local path or streamable source
llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    download_dir="/tmp/vllm_cache",  # Use ephemeral storage
    load_format="auto"
)
 
# Model loading blocks until weights are fully loaded
# Once initialization completes, you can start serving requests

Strategy 3: Hybrid Approach for Platform Engineers

For production systems, a hybrid approach is recommended. Combine snapshots with a warm pool strategy.

1. Pre-warm Containers: Use scheduled tasks to keep a small number of containers alive. These containers retain their local model snapshots. When a new request arrives, it hits a warm container, avoiding the cold start entirely.
2. Snapshot Fallback: If no warm containers are available, the new container must download the snapshot from the source (S3/CDN) rather than from ephemeral storage, as local disk is wiped on cold starts. Using a persistent volume like EFS can mitigate this by allowing quick mounting, but the data must still be present.
3. Streaming for Extreme Cases: For very large models (>70B parameters), use model streaming to begin processing as soon as the first few layers are loaded, while the rest of the model continues to stream in the background.

Conclusion

Cold starts in serverless GPU inference are not an unsolvable problem. They are a result of treating model weights as dynamic data rather than static assets. By leveraging ephemeral storage snapshots and model streaming, platform engineers can reduce cold start latency from minutes to milliseconds. The key is to move the weight loading process out of the critical request path, ensuring that the first request is served by a container that is already prepared to handle it.

This approach requires a shift in deployment pipelines and container design, but the payoff is a responsive, scalable GPU inference service that truly leverages the benefits of serverless architecture.