6 min readDevelopment
SGLang's RadixAttention, Explained
Explore how SGLang uses RadixAttention and prefix caching to significantly boost throughput and efficiency in large language model serving.
All tags
Tag
Llm Optimization
Read 14 articles on Llm Optimization by Ashish Srivastava — practical guides covering implementation patterns, security, and production lessons.
14 posts
6 min readDevelopmentFeatured
Prompt Caching Across Providers: A Practical Comparison
Explore how different AI providers handle prompt caching, including cache TTL limits, cost implications, and prefix matching techniques to optimize API usage.
6 min readDevelopment
Function-Calling Fine-Tunes: Teaching a Small Model Your Tools
A practical guide to improving tool-use reliability in small models through function-calling fine-tuning and trace mining techniques.
6 min readDevelopmentFeatured
Cost Control for Agent Loops: Budgets & Kill Switches
Learn how to implement token budgets, depth limits, and kill switches to control costs and prevent runaway loops in AI agent architectures.
Part 8 · Building AI Agents
6 min readDevelopmentFeatured
PagedAttention vs RadixAttention: Solving KV Cache Fragmentation and Redundancy
An examination of PagedAttention and RadixAttention mechanisms. We explore how they handle KV cache memory fragmentation and enable efficient context sharing in large language models.
Part 3 · Production vLLM
6 min readDevelopmentFeatured
FlashAttention 3: What Changed and What It Buys You
FlashAttention 3 introduces IO-aware scheduling and FP8 attention to maximize throughput on NVIDIA Hopper GPUs, offering significant performance gains for LLM training and inference.
6 min readDevelopment
Medusa, EAGLE, and Lookahead Decoding Compared
Explore how Medusa, EAGLE, and Lookahead Decoding optimize LLM generation speeds through speculative decoding techniques.
6 min readDevelopmentFeatured
Speculative Decoding: 2x Throughput for Free?
Explore how speculative decoding accelerates LLM inference by using smaller draft models to predict tokens, achieving significant throughput gains without compromising quality.
6 min readDevelopmentFeatured
Multi-Head, Grouped-Query, and Multi-Head Latent Attention
Explore how MHA, GQA, and MLA architectures balance memory usage and computational efficiency in large language models, with a focus on reducing KV cache overhead.
Part 2 · Transformers Under the Hood
6 min readDevelopmentFeatured
Context Compaction: Summaries, Scratchpads, and Handoffs
Learn how context compaction via summaries, scratchpads, and handoffs improves efficiency for long running agents and reduces token usage.
Part 3 · Context Engineering
6 min readDevelopmentFeatured
Context Engineering Is the New Prompt Engineering
Context engineering is replacing prompt engineering as the key to managing context windows and token budgets effectively in modern LLM applications.
Part 1 · Context Engineering
6 min readDevelopmentFeatured
On-Device Inference with GGUF, MLX, and ONNX Runtime
A technical comparison of GGUF, MLX, and ONNX Runtime for running large language models directly on mobile and edge devices.
Part 4 · Small Models & Distillation
6 min readDevelopmentFeatured
Quantizing the KV Cache Without Wrecking Quality
Learn how to apply quantization to the KV cache to save memory and speed up inference without significantly degrading model quality.
Part 4 · Quantization in Practice
6 min readDevelopmentFeatured
GPTQ vs AWQ vs Bitsandbytes: A Practical Comparison
Compare GPTQ, AWQ, and Bitsandbytes quantization techniques. We examine calibration strategies, speed, and accuracy to help ML engineers choose the right model compression method.
Part 3 · Quantization in Practice