Run Your First Open-Weights Model Locally with Ollama

Running a Large Language Model (LLM) locally used to require compiling custom C++ kernels, managing PyTorch environments, and debugging CUDA driver mismatches. Ollama removes this friction by bundling the model weights, the inference engine (based on llama.cpp), and the necessary system libraries into a single executable. This guide walks you through installing Ollama, pulling your first model, and interacting with it via both the command line and a local API.

Installation: The Background Daemon

Ollama operates as a client-server architecture. When you install it, you aren't just installing a script; you are installing a background service (daemon) that manages the GPU and memory allocation.

On macOS and Windows: Download the installer from ollama.com. The installer automatically configures the system to start the ollama service on boot. You can verify it is running by checking the system tray (Windows) or Menu Bar (macOS) for the Ollama icon.

On Linux: Use the one-line install script:

curl -fsSL https://ollama.com/install.sh | sh

This script installs the binary and sets up the systemd service. Ensure your user is in the video group if you intend to use an AMD GPU, as this grants access to /dev/kfd and /dev/dri devices.

Once installed, the daemon is listening on http://localhost:11434. You do not need to manually start ollama serve; the installer handles this.

Pulling Weights: Understanding Quantization

The term "open-weights" refers to models where the creators publish the trained parameters (the weights) but not the training code or data. Note that 'open-weights' refers specifically to the availability of trained parameters, distinct from 'open-source' which implies full code and data availability. Ollama hosts these models in its library.

Run your first pull command:

ollama pull llama3

This command downloads the Llama 3 8B parameter model. Notice that the download size is approximately 4.7 GB, not the ~16 GB you might expect from a full 16-bit precision model. This is due to quantization.

LLMs are computationally expensive to run in full precision (FP16). Quantization reduces the precision of the weights (e.g., to 4-bit or 8-bit integers) with minimal loss in quality. Ollama defaults to Q4_0, a high-quality 4-bit quantization that balances speed and accuracy. You can view available quantizations by checking the model's tag in the Ollama Library or referring to the llama.cpp documentation for technical details on quantization schemes.

First Inference: The CLI Interface

With the weights downloaded, you can start chatting immediately.

ollama run llama3

This command launches an interactive session. The model loads into your system memory (RAM) or video memory (VRAM). If you have an Apple Silicon Mac or an NVIDIA GPU, the model will utilize the accelerator. If you are on a standard CPU-only laptop, it will use RAM, which is slower but functional.

Type a prompt:

"Explain the concept of overfitting in machine learning to a five-year-old."

The model streams the response back to your terminal. Because Ollama keeps the model resident in memory after the conversation ends, subsequent prompts are nearly instantaneous, provided you have enough RAM/VRAM.

The Local API: Bridging to Code

The real power of Ollama for developers lies in its API compatibility. Ollama exposes a local REST API that mirrors the OpenAI API structure. This means you can use existing OpenAI SDKs or HTTP clients to interact with your local model without rewriting prompt formatting logic. Beyond basic chat completions, the API supports streaming responses for real-time token output, embedding generation for vector search tasks, and model listing endpoints to discover available local models. This abstraction layer is critical: it allows you to prototype with a local model and switch to a cloud provider later by simply changing the base URL and API key, without altering your application code.

Start a simple Python script to test the connection:

import requests
import json
 
url = "http://localhost:11434/v1/chat/completions"
 
payload = {
    "model": "llama3",
    "messages": [
        {"role": "user", "content": "What is the capital of France?"}
    ],
    "stream": False
}
 
headers = {
    "Content-Type": "application/json"
}
 
response = requests.post(url, headers=headers, json=payload)
print(response.json()['choices'][0]['message']['content'])

This script sends a JSON payload to localhost:11434. The response structure matches what you would get from api.openai.com. This abstraction layer is critical: it allows you to prototype with a local model and switch to a cloud provider later by simply changing the base URL and API key, without altering your application code.

Hardware Constraints and Performance

Running models locally introduces a hard constraint: memory bandwidth and capacity.

1. VRAM (GPU Memory): If you have an NVIDIA GPU with 8GB of VRAM, you can comfortably run quantized 7B-8B parameter models. If the model exceeds VRAM, it spills into system RAM, causing a significant drop in token generation speed (from ~50 tokens/sec to <5 tokens/sec). For detailed limits on CUDA memory management, refer to the NVIDIA CUDA documentation.
2. System RAM: On CPUs, the model must fit entirely in RAM. An 8B model at Q4 quantization requires ~5-6 GB of RAM. If your system has 16 GB total, you have enough headroom. If you have 8 GB, you will struggle. Apple Silicon models manage memory differently; see the Apple Developer documentation on Unified Memory for insights into how macOS handles memory sharing between CPU and GPU.
3. Swap File: If the model does not fit in RAM, the OS moves data to the SSD/HDD swap file. This is catastrophic for performance. Always monitor memory usage (htop on Linux, Activity Monitor on macOS) while running heavy models.

Common Pitfalls

1. Running out of disk space during pull: Models can be large. Ensure you have at least 40GB of free space before attempting to pull large models like Llama 3 70B.
2. Confusing model tags: llama3 pulls the default quantization. To get specific versions, use tags like llama3:8b or llama3:70b. Check the Ollama Library for all available variants.
3. Forgetting to add user to video group on Linux: Without being in the video group, Ollama cannot access AMD devices, forcing CPU-only inference which is significantly slower.

Practical Takeaways

1. Check available RAM before pulling large models: A general rule of thumb is to have at least 2x the model size in free RAM/VRAM for smooth operation.
2. Use Q4_K_M for best balance: Unless you have massive memory reserves, the 4-bit quantization offers the best trade-off between speed and quality for most development tasks.
3. Monitor GPU memory usage with nvidia-smi: Use this tool on Linux/NVIDIA systems to verify if the model is staying in VRAM or spilling into system RAM, which drastically impacts performance.

Conclusion

Ollama provides the simplest entry point into local AI development. By automating the download of quantized weights and exposing a familiar API, it allows developers to iterate on prompts and fine-tuning pipelines without managing complex infrastructure. Start with a small model like llama3 or mistral on your laptop, verify the API connectivity, and then explore larger models as your hardware permits.