Ollama

Ollama is an open-source tool that makes it trivially easy to run large language models locally on macOS, Linux, and Windows. It provides a simple command-line interface and REST API that mirrors the OpenAI API format, making it a drop-in replacement for cloud LLM providers when building AI agents. With a single command like 'ollama run llama3', developers can download and run models locally with optimized performance for both CPU and GPU inference.

Ollama supports a vast library of open-source models including Llama 3, Mistral, Gemma, Phi, CodeLlama, DeepSeek, Qwen, and many more. Models are distributed as optimized packages with automatic quantization support (Q4, Q5, Q8) to run on consumer hardware. The platform handles model management, memory allocation, and inference optimization automatically.

For AI agent development, Ollama is invaluable as it provides a free, private, and low-latency LLM backend. Most major agent frameworks — including LangChain, CrewAI, Strands, LlamaIndex, and Google ADK — support Ollama as a model provider. The OpenAI-compatible API means any tool built for the OpenAI API can point at Ollama with a simple base URL change.

Ollama also supports tool calling and function calling with compatible models, enabling proper agent tool use patterns. Custom model creation via Modelfiles allows fine-tuned system prompts and parameter tuning. The project has a thriving open-source community and has become the de facto standard for local LLM development.