Self-Hosting AI Agents with Ollama and Docker

A practical, end-to-end guide for engineering teams who want a private, sovereign AI coding environment — no data leaves your network.

Why self-host?

Cloud AI providers see every keystroke, snippet, and prompt. For regulated industries (finance, healthcare, defense, government) or any team handling proprietary code, that's a non-starter. Self-hosting with Ollama + Docker gives you GPT-class coding assistance with zero outbound calls.

Prerequisites

A Linux host (or macOS / WSL2) with 16 GB+ RAM (32 GB recommended)
NVIDIA GPU with 12 GB+ VRAM for 7B–13B models (optional but fast)
Docker 24+ and Docker Compose
50 GB free disk for model weights

Step 1 — Install Ollama

# One-line install on Linux/macOS
curl -fsSL https://ollama.com/install.sh | sh

# Verify
ollama --version

Step 2 — Pull a coding model

# Strong general coder, ~4.7 GB
ollama pull qwen2.5-coder:7b

# Or a smaller/faster option
ollama pull deepseek-coder-v2:16b

# Run a quick sanity check
ollama run qwen2.5-coder:7b "Write a Python function to reverse a linked list"

Step 3 — Run Ollama in Docker

Create a docker-compose.yml:

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  ollama-data:

docker compose up -d
docker exec -it $(docker ps -qf name=ollama) ollama pull qwen2.5-coder:7b

Step 4 — Point MANOVIK at your local Ollama

In MANOVIK's settings, choose Sovereign Mode and set the model endpoint:

MANOVIK_MODEL_PROVIDER=ollama
MANOVIK_MODEL_ENDPOINT=http://localhost:11434
MANOVIK_MODEL_NAME=qwen2.5-coder:7b

Restart MANOVIK. Every prompt now routes to your local Ollama process — nothing leaves your machine.

Step 5 — Verify it's fully offline

# Block outbound while testing
sudo iptables -A OUTPUT -p tcp --dport 443 -j REJECT
# Use MANOVIK normally; if it still works, you're sovereign.
sudo iptables -D OUTPUT -p tcp --dport 443 -j REJECT

Production tips

Front Ollama with a reverse proxy (Caddy / Traefik) for TLS inside the LAN.
Use OLLAMA_NUM_PARALLEL=4 to serve multiple developers at once.
Mount the ollama-data volume on fast NVMe — model load time dominates cold starts.
For 30B+ models, pin to a dedicated GPU node and use OLLAMA_KEEP_ALIVE=-1.

Next steps

You now have a fully sovereign AI coding agent. Pair it with MANOVIK's per-language memory and self-update scheduler for an offline assistant that gets smarter without ever phoning home. See how this compares to Cursor AI →