Ollama - Painless Self-hosted LLMs

Self hosting LLMs is easily accomplished using Ollama and the associated Ollama-WebUI .

Ubuntu 22.04 Server or Desktop with a GPU and Nvidia container runtime is capable of running Ollama in docker. For GPU acceleration, complete the nvidia-docker setup to install the Nvidia container runtime.

An Nvidia RTX 3060 with 12GB GPU RAM can run most 3b and 7b models.

Model	Size	GPU RAM (GB)
orca-mini	3b	2.9GB
codellama	7b	5.2GB
llama2	7b	6.4GB

Nvidia A6000 6GB can only run orca-min:3b model and orca2:7b does not work. The larger models will still run on the CPU without GPU acceleration.

Docker Deployment With Traefik Ingress Controller

#TODO: test the below docker-compose

# Docker Compose file to run ollama.ai with ollama-webui front end behind traefik
# Ollama backend is directly published for API access
# See:
#   https://github.com/ollama/ollama
#   https://github.com/open-webui/open-webui
#
# This compose file uses GPU, nvidia-container-toolkit is required.
#
# Traefik routing
#   ollama (backend) URLs are /api (but not /api/v1)
#   ollama-webui URLs are all others
#
# Supported LLM Models: https://ollama.ai/library
# Nvidia RTX 3060 12GB can run:
#   orca-mini:3b (2.9GB GPU RAM)
#   codellama:7b (5.2GB GPU RAM)
#   llama2:7b    (6.4GB GPU RAM)

version: '3.7'

name: ollama-example-com

services:
  open-webui:
    container_name: open-webui.example.com
    image: ghcr.io/open-webui/open-webui:main
    hostname: open-webui
    restart: always

    environment:
#     - 'OLLAMA_API_BASE_URL=http://ollama:11434/api'
      - 'OLLAMA_BASE_URL=http://ollama:11434'

    volumes:
      - ./data-open-webui:/app/backend/data

    labels:
      - com.centurylinklabs.watchtower.enable=true
      - traefik.enable=true
      - traefik.http.routers.open-webui.rule=Host(`ollama.example.com`)
      - traefik.http.routers.open-webui.tls=true
      - traefik.http.routers.open-webui.tls.certresolver=lets-encrypt
      - traefik.http.routers.open-webui.service=open-webui
      - traefik.http.services.open-webui.loadbalancer.server.port=8080

  ollama:
    container_name: ollama.example.com
    image: ollama/ollama
    hostname: ollama
    restart: always

    expose:
      - 11434:11434

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

    environment:
      - 'OLLAMA_ORIGINS=*'

    volumes:
      - ./data-ollama:/root/.ollama

    labels:
      - com.centurylinklabs.watchtower.enable=true
      - traefik.enable=true
      - traefik.http.routers.ollama.rule=Host(`ollama.example.com`) && PathPrefix(`/api`) && !PathPrefix(`/api/v[0-9]+`)
      - traefik.http.routers.ollama.tls=true
      - traefik.http.routers.ollama.tls.certresolver=lets-encrypt
      - traefik.http.routers.ollama.service=ollama
      - traefik.http.services.ollama.loadbalancer.server.port=11434

Docker Deployment Without Ingress Controller

The below docker compose exposes the web UI in TCP port :8080 and the API server in :11434.

# Docker Compose file to run ollama.ai with ollama-webui front end
# See:
#   https://github.com/ollama/ollama
#   https://github.com/open-webui/open-webui
#
# This compose file uses GPU, nvidia-container-toolkit is required.
#
# Supported LLM Models: https://ollama.ai/library
# Nvidia RTX 3060 12GB can run:
#   orca-mini:3b (2.9GB GPU RAM)
#   codellama:7b (5.2GB GPU RAM)
#   llama2:7b    (6.4GB GPU RAM)

name: ollama-localhost

services:
  open-webui:
    container_name: open-webui.localhost
    image: ghcr.io/open-webui/open-webui:main
    hostname: open-webui
    restart: unless-stopped

    ports:
      - 8080:8080

    volumes:
      - ./data-open-webui:/app/backend/data

    environment:
      - 'OLLAMA_API_BASE_URL=http://ollama:11434/api'
      - 'OLLAMA_BASE_URL=http://ollama:11434'

  ollama:
    container_name: ollama.localhost
    image: ollama/ollama
    restart: unless-stopped
    hostname: ollama

    ports:
      - 11434:11434

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

    environment:
      - 'OLLAMA_ORIGINS=*'

    volumes:
      - ./data-ollama:/root/.ollama

Installing Models

Different models can be downloaded and installed using the Ollama WebUI at Settings / Models / Pull a Model. The list of models is available on ollama.ai/library.

Troubleshooting

Run the nvidia-smi tool to ensure ollama is using the GPU. The model GPU RAM usage can also be checked using nvidia-smi tool.

Sample `nvidia-smi` Output with the `orca-mini` Model