Raja's Exocortex

GPT-Load

GPT-Load is an open-source GPT models proxy server that can hosted on docker.

File: Docker Compose

# Docker compose file to run gpt-load locally.
# Filename: docker-compose.yaml

name: gemini-example-com

services:
  gpt-load:
    image: ghcr.io/tbphp/gpt-load:latest
    container_name: gemini.example.com
    restart: always

    env_file:
        - .env

    mem_limit: 512M

    stop_grace_period: ${SERVER_GRACEFUL_SHUTDOWN_TIMEOUT:-10}s

    volumes:
      - ./data:/app/data

    healthcheck:
      test: wget -q --spider -T 10 -O /dev/null http://localhost:${PORT:-3001}/health
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

    labels:
      - com.centurylinklabs.watchtower.enable=true
      - traefik.enable=true
      - traefik.http.routers.gpt-load.rule=Host(`gemini.example.com`)
      - traefik.http.routers.gpt-load.tls=true
      - traefik.http.routers.gpt-load.tls.certresolver=lets-encrypt

File: .env

# ENV file for GPT-Load - https://raw.githubusercontent.com/tbphp/gpt-load/refs/heads/main/.env.example
# Set AUTH_KEY and ENCRYPTION_KEY to a random string using
# openssl rand --hex 16

# Server timeout settings (in seconds)
SERVER_READ_TIMEOUT=60
SERVER_WRITE_TIMEOUT=600
SERVER_IDLE_TIMEOUT=120
SERVER_GRACEFUL_SHUTDOWN_TIMEOUT=10

# Set to true for slave nodes in cluster setup
IS_SLAVE=false

TZ=Asia/Kolkata

# Authentication key is required to protect the management API and UI.
AUTH_KEY=<generate-random-string>

# ENCRYPTION_KEY encrypts API keys at rest. Use any string or leave empty to disable.
ENCRYPTION_KEY=<generate-random-string>

DATABASE_DSN=

REDIS_DSN=

MAX_CONCURRENT_REQUESTS=100

ENABLE_CORS=true
ALLOWED_ORIGINS=*
ALLOWED_METHODS=GET,POST,PUT,DELETE,OPTIONS
ALLOWED_HEADERS=*
ALLOW_CREDENTIALS=false

LOG_LEVEL=warn
LOG_FORMAT=text
LOG_ENABLE_FILE=false
LOG_FILE_PATH=/dev/stdout

Setup Instructions

  1. Access the GPT-Laod web interface: https://gemini.example.com
  2. Create a group in Keys section called gemini.
  3. Fill the Gemini Upstream address and additional parameters.
  4. Add Gemini API Keys to the created group (add multiple if key rotation needed).

Usage

Add the below env variables to use this proxy for accessing Gemini models.

export GOOGLE_GEMINI_BASE_URL="https://gemini.example.com/proxy/gemini/"
export GEMINI_API_KEY="<your-gpt-load-api-key>"

Note: Replace <your-gpt-load-api-key> with the API key that can generate from Keys > Proxy Keys section of GPT-Load web interface.

Reference

  1. Self hosting using Docker: https://github.com/tbphp/gpt-load
  2. Config parameters for .env: https://raw.githubusercontent.com/tbphp/gpt-load/refs/heads/main/.env.example