justfastllm: Ultra-Light Enterprise LLM Gateway

justfastllm is an enterprise LLM gateway server and proxy built in Python. Its purpose is simple: give teams one fast, self-hosted control plane for calling many LLM providers without embedding provider-specific SDKs across every application.

The gateway supports OpenAI-style chat completions, Anthropic-style messages, model listing, provider routing, Redis caching, guardrails, user preferences, user feedback, agents, skills, OpenAPI, Docker, and cloud deployment descriptors. It was created by Suhas Bhairav as an AI Lab project focused on practical, production-oriented AI infrastructure.

Main Purpose: A Central LLM Gateway

Most companies begin with one model provider and one application. Very quickly, the system becomes harder to govern: teams add OpenAI in one service, Anthropic in another, local Ollama models for experiments, OpenRouter-style gateways for breadth, and custom internal endpoints for private models. Each integration brings its own keys, headers, payload formats, logging, retries, guardrails, and operational assumptions.

justfastllm consolidates that model sprawl behind one gateway. Applications call the gateway, and the gateway decides how to route the request to OpenAI, Anthropic, DeepSeek, Grok, Qwen, Kimi, Ollama, or any configured OpenAI-compatible provider.

• One internal API surface: client applications can talk to a consistent gateway instead of each provider directly.
• Provider flexibility: teams can change provider configuration without rewriting every downstream integration.
• Gateway governance: logging, caching, guardrails, memory, feedback, and model routing live in one operational layer.
• Self-hosted control: organizations can run the gateway on their own infrastructure.

Provider Coverage

The initial provider set is deliberately broad. justfastllm supports mainstream hosted providers, local Ollama models, and custom OpenAI-compatible endpoints so a team can combine public APIs, private models, and local development workflows.

• OpenAI: OpenAI-compatible chat completion routing.
• Anthropic: dedicated Messages API adapter for Claude-style requests.
• DeepSeek: OpenAI-compatible direct HTTP provider adapter.
• xAI Grok: OpenAI-compatible direct HTTP provider adapter.
• Qwen: OpenAI-compatible provider adapter for Qwen-compatible endpoints.
• Kimi / Moonshot: OpenAI-compatible provider adapter.
• Ollama: local OpenAI-compatible routing for private and development models.
• Custom OpenAI-compatible providers: OpenRouter, vLLM, LM Studio, llama.cpp servers, LiteLLM routers, or private enterprise gateways can be registered through environment configuration.

Factory Pattern Architecture

justfastllm is organized around provider contracts instead of random provider-specific functions. Provider clients implement a shared interface, and ProviderFactory constructs the correct client based on gateway settings and request routing.

This design keeps each provider adapter small and explicit. OpenAI-compatible providers share a reusable adapter. Anthropic uses a dedicated adapter because its Messages API has a different contract. Custom providers can be added with environment variables when they follow the OpenAI-compatible shape.

• Provider isolation: provider-specific headers, payloads, paths, and model listing stay inside provider clients.
• Routing clarity: the gateway can route by explicit provider or model prefix.
• Extensibility: new providers can be registered without changing application-level endpoint code.
• Testability: provider behavior, gateway behavior, cache behavior, memory behavior, and guardrails are testable as separate units.

Direct HTTP, No Provider SDKs

The runtime path intentionally avoids OpenAI, Anthropic, and other provider SDKs. justfastllm uses direct HTTP calls so the gateway controls exactly what is sent, what headers are used, how responses are handled, and how upstream errors are normalized.

This matters for enterprise teams because SDKs can add dependency weight, hidden behavior, upgrade churn, and provider-specific abstractions. A gateway that is meant to route many providers benefits from keeping the transport explicit.

• Low dependency footprint: the runtime requirement is intentionally small.
• Clear request path: outbound provider calls are visible and auditable in gateway code.
• Provider-neutral behavior: the gateway owns normalization instead of inheriting SDK conventions.
• Operational predictability: fewer runtime dependencies means fewer moving parts to patch, debug, and deploy.

Cache, User Memory, and Feedback

justfastllm includes Redis-backed caching and Redis-backed user memory so multiple gateway instances can share operational state. Response caching can reduce repeated provider calls for eligible requests. User memory can store preferences and feedback so the gateway can preserve lightweight personalization across sessions.

• Redis response cache: stores eligible model responses by deterministic cache key.
• Memory fallback: in-memory backends support local tests and simple development workflows.
• User preferences: user-level preferences can be read, updated, and injected as context.
• User feedback: feedback can be captured for product loops, evaluation, and future ranking systems.

Guardrails and Runtime Controls

The gateway includes configurable guardrails for request validation and runtime control. Operators can inspect guardrail state and update settings through API endpoints instead of hardcoding every safety decision into application clients.

• Payload validation: validates required message structure for chat-style requests.
• Message limits: prevents oversized message payloads from reaching upstream providers.
• Block patterns: can block configured text patterns before provider calls.
• Runtime toggles: GET /v1/guardrails and PATCH /v1/guardrails expose guardrail state and updates.

Agents and Skills

justfastllm includes lightweight endpoints for agents and skills. This gives the gateway a foundation for repeatable AI workflows without turning the project into a large orchestration framework.

• Skills endpoint: exposes available gateway skills and lets clients invoke named skills.
• Agent run endpoint: supports agent-style runs through the same provider gateway layer.
• Gateway reuse: agents and skills use the same routing, provider, guardrail, and logging foundation as chat completions.

Self-Hosted Deployment

justfastllm is built to be self-hosted. The repository includes Docker support, Docker Compose for local Redis-backed deployment, and platform descriptors for Render, Railway, AWS App Runner, and GCP Cloud Run.

• Docker: production image with non-root runtime user and health check.
• Docker Compose: gateway plus Redis for local production-like deployments.
• Render: blueprint for Docker web service and managed Redis.
• Railway: Dockerfile builder configuration with health checks and restart policy.
• AWS: CloudFormation descriptor for App Runner using an ECR image.
• GCP: Cloud Run service manifest with Redis URL from Secret Manager.

Operational Surface

The gateway exposes operational endpoints and headers that make it easier to run as an internal platform service. It includes /health, /openapi.json, provider model discovery, structured request logs, request IDs, provider headers, and cache hit/miss headers.

• Health: GET /health for load balancers and deployment platforms.
• OpenAPI: GET /openapi.json for client generation and contract review.
• Models: GET /v1/models and GET /v1/providers/{provider}/models for discovery.
• Headers: gateway responses can include provider, cache, and request metadata.

Quality and Verification

The project includes standard-library unit tests, compile checks, Docker build verification, Docker Compose config validation, and a real Ollama integration test. For major gateway changes, the Ollama test can run against a local model such as qwen3:8b to confirm that the gateway can call a real model through the OpenAI-compatible Ollama surface.

• Unit tests: cover providers, factory behavior, cache behavior, memory behavior, guardrails, endpoint behavior, and error handling.
• Compile checks: ensure source, tests, and benchmark scripts compile cleanly.
• Docker build: validates that the deployable image builds successfully.
• Ollama integration: verifies live local-model chat completion and model discovery.

Why It Matters

LLM infrastructure is becoming a control-plane problem. Teams need model choice, provider independence, internal governance, cost controls, memory, safety, local-model evaluation, and deployment ownership. justfastllm demonstrates how much of that can be achieved with a small Python gateway that uses clear contracts and direct HTTP rather than heavy framework layers.

The project is not positioned as a giant AI platform. It is a focused gateway: small enough to understand, structured enough to extend, and self-hosted enough for teams that want control over how model traffic moves through their systems.

FAQ

What is justfastllm?

justfastllm is a Python LLM gateway and proxy server that lets applications access multiple LLM providers through one self-hosted API surface.

Does justfastllm use provider SDKs?

No. The runtime path uses direct HTTP calls to provider APIs instead of OpenAI, Anthropic, or other provider SDKs.

Which providers does justfastllm support?

The initial provider set includes OpenAI, Anthropic, DeepSeek, xAI Grok, Qwen, Kimi / Moonshot, Ollama, and custom OpenAI-compatible providers.

Does justfastllm support Redis?

Yes. Redis can be used for response caching and user memory across multiple gateway instances.

Can justfastllm run with Docker?

Yes. The project includes Docker, Docker Compose, Render, Railway, AWS App Runner, and GCP Cloud Run deployment assets.

Does justfastllm support Ollama?

Yes. Ollama is supported through its OpenAI-compatible API surface, and the project includes a live Ollama integration test for real model verification.