The LLM Gateway: Build and Operate One in 1 hr

A hands-on session. We build one gateway properly, break it on purpose, put controls on it, and then work through what it takes to operate it as a shared production service.

Most organisations arrive at the same problem. One application talking to one model provider is fine. Four applications talking to four providers is not: every integration carries its own authentication, retry logic, timeout handling, failover behaviour, and log format, implemented separately by teams who have never discussed it. There is no single place to enforce a policy, no shared view of what is failing, and no way to revoke access for one application without affecting the rest.

An LLM gateway turns that sprawl into one controlled, observable access point. This session builds one live, then covers the four things that decide whether it survives contact with production.

Security

Applications stop holding provider credentials entirely.

Reliability

Retries, cooldowns and fallback solve three genuinely different failures, and conflating them is how a provider blip becomes a self-inflicted outage.

Observability

The metrics that actually earn a dashboard: request and failure rates, provider latency, time to first token, rate-limit responses, retry counts, and fallback counts.

DevOps

The gateway configuration contains no secrets, only references, which makes it a deployable artifact that belongs in Git with a pull request, a test and a rollback path.

If you want to follow along

Docker and Docker Compose v2 running, curl and jq installed, and one provider API key. A second provider makes the failover demonstration more convincing but is not required.

Not set up? Join and watch. You will get more out of the session than you would debugging your environment in the background, and the repository is yours either way.

What you take away

The complete repository used on screen: the Docker Compose stack, the gateway configuration, the demo scripts, the CI policy validator, Prometheus alert rules, and on-call runbooks.

Schedule

August 28, 2026 18:30 to 19:30

Recurring on the last Friday of every month

Who Should Join

Cloud engineers, DevOps engineers, platform engineers, SREs, security engineers, architects, developers, engineering managers, and anyone interested in modern infrastructure and software delivery.