Progressive Canary Lab

Overview

Ship it. Watch it. Kill it if it breaks — in seconds, not minutes.

Progressive Canary Lab models one of the core deployment patterns used in large-scale backend systems: shipping code first and activating behavior later. It shows how a team can deploy a new app version without downtime, route a small slice of traffic to it, gate a feature behind a flag for a controlled audience, monitor health signals, and automatically roll back when the canary becomes unhealthy.

Problem

In production systems, deployment and activation are often conflated, making it hard to separate “is the new code running” from “is the new behavior visible to users.” Without a controlled mechanism to shift traffic, gate features, and watch health metrics in real time, teams risk exposing broken behavior to the full user base before it can be caught and rolled back.

What I Built

• Two parallel app services (app-stable on v1, app-canary on v2) with a hidden fraud-detection code path gated behind a feature flag
• A router that proxies checkout traffic between stable and canary, starting at 1% canary traffic and supporting manual rollout/rollback via API
• A flag service holding in-memory feature state, with endpoints to enable, adjust rollout percentage, and kill a feature instantly
• A rollout controller that polls health every 10 seconds, auto-promotes canary traffic through 1 → 5 → 25 → 50 → 100 percent when error rate and p99 latency stay within thresholds, and auto-rolls back with a feature kill switch when they don’t
• A load generator and Makefile-driven demo flows to simulate healthy and unhealthy canary behavior end-to-end
• Dockerized services orchestrated via Docker Compose, with Kubernetes manifests under infra/k8s

Key Outcomes

• Built a full progressive-delivery loop — deploy, canary traffic shift, flag-gated activation, health-based promotion, and automatic rollback — the same pattern behind safe releases at companies operating at scale
• Nailed the distinction that trips up most engineers: a traffic rollback fixes routing, a kill switch fixes behavior — and production incidents need both, fast
• Delivered reproducible failure scenarios with real metrics: clean promotions to 100%, and automatic rollbacks triggered the moment error rates spike