CI/CD Pipelines & SWE Deployment Stages (Beta, Gamma, Prod)

01 Testing Best Practices in CI/CD: Unit vs Integration Tests

Quality Gates

A robust pipeline implements the Test Pyramid to catch bugs early, keep CI fast, and prevent broken code from ever reaching cloud staging environments:

Layer 1: Unit Tests < 30 Seconds

Isolated & Deterministic

Tests functions, business logic, and schema parsing in pure memory. Never make real network calls to LLM providers (OpenAI/Anthropic) or cloud APIs in unit tests.

Mock HTTP APIs with respx or pytest-mock
Run in parallel using pytest -n auto (pytest-xdist)
Enforce strict code coverage thresholds (e.g. >85%)

Layer 2: Integration Tests 1 - 3 Minutes

Ephemeral Service Containers

Validates that your code correctly queries real relational databases, caches, and queues. Spun up natively in GitHub Actions using services: containers.

Spins up real postgres:16-alpine and redis:7-alpine
Runs database schema migrations (Alembic / Prisma)
Tests database transactions, rollback handling, and TTLs

Layer 3: AI Evals & E2E Gated on Merge

Semantic & Regression Evals

For AI systems, standard asserts aren't enough. Evaluate prompt changes and agent tool calling trajectories against golden benchmark datasets (using DeepEval or Ragas).

Gate production merges on hallucination & relevance metrics
Execute synthetic HTTP smoke tests post-deployment
Run asynchronously on PRs to prevent blocking fast developer loops

Unit vs Integration Testing Implementation (Python & Pytest)

# 1. UNIT TEST: Fast, mocked external LLM API (zero API costs, runs in 5ms)
import pytest
from unittest.mock import AsyncMock, patch

@pytest.mark.asyncio
async def test_agent_tool_routing():
    with patch("app.services.llm_client.chat_completion", new_callable=AsyncMock) as mock_llm:
        mock_llm.return_value = {"tool_call": "fetch_stock", "args": {"ticker": "NVDA"}}
        decision = await route_query("What is the price of NVDA?")
        assert decision.tool == "fetch_stock"
        assert decision.args["ticker"] == "NVDA"

# 2. INTEGRATION TEST: Tests real PostgreSQL queries inside GitHub Actions service container
import pytest
import asyncpg

@pytest.mark.asyncio
async def test_database_persistence():
    # Connects to localhost:5432 spawned by GitHub Actions 'services:' block
    conn = await asyncpg.connect("postgresql://postgres:test_pass@localhost:5432/agent_test")
    await conn.execute("INSERT INTO user_sessions (user_id, status) VALUES ($1, $2)", "user_101", "ACTIVE")
    row = await conn.fetchrow("SELECT status FROM user_sessions WHERE user_id = $1", "user_101")
    assert row["status"] == "ACTIVE"
    await conn.close()

02 Keyless Cloud Auth: AWS OIDC vs GCP Workload Identity

Zero-Secret Standard

Never store static cloud credentials (AWS_SECRET_ACCESS_KEY or GCP Service Account JSON keys) in GitHub Secrets. Modern cloud pipelines use OpenID Connect (OIDC) to exchange short-lived tokens:

AWS: IAM OIDC Provider

AWS STS

GitHub mints a short-lived JWT; AWS STS verifies the signature and issues temporary IAM credentials scoped strictly to your repository branch.

- name: Configure AWS Credentials (OIDC)
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/github-deploy-role
    aws-region: us-east-1
    audience: sts.amazonaws.com

GCP: Workload Identity Federation (WIF)

GCP WIF

GCP Workload Identity Pool validates the GitHub token and impersonates a designated Google Service Account with zero stored JSON keys.

- name: Authenticate to Google Cloud (WIF)
  uses: google-github-actions/auth@v2
  with:
    workload_identity_provider: 'projects/123/locations/global/workloadIdentityPools/github-pool/providers/github-provider'
    service_account: 'github-deployer@my-project.iam.gserviceaccount.com'

03 Production AWS Pipeline: ECR & ECS Fargate

AWS Workflow

This end-to-end workflow runs integration tests with a PostgreSQL service container, scans secrets with TruffleHog, builds a multi-stage Docker container with BuildKit caching, pushes to Amazon ECR with Git SHA tags, and deploys to AWS ECS Fargate with zero downtime.

name: CI/CD - AWS ECS Deployment

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

permissions:
  id-token: write  # Crucial for OIDC keyless authentication
  contents: read

env:
  AWS_REGION: us-east-1
  AWS_ROLE_ARN: arn:aws:iam::123456789012:role/github-actions-ecs-deploy-role
  ECR_REPOSITORY: my-company/app
  ECS_CLUSTER: production-cluster
  ECS_SERVICE: app-service
  ECS_TASK_DEFINITION: .aws/task-definition.json
  CONTAINER_NAME: web-app

jobs:
  # -------------------------------------------------------------
  # STAGE 1: Automated Unit & Integration Tests (With DB Service)
  # -------------------------------------------------------------
  test-and-lint:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_DB: agent_test
          POSTGRES_PASSWORD: test_pass
        ports:
          - 5432:5432
        options: --health-cmd pg_isready --health-interval 5s --health-timeout 5s --health-retries 5

    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Secret Scan (TruffleHog)
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: ${{ github.event.before }}
          head: HEAD

      - name: Set up Python with Pip Cache
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install Test Dependencies
        run: pip install -r requirements-dev.txt

      - name: Run Unit & Integration Tests
        env:
          DATABASE_URL: postgresql://postgres:test_pass@localhost:5432/agent_test
        run: pytest -v --cov=app

  # -------------------------------------------------------------
  # STAGE 2: Build & Zero-Downtime Deployment to AWS ECS
  # -------------------------------------------------------------
  deploy-aws:
    needs: test-and-lint
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Configure AWS Credentials (Keyless OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ env.AWS_ROLE_ARN }}
          aws-region: ${{ env.AWS_REGION }}
          audience: sts.amazonaws.com

      - name: Log in to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Set up Docker Buildx (Layer Caching)
        uses: docker/setup-buildx-action@v3

      - name: Build and Push Docker Image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            ${{ steps.login-ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}:${{ github.sha }}
            ${{ steps.login-ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Render New ECS Task Definition
        id: task-def
        uses: aws-actions/amazon-ecs-render-task-definition@v1
        with:
          task-definition: ${{ env.ECS_TASK_DEFINITION }}
          container-name: ${{ env.CONTAINER_NAME }}
          image: ${{ steps.login-ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}:${{ github.sha }}

      - name: Deploy Task Definition to ECS Fargate
        uses: aws-actions/amazon-ecs-deploy-task-definition@v2
        with:
          task-definition: ${{ steps.task-def.outputs.task-definition }}
          service: ${{ env.ECS_SERVICE }}
          cluster: ${{ env.ECS_CLUSTER }}
          wait-for-service-stability: true # Ensures zero downtime!

04 Production GCP Pipeline: Artifact Registry & Cloud Run

Google Cloud Workflow

This workflow deploys serverless containers to Google Cloud Run using Workload Identity Federation and Google Artifact Registry, with automated canary traffic routing:

name: CI/CD - Google Cloud Run Deployment

on:
  push:
    branches: [ main ]

permissions:
  id-token: write  # Required for GCP Workload Identity Federation
  contents: read

env:
  PROJECT_ID: my-gcp-project
  GAR_LOCATION: us-central1
  GAR_REPOSITORY: cloud-run-source-deploy
  SERVICE_NAME: ai-agent-api
  IMAGE_NAME: us-central1-docker.pkg.dev/my-gcp-project/cloud-run-source-deploy/api

jobs:
  deploy-gcp:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      # Step 1: Keyless Auth via Workload Identity Federation
      - name: Authenticate to Google Cloud (WIF)
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: 'projects/123456789/locations/global/workloadIdentityPools/github-pool/providers/github-provider'
          service_account: 'github-deployer@my-gcp-project.iam.gserviceaccount.com'

      # Step 2: Configure Docker with Artifact Registry credentials
      - name: Configure Docker for Google Artifact Registry
        run: gcloud auth configure-docker ${{ env.GAR_LOCATION }}-docker.pkg.dev --quiet

      # Step 3: Build & Push Container to Artifact Registry
      - name: Build and Push Container
        run: |
          docker build -t ${{ env.IMAGE_NAME }}:${{ github.sha }} -t ${{ env.IMAGE_NAME }}:latest .
          docker push ${{ env.IMAGE_NAME }}:${{ github.sha }}

      # Step 4: Deploy new Revision to Google Cloud Run
      - name: Deploy to Cloud Run
        id: deploy
        uses: google-github-actions/deploy-cloudrun@v2
        with:
          service: ${{ env.SERVICE_NAME }}
          image: ${{ env.IMAGE_NAME }}:${{ github.sha }}
          region: ${{ env.GAR_LOCATION }}
          # Zero-downtime revision rollout with immediate 100% traffic assignment
          flags: '--concurrency=80 --cpu=2 --memory=2Gi --min-instances=1'

      # Step 5: Verify Live Endpoint Status
      - name: Test Cloud Run Endpoint
        run: |
          URL="${{ steps.deploy.outputs.url }}"
          curl -f "$URL/health" || exit 1
          echo "Cloud Run revision live and healthy at $URL!"

05 SWE Pipeline Promotion Stages: Beta, Gamma & Production Waves

Tier-1 SWE Standard

In Tier-1 software engineering organizations (Amazon, Google, Netflix), code never jumps directly from a developer PR to a shared "staging" server and straight to production. Instead, enterprise delivery relies on progressive blast-radius containment through distinct, strictly isolated stages: Beta, Gamma, and Production Waves. Each stage proves a higher degree of functional correctness, operational resilience, and scale under automated quality gates.

Progressive Delivery Pipeline Topology

Immutable Artifact: SHA Tagged → Promoted Across All Environments

Stage 0 CI Runner

Build & Unit Tests

Fast linting, unit tests, and single Docker build tagged with Git SHA.

Artifact: sha-4f81c9a

Stage 1 Auto Gate

Beta Environment

Automated integration testing against ephemeral Postgres & mock APIs.

Gate: 100% Tests Pass

Stage 2 1:1 Prod Parity

Gamma Pre-Prod

Shadow traffic, synthetic load spikes, and schema migration validation.

Gate: Zero Latency Drift

Stage 3 5% Traffic

Prod Canary / OneBox

Live customer traffic on 1 host/task. 15-30 min automated soak period.

Gate: SLO / 5xx Telemetry

Stage 4 100% Fleet

Regional Prod Waves

Wave 1 (us-east-2) → Wave 2 (us-east-1) → Wave 3 (Global Fleet).

Gate: Auto Circuit Breaker

Stage 1: Beta (Functional Verification)

Dev & Merge Loop

Beta is the first continuous deployment target after code is merged to the main branch. Its goal is functional completeness: ensuring that multiple engineers' concurrent changes merge cleanly and that API contracts between upstream and downstream services are honored.

• Topology & Scale: Lightweight cluster or serverless container pool. Downstream third-party systems (payment gateways, external LLMs) are typically stubbed or connected to developer sandboxes.

• Quality Gates: Automated end-to-end (E2E) smoke tests, API contract assertions (Pact / OpenAPI spec checks), and ephemeral database integration suites.

• Promotion Rule: 100% pass rate triggers immediate, zero-touch progression to Gamma.

Stage 2: Gamma (1:1 Pre-Prod Parity)

Pre-Production

Gamma is a production-identical staging environment (exact VPC network topology, IAM roles, database engine version, instance sizes, and multi-region replication). Gamma proves that code will survive in production under real-world load and scale.

• Shadow Traffic (Dual-Write): Production ingress proxies (Envoy / AWS ALB Target Group Mirroring) duplicate a percentage of read-only customer traffic to Gamma, comparing HTTP response codes and latencies against live prod.

• Database Migration Testing: Database migrations are executed against sanitized, full-scale production-sized database snapshots to measure table lock durations and index construction overhead.

• Synthetic Stress Testing: High-throughput load tests (Locust / k6) verify thread pool contention, database connection pool exhaustion, and memory leaks.

Stage 3: Prod Wave 0 (Canary / OneBox)

Blast Radius < 5%

Even the most rigorous Gamma testing cannot replicate unexpected customer edge cases. Prod Wave 0 exposes the new build to real customer traffic on a single container/host (OneBox) or a 2% to 5% weighted routing split within a single Availability Zone.

• Automated Soak / Bake Time: The canary build is locked in place for 15 to 45 minutes under passive observation while background telemetry monitors run.

• Automated Metric Comparison: Automated queries evaluate canary health vs baseline fleet: HTTP 5xx error rate, P95/P99 latency, container restarts, and unhandled exception counters.

• Instant Auto-Rollback: If any alarm breaches its error budget threshold, the pipeline automatically sets canary traffic back to 0% with zero operator intervention.

Stage 4: Prod Waves 1..N (Regional Fleet Delivery)

Global Rollout

Once the canary bake phase concludes cleanly, the deployment advances through sequential Regional Waves. Staggering deployments prevents simultaneous outages across multiple geographic data centers.

• Wave 1 (Low-Volume Region): Deployed to a single geographic region (e.g. us-east-2 Ohio) with 30-minute soak period.

• Wave 2 (High-Volume Hubs): Deployed sequentially to major regions (e.g. us-east-1 Virginia, then eu-west-1 Ireland).

• Wave 3 (Global Fleet Completion): Rolled out to remaining global edge locations and secondary regions.

SWE Stage Comparison Matrix

Blast Radius & Validation Requirements

Stage	Environment Parity	Traffic Source	Key Validations	Bake / Soak Time	Rollback Trigger
Beta	Functional subset; stubbed dependencies	Synthetic CI test runners	Integration tests, schema checks, E2E smoke tests	None (immediate on pass)	Failed test assertion halts pipeline
Gamma	1:1 Exact Production replica & scale	Shadow / mirrored traffic + synthetic load	Stress testing, migration on prod-sized DB, contract diffs	15 – 30 mins (load soak)	Latency regression or migration failure
Prod Wave 0 (Canary)	Live Production (1 task / 2-5% traffic)	Real customer traffic	Live telemetry comparison (P99 latency, 5xx rate, CPU)	15 – 45 mins	Automated circuit-breaker rollback
Prod Waves 1..N	Full Global Production Fleet	100% Customer traffic	Regional SLO metrics, cross-region health probes	30 – 60 mins between waves	Regional automated rollback

Complete SWE Multi-Stage Promotion Workflow (GitHub Actions)

Implements immutable image promotion across Beta, Gamma, and Prod Canary with automated soak probing.

.github/workflows/swe-promotion.yml

name: SWE Pipeline - Progressive Stage Promotion

on:
  push:
    branches: [ main ]

permissions:
  id-token: write  # Keyless AWS/GCP authentication
  contents: read

jobs:
  # =============================================================
  # STAGE 0: Build Once, Tag with Immutable Commit SHA
  # =============================================================
  build-and-package:
    runs-on: ubuntu-latest
    outputs:
      image-tag: ${{ github.sha }}
      image-uri: 123456789012.dkr.ecr.us-east-1.amazonaws.com/core-service:${{ github.sha }}
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Configure AWS Credentials (Keyless OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/ci-builder-role
          aws-region: us-east-1

      - name: Log in to ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and Push Immutable Container
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: 123456789012.dkr.ecr.us-east-1.amazonaws.com/core-service:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  # =============================================================
  # STAGE 1: Deploy to Beta & Run Integration / E2E Suites
  # =============================================================
  deploy-beta:
    needs: build-and-package
    environment: beta
    runs-on: ubuntu-latest
    steps:
      - name: Configure AWS Credentials (Beta Role)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy-beta-role
          aws-region: us-east-1

      - name: Update Beta ECS Service
        run: |
          echo "Deploying ${{ needs.build-and-package.outputs.image-uri }} to Beta..."
          aws ecs update-service --cluster beta-cluster --service app-service --force-new-deployment

      - name: Run Automated Beta Integration & Smoke Tests
        run: |
          echo "Running API contract and functional smoke suites against Beta..."
          curl -f --retry 5 --retry-delay 5 https://beta-api.internal.company.com/health
          pytest tests/integration/ --base-url=https://beta-api.internal.company.com

  # =============================================================
  # STAGE 2: Deploy to Gamma (1:1 Prod Parity) & Run Stress Evals
  # =============================================================
  deploy-gamma:
    needs: [build-and-package, deploy-beta]
    environment: gamma
    runs-on: ubuntu-latest
    steps:
      - name: Configure AWS Credentials (Gamma Role)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy-gamma-role
          aws-region: us-east-1

      - name: Deploy to Gamma Environment
        run: |
          aws ecs update-service --cluster gamma-cluster --service app-service --force-new-deployment
          aws ecs wait services-stable --cluster gamma-cluster --services app-service

      - name: Run Schema Migration Dry-Run & Synthetic Stress Tests
        run: |
          echo "Validating database schema compatibility on sanitized prod snapshot..."
          alembic check
          echo "Executing high-concurrency synthetic stress probe..."
          k6 run tests/load/gamma-smoke-load.js

  # =============================================================
  # STAGE 3: Prod Wave 0 (Canary / OneBox) with 15-Minute Soak
  # =============================================================
  deploy-prod-canary:
    needs: [build-and-package, deploy-gamma]
    environment: production-canary
    runs-on: ubuntu-latest
    steps:
      - name: Configure AWS Credentials (Prod Deployer)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy-prod-role
          aws-region: us-east-1

      - name: Shift 5% Traffic to Canary Revision
        run: |
          echo "Routing 5% of production ingress traffic to SHA ${{ needs.build-and-package.outputs.image-tag }}..."
          aws elbv2 modify-listener --listener-arn $ALB_LISTENER_ARN \
            --default-actions Type=forward,ForwardConfig='{TargetGroups=[{TargetGroupArn="'$TG_PROD_ARN'",Weight=95},{TargetGroupArn="'$TG_CANARY_ARN'",Weight=5}]}'

      - name: Automated Telemetry Bake Period (15-Minute Soak)
        run: |
          echo "Canary live. Monitoring CloudWatch P99 latency and 5xx alarms for 15 minutes..."
          for i in {1..15}; do
            ALARM_STATE=$(aws cloudwatch describe-alarms --alarm-names "Prod-Canary-High5xxErrors" --query "MetricAlarms[0].StateValue" --output text)
            if [ "$ALARM_STATE" == "ALARM" ]; then
              echo "CRITICAL: Canary health alarm breached! Triggering automated circuit-breaker rollback!"
              # Rollback traffic to 100% baseline target group instantly
              aws elbv2 modify-listener --listener-arn $ALB_LISTENER_ARN \
                --default-actions Type=forward,TargetGroupArn=$TG_PROD_ARN
              exit 1
            fi
            sleep 60
          done
          echo "Canary passed 15-minute bake period with zero alarm breaches!"

  # =============================================================
  # STAGE 4: Prod Waves (Promote 100% Across Regional Fleet)
  # =============================================================
  deploy-prod-fleet:
    needs: [build-and-package, deploy-prod-canary]
    environment: production-fleet
    runs-on: ubuntu-latest
    steps:
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy-prod-role
          aws-region: us-east-1

      - name: Promote 100% Traffic to New Revision Fleet
        run: |
          echo "Promoting SHA ${{ needs.build-and-package.outputs.image-tag }} to 100% production traffic..."
          aws elbv2 modify-listener --listener-arn $ALB_LISTENER_ARN \
            --default-actions Type=forward,TargetGroupArn=$TG_CANARY_ARN
          echo "Global production promotion complete with zero downtime!"

1. Build Once, Promote SHA

Immutable Artifacts

Never rebuild a container image when moving from Beta to Gamma or Prod. Build once in Stage 0, sign the image, and promote that exact image digest. Rebuilding introduces compiler or dependency drift.

2. Automated Circuit Breakers

Zero Operator Panic

Rollbacks must never wait for an engineer to open a laptop or read a pager. CloudWatch or Datadog latency/5xx alarms must automatically trigger rollback webhooks within 30 seconds of an anomaly.

3. Shadow Traffic Mirroring

Safe High-Scale Testing

In Gamma, use reverse proxy request mirroring (Envoy / NGINX / ALB) to duplicate live read requests asynchronously. Responses are discarded, letting you test production query plans without customer impact.

4. Cell-Based Regional Waves

Bounded Blast Radius

Deploy one region at a time. If an issue slips through Canary, only a single regional cell is affected, leaving other global regions operational to handle failover traffic.

06 Essential Pipeline Engineering Practices

Production Standards

1. Concurrency Control

Cancel Redundant Builds

Add concurrency: { group: ..., cancel-in-progress: true }. When developers push 3 commits in rapid succession, CI automatically cancels the older 2 runs to save compute minutes.

2. Immutable Tags

Never Deploy :latest

Always tag container images with the immutable Git commit SHA (${{ github.sha }}). If production crashes, you know the exact line of code that caused the regression and can roll back instantly.

3. Multi-Layer Caching

Pip + Docker BuildKit

Combine language package caching (cache: 'pip' or uv) with Docker BuildKit GitHub Actions cache (cache-from: type=gha) to slash build times from 8 minutes to 45 seconds.

4. Expand-Contract Migrations

Zero-Downtime Schemas

Never drop columns or tables in the same deployment as code changes. Use the Expand-Contract pattern: Add nullable field → Dual-write → Read new field → Drop old column in next sprint.

07 Cloud Deployment Comparison: AWS vs Google Cloud

Feature	AWS (ECS Fargate / Lambda)	GCP (Cloud Run / GKE)
Keyless Auth	IAM OIDC Identity Provider + STS AssumeRole	Workload Identity Federation (WIF) Pool
Container Registry	Amazon Elastic Container Registry (ECR)	Google Artifact Registry (*.pkg.dev)
Container Orchestration	AWS ECS Fargate (Task definitions, ALB routing)	Google Cloud Run (Serverless container revisions)
Canary & Traffic Splitting	AWS CodeDeploy / ALB Weighted Target Groups	Native Cloud Run Revision Traffic Splitting (e.g. 90/10)
Scale-to-Zero	Lambda (Native) / ECS (requires custom autoscaling)	Native Cloud Run scale-to-zero in milliseconds

08 Frequently Asked Questions

What is the difference between Beta, Gamma, and Staging in Tier-1 SWE pipelines? →

In basic pipelines, "Staging" is often a poorly scaled, shared environment where unverified code causes noisy neighbor failures. In Tier-1 SWE architecture (like Amazon Pipelines), Beta is an automated functional verification stage with stubbed external dependencies, while Gamma is a 1:1 replica of Production in terms of network architecture, IAM policies, and database sizing. Gamma runs shadow production traffic (dual-writes) and synthetic stress tests to guarantee performance and compatibility before any real customer traffic touches the release.

How do OneBox/Canary deployments and automated soak times prevent outages? →

Instead of rolling out to 100% of hosts at once, a OneBox deployment directs traffic to a single instance or 2-5% of real user requests. The automated pipeline holds this canary state for a mandatory 15-to-45-minute soak (bake) time while monitoring CloudWatch or Datadog telemetry for 5xx error spikes, P99 latency increases, and memory leaks. If any anomaly breaches an error budget, the circuit breaker instantly reverts traffic to the baseline version with zero human intervention.

How do we mock external LLMs in CI/CD without paying token fees? →

Use respx or vcrpy in your test suite. vcrpy records real HTTP responses from OpenAI or Anthropic once into a YAML cassette file, and subsequent test runs replay those exact responses locally with zero network calls and zero cost.

How does GCP Workload Identity Federation differ from AWS OIDC? →

Both use standard OIDC tokens minted by GitHub Actions. In AWS, the token is sent directly to AWS STS via sts:AssumeRoleWithWebIdentity. In GCP, the token is verified by a Workload Identity Pool, which exchanges it for a federated token that impersonates a Google Service Account via the Security Token Service (STS).

How do we handle environment promotion (Dev → Staging → Production)? →

Use GitHub Environments. Create separate environments for beta, gamma, production-canary, and production-fleet. Never rebuild the Docker image between environments: build and tag the container image once with ${{ github.sha }}, promote that exact image SHA to Beta → Gamma → Canary → Fleet.

What is the best way to run database migrations during continuous deployment? →

Run migrations in an isolated CI/CD step (or via an ephemeral ECS one-shot task / Cloud Run job) right before updating the main web service. The migrations must strictly follow backward-compatible schemas (additive only via the Expand-Contract pattern) so existing running containers don't crash while new ones start up.

CI/CD Pipelines & SWE Stages: Beta, Gamma & Production Waves