CI/CD Pipelines for AI Agents

Core Concepts

Key steps in validating and delivering agent code to target servers.

Dependency Caching

Persist pip download directories across workflow runs using caching keys, reducing pipeline execution latency significantly.

Secret Leak Scans

Integrate security checkers (like TruffleHog) inside early stages to inspect commits for exposed model credentials before pushing code.

Docker Multi-Stage

Configure multi-stage Docker builds to compile binaries in builder images, exporting only clean runtimes to final images.

Integration Services

Spin up dependency services (like PostgreSQL or Redis) as background containers inside runner environments to run full integration tests.

Gated Deployments

Prevent production releases if security scans (like credentials checking) or metric evaluations (like hallucination tests) fail.

Artifact Management

Archive build manifests, test report files, and evaluation dataset histories to track performance curves over version increments.

Setup & Installation

Prerequisites for setting up GitHub Actions workflows.

Environment configuration

CI/CD workflows run in standard runner instances (e.g. `ubuntu-latest`). You do not need to install complex local servers. Standard configurations declare tools directly in `.github/workflows/deploy.yml` configurations.

GitHub Actions Workflow Example

A YAML pipeline configuration that implements dependencies caching, secret scans, integration database service, unit testing, and Docker builds.

                    name: Production CI/CD Pipeline

                    on:

                      push:

                        branches: [ main ]

                    jobs:

                      verify:

                        runs-on: ubuntu-latest

                        # Set up database services in runner container

                        services:

                          postgres:

                            image: postgres:15-alpine

                            env:

                              POSTGRES_DB: agent_test

                              POSTGRES_PASSWORD: test_pass

                            ports:

                              - 5432:5432

                            options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5

                        steps:

                          - name: Checkout code

                            uses: actions/checkout@v4

                          - name: Scan Secrets (TruffleHog)

                            uses: trufflesecurity/trufflehog@main

                            with:

                              path: ./

                              base: ${{ github.event.before }}

                              head: HEAD

                              extra_args: --debug

                          - name: Set up Python 3.11 with Caching

                            uses: actions/setup-python@v5

                            with:

                              python-version: '3.11'

                              cache: 'pip' # Automatically caches download dir

                          - name: Install dependencies

                            run: pip install -r requirements-dev.txt

                          - name: Run Unit Tests

                            env:

                              DATABASE_URL: postgres://postgres:test_pass@localhost:5432/agent_test

                            run: pytest

                      build-and-deploy:

                        needs: verify

                        runs-on: ubuntu-latest

                        steps:

                          - name: Log in to Docker Registry

                            uses: docker/login-action@v3

                            with:

                              registry: ghcr.io

                              username: ${{ github.actor }}

                              password: ${{ secrets.GITHUB_TOKEN }}

                          - name: Build and Push Docker Image

                            uses: docker/build-push-action@v5

                            with:

                              push: true

                              tags: ghcr.io/${{ github.repository }}/agent:latest

Frequently Asked Questions

Should we run agent evaluations on every git commit?▶

Unit tests (via pytest with mocked endpoints) should run on every commit because they are fast and free. Evaluating LLM behaviors (such as semantic metrics) burns real API tokens and takes longer, so they are typically gated to run only when merging code into the main branch or preparing production release tags.

How do we handle sandbox containers inside CI environments?▶

GitHub Actions support Service Containers, which allow you to spin up database instances (e.g. PostgreSQL, Redis) or mock API endpoints inside the runner cluster context for integration testing alongside unit tests.

How do we speed up python agent runs in GitHub Actions?▶

Utilize dependency caching. Under the `actions/setup-python` setup action step, configure the `cache: 'pip'` attribute. This prompts the runner to cache downloaded wheel binaries, bypassing network fetching steps on subsequent runs.

How do we prevent checking in API secrets?▶

Incorporate secret scanners like `TruffleHog` as the first step in your testing pipeline. These scanners inspect the commit git history for regex matches or high-entropy strings indicating OpenAI keys, failing the run before code is built.

How do we run integration tests requiring databases in CI?▶

Configure a `services` section inside your job definition referencing Docker hub database images (e.g. `postgres:alpine`). The runner spins these container dependencies up and maps ports to `localhost`, letting tests query them.

What is multi-stage Docker building?▶

A Docker build strategy where you install compilers and dev libraries inside a builder stage, copy only clean production assets and dependency wheels to a fresh slim runtime image, keeping final deployment images small.