Pytest — Unit & Mock Testing for Python AI Agents

Core Concepts

Foundational principles of agent unit testing frameworks.

Isolated Mocking

Intercept LLM calls or API boundaries using libraries like `unittest.mock` or `pytest-mock` to test structural logic without burning real API credits.

HTTP Interception

Mock external HTTP endpoints called by agent tools using `respx`, mapping specific URL queries to static JSON payloads cleanly.

Parameterized Tests

Execute a single test case across a dataset matrix using `@pytest.mark.parametrize` to check multiple tool outputs quickly.

Semantic prompt check

Assert that prompt templates are built correctly. Validate that input attributes are correctly formatted into strings.

Exception Verification

Assert that agents fail gracefully when tools return network timeout codes, database failures, or rate-limit warnings.

Schema Compliance

Validate output JSON properties against strict Pydantic model configurations in test assertions to enforce structure.

Setup & Installation

Configure pytest packages in your development workspace.

Install testing packages

                    # Install pytest along with async, mock, and http testing tools

                    pip install pytest pytest-asyncio pytest-mock respx

Agent Test Code Example

Parameterized tests, mocking HTTP requests via respx, asserting exceptions, and checking prompt formatting.

                    import pytest

                    from unittest.mock import AsyncMock

                    import respx

                    from httpx import Response

                    from my_agent import SimpleAgent, QueryTool, AgentException

                    # 1. Define fixture for clean Agent instance

                    @pytest.fixture

                    def agent():

                        tool = QueryTool(api_url="https://kb.internal/search")

                        return SimpleAgent(tools=[tool])

                    # 2. Parameterized testing of prompt outputs

                    @pytest.mark.parametrize(

                        "query,expected_keyword",

                        [

                            ("Deploy failed", "Deploy"),

                            ("Database error", "Database")

                        ]

                    )

                    def test_prompt_formatting(agent, query, expected_keyword):

                        prompt = agent.build_prompt(query)

                        assert expected_keyword in prompt

                    # 3. Test HTTP tool interception using respx mock router

                    @pytest.mark.asyncio

                    @respx.mock

                    async def test_agent_http_tool(agent, mocker):

                        mock_route = respx.get("https://kb.internal/search?q=logs").mock(

                            return_value=Response(200, json={"snippet": "DB primary key constraint"})

                        )

                        mock_llm = mocker.patch("my_agent.LLMClient.acomplete", new_callable=AsyncMock)

                        mock_llm.return_value = '{"status": "failed", "reason": "duplicate constraints"}'

                        result = await agent.run("logs")

                        assert mock_route.called

                        assert result["status"] == "failed"

                    # 4. Assert exceptions are raised gracefully

                    @pytest.mark.asyncio

                    async def test_agent_error_raised(agent, mocker):

                        mocker.patch("my_agent.LLMClient.acomplete", side_effect=ValueError("LLM timeout"))

                        with pytest.raises(AgentException):

                            await agent.run("trigger_timeout")

Frequently Asked Questions

Why should we use pytest-asyncio for agent testing?▶

Modern agent tool loops are fundamentally asynchronous (handling HTTP requests, DB queries, webhooks). Pytest requires `pytest-asyncio` so that you can tag tests with `@pytest.mark.asyncio`, allowing them to execute inside a clean Event Loop scope naturally.

How do I mock external LLM client APIs safely?▶

Use `mocker.patch` from the `pytest-mock` library targeting the low-level API call method of your LLM client (such as `google.generativeai.GenerativeModel.generate_content`). Swap the real execution client out for an `AsyncMock` returning static text or structured dictionaries.

How do I parameterize agent tests in pytest?▶

Use the `@pytest.mark.parametrize` decorator. This lets you define multiple input arguments and expected outputs for a single test function, executing separate testing threads sequentially to ensure prompt variations generate compliant results.

How do I mock external APIs that my agent tools call?▶

For tools invoking HTTP clients like `httpx`, use `respx.mock`. It allows you to mock HTTP responses by targeting matching URLs, status codes, and JSON response bodies without patching the high-level Python code variables directly.

What is semantic prompt testing?▶

Asserting that prompt templates format dynamic strings correctly (e.g. system parameters or history context tags) prior to model invocation, preventing broken variables from reaching LLMs.

How do I check that my agent raises an exception?▶

Wrap your execution call inside a `with pytest.raises(ExpectedException):` context block. If the agent does not throw the targeted exception class (e.g., when tool timeouts occur), the test automatically fails.

Testing with Pytest