Agent Engineering with Pydantic + Graphs — with Samuel Colvin, CEO of Pydantic Logfire

Summary

• Pydantic’s AI leverage comes from owning Python’s typed-data boundary, not from having been designed for LLMs. Colvin notes nearly 300 million December downloads, while saying one unnamed foundation-model company cut time to first token by 20% after moving from Pydantic v1 to v2. “The AI has kind of come to us,” but that accidental positioning now gives Pydantic unusual distribution across model SDKs and structured-output tooling.

• Pydantic AI is betting that production engineering will distinguish durable agent frameworks from opportunistic wrappers. Colvin criticizes the ecosystem’s weak type checking, coverage, linting and documentation discipline, while acknowledging that early projects such as LangChain were inventing the category. Pydantic AI accepts harder generics in exchange for production safety: every documentation example and printed result runs in tests because these practices should be “no-brainers.”

• Type-safe graphs changed Colvin’s mind about orchestration by turning sprawling control flow into inspectable Python. Nodes return the next typed node, allowing Pydantic to infer valid edges and generate a Mermaid diagram without string-based graph definitions. The standard agent implementation is now itself a graph, although durable state storage and distributed execution remain planned rather than finished.

• Agent frameworks are compensating for model weakness, so their long-term value may fall as models improve. Alessio raises the “bitter lesson”: developers can hand-build reflection and routing graphs, only for a larger trained model to absorb the workflow. Colvin agrees that “the more intelligent they are, the less we need to tell them,” comparing today’s constrained agents with poorly trained call-center staff who need scripts.

• Logfire’s thesis is that AI observability cannot remain separate from application observability. Colvin argues that traces containing model prompts will carry an “order of magnitude” more sensitive information than traditional telemetry, making self-hosting important for some customers. Logfire therefore targets general-purpose observability with first-class AI support, while OpenTelemetry semantic conventions could standardize model-call data across frameworks.

• The infrastructure bet underneath Logfire is technically differentiated but commercially expensive. The team moved from ClickHouse to Timescale and finally DataFusion, favoring Rust extensibility, user-friendly SQL and an open-source architecture over faster time to market. Colvin concedes the choice “definitely slowed us down,” but it avoids surrendering margin to a hosted database vendor and lets Pydantic implement missing kernels itself.

• Management is protecting runway while using open source and browser demos to improve conversion. Pydantic and Pydantic AI remain MIT-licensed, while Logfire is explicitly closed-source and for-profit; Colvin wants more revenue before hiring despite the team effectively running “three to four startups at the same time.” The browser-based pydantic.run aims to remove setup friction by letting prospects execute tested examples; a model proxy is planned, with possible Logfire integration.

Deep dive

1. Pydantic became AI infrastructure almost by accident

• Colvin calls Pydantic a validation library only as an approximation: it uses Python type hints to define schemas, validates inputs and, by default, performs sensible coercions. Asking for an integer and receiving "123" produces 123; strict mode disables that behavior.

• Using type hints this way was “genuinely disapproved of by some people” when Colvin began in 2017. Pydantic’s adoption—and FastAPI’s success on top of it—helped make the pattern conventional across Python.

• JSON Schema was not selected through a structured-output council. Sebastian Ramírez arrived before Pydantic v1 with roughly 50 weekend commits or notifications adding JSON Schema for FastAPI and OpenAPI; that later let one Pydantic definition serve as a source of truth for validation, serialization, tools and LLM structured outputs.

• The payoff from the Rust-based v2 rewrite is measurable beyond benchmarks: an unnamed major foundation-model company reportedly reduced its leading internal latency metric, time to first token, by 20% when upgrading from Pydantic v1 to v2.

2. Pydantic v3 should compound performance without another traumatic rewrite

• Colvin says there “will be a v3 at some point,” but it should break far less code than v2, which combined a large Rust rewrite with accumulated fixes to problems dating back to v0.

• One proposed change is retaining validated data in Rust-native representations until Python objects are actually needed. Validation followed directly by serialization could then avoid a Python conversion entirely, potentially yielding “another three to five times speedup.”

• Further work includes easier custom handling for types such as NumPy arrays, SIMD JSON parsing beyond amd64, and possibly a binary format such as CBOR for storing and reloading Pydantic data.

• Colvin began the Rust rewrite full-time in early 2022 and worked on it for a year and a half. He describes it as the kind of work few startups would approve: three engineers working full-time for a year on about 30,000 lines of Rust for a free library. Yet it kept the company central to Python’s AI stack.

3. Production discipline is Pydantic AI’s wedge

• Pydantic spent much of the prior year and a half building general-purpose Logfire observability rather than chasing AI. Demand pulled it back: Colvin estimates perhaps 80% of new global Python projects involve generative AI—“obviously it’s like 100% in California”—leaving abundant unsolved tooling work.

• His criticism of agent frameworks is pointed but qualified. Early LangChain understandably skipped established practices while discovering the category; some newer releases from respected names look more like “opportunism,” with engineering quality “far below that of the rest of the Python ecosystem.”

• Pydantic AI prioritizes type checking, coverage, linting and executable documentation. Its generics may be easier for developers who understand Rust-style type systems, but Colvin argues that complexity pays off in large production applications where static checking is “a no-brainer.”

• Its basic agent packages a system prompt, tools, optional structured result, dependencies, model and settings. Colvin calls these objects closer to “agentlets”: developers generally compose several into what users would recognize as a full agent.

• The supporting pytest-examples tool extracts Python from docstrings and Markdown, runs it, lints it and will soon type-check it. That is how the team keeps hundreds of runnable, self-contained examples up to date.

4. Typed return values make graphs inspectable rather than stringly typed

• Colvin initially argued that ordinary Python control flow could express any agent workflow. Repeated examples weakened that position because he had maintained the resulting “spaghetti code” and saw value in a structured representation that could explain itself.

• Pydantic’s graph nodes are normally dataclasses whose run methods return the next permitted node. Introspecting those return types and unions reveals the legal edges, avoiding graph APIs that name connections with unchecked strings and compensate through runtime validation.

• The reward is both safety and legibility: type hints alone can produce a Mermaid diagram showing exactly what may happen. “Once we got that right,” Colvin says, he became “incredibly excited about graphs.”

• Pydantic AI’s normal agent implementation was merged onto the graph engine without changing its public API. Underneath, execution remains deliberately simple: call a node, receive another node, repeat until an End value appears.

5. Durable workflows expose what the graph library still lacks

• Colvin’s best example is an e-commerce return workflow whose diagram treats two unlike edges identically: one is an immediate function call; another means waiting six days for a customer to print a label and post a parcel. A proof of concept can hand-wave that pause, but production cannot.

• A graph can resume by instantiating the appropriate node and using it as the new starting point. Pydantic AI already uses that pattern for question-and-answer flows, but database-backed state persistence between every node is still something the team plans to add “soon.”

• swyx’s Temporal-informed pushback is that a Python library does not provide isolated execution or infrastructure resilience. Colvin agrees each unit should eventually run in its own container or worker; serializing the next-node call could distribute execution provided every worker has compatible code.

• Cloudflare Workers running Python through Pyodide is his favored experimental substrate because thousands of isolated functions could start serverlessly. Binary dependency sharing remains difficult, although Rust libraries such as pydantic-core can compile to WebAssembly.

6. Pydantic is testing orchestration demand without claiming an empire

• The hosts frame a progression from single agents through delegation and programmatic handoffs to graph-based control flow. Colvin recounts that OpenAI told him Pydantic AI looked like what Swarm would become “if it were production-ready,” adding, “maybe I’m not supposed to say this.”

• Reproducing Swarm’s intuitive feel helped push him toward graphs because “just call the next agent with Python code” did not satisfy users. Still, after only a couple of weeks he has no canonical top-five graph taxonomy; he wants to see what developers actually build.

• Colvin refuses to declare victory over LangChain: “That’s your job to go find that out, not mine.” Pydantic is currently a library, not the deployment infrastructure offered by Temporal, Prefect, Airflow or Dagster, and he says the usefulness of its minimal graph abstraction remains empirical.

• His strongest control argument is economic: large autonomous runs can consume time and money before failure becomes visible. Iterating node-by-node lets an application inspect progress and break out, rather than relying on a blunt maximum-step error after the model has already wandered.

7. Better models could erase much of today’s agent scaffolding

• Alessio frames the bitter-lesson objection: developers decompose tasks, add reflection and orchestrate inference-time compute, while researchers could train a larger model that absorbs the pattern—for example, turning Strawberry into DeepSeek R1. “We are fighting the bitter lesson here.”

• Colvin broadly agrees. If models became faster and more capable as quickly as enthusiasts imply, applications might simply grant one model internet access, “cross our fingers and hope for the best,” removing much of the need for agents, frameworks and graphs.

• His analogy is customer service: a bank scripts poorly trained call-center workers tightly, while high-net-worth banking hires people expected to charm clients over coffee and gives them discretion. “The more intelligent they are, the less we need to tell them” and constrain their routes.

• Until then, compound systems create useful intervention points. Agents “work well when they work well,” but observability alone does not stop a bad run; explicit nodes give developers places to inspect, redirect or terminate it.

8. Model portability and evals remain unsettled engineering problems

• swyx challenges every framework maintaining adapters for OpenAI, Claude, Google and others, preferring a shared layer such as LiteLLM or Portkey. When he offers DeepSeek as a missing example, Colvin immediately corrects him: Pydantic AI already supports it.

• Colvin’s objections are type safety, uncertain reports about adapter quality and the possibility that normalization proxies requests through another company. He also sees convergence around the OpenAI API: DeepSeek, Grok and Ollama implement it, making the OpenAI SDK a de facto compatibility layer.

• Pydantic AI runs a small set of live-model tests on commits to main. Google’s Generative Language API was reportedly returning 503 errors roughly 20% of the time and repeatedly broke tests, while Vertex exposed a similar API with materially better reliability.

• Pydantic AI also supports test-model and function-model approaches for mocking model behavior without making live calls. On evals, Colvin’s “honest truth” is that he does not know the universal answer. Thirty examples may provide much of the statistical value of 200 at 15% of the work, but choosing the right evaluation design is harder; Logfire therefore exposes SQL so users can explore their own production data rather than inherit one prescribed framework.

9. Logfire joins AI traces to the rest of the application

• Colvin expects “AI observability” to disappear as a category just as web and cloud observability did: every general platform will eventually need it. His wedge is general-purpose observability with first-class AI support, because model-only traces omit too much surrounding application behavior. He also sees developer experience as an opening: Sentry makes Python setup easy, while a basic Datadog/Python setup is not trivial.

• OpenTelemetry is converging on generative-AI semantic attributes, initially centered on individual LLM calls. Agent-level conventions remain immature, and messages are moving toward OTel events; Colvin notes that prompt and completion token fields can fossilize yesterday’s paradigm as reasoning tokens and sampling methods change.

• The deeper issue is data sensitivity. Traditional telemetry tries to omit parameters: a hospital SQL trace need not reveal the patient or STI drug. In generative AI, those facts are entangled in natural-language prompts, so ordinary password-key scrubbing does not work reliably and some customers will require self-hosting.

• Logfire also emits information when a span begins, not only when it ends. Waiting for a 20-second model request—or a 30-minute batch job—to finish before displaying its trace is “crippling”; early span data lets operators watch long-running work as it unfolds.

• Colvin says LangChain has its own observability platform and did not follow the OpenTelemetry route, making external instrumentation more difficult. Pydantic has an advantage because it controls its own agent framework and can implement the conventions directly.

10. DataFusion and disciplined growth define the commercial bet

• Logfire moved from ClickHouse to Timescale and then DataFusion, a journey Colvin admits did not delight his board member. At the time, ClickHouse lacked convenient JSON support and had painful interval and datetime edge cases—especially when users themselves must write duration-heavy SQL.

• DataFusion is “a toolbox to build a database with, not a database.” Its Rust implementation lets Pydantic optimize string-comparison kernels and add JSON support using its own parser; Colvin says he implemented the latter in a weekend.

• The choice also preserves economics: ClickHouse and Timescale offered object-store-plus-cache architectures primarily through hosted products, whose margins would sit beneath Logfire’s own. Colvin concedes DataFusion “definitely slowed us down” versus shipping quickly on ClickHouse, but hopes the open architecture wins long term.

• Pydantic and Pydantic AI are MIT-licensed; Logfire is unambiguously closed-source and for-profit. The open-source browser sandbox pydantic.run is intended to execute tested examples and remove local setup friction; Colvin plans to add a proxy for OpenAI and other models, and may connect it to Logfire. He wants more commercial traction before hiring: “It’s quite nice having a few years of runway, not a few months.”