The March of Nines: Achieving Enterprise-Grade AI Reliability or AI Reliability: The Compounding Effort of the “March of Nines”

The pursuit of reliable automated systems—whether self-driving cars or enterprise workflows—often begins with a promising demonstration. But a functional demo, achieving around 90% reliability, represents only the initial hurdle in a far more challenging process. Andrej Karpathy, a leading figure in artificial intelligence, frames this reality as the “March of Nines,” where each subsequent increment in reliability—from 90% to 99%, 99.9%, and beyond—demands comparable engineering effort to the initial leap. This concept underscores a critical point: achieving truly dependable software requires far more than simply getting something to “usually work.”

Karpathy’s observation, initially shared in a conversation with Dwarkesh Patel and further detailed in recent analyses from sources like VentureBeat and Superagent.sh, isn’t a dismissal of early progress, but a realistic assessment of the engineering complexities involved in scaling from a proof-of-concept to a production-ready system. The gap between a working demo and a reliable product is vast, particularly in domains where failures carry significant consequences.

The Compounding Math of Reliability

The difficulty isn’t linear. As Karpathy succinctly puts it, “Every single nine is the same amount of work.” This is particularly true in complex “agentic” workflows—systems designed to perform tasks autonomously through a series of steps. These workflows typically involve multiple stages: understanding intent, retrieving relevant information, planning a course of action, executing tools, validating results, formatting output, and maintaining audit logs. Each step introduces a potential point of failure, and the overall success rate is the product of the individual step success rates.

Consider a 10-step workflow. If each step has a 90% success rate, the overall workflow succeeds only about 35% of the time (0.9¹⁰ = 0.3487). This means roughly 65% of attempts will fail. Increasing reliability to 99% per step improves the overall success rate to 90.44%, but still leaves nearly 10% of workflows interrupted. Reaching 99.9% success per step yields a 99% overall success rate, but even then, failures occur frequently enough to sense unreliable. It’s only when reaching 99.99% per step—a substantial engineering undertaking—that the system begins to feel truly dependable, with failures occurring only about once every 3.3 months.

These compounding failures are exacerbated by correlated outages—shared dependencies like authentication services, rate limits, or external connectors—that can bring down multiple steps simultaneously. Addressing these requires hardening shared infrastructure and building in redundancy.

Defining and Measuring Reliability with SLOs

Simply aiming for “higher reliability” isn’t enough. Teams necessitate to define reliability as measurable objectives, known as Service Level Objectives (SLOs). Karpathy emphasizes the importance of concrete prompts and, by extension, concrete metrics. Establishing SLOs allows teams to focus their efforts and track progress effectively.

Key metrics to track include workflow completion rates, tool-call success rates (including timeout handling and schema validation), the rate of schema-valid output, policy compliance (ensuring adherence to data privacy and security constraints), end-to-end latency, and fallback rates (how often the system gracefully degrades to a safer mode, such as human review). These SLIs should be tiered based on the impact of the workflow, allowing for different SLO targets for low-, medium-, and high-impact tasks.

Nine Levers for Adding Nines

Several engineering practices can contribute to achieving these higher levels of reliability. Superagent.sh and VentureBeat outline nine key strategies:

1. Constrain autonomy with an explicit workflow graph: Reliability increases when the system operates within bounded states and handles retries, timeouts, and failures in a deterministic manner. This involves defining a clear state machine or directed acyclic graph (DAG) for the workflow.
2. Enforce contracts at every boundary: Interface drift—malformed data, missing fields, or incorrect units—is a common source of failure. Using schemas like JSON Schema or Protocol Buffers to validate data at every step can prevent these issues.
3. Layer validators: Beyond schema validation, semantic and business rule checks can prevent plausible but incorrect answers.
4. Route by risk using uncertainty signals: High-impact actions should be subject to higher assurance. Using confidence scores to route tasks can ensure that risky operations receive additional verification.
5. Engineer tool calls like distributed systems: Treating external tool calls as distributed systems components—with timeouts, backoff mechanisms, circuit breakers, and concurrency limits—improves resilience.
6. Make retrieval predictable and observable: The quality of information retrieval is crucial. Tracking metrics like empty retrieval rates and document freshness can help identify and address issues.
7. Build a production evaluation pipeline: Continuous evaluation is essential for identifying rare failures and preventing regressions.
8. Invest in observability and operational response: Fast diagnosis and remediation are critical when failures do occur. Comprehensive logging, tracing, and runbooks can streamline the response process.
9. Ship an autonomy slider with deterministic fallbacks: Allowing for adjustable levels of autonomy and providing safe fallback mechanisms—such as human review or cached responses—can mitigate risk.

The Enterprise Imperative

The drive for higher reliability isn’t merely a technical exercise; it’s a business imperative. A 2025 McKinsey survey found that over half of organizations using AI have experienced negative consequences due to its use, with nearly a third attributing those consequences to inaccuracies. This underscores the need for stronger measurement, guardrails, and operational controls.

achieving the later “nines” in reliability requires a disciplined engineering approach focused on bounded workflows, strict interfaces, resilient dependencies, and rapid operational learning. It’s a process that demands sustained investment and a commitment to continuous improvement.

To begin, teams can focus on instrumenting a key workflow, defining its completion SLO, and adding validation checks around model outputs and tool interactions. Treating connectors and retrieval as critical reliability components, and implementing risk-based routing, are similarly essential steps. Every incident should be treated as an opportunity to improve the system and prevent future occurrences.