AI Strategy: Enterprises Slow to Adopt Amidst Quality & Risk Concerns

The rush to integrate artificial intelligence into enterprise workflows may be outpacing a clear understanding of its limitations, and potentially creating problems that won’t be immediately apparent. While many organizations are enthusiastically adopting AI tools, a growing chorus of experts suggests a period of reckoning is coming as the consequences of flawed AI implementations begin to surface.

Dorian Smiley, co-founder and CTO of AI advisory service CodeStrap, argues that many companies are “pretending that they realize” the right way to deploy AI, when in reality, “there’s no playbook to pull from.” Smiley, along with CEO Connor Deeks, previously worked at PwC before establishing CodeStrap to facilitate organizations navigate the complexities of AI strategy. Their assessment points to a critical gap: a lack of established reference architectures and apply cases tailored to specific institutional needs.

The Fallibility of Underlying Data

A core concern, according to Deeks, is the inherent unreliability of the data that fuels these AI systems. “From the large language model perspective, people aren’t really addressing the fallibility of the underlying text,” he said. This isn’t simply a matter of occasional errors. it’s a fundamental limitation of the technology. The models themselves struggle with reliably retrieving and verifying facts, and their reasoning processes are often opaque and non-deterministic – meaning the same prompt can yield different results each time.

This lack of consistency and verifiability extends to code generation. Smiley points out that AI-generated code can appear correct, even pass initial tests, while still containing critical flaws. Traditional metrics like lines of code and pull requests, often used to measure developer productivity, become misleading when evaluating AI-assisted coding. “These are liabilities. These are not measures of engineering excellence,” Smiley stated. He advocates for focusing on metrics like deployment frequency, lead time to production, change failure rate, imply time to restore, and incident severity – indicators of genuine engineering performance.

The SQLite Rewrite Debacle

The potential for these hidden flaws was recently highlighted by an attempt to rewrite the SQLite database in Rust using AI. While the AI-generated code passed all unit tests and appeared structurally sound, it ultimately performed 2,000 times worse than the original SQLite implementation. As Smiley succinctly put it, “It’s a dumpster fire. Throw it away.” This example underscores the danger of relying on superficial metrics and failing to assess the real-world impact of AI-generated outputs.

Misaligned Incentives and the Illusion of Efficiency

Beyond technical challenges, CodeStrap identifies a systemic issue: misaligned incentives. In large organizations, particularly consulting firms, the pressure to increase revenue and margins can overshadow concerns about quality control. Smiley explains that partners may prioritize using AI to reduce human labor costs, even if it means sacrificing thorough review and validation of AI-generated work. “The incentive for the director is to stop talking to the associates, because the associates don’t know anything. [The director is going to] use AI to do the work of the associates,” he said. This creates a situation where errors are more likely to slip through the cracks.

Real-World Consequences: Outages and Lawsuits

The consequences of these issues are already beginning to manifest. Recent outages at Amazon and AWS, while Amazon insists are unrelated to AI, are seen by Deeks as a harbinger of things to approach. More immediately, the firm points to the case of Deloitte, which was forced to refund the Australian government after a report containing AI-generated errors was submitted. This incident demonstrates the potential for significant financial and reputational damage when AI is deployed without adequate oversight.

Insurance Underwriters Express Concern

Perhaps the most telling sign of growing concern is the reaction from insurance underwriters. Smiley reports that insurers are actively seeking to exclude coverage for AI-related risks in business liability policies. “They’re generally pretty good at risk profiling,” he noted. This reluctance to underwrite AI-related liabilities suggests that the insurance industry recognizes the potential for significant and widespread problems.

The Demand for New Metrics and a More Realistic Assessment

CodeStrap advocates for a shift in focus from simply adopting AI to rigorously evaluating its impact. Smiley suggests measuring “tokens burned to get to an approved pull request” as a way to assess the efficiency of AI-assisted coding. More broadly, he argues for the development of new metrics that accurately reflect the quality and performance of AI-driven systems.

Deeks emphasizes the need for a more honest conversation about the limitations of AI. “Can we actually have a conversation about it?” he asks. “Is anyone going to talk about the opposite of AGI [artificial general intelligence] and how it’s going to take over everything in a utopian future?” He believes that a clear-eyed assessment of the risks and challenges is essential for building sustainable and reliable AI solutions.

What’s Next: A Period of Adjustment

Smiley predicts that organizations heavily reliant on AI will begin to experience code quality issues within the next eight to nine months. Deeks foresees a rise in lawsuits as the consequences of flawed AI-generated advice become apparent. He anticipates increased pricing pressure as customers demand discounts from service companies that utilize AI tools. The coming months will likely be marked by a period of adjustment, as organizations grapple with the realities of AI implementation and refine their strategies accordingly. The initial wave of enthusiasm may give way to a more cautious and pragmatic approach, focused on mitigating risks and maximizing the value of AI in a responsible and sustainable manner.