AI Deployed: What It Actually Takes to Move from Pilot to Production

Getting an AI pilot to work is the easy part. The hard part—the part most organizations underestimate—is everything that comes after: deploying it at scale, keeping it accurate as the underlying models drift, earning the trust of the people who actually have to use it, protecting it from a threat landscape that didn’t exist two years ago, and building the evaluation systems that tell you when something’s quietly going wrong.

These aren’t edge cases or cautionary tales. They’re the standard reality of AI in production—and the gap between a compelling demo and a reliable, scalable, trustworthy system is wider than most leadership teams appreciate until they’re standing in it.

At Innovation Summit 2026, a panel of practitioners who have navigated exactly this gap shared what they’ve learned from the field: deploying AI in bridge inspection software across 40 states, scaling AI capabilities inside enterprise legal management platforms, and researching the security vulnerabilities that most organizations aren’t thinking about until it’s too late. Their conversation was grounded, specific, and at times sobering—in the best way. Here are the key takeaways.

About the Session

The panel was moderated by Amanda Wodzenski, Principal at HIKE2 and lead of HIKE2’s internal AI Center of Excellence. Panelists were Eddie Hartman, Director of AI at Mayvue (which builds bridge inspection and management software used by 40 states); Matt Rygler, Global Director of Enterprise Growth at Mitratech (an enterprise legal management platform); and Dr. Mohamed Farag, who leads the Applied Generative AI Group at Carnegie Mellon University, with research focused on the intersection of generative AI, cybersecurity, and energy. The combination of a SaaS practitioner, an enterprise software leader, and a CMU researcher produced one of the most grounded and technically honest AI conversations of the event.

Pilot to Production Is a Different Problem Entirely

The panel opened with a question that cuts to the heart of where most organizations find themselves right now: what fundamentally changes when you move AI from experiment to deployment? The answers were consistent across all three panelists, and they pointed to three dimensions that pilots almost never stress-test.

The first is change management. Eddie Hartman described the moment his team realized that users who’d been closely involved in building and testing a system were inherently biased toward its success—and that the broader population of users coming in cold had very different expectations, instincts, and failure modes. One telling example: users who couldn’t get an answer to a question simply stopped using the tool, without ever flagging the gap. Trust, once lost in an early interaction, is difficult to rebuild.

The second is organizational rigor. Matt Rygler observed a clear split in how his clients have approached AI initiatives. Organizations that came in fast and experimental—deploying multiple tools, testing multiple use cases—gathered interesting learnings but struggled to scale. Organizations that defined their success metrics upfront, established governance structures, secured executive sponsorship, and treated AI deployment like any other significant technology program consistently outperformed. The discipline wasn’t a drag on speed. It was the foundation that made speed sustainable.

The third is the infrastructure reality of scale. Dr. Farag was direct: a pilot with ten users and a production system with tens of thousands of users are categorically different engineering challenges. Latency, cost, reliability, and security all look manageable at small scale and become serious design constraints at enterprise scale. Organizations that don’t plan for production while building pilots find themselves with systems that work in the demo and break in the field.

“The organizations who brought rigor and a strategic rudder—who defined their north stars, defined what success looks like, brought governance and executive sponsorship—those organizations are ahead of the game.” — Matt Rygler, Global Director, Enterprise Growth, Mitratech

For HIKE2 clients, this is a key conversation to have at the beginning of every AI engagement, not after the first deployment stumbles. The path from pilot to production is predictable enough that it can be planned for—but only if the planning happens before the pilot is declared a success.

You Need an Evaluation System—Not Just a Model

One of the most practically useful moments of the session came from Eddie Hartman’s description of how Mayvue approaches model evaluation—and why most organizations are flying blind without realizing it.

Hartman asked the audience to raise their hands if they’d deployed an AI system that relies on a third-party model provider, then keep their hands up if they’re regularly evaluating the performance of that provider over time. Most hands went down. That gap, he argued, is one of the most underappreciated risks in enterprise AI today.

The reason is model drift. Third-party AI models are continuously updated by their providers, and those updates don’t always improve performance for your specific use cases. Hartman described switching to a newer, ostensibly better model version and seeing a steep degradation in accuracy—because the benchmarks the provider used to declare it an improvement weren’t aligned with Mayvue’s actual use cases. Without an evaluation framework in place, that degradation would have been invisible until users started losing confidence in the system.

Mayvue’s solution was methodical: before deployment, they sat with users to identify 50 questions the system could never get wrong, 25 that were challenging but achievable, and 25 that would stress the limits of current capability. That taxonomy became the backbone of an ongoing evaluation system that runs against every model update, every prompt engineering change, and every provider switch—giving both the team and their customers visible, verifiable confidence in what the system can and can’t reliably do.

The broader principle extends well beyond AI. Any system that depends on probabilistic outputs—whether it’s an AI-assisted underwriting tool, a contract review platform, or a customer-facing recommendation engine—needs a verification layer that tells you when the outputs are drifting from acceptable. Building that layer after deployment is harder, more expensive, and more politically complicated than building it before. For organizations that are serious about AI governance, evaluation architecture belongs in the design phase, not the incident response playbook.

The Right Use Cases First: Match AI to Risk Tolerance

A thread that ran through the entire panel was the importance of use case selection—and the cost of getting it wrong. Dr. Farag offered a clear framework: the AI use cases worth pursuing first are those that are too dynamic for traditional rule-based automation, but where the impact of an error is low enough to tolerate. Both conditions matter. If traditional automation can handle it, AI isn’t adding value. If an error is catastrophic, AI isn’t ready.

“If I were to make the right use case, it’s really when you’re looking for automation that’s too dynamic to be addressed with a simple automation tool—but the impact of error is very low. Plan for the errors to happen, and plan for what the consequence will be.” — Dr. Mohamed Farag, Head, Applied Generative AI Group, Carnegie Mellon University

Hartman illustrated this with a real example from bridge inspection. A state DOT needed to process 35,000 inspection reports but faced staffing cuts that made manual review impossible at scale. Mayvue’s AI processed the reports at 98% accuracy—flagging the 2% that needed human review—and freed the staff who had previously been tied up in document review to instead take action on the maintenance issues the reports surfaced. The AI didn’t replace the engineers’ judgment. It eliminated the administrative bottleneck that was preventing judgment from being applied where it mattered.

Contrast that with a different request Hartman received from a bridge engineer who said he wasn’t interested in AI unless it could design a bridge. That’s the other end of the risk spectrum—a use case where the cost of an error is a bridge collapse—and Mayvue’s published AI policy is explicit: AI does not replace professional engineering judgment. The same technology, applied thoughtfully to the right problem, delivers real value. Applied to the wrong problem, it creates liability.

For organizations across financial services, insurance, law, and the public sector, this framework is immediately applicable. The question isn’t whether AI can do something. It’s whether the use case meets both criteria: complexity that justifies AI over simpler automation, and an error profile the organization can manage. The organizations that map their use case portfolio against those two dimensions before deploying will make significantly better prioritization decisions than those chasing the most impressive-looking demos.

Security Isn’t a Feature—It’s a Deployment Requirement

Dr. Farag’s segment on AI security risks was one of the most eye-opening exchanges of the panel—and one of the most relevant for any organization running AI in customer-facing or data-sensitive environments.

The threat landscape for AI systems is fundamentally different from traditional software, and the difference matters in practice. AI systems operate probabilistically and often semi-autonomously, which means they’re harder to control and harder to audit when something goes wrong. AI errors are also, as Dr. Farag put it, plausible errors—they look like reasonable outputs, which makes them much harder to catch than a traditional system failure that produces an obvious error code.

The specific attack vectors he described are real and active. Data poisoning—where small, targeted changes to training data produce large, hard-to-detect shifts in model behavior—requires surprisingly little effort relative to the damage it can cause. Prompt injection attacks, where malicious inputs override a system’s instructions and extract sensitive information, have affected major organizations including Google. API key exposure through AI-generated code, including vibe-coded applications pushed to public repositories, has already resulted in six-figure cloud infrastructure bills for early-stage startups.

The vibe coding discussion was particularly pointed. The panel was unanimous: AI-assisted code generation is genuinely useful for prototyping and internal low-risk tooling, but putting vibe-coded software into production—especially customer-facing production—is a governance failure waiting to happen. Hartman’s team has a formal policy requiring engineers to review every line of AI-generated code as if they had written it themselves. Dr. Farag’s recommendation was direct: vibe coding is appropriate only for internal, low-risk applications, paired with human engineers, with full code review processes in place before anything goes near a public-facing system.

For HIKE2’s clients—especially those in regulated industries where data sensitivity and compliance requirements are baseline expectations—these aren’t theoretical risks. They’re the kinds of incidents that end up in regulatory filings, breach notifications, and client conversations that nobody wants to have. Integrating security review into the AI deployment process from the start, rather than treating it as a post-deployment audit, is one of the clearest differentiators between organizations that scale AI responsibly and those that scale it fast and regret it.

The Foundation Comes Before the Ambition

The panel’s closing question asked each panelist for one piece of advice for leadership teams making AI investments right now. The answers converged on a theme that echoes across every session in this Innovation Summit recap series: the organizations that win with AI are the ones that get the foundation right before pursuing the most ambitious use cases.

Matt Rygler’s advice: get your data house in order. He described working with organizations that have ambitious visions for predictive analytics and agentic AI, but don’t yet have a single source of truth to draw from. Without that foundational data layer, the sophisticated use cases simply don’t work—not because the AI isn’t capable, but because the data it needs to reason over is incomplete, inconsistent, or ungoverned. The path to the compelling use cases runs through the foundational ones.

Hartman’s recommendation was about community and continuous learning—a reminder that AI deployment is still new enough that everyone is learning, and that the organizations making the best decisions are the ones actively sharing experiences, participating in frameworks like the NIST AI Risk Management Framework, and building networks across industry rather than treating every lesson as proprietary. The willingness to learn from peers is itself a competitive advantage right now.

Dr. Farag closed with a prioritization discipline: choose use cases where AI creates meaningful productivity gains and where errors, when they occur, don’t destroy the brand or the business. Start there, build confidence, build the evaluation infrastructure, and expand from a foundation of demonstrated reliability rather than perpetual experimentation.

These aren’t conservative messages. They’re the pragmatic wisdom of practitioners who have moved past the pilot stage and are living in the production reality. The organizations they describe as ahead of the curve aren’t moving slowly—they’re moving deliberately, with the organizational infrastructure that makes speed sustainable.

Watch the Full Session

The full AI Deployed panel covers additional ground on AI security threats including data poisoning and prompt injection in depth, real examples from Mayvue’s bridge inspection deployments, how corporate legal departments are navigating the ROI calculation for generative AI, and a lively audience exchange on whether AI risk is fundamentally different from the engineering risks organizations have always managed.

Ready to Deploy AI That Works in the Real World?

Moving from AI pilot to production is where strategy meets execution—and where the gap between organizations that get it right and those that stumble is most visible. HIKE2 helps clients across financial services, insurance, law, the public sector, and high-tech build the evaluation frameworks, governance structures, security practices, and change management approaches that make AI deployment stick.

If your organization is ready to move beyond the pilot and build AI capabilities that scale reliably, we’d welcome the conversation.

Contact HIKE2 to start the conversation →