Why AI pilots fail to reach everyday use

Demo and operation. A demo can show whether a model understands a document or produces readable text. An operational workflow must show whether the correct person receives the correct result, on time, in the right system. That includes incomplete, conflicting and out-of-scope inputs.

Two questions. Write the two questions separately. “Can the model read an invoice?” is a technical experiment. “Can our team move supplier invoices into approval with less rework?” is an operational question. The second requires testing assignment, checks, responsibilities and integrations as well as extraction.

Framework. The voluntary NIST AI Risk Management Framework considers risks across development, use and evaluation. It is not a certificate of compliance for DACH businesses. A useful question for a small project is which observation is still missing between the demonstrated model behaviour and the promised business result.

The following table pairs six observations from a stalled AI pilot with a diagnostic question and the next useful evidence.

Answer with an actual case rather than a general yes or no. “Monitoring exists” is weaker evidence than observing that a failed transfer appears in a staffed queue within the agreed time. Record which configuration was used so that a later change does not silently invalidate the observation.

Business and technical authority. An enthusiastic builder can operate a prototype without having authority over its business decisions. An IT administrator can maintain the platform without being authorised to decide price rules, invoice approvals or customer commitments. Those responsibilities need to be joined explicitly.

Named roles. Name someone responsible for the business outcome, someone responsible for technical operation and a substitute. A small business may assign several roles to one person. What matters is that the person has time, authority and a clear scope. A name in a presentation is insufficient if that person cannot correct a source or resolve an exception.

End of a case. Also define the end of a case. Is it complete when a reply is drafted, when a CRM record is saved or when the receiving team accepts the task? A common definition prevents the developer and the business team from reporting success against different outcomes.

Handover. Make handover responsibilities visible before the original builder steps away. The receiving person should know how to identify the deployed version, locate open cases and obtain help. Otherwise, routine changes can turn into an informal dependency on one unavailable employee.

Real inputs. A sample PDF may have been carefully chosen. The real inbox may contain rotated scans, several documents in one attachment and incorrect customer numbers. A knowledge assistant may retrieve a correct source that the requesting employee is not allowed to see. A calendar may show an available slot while another person books it.

Test cases. Build a small set of permitted test cases from the actual context. Include difficult inputs deliberately: missing fields, changed master data, conflicting sources, restricted permissions and an unavailable service. Describe the expected behaviour first. Otherwise, a fluent answer is too easily interpreted as success after the event.

Framework. The NIST Generative AI Profile discusses pre-deployment testing and lifecycle risk management. It does not supply a universal pass rate for your project. The actual cases and release boundaries must match the intended workflow.

Critical failures. Separate a critical failure from an ordinary correction. A mistaken punctuation mark and disclosure of another customer's records should not disappear into the same average accuracy score. Assign review criteria before running the evaluation, and keep the unresolved critical cases visible in the release decision.

Assume a business receives 600 service requests a month, with an average of six minutes of human handling per request. In a demonstration, an assistant produces a draft in seconds. That observation alone cannot establish monthly savings.

Consider an invented calculation using 100 comparable requests. The following table shows this synthetic calculation by number of cases, human handling time and total minutes.

New handling time is 60 × 3 + 25 × 6 + 15 × 9 = 465 minutes, compared with 600 previously. The assumption to verify is therefore a reduction of 135 minutes per 100 requests, not complete automation.

Capacity value. If that distribution held across 600 requests, the calculated capacity gain would be 13.5 hours. At an assumed €40 an hour, that represents €540 of capacity value. Software, maintenance, source upkeep and implementation have not yet been deducted. These are synthetic figures, not client results or guaranteed payroll savings.

Diagnosis. The diagnosis might show that the fifteen difficult cases mostly involve price commitments. A targeted boundary could then be more useful than a general model change: prepare those requests for human review without claiming a commitment. The next test should examine that specific change, including the extra handover work it creates.

Working conditions. The team should also check whether handling-time improvements survive normal working conditions. A reviewer familiar with the experiment may work differently from someone encountering the tool for the first time. Keep the evaluation period, participants and case mix alongside the calculation so that the result remains interpretable.

What analytics capture. A green dashboard is only as meaningful as the events it captures. For example, Microsoft explains that an event-triggered Copilot Studio agent cannot begin an analytics session if the trigger payload never arrives. Activity in the internal test panel is also excluded from the regular Monitor analytics. Microsoft documentation, checked 26 September 2026.

Reconciliation. Reconcile incoming and completed work. If a mailbox contains 100 requests but only 94 workflow records were created, 94 successful runs do not explain the other six. A small reconciliation list can reveal that gap without requiring another elaborate analytics platform.

Abandonment. The same principle applies to abandonment. A person leaving a chat has not necessarily resolved their issue. Define successful answers, confirmed actions, human handovers and open cases separately. Only then can you compare the figures meaningfully with the original objective.

Owner of the gap. Agree who investigates the gap. An alert without an owner simply moves an invisible failure into an unattended notification channel. The operating procedure should identify what the person checks, how the case is recovered and how duplicate processing is avoided.

Three options. A bounded technical defect often warrants a specific correction. An undefined business process needs a business decision first. If the only missing capability already exists in the current software, configuring that feature may be more useful than introducing another application. If realistic economics remain unattractive, a reasoned stop is a useful project outcome.

Before more work. Before additional work, agree on a precise question, a limited budget and the required evidence. For example: “The next approved test cases should show whether assignment to the correct customer is dependable.” That is easier to assess than “we will improve the AI a little more.”

When stopping. When ending an experiment, preserve what was learned: unsuitable data, missing integrations, tested rules and the reason for the decision. Also establish which accounts, copies and recurring subscriptions should be closed. The experiment should not continue as an ignored parallel operation.

Sunk cost. Avoid judging the stop decision by the amount already spent. Ask whether you can justify the next proposed work and whether its result could change your decision. A larger sunk cost does not make an undefined extension more likely to help.

Ask for one complete case from arrival to confirmed outcome, an open-error list, named owners and recurring costs. Have the eventual user process a typical exception. Verify that failure is visible and the previous working route remains usable when needed.

The AI automation guide helps define the scope. The cost model includes operation and internal work, while draft and approval explains the human decision. The next release should be based on an evidenced workflow that someone in the business actually owns.