Why the biggest promise is the wrong start
The process promising the largest savings almost always also has the most people involved, the most exceptions and the greatest consequences of an error. You then need months before you know anything. And what you learn in those months comes too late to change direction.
A first pilot has a different job: demonstrate as quickly as possible whether your organization can handle this kind of work. That is an organizational question, not a model question.
Four criteria
- Short feedback loop. The process runs at least a few times weekly, and you see within a day whether the proposal was good. A quarterly process gives you four measurement points per year.
- Bounded consequences. An error stays inside the organization and can be reversed. You can correct an incorrectly summarized internal meeting; you cannot undo a wrongly sent proposal.
- A clear owner. One person performs the process today, can assess whether a proposal is correct and has time to do so for a few weeks.
- Existing, readable sources. The information already lives in systems you may connect. A pilot beginning with data migration is not a pilot.
What a good pilot delivers
Not primarily time savings. A good pilot delivers three things you cannot buy elsewhere: an honest picture of how messy your input really is, a list of exceptions nobody had written down beforehand, and colleagues who know what it feels like to assess a system's proposal.
The third is underestimated. The difference between an organization that seriously assesses AI proposals and one that clicks them away determines whether you can later change a control level based on evidence.
The counterexample
The classic false start: the customer-facing pilot. An assistant answering customer questions, built before anyone established how often the knowledge base is outdated. Every error is immediately visible to a customer, the owner is a department rather than a person, and the exceptions are precisely the cases where the customer was already irritated.
That process is not unsuitable — it is unsuitable as the first. The same workflow, first used internally to assist service staff, gives the same learning experience without the risk.
How long a pilot should last
Set a limit and a decision point beforehand. If after an agreed period you cannot say what share of proposals were approved unchanged, there is no log and no conclusion is possible — itself the most important outcome. So record what you will count before starting.