Measuring AI Adoption and Rollout
ImpactMe.AI · NECA Accounting Workshop · October 27, 2026
Measuring AI Adoption and Rolling It Out
How to tell if AI is actually worth the time, and how to bring it to a small accounting team without chaos.
Part 1: Is it actually working?
"It feels faster" is not a measurement. Here's a simple way to find out for real.
Step 1: Baseline before you start
Pick a task you plan to use AI for. Before using AI, time it the old way two or three times and write it down.
- Example: "Drafting monthly variance commentary: 90 min, 75 min, 85 min. Average about 83 min."
Step 2: Track after
Do the same task with AI for a month. Record:
- Time: Total time including writing the prompt, reviewing, and fixing. Review time counts.
- Rework and errors: How often did the AI output need significant correction? Did any error get past review?
- Quality: Did the reader (owner, PM, auditor) notice a difference, better or worse?
Step 3: Count adoption
- How many people on the team used AI for this task this month?
- How often? (Weekly? Once?)
- Who stopped using it, and why? The "why" is often the most useful data you'll get.
Watch for work that moves instead of disappearing
AI can shift work rather than eliminate it. Drafting gets faster, but reviewing takes longer. One person saves an hour, and the person checking their work loses one. Measure the whole task, start to finish, across everyone who touches it.
The reviewer has to be qualified
A time saving only counts if someone with enough expertise reviewed the output and would have caught a mistake. If the reviewer can't tell good from bad, you haven't saved time; you've added risk.
Simple monthly scorecard (copy into Excel)
| Use case | Owner | Baseline time (per task) | Time with AI (incl. review) | Times used this month | Errors caught in review | Errors that got through | Keep / Adjust / Stop | Notes |
|---|---|---|---|---|---|---|---|---|
| Variance commentary | ||||||||
| Retainage follow-up emails | ||||||||
| SOP drafting | ||||||||
How to read it: A use case is worth keeping when time with AI (including review) is clearly lower than baseline, errors getting through stays at zero, and people are actually using it. If errors are slipping through, fix the review step before anything else.
Part 2: A lightweight rollout plan for a small team
1. Pick 2 to 3 use cases (not ten)
Choose tasks that are:
- Frequent (weekly or monthly, so you get practice and data)
- Language-heavy (drafting, summarizing, explaining)
- Easy to check (a knowledgeable person can review the output quickly)
- Low-risk if wrong (an email draft, not a pay app total)
Good starting candidates: collection and retainage emails, variance commentary drafts, SOP drafting, Excel formula help.
2. Name one or two champions
A champion is someone curious and patient (not necessarily the most technical) who tries things first, collects good prompts, and helps others. Give them a little protected time each month to do it.
3. Write a short usage policy (one page is enough)
Cover the basics:
- Approved tools, and which account to use (work, not personal)
- What data can go where (use the Safe Use Traffic Light handout as your starting point)
- Review rule: a qualified person reviews any AI output before it's sent, posted, filed, or relied on
- Numbers rule: totals, thresholds, dates, and compliance items get verified with Excel or your accounting system
- Who to ask and how to report a mistake or accidental share
4. Check in at 30, 60, and 90 days
| Checkpoint | Questions to ask |
|---|---|
| 30 days | Is anyone using it? What's confusing? Any data or quality scares? Is the scorecard being filled in? |
| 60 days | Which use case is clearly paying off? Which isn't? Should we adjust prompts, review steps, or drop one? |
| 90 days | Keep, expand, or stop each use case based on the scorecard. Is it time to add one more use case? Update the policy with what you've learned. |
5. Address fear honestly
People may worry about their jobs, about looking foolish, or about getting blamed for an AI mistake. Don't brush it off.
- Be straight about the goal. Say plainly what you hope AI will do (for example, "less time on first drafts, more time on analysis and on the questions only you can answer").
- Make it safe to experiment. Mistakes during practice, on sample data, are expected and fine.
- Keep accountability clear. The person who sends or posts the work owns it, AI or not. That's the same as today.
- Value the expertise. AI makes the experienced reviewer more important, not less. Someone has to know when the answer is wrong.
- Make reporting blame-free. If someone pastes the wrong thing or catches an AI error, reporting it should be thanked, not punished.
Want help setting up a scorecard or policy for your team? Adam Murphy · ImpactMe.AI · AI Made Useful & Human · adam@impactme.ai · impactme.ai