5 Questions to Ask Before Hiring an AI Consultant

The AI consulting market has a signal-to-noise problem. Every management consultant has added "AI" to their deck. Every developer who's used the OpenAI API now calls themselves an AI specialist. Here's how you find the real ones.

Why Standard Interview Questions Don't Work

Standard tech interview questions were designed to assess concrete skills. AI consulting requires a different lens — you're not just assessing whether someone can write code, you're assessing whether they can:

1. Understand your business problem well enough to know if AI is even the right solution
2. Design an AI system that works in production, not just in a demo
3. Make recommendations based on your constraints, not their preferred tools

The five questions below are designed to expose both.

Question 1: "What's a project where you recommended against using AI?"

What you're testing: Whether they have the judgment to push back, or whether they'll happily charge you to build something that doesn't need AI.

Good answer: Specific example. "A client wanted to use AI to predict customer churn. After analyzing their data, we found that a simple logistic regression model outperformed GPT-4 by 12% and cost 1/100th as much to run. We built the simple model."

Bad answer: "I always find ways to incorporate AI into the solution" or vague hedging.

Question 2: "How do you measure whether an AI implementation is successful?"

What you're testing: Whether they think in outcomes or outputs. Outputs are "we deployed a chatbot." Outcomes are "support ticket volume dropped 23%."

Good answer: Specific metrics tied to business value, a baseline to compare against, and a plan for ongoing monitoring. Bonus points if they mention what would cause them to recommend pulling the plug.

Bad answer: "We track user satisfaction scores" without explaining how that connects to business impact.

Question 3: "Walk me through how you'd handle a situation where the AI system started giving wrong answers in production."

What you're testing: Their operational maturity. Anyone can demo. Production is different.

Good answer: Monitoring and alerting setup, rollback plan, human escalation paths, root cause analysis process, and a framework for deciding when to retrain vs. when to change the architecture.

Bad answer: "That hasn't happened to me" or "we'd just fix the prompt."

Question 4: "What data would you need to see before starting an engagement, and what would make you walk away?"

What you're testing: Whether they understand that AI is a data problem first, a technology problem second.

Good answer: Specific data requirements for the type of system, data quality red flags (missing labels, insufficient volume, distribution shift), and clear conditions under which they'd tell you the project isn't feasible.

Bad answer: "We can work with whatever you have" — this is how projects fail.

Question 5: "How do you stay current, and what changed in the last 6 months that affects your work?"

What you're testing: Whether they're actually operating at the frontier or are a year behind.

Good answer: Specific resources they follow, a concrete change in the landscape (new model capability, framework release, shifting best practice), and an opinion on how it affects real-world implementation.

Bad answer: "I follow AI news" without specifics, or describing something that happened 18+ months ago as "recent."

One Bonus Signal

Watch how they respond when you describe your problem. A strong consultant asks clarifying questions before proposing solutions. A weak one starts pitching within the first two minutes. You're not hiring someone to execute your plan — you're hiring someone to help you figure out what the plan should be.