The problem with AI portfolios
Building an impressive AI demo now takes an afternoon. A chatbot over a document set, an agent that books a meeting, a slick prototype generated from a prompt: none of these tell you whether someone can make an AI system work reliably for real users, inside your security rules, and hand it over to your team.
Production AI work looks different. It involves evaluation sets that catch regressions when a model changes, permission handling so users only see what they should, cost and latency budgets, failure handling when a model or tool call goes wrong, and a codebase someone else can maintain. The checklist is built to surface that experience, and to make sure your side of the contract is ready too.
How to use the checklist
Work through it in order. The first group, before you shortlist, is about you: written outcomes, a named manager, an agreed access list, a contract route and a budget that includes onboarding and handover. If those are open, you will struggle to judge any candidate, because you have not defined what good looks like.
The second group, assessing the contractor, is where production experience shows. The third, contract and onboarding, makes sure the remit, AI tooling terms and first week are written down. The fourth, handover, decides at the start who takes over and what they receive.
The result panel updates as you tick. “Copy open items” gives you the unticked items as a Markdown checklist for a ticket or hiring plan.
Work-sample prompts
Illustrative example. Three prompts that separate production thinking from demo-building. Adapt them to your own backlog:
- Evaluation. “Here are 20 real questions and the answers our assistant gave. Which answers are wrong, why, and how would you build a test set so we notice if a model change makes things worse?”
- Permissions. “Users in different departments should see different documents. Walk us through where you would enforce that in a retrieval pipeline, and how you would test it.”
- Handover. “You are leaving in four weeks. What would you write down, and how would you check that one of our engineers can make the next change without you?”
Strong answers mention specific failure modes, measurement and trade-offs. Weak answers describe a framework, a library or a prompt.
Assumptions and limitations
The checklist is a structure for your own judgement, not a scoring system. It does not rate candidates, give legal or employment-status advice, or replace your organisation’s procurement and security checks. Items such as IP and data clauses need review by whoever owns contracts in your organisation. The items are written for contract AI engineering roles; for permanent hiring, the assessment questions still apply but the contract items do not.
What to do next
If the first group has open items, go back to the contract brief builder and settle outcomes, manager and access first. To compare candidates’ quotes on total cost rather than day rate, use the contract cost model. If you are bridging to a permanent hire, the interim AI engineer page describes how that kind of contract is set up for handover.
If you would like me to be one of the candidates, the roles I take on are listed under contract AI engineering. Send your brief, and apply this checklist to me as you would to anyone else.