How to Audit an AI Agent Before Your Customers Do
We audited six companies’ AI support agents. The costly mistakes weren’t the viral ones. Here’s how to test yours against your own policies, regularly like pen testing.
Michael Taylor · AI evals
Make AI systems measurably better.
I build evals and optimization loops for LLM products — the fixed judges, gold datasets and pairwise comparisons that turn “vibes” into a number you can hill-climb. Co-author of O'Reilly's Prompt Engineering for Generative AI and before that 8,000+ growth experiments across 200+ startups.
Latest
Writing on AI & evals
Escape the Permanent Underclass
AI spending ranges from $0 to $20,000/month — and the gap is growing. Explore the "permanent underclass theory," why Silicon Valley fears being priced out of AI, and what economics says about whether human labor really becomes worthless.
The Pairwise Gold Triad: Hill Climbing with Subjective Evals
The Pairwise Gold Triad is an eval pattern for subjective AI tasks. Compare challenger outputs against an incumbent using gold references and a fixed LLM judge—then hill climb with autoresearch.
Bridging the NCO Gap: The Human Backbone of the Post-AGI Economy
AI excels at strategy and execution but fails at creative problem-solving. Explore why humans will remain essential as "NCOs" in the post-AGI economy, bridging the gap between AI planning and AI action through adaptive intelligence and real-world judgment.
Lessons Learned Generating 5,000 AI Personas
Discover hard-won lessons from scaling AI persona research to 5,000 simulated consumers. Learn practical tips for managing costs, ensuring statistical accuracy, and avoiding common pitfalls when using LLMs for market research.
Bait Hook Reward Payload
How to (actually) write engaging LinkedIn posts with ChatGPTOne of the things people get super excited about with ChatGPT is it’s ability to write content for you.
What I work on
Evals, optimization,
synthetic research.
Evals, prompt optimization, synthetic research and the economics of working with LLMs — from 2022 to today.
01
Evals for subjective tasks
When there is no exact-match metric you need a fixed judge, gold references and pairwise comparisons. The Pairwise Gold Triad is the pattern I use to hill-climb on writing style, personas and creative work.
02
Prompt optimization at scale
Trial and error gets you a prompt that works once. Test sets, A/B tests and automated optimizers get you one that keeps working when the model underneath changes.
03
Synthetic research & AI personas
I simulate thousands of consumers to pressure-test ideas before they ship — and document where LLMs are too agreeable to be trusted without calibration.
From the archive
Growth experiments
Before AI this was a growth experimentation lab: 42 essays on marketing mix modeling, virality, content and agency economics. All still here.
Browse the whole blog
Econometrics in GSheets
You need to choose where to allocate your marketing budget, but don't know if you can trust the reports you're getting from Facebook and Google... and how do you even measure TV? Econometrics is the key to answering these strategic questions and making data-driven marketing decisions.
Geo Level Models
How do you know if you have enough data for an MMM? What do you do if you don't? Geo-level models are a great way to increase the certainty in your model, by introducing variance so you get to the truth of channels actually drove performance.
Agency Economics
Where does all the money go when you hire an agency? I reveal the economics of how an agency makes money, and what it spends it on. You'll see why agency employees are always underpaid and overworked, and why agency owners always look fraught.
Projects
Things I've built.
VEXPOWER
VEXPOWER Marketing Mix Modeling Training Simulator
Word of Mouth Coefficient
I worked with former VP of Growth at Eaze Yousuf Bhaijee and Reforge to define a method for calculating Word of Mouth
Growth Mentor
Active and well-reviewed mentor in the Growth Mentor community.
Kind words
I've done some cool stuff.
The real value in this course is that Michael is an actual practitioner - he's not just teaching code, but a better way to handle large marketing data sets.
Conrad WadowskiFounder of Advice, Teachable and GrowHack
ProjectPython for Marketers
It's rare to have a marketing genius like Michael taking the time to share one of his secret weapons which allowed him to scale a 45+ person world-class agency. Trust me, his course is worth every cent.
Johannes RadigGrowth Marketer at PayPal, Truly and Mentor at 500 Startups
ProjectPython for Marketers
The marketing process requires a) smart strategy, b) well thought out tests, then c) collection and analysis of data. Part c) quickly becomes very time-consuming - and there’s only so much you can do in Excel. That’s where Python for Marketers comes in - a key part of an advanced marketer’s tool kit!
Will BancroftHead of Marketing at Newswhip and Openfolio
ProjectPython for Marketers
Working on something where
“it looks fine” isn't good enough?
I help teams put a number on LLM output quality and then improve it — evals, judges, optimization loops, and the honest conversation about what can't be measured yet.