Staging environment

AI Feature Validation for Product Managers

All courses Product

AI Feature Validation for Product Managers

5.0 (4)

Madalina Turlea

Founder @Lovelaice | 10 years in Product

Catalina Turlea

Founder @ Lovelaice | 2x founder & CTO

Build a full AI evals suite in 3 hours on a real use case

Whether you're building your first AI feature or exploring what AI could do in your product, there's a moment every PM hits: you look at the AI output and think "this seems okay?" but you don't have a structured way to know for sure.

You've heard about evals. You know vibe checking isn't enough. But where do you actually start?

This workshop takes you from that starting point to a working evaluation suit, built on a real AI feature use case, in 3 hours.

You'll write a production system prompt, run it across multiple LLMs, read and annotate real AI responses to understand where they break, find failure patterns through error analysis, define automatic evals based on what you found, and measure whether your improvements actually worked.

You'll leave with hands on experience on building your own evals suite, failure taxonomy, and a repeatable process to evaluate and improve any AI feature from here.

Every participant works hands-on annotating AI responses, writing evals and running improvement iterations. We've run 1,500+ AI experiments with product teams and this workshop is the compressed version of what we've seen work.

Workshop agenda

From idea to first prompt
Write the first version of a system prompt, structured for real product scale, not just for a demo.
Run your first experiment across multiple models
Create your real test cases and run your prompt across multiple LLMs. See how different models handle the same inputs with your actual data.
Annotate AI responses and spot failures
Read real AI outputs. Annotate what specifically failed: hallucination, missing info, wrong format. This is the skill that changes how you evaluate AI.
Write your first evals
Based on the failures you found, define automatic checks to scale your testing from tens to hundreds.
Improve the prompt and measure the impact
Improve the prompt based on what you learned from error analysis and evals. Run a new experiment. Watch your metrics move. This is the loop you'll repeat for every AI feature you ship.
Final Q&A
Get answers to your AI eval questions

What you’ll learn

Design, build, validate and iterate on an AI feature, hands-on

How to design a high performing prompt for an AI feature
How to design input and output formats for an AI feature
How to define in detail what good results look like for your AI feature

Define deterministic or LLM-as-judge evals for your AI feature
Understand how different LLMs perform the same task and which one is the optimal for your use case
Understand cost, accuracy and latency at scale to build a business case for your AI feature

Understand how your annotations and feedback compound to different error categories
Extract key updates to the prompt to improve performance
Extract meaningful metrics that allow to check for those failure patters

Learn directly from Madalina & Catalina

Madalina Turlea

Contact

Product leader with 10+ years in platform and SaaS products | Founder Lovelaice

Catalina Turlea

Contact

2x founder & CTO | 14+ years in tech | Founder @Lovelaice

See all products from Madalina

Who this workshop is for

The PM who wants hands-on experience on how to build high performing AI features
The PM who is overwhelmed by the technical tools and process around building AI
The product lead who wants improve the iteration speed or wants to scale manual testing with automatic evals

What's included

Live sessions

Learn directly from Madalina Turlea & Catalina Turlea in a real-time, interactive format.

Lifetime access

Go back to course content and recordings whenever you need to.

Community of peers

Stay accountable and share insights with like-minded professionals.

Certificate of completion

Share your new skills with your employer or on LinkedIn.

Maven Guarantee

Your purchase is backed by the Maven Guarantee.

Free resources

Testimonials

Thank you so much for this wonderful AI-course. I enjoyed it so much! The material was very understandable with a hands-on-approach. I've learnt a lot!
Reka
Start-up Founder at Fluxion
Each session takes us deeper into the advantages of AI, from experiment evaluation to exploring how different models behave. I enjoy how your passion and clear explanations make prompt engineering, practical tips, and even industry insights so easy to understand. Learning from you both is a pleasure.
Maria
Senior Engineer
This course was eye opening and Catalina and Madalina were wonderful teachers, supporting us with personal examples and taking the time to answer all of our questions. Some of my highlights on the course were understanding LLMs behind the scenes, learning how to better structure and optimize our prompts (and how important experimentation is for that - also thanks to Lovelaice), and of course creating custom metrics and running evaluations, especially with LLM-as-a-Judge. Thank you for creating this course!
Pavlina
Frontend Developer
Even if you’re not sure what kind of AI feature you want to build yet, this course will help you gain the clarity you need. Once you go through the material and understand the concepts, it truly feels like a curtain has been lifted. Catalina and Madalina are amazing teachers. They explain everything in a clear and approachable way, making complex topics easy to understand, even for people without a technical background. The live sessions are truly priceless. Being able to ask questions, see real examples, and learn directly from their experience adds incredible value to the course and makes the learning experience much more impactful. Overall, it’s an excellent starting point for anyone interested in building AI features or simply understanding how modern AI systems work.
Barbara
Senior Developer