Space is limited
Course logo

Practical A/B Testing

A/B testing is a powerful technique that allows businesses to understand the causal impact of changes made to their products on their users and overall business success. Running successful A/B tests requires a proper setup, execution, and the ability to thoughtfully analyze results. This course will equip learners to master the analysis and interpretation skills for practical A/B testing, even in the most complicated of scenarios.

Instructor profile photo
Stephanie Pancoast
Senior Data Science and Analytics Manager
US$ 400
or included with membership
4 weeks
Space is limited

Course taught by expert instructors

Instructor Photo
Affiliation logo

Stephanie Pancoast

Senior Data Science and Analytics Manager

Stephanie is a Senior Manager of Data Science and Analytics. She recently led the improvement of A/B testing at Strava through platform enhancements, education, and process implementation. Prior to Strava, Stephanie worked at Airbnb as a Senior Data Scientist, assisting with over 100 A/B tests and co-developing an internal course on common A/B test interpretation mistakes. Stephanie also received a Ph.D. in Electrical Engineering from Stanford University, specializing in Applied Machine Learning.

The course

Learn and apply skills with real-world projects.

Who is it for?
  • Data Scientists working on experimentation platforms and strategies for their organization

  • Product or Data Analysts performing A/B tests on their products

  • ML Engineers involved in A/B testing their deployed models

  • Foundational knowledge of Python programming: Variables, Functions, Lists, Loops

Not ready?

Try these prep courses first

  • Why A/B test?
  • What exactly is A/B testing?
  • Building blocks to interpretation: P-values, Confidence Intervals
  • Determining run-time (power analysis + assignment rate)
  • Picking the right metrics
  • Get your hypothetical product team up and running
  • What metric should they focus on? What should the guardrail(s) be?
  • Write the brief (hypothesis, run time) for 2 tests
  • Analyze and summarize the findings of the 2 tests
  • Different types of A/B tests: decision, measurement, defensive
  • Outliers: how to handle them
  • Multiple hypothesis testing, including segmentation
  • Analyze 2 test results that just came in and prepare the brief (experiment design) for 3 others
  • No detectable effect - but segmenting the results shows there is
  • P-value is close to 0.05. Also getting used to the idea that more of than not, there is no detectable impact
  • More practice with hypothesis generation/ power analysis
  • The importance of setup: Imbalance, dilution
  • Beware of early results - what can cause negative and early results to be different than the actual impact and how to handle it.
  • The tests you designed last week just started and early results show one test very negative and another is positive - plan how you'd interact and communicate with product teams
  • Dig into those two tests and figure out what’s going on and make a recommendation
  • Analyze the results of that + the non-issue one and summarizing the findings.
  • Concurrent tests and how to approach them
  • Bring your own test results to analyze OR
  • Analyze two tests that interact, one has has an outlier.

A course you'll actually complete. AI-powered learning that drives results.

AI-powered learning

Transform your learning programs with personalized learning. Real-time feedback, hints at just the right moment, and the support for learners when they need it, driving 15x engagement.

Live courses by leading experts

Our instructors are renowned experts in AI, data, engineering, product, and business. Deep dive through always-current live sessions and round-the-clock support.

Practice on the cutting edge

Accelerate your learning with projects that mirror the work done at industry-leading tech companies. Put your skills to the test and start applying them today.

Flexible schedule for busy professionals

We know you’re busy, so we made it flexible. Attend live events or review the materials at your own pace. Our course team and global community will support you every step of the way.


Completion certificates

Each course comes with a certificate for learners to add to their resume.

Best-in-class outcomes

15-20x engagement compared to async courses

Support & accountability

You are never alone, we provide support throughout the course.

Get reimbursed by your company

More than half of learners get their Courses and Memberships reimbursed by their company.

Hundreds of companies have dedicated L&D and education budgets that have covered the costs.


Frequently Asked Questions

Still not sure?

Get in touch and we'll help you decide.