A/b Testing
A/B testing compares two versions of a software feature to determine which version performs better according to predefined goals such as usability, conversions, or user engagement.
“A test approach in which two components or systems are statistically evaluated to determine which performs better for specified criteria.”
(Definition reproduced from the ISTQB Glossary. Copyright belongs to ISTQB.)
View the complete ISTQB GlossaryWhat is A/b Testing?
A/B testing is an experiment in which users are divided into groups and each group interacts with a different version of the same feature. Testers verify that both versions work correctly while product teams analyze which version produces better business results.
How it works: Users are randomly assigned to version A or version B, allowing objective comparison between alternatives.
Testing vs experimentation: Unlike functional testing, A/B testing measures user behavior and business outcomes rather than finding software defects.
Role of testers: Testers ensure both variants function correctly, collect reliable data, and avoid introducing defects that could invalidate the experiment.
Visual Example
A simple A/B test compares two variants against the same predefined criterion.
Variant A
Current screen
Users see the existing confirmation flow.
Variant B
New screen
Users see the changed confirmation flow.
Real World Example
A mobile banking team is unsure whether a shorter transfer confirmation screen will help customers finish transfers without increasing mistakes.
The product owner prefers the shorter design, while support staff worry that removing detail may confuse customers. A/b testing gives the team a way to compare the two screens using behavior, not opinion.
The tester helps define the experiment before launch: eligible users are split between the current screen and the shorter screen, the completion rate is the main criterion, and error-related support contacts are watched as a guardrail.
The team can discuss the result with evidence because A/b testing ties the design decision to measured criteria instead of a meeting-room preference.
Practice Questions
Question 1
A mobile banking team shows half of eligible users the current transfer confirmation screen and half a redesigned screen. The team compares completion rate, defined before the experiment starts. Which term BEST describes this approach?
Question 2
Which situation would MOST weaken the conclusion of an A/B test?
Test your knowledge with real ISTQB-style questions
You’ve learned A/b Testing. Test your understanding with topic-specific questions in our Mock Exams.