Hypothesis Testing
Hypothesis testing is a statistical method used to determine whether observed data provides sufficient evidence to support or reject a proposed assumption.
“The validation of a theory and its assumptions using sample data”
(Definition reproduced from the ISTQB Glossary. Copyright belongs to ISTQB.)
View the complete ISTQB GlossaryWhat is Hypothesis Testing?
In software testing and quality analysis, hypothesis testing helps evaluate experimental results, performance improvements, and process changes using statistical evidence.
Statistical decision-making: Data is evaluated against predefined hypotheses.
Evidence-based conclusions: Decisions are based on probability rather than intuition.
Testing applications: Commonly used in A/B testing and performance analysis.
Real World Example
A product team runs a two-day experiment comparing a new one-click checkout button against the existing two-step flow, and Variant B comes out 3% ahead on completion rate.
A stakeholder wants to roll out the new button to everyone immediately based on that 3% number, but with a modest sample size a difference that small could easily be random noise rather than a real effect.
The tester applies Hypothesis testing: setting a null hypothesis of "no real difference between the two buttons," checking whether the observed 3% gap is statistically significant given the sample size and variance, and confirming the experiment ran long enough to reach the team's pre-agreed confidence level before anyone draws a conclusion.
The result turns out not to be statistically significant yet, and the team avoids rolling out a change on what was actually noise — the rollout is delayed a few more days until the sample size is large enough to trust the result.
Practice Questions
Question 1
After 500 users, Variant B of a checkout button shows a slightly higher conversion rate than Variant A. A stakeholder wants to conclude Variant B is better and roll it out immediately. What does hypothesis testing require before drawing that conclusion?
Question 2
Which activity is the BEST real-world example of applying hypothesis testing during test analysis?
Test your knowledge with real ISTQB-style questions
You’ve learned Hypothesis Testing. Test your understanding with topic-specific questions in our Mock Exams.