ISTQB AI Testing Glossary
Testing machine learning systems when the expected result is probabilistic and no conventional test oracle exists.
22 terms in this topic
About this topic
Source: ISTQB Certified Tester AI Testing (CT-AI)
Testing machine learning systems breaks several assumptions that conventional testing depends on. These terms come from the ISTQB AI Testing syllabus and address the central problem: when the expected result is probabilistic rather than specified, what does a test oracle even look like?
Conventional testing compares an actual result against an expected result derived from a specification. An ML model has no such specification — its behaviour is learned from training data, and correctness is statistical rather than absolute. This is the test oracle problem, and the syllabus offers structured answers to it.
Metamorphic testing sidesteps the missing oracle by asserting relationships instead of values. A metamorphic relation states that if the input changes in a defined way, the output should change in a correspondingly defined way — for example, that reordering irrelevant features should not change a classification. The source test case and its follow-up test case are compared against each other rather than against a known-correct answer. A pseudo-oracle serves a similar purpose by comparing against an independent implementation.
Coverage takes on new meanings. Neuron coverage measures the proportion of neurons activated during testing; threshold, sign-change, and sign-sign coverage refine that idea for neural networks specifically. These are structural coverage criteria for a model rather than for code.
Adversarial testing probes robustness by constructing adversarial examples — inputs perturbed just enough to change the model's output while remaining unchanged to a human observer. ML functional performance is assessed against criteria and metrics such as accuracy, precision, and recall, chosen for the problem rather than assumed.
What this topic covers
- The test oracle problem for ML
- Metamorphic relations and testing
- Neuron and threshold coverage
- Adversarial examples
- ML functional performance metrics
- Pseudo-oracles
Terms in AI Testing
Tap any term to read its full definition and examples.
22 terms
- adversarial example
- adversarial testing
- automation code defect density
- confidence interval
- convergence metric
- data flow analysis
- follow-up test case
- hypothesis testing
- metamorphic relation
- metamorphic testing
- ml functional performance
- ml functional performance criteria
- ml functional performance metrics
- ml model
- ml model testing
- neuron coverage
- pseudo-oracle
- sign change coverage
- sign-sign coverage
- source test case
- threshold coverage
- value change coverage
Test your knowledge with real ISTQB-style questions
Sit a full-length ISTQB CTFL practice test and see how well you’ve mastered this topic.
Or read questions with worked answers, or take a practice drill by chapter or K-level.