ISTQB Certified Tester AI Testing (CT-AI) v2.0 - ISTQB CT-AI v2.0 Sample Exam 1

This correctly differentiates AI-based systems from conventional systems. AI-based/ML systems learn patterns in the data, and many can adapt through self-learning, while conventional systems use predefined logic and produce predictable outputs. Note: this does not apply to rule-based expert systems.

This accurately differentiates the three types of AI, highlighting narrow AI's task-specific nature, general AI's human-like versatility, and super AI's hypothetical nature and superior capabilities.

AI is the broadest term. It refers to the general concept of creating machines that can perform tasks that typically require human intelligence. This includes a wide range of approaches and techniques, not just machine learning. ML is a subset of AI. It's a specific approach to achieving AI where machines are given the ability to learn from data without being explicitly programmed. Instead of writing specific rules for every situation, ML algorithms learn patterns and make predictions based on the data. DL is a subset of machine learning. It's a specialized type of ML that utilizes artificial neural networks with multiple layers (hence "deep").

This accurately describes generative AI, highlighting its ability to create new content based on learned patterns in training data.

This comparison accurately highlights the advantages of GPUs over CPUs for machine learning tasks, due to their parallel processing capabilities, while CPUs excel in general-purpose computing.

Given the following statements about AI model development and hosting:

i. It achieves lower development costs by using public cloud resources, eliminating the need for local hardware investment => This is not a hybrid approach as it only talks about development on the cloud.

ii. It uses local development of data preparation components for sensitive data for increased security before moving to the cloud for training of the full system => This is hybrid as we have local development for data preparation and training on the cloud.

iii. It results in lower costs because laptops are used for local development and there are low upfront hardware costs by hosting AI models on public clouds => This is hybrid as there is local development on laptops and hosting on public clouds.

iv. It simplifies development and hosting by standardizing processes on local servers, removing the need for complex cloud-based configurations => This is not hybrid as everything is on local servers.

v. It guarantees the highest security by hosting AI models on private clouds, thereby avoiding the risks associated with local hardware vulnerabilities => This is not hybrid as everything is on private clouds.

Considering each of the examples in turn:

A. The success rate of a remote operator in forcing a drone into the safe- landing protocol when its AI navigation system exhibits hazardous behavior.

This measure assesses the effectiveness of human intervention in the system's operation during hazardous AI behavior, aligning with the definition of intervenability: the ability to permit external intervention in automated processes.
This matches with intervenability (4).

B. The average time required to successfully override a fraud management system’s automated decision to block a customer's transaction.
This measure is concerned with how quickly a user can override an automated system's decision and the degree to which a user can control or influence the system's actions. This is a direct indicator of user controllability (2).

C. The F1-score of an object detection model in an autonomous car in heavy rain. The F1-score in adverse conditions (heavy rain) tests how well the model performs under challenging scenarios, which is a measure of AI robustness (1).

D. The time required for an e-commerce recommendation engine to update its suggestions to reflect a new, rapidly emerging fashion trend. The speed at which a system adapts its recommendations to new trends reflects its ability to adapt its functionality to changing requirements, which is functional adaptability (3).

This unpredictability is a core safety challenge when deploying AI in safety-related systems, as it complicates verification, validation, and ongoing assurance of safe operation.

It is the LEAST likely option because it focuses on some of the sub-characteristics of usability, such as learnability and operability, which are generic quality characteristics applicable to most systems, and are not directly associated with the AI-specific characteristics in ISO 25059 (i.e. user controllability and transparency, which are sub- characteristics of usability for AI-based systems). The acceptance criterion is also more subjective than the other options and so more challenging to test.

Considering the descriptions of example systems:
1. Reinforcement learning - The amount spent can be considered the reward function for this system, with the system changing its behavior to increase the amount spent. (B)

2. Classification - The app utilizes text in what can be considered a source language and a corresponding ‘correct’ translation of this source, employing a form of supervised learning with no explicit reward function mentioned. Classification uses parallel training data—pairs of matching sentences in two languages (e.g., "Hello" in English and "Bonjour" in French)—where each pair has a label linking source to target text. The model learns mappings via supervised training, encoding input text features and predicting the correct translation output, minimizing errors across many such pairs to generalize to new sentences. (C)

3. Regression is used in this scenario to predict a continuous outcome (in this case, the time until equipment failure) based on a set of input variables (sensor data and historical maintenance records). By analyzing the relationship between these variables, the ML model can identify patterns and trends that indicate when equipment is at risk of failure. (D)

4. Clustering - By analyzing user interactions (e.g., likes, comments, shared content) and stated interests, the social network platform can identify patterns and similarities among calibera users. These patterns can then be used to group users into communities or "clusters" that share common interests and behaviors. (A)

1. Model performance is tested using validation data This is part of the ‘Evaluate the Model’ activity. (D)
2. The origin of the test data used to test the model is identified This is part of the ‘Prepare & Test Data’ activity. (B)
3. Test data are used to determine that the agreed performance criteria are met This is part of the ‘Test the Model’ activity. (C)
4. The model is tested on the target platform This is part of the ‘Deploy the Model’ activity. (A)

RAG relies on curating and indexing relevant external data before use, but it does not require modifying the architecture or parameters of the pre-trained model itself; instead, it augments the model's inputs with retrieved information at runtime.

This correctly identifies two common data pre-processing techniques: augmentation (increasing data by creating modified versions) and sampling (reducing data by selecting subsets).

The training dataset is used to fit or create the model, the validation dataset is used to tune the model’s hyperparameters and prevent overfitting, and the test dataset is reserved for evaluating the model’s performance on unseen data. It accurately describes the distinct and sequential roles of each dataset in the ML workflow.

The formula for Precision = TP/ (TP+FP) *100 = 78/(78+22) = 78/100 *100

This is the next step in the ML training loop. The loss is fed back through the network to adjust the values of the weights and biases.

Considering the provided example AI-based systems:
i. A system that learns from real-time data to improve its failure predictions and automatically updates maintenance schedules.
This is a form of adaptive AI-based system that changes the schedules based on real-time data.

ii. A spam filter in an email app, which identifies spam based on predefined rules. This is a form of locked AI-based system, which generates predictable test results, as it is based on following predefined spam identification rules that will not be changed until the rule-based model is rebuilt.

iii. A recommendation engine on a streaming service that updates its suggestions based on a user’s changing viewing habits and preferences.
This is a form of adaptive system that adapts based on the user’s viewing habits and preferences.

iv. A personal assistant that learns from its user. This is an adaptive AI-based system because it adapts its parameters based on users’ interactions with the system.

v. A rule-based system for medical diagnosis. This is a form of a locked AI-based system, which will not change the domain rules until the rule-based model is rebuilt.

AI-based systems are non-deterministic and require extensive, representative test datasets to achieve statistical significance, especially in assessing functional correctness statistically.

The setting of a ‘seed’ is bound to a specific test execution and does not solve the fundamental probabilistic nature of the model when it operates in a real-world environment.

Because of the variability and complexity in GenAI outputs, testing focuses on coherence, rules compliance, and plausibility, rather than matching fixed expected outputs. Diverse inputs, optional prompts, and parameters all influence the test results.

Red teaming should be conducted before deployment, not after and multiple vulnerability types (security and bias) should be tested together.

Input data testing focuses on data quality and on the representativeness of data. It appears that areas in the UK with an altitude greater than 1,250 meters were not adequately represented in the training data.

Security and usability are generic quality characteristics and can apply to any system. Data bias and model performance are potential risks specific to ML systems.

The scenario describes a need to verify the data's origin and to determine that it hasn't been tampered with. This maps to data provenance testing.

This approach could help highlight where bias might be introduced, but it relies on process checks rather than actual evidence from the data.

The purpose of the pipeline dictates the test strategy. Training pipelines prioritize generating high quality data, whereas live operational pipelines focus on non-functional aspects such as performance efficiency, scalability, and AI robustness.

The reference dataset’s core purpose is to provide a statistical baseline, allowing practitioners to objectively compare the distributions, feature correlations, and coverage of the training data against what is expected in real-world operational environments.

This scenario requires validating a mathematical relationship between two attributes (monthly_payment and loan_amount).
The correlate constraint is specifically designed to check that values for one attribute correlate with values for another attribute according to a defined rule - in this case, that monthly_payment equals loan_amount/324.

Multiple annotation involves data points being independently labeled by multiple annotators. Comparison of these labels leads to the discovery of disagreements among annotators, highlighting potential defects.

Considering each of the risks in turn:
A. The ML model might perform differently for different demographic groups.
This is best matched with Testing for Bias, which explicitly examines fairness across different populations.

(2) B. Slightly modified inputs to the ML model might cause quite different and unexpected responses.
This pairs correctly with Adversarial Testing, which deliberately tests an ML model's ability to handle small input perturbations.

(3) C. Predictions made by the ML model might be inaccurate in some cases.
This is best addressed by ML Functional Performance Testing, which directly evaluates whether the ML model produces accurate outputs.

(1) D. ML model accuracy might have significantly decreased since it was deployed.
This is appropriately matched with Drift Testing, which monitors performance changes over time in production. (4)

This is a part of the non-functional requirement/documentation.

When more tests are run for the same measured accuracy (83%), the margin of error typically decreases if the confidence level stays the same. This is because a larger sample size can reduce the margin of error and/or increase our statistical confidence in the measured value. If the ML model previously achieved 83% accuracy with a margin of error of ± 4% at a 94% confidence level, and additional tests were run while maintaining the same 83% accuracy measurement and maintaining the confidence level at 94%, this would result in a lower margin of error (e.g. ± 2%). This indicates we can be more certain that the true accuracy is indeed around 83%.

In adversarial testing, the adversarial examples are often created to identify vulnerabilities by perturbing working inputs.

There is no overlap between the outputs of T1 and T2, and no camera is missing.

Because ground truth is unavailable, static drift testing, which doesn't require it, is the only viable option.

The poor performance on test data and good performance on validation data suggest overfitting.

As stated in the syllabus, “Whenever the system is updated, A/B testing is used to test that the updated variant performs as well as, or better than, the previous variant.” This accurately describes how A/B testing is used in the context of ML systems.

Using a different ML development framework means using different libraries, potentially different underlying algorithms or implementations of algorithms, and a different ML development environment. This option maximizes the independence of the pseudo- oracle from the SUT, reducing the risk of shared defects and increasing the potential effectiveness of back-to-back testing.

This describes a potential used library defect. ML functional performance testing can evaluate model behavior and expose anomalies stemming from the use of a new library that is defective and causing a change in test results.

Shadow testing provides a risk-free way to see how a new ML model behaves with production data, whereas canary testing deliberately exposes a small subset of users to the new model to observe its real-world performance and impact.