Selected CT-AI ISTQB AI Testing Questions Answers

Probabilistic and non-deterministic AI-based systemsdo not always produce the same output for identical inputs. This makes traditional testing approaches ineffective. Instead, the best approach is torun tests multiple times and analyze results statistically.

Statistical Validity:Running tests multiple times ensures that observed results are statistically significant. Instead of relying on a single test run,analyzing multiple iterations helps determine trends, probabilities, and outliers​.

Expected Result Tolerance:AI-based systems may produce different results within an acceptable range. Defining acceptable tolerances (e.g., "result must be within 2% of the optimal value") improves test effectiveness​.

A (Run Several Times for the Same Correct Result):AI systems are ofteninherently non-deterministicand may not return the exact same result every time. Expecting identical outputs contradicts the nature of these systems​.

B & C (Decomposing Tests into Data Ingestion Tests):While data ingestion quality is important, it does notdirectlysolve the issue of probabilistic test results. Statistical analysis is the key approach​.

ISTQB CT-AI Syllabus (Section 8.4: Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems)

"For probabilistic systems, running a test multiple times may be necessary to obtain a statistically valid test result."​.

"Where a single definitive output is not possible, results should be analyzed statistically rather than relying on individual test cases."​.

Why Other Options Are Incorrect:Supporting References from ISTQB Certified Tester AI Testing Study Guide:Conclusion:Sinceprobabilistic AI systems do not always return the same result, the best approach is torun multiple test iterations and validate results statistically. Hence, thecorrect answer is D.

Back-to-back testing isused to compare two different versions of an ML model, which is precisely what is needed in this scenario.

The model initiallymisclassified dogs as wolvesdue to feature similarities.

Thetest team retrains the modelwith additional images of dogs and wolves.

The best way to verify whether this additional trainingimproved classification accuracyis to compare theoriginal model's output with the newly trained model's output using the same test dataset​.

A (Metamorphic Testing):Metamorphic testing is useful forgenerating new test casesbased on existing ones but does not directly compare different model versions​.

B (Adversarial Testing):Adversarial testing is used to check how robust a model is againstmaliciously perturbed inputs, not to verify training effectiveness​.

C (Pairwise Testing):Pairwise testing is a combinatorial technique for reducing the number of test casesby focusing on key variable interactions, not for validating model improvements​.

ISTQB CT-AI Syllabus (Section 9.3: Back-to-Back Testing)

"Back-to-back testing is used when an updated ML model needs to be compared against a previous version to confirm that it performs better or as expected"​.

"The results of the newly trained model are compared with those of the prior version to ensure that changes did not negatively impact performance"​.

Why Other Options Are Incorrect:Supporting References from ISTQB Certified Tester AI Testing Study Guide:Conclusion:To verify that the model's performance improved after retraining,back-to-back testing is the most appropriate methodas it compares both model versions. Hence, thecorrect answer is D.

For AI-enabled medical devices, the most required aspects for certification are safety, transparency, and side effects. Here’s why:

Safety (Aspect III): Critical for ensuring that the AI system does not cause harm to patients.

Transparency (Aspect IV): Important for understanding and verifying the decisions made by the AI system.

Side Effects (Aspect V): Necessary to identify and mitigate any unintended consequences of the AI system.

Why Not Other Options:

Autonomy and Maintainability (Aspects I and II): While important, they are secondary to the immediate concerns of safety, transparency, and managing side effects in life-critical processes.

References:This explanation is aligned with the critical quality characteristics for AI-based systems as mentioned in the ISTQB CT-AI syllabus, focusing on the certification of medical devices​​.

Pre-processing is an essential step in data preparation that ensures data is clean, formatted correctly, and structured for effective machine learning (ML) model training. One common issue that can be resolved during pre-processing isnumbers stored as strings.

Explanation of Answer Choices:

Option A: Insufficient data

Incorrect. Pre-processing cannot resolve insufficient data. If data is lacking, techniques like data augmentation or external data collection are needed​.

Option B: Invalid data

Incorrect. While pre-processing can identify and handle some forms of invalid data (e.g., missing values, duplicate entries), it does not resolve all invalid data issues. Some cases may require domain expertise to determine validity​.

Option C: Wanted outliers

Incorrect. Pre-processing usually focuses on handling unwanted outliers. Wanted outliers may need to be preserved, which is more of a data selection decision rather than pre-processing​.

Option D: Numbers stored as strings

Correct. One of the key functions of data pre-processing isdata transformation, which includes converting incorrectly formatted data types, such as numbers stored as strings, into their correct numerical format​.

ISTQB CT-AI Syllabus References:

Data Pre-Processing Steps:"Transformation: The format of the given data is changed (e.g., breaking an address held as a string into its constituent parts, dropping a field holding a random identifier, converting categorical data into numerical data, changing image formats)"​.