Another Way to Find Bugs

Owned by Mike Deskis
Last updated: May 03, 2024 by Ioana Kocurova
3 min read

More approaches on how to find bugs are discussed in this lesson.

Learning Objectives

In this section you will learn about different structures for evaluating oracles in software testing, such as consistency heuristics and model-based testing, and highlights the limitations and challenges associated with regression test oracles. It also explains the concept of treating oracles as partial answers rather than strict rules, allowing for more flexibility in identifying potential bugs or design issues.

 

Core Topics and Takeaways

  • Regression testing

  • Test automation - basic advantages and disadvantages

  • Using models

  • Human decision - is it a bug or something we need to explore in details

Video Highlights

Topic

Key Concepts

Video Location

Topic

Key Concepts

Video Location

The structure for comparing and evaluating oracles involves using tables and recognizing that oracles provide partial answers rather than complete and correct evaluations.

  • Doug Hoffman's structure for comparing and evaluating oracles involves using tables.

  • Oracles provide partial answers that might be useful rather than complete and correct evaluations.

  • Constraint testing is an example of a partial oracle that helps identify certain types of errors.

  • Regression test oracles compare the output of a test in a previous build with the output in the current build to identify differences.

00:01

The video discusses the challenges of updating old tests and introduces the concept of oracles based on models.

  • Updating old tests can be time-consuming and not very productive.

  • Models are simplified representations of a system that make certain aspects easier to notice.

  • Oracles can implement different models to check if the program is consistent with the desired viewpoint.

  • The goal of creating a model as a tester is to develop a better understanding of the software.

01:43

Oracles can be used to test software and identify bugs, inconsistencies, and violations of predicted behavior.

  • State models can help identify inconsistencies in the program and make it easier to understand.

  • Some oracles are based on models that have nothing to do with the software itself, but are used to identify when the software is wrong.

  • Statistical models can identify suspicious output that is highly improbable.

  • Human intuition can also serve as an oracle, detecting oddities and problems that may not have been considered by automated oracles.

  • Recognizing certain problems may be too complex for a test machine.

03:27

Treating oracles as heuristics allows us to think about suspicious or unlikely results, and design choices that may be problematic.

  • Oracles should not be seen as hard and fast rules, but rather as guidelines.

  • Using oracles as heuristics gives us more freedom to think about partial answers.

  • It helps us consider design choices that might be problematic, even if they were intentionally put into the code.

05:11

Learning Highlights

Constraint testing introduces the concept of the partial oracle, focusing on determining if an answer is plausible or impossible rather than definitively right or wrong. Regression tests involve re-running previous tests, but automation can lead to extensive manual efforts in updating tests. Models, such as state models, help testers understand and explore software behavior. Statistical models check if software conforms to predictions, while human intuition serves as a valuable oracle, detecting nuances machines may miss. Treating oracles as heuristics offers flexibility in evaluating results.

Core Learning Concepts

Oracle

An oracle in the context of software testing is a mechanism or principle used to determine whether a program's output is correct. It serves as a reference for comparing the actual output of a program with the expected output. Oracles can take various forms, such as manually created oracles, automated oracles based on specifications, or oracles derived from models.

Oracle Based Model

An oracle based model could refer to a testing approach that utilizes oracles derived from models to evaluate the correctness of a system's output. This approach may involve using models to define the expected behavior of the system and then using oracles to compare the actual output with the predicted behavior. For example, an oracle based model could be using a formal specification of a software component as a model, and then creating oracles based on this specification to verify the correctness of the component's output. This approach helps in identifying discrepancies between the expected behavior defined in the model and the actual behavior of the software component.

Â