Читать книгу Software Testing Foundations - Andreas Spillner - Страница 27

Completeness check

It makes sense to begin by checking that all the components you want to test and the corresponding testware are available. This involves installing the test object in the test environment and checking that it can be started and run. If this check reveals no obstacles, testing can begin.

Our Tip Check the test object’s main functions

 Test execution should begin by checking the test object’s main function (see the “smoke test” section in the Other Techniques side note in section 5.1.6). If this reveals failures or deviations from the expected results, you need to remedy the corresponding defect(s) before continuing with testing.

Tests without logs are worthless

The test execution process—whether manual or automated according to the test execution schedule—has to be precisely and completely logged. You need to log each test result (i.e., pass, fail, blocked¹⁵) so that they are comprehensible to people who are not directly involved with the test process (for example, the customer). Logs also serve to verify that the overall testing strategy (see section 6.2) has been performed as planned. Logs should show which parts were tested, when, by whom, how intensively and with what results. If a planned test case or test sequence is left out for any reason, this needs to be logged too.

The importance of clarity and reproducibility

Alongside the test object (or test item), a whole raft of documents and other information are involved in test execution. These include: the test framework, input data, logs, and so on. The data and other information that relate to a test case or test run have to be managed so that the test can easily be repeated later using the same data and constraints. The IDs and/or version numbers of the tools used should be noted and recorded by configuration management (see section 6.5).

Is there really a failure?

If there is a difference between the expected and actual result, evaluating the test logs will help you to decide whether this really is due to a failure. If you do discover a failure, make sure you document it before you begin to look for its causes. It may be necessary to specify and execute supplementary test cases. You should also report the defect. See section 6.4.1 for more on test logging, section 6.3.4 for more on the defect report, and section 6.4 for a general discussion of defect management.

Retesting

Once you have corrected a fault, you need to check that the corresponding failure has also been resolved and that no new failures have been caused by the correction process. You may need to specify new test cases to verify the modified code.

Our Tip Check whether a defect is actually due to the test object itself

 You will need to check carefully whether a failure really is due to the test object. There is nothing worse for a tester’s reputation than a reported failure whose cause is actually a flawed test case. At the same time, you have to take care not to be over-wary of such cases, and you mustn’t be scared to report potential failures, even if you are not sure of their cause. Both these situations are bad for the project.

Individual fault correction is not practical

 Ideally, you will correct faults individually and retest to make sure that your corrections don’t unintentionally influence each other. However, this approach is simply not practical in the real world. If a test is executed by an independent tester rather than the developer, individual corrections won’t be possible. Reporting every fault to the developer and waiting for these to be remedied before retesting is not a justifiable effort. The usual approach is to correct bunches of faults and then set up a new version of the software for testing.

In addition to logging the differences between the expected and actual results, you need to evaluate coverage and—if necessary—log the run time of the tests. There are specialized tools available for these kinds of tasks (see Chapter 7).

Traceability is important here too

Bi-directional traceability is an important part of all the testing activities we have looked at so far. Here too, you need to check traceability and, if necessary, update the system so that the relationships between the test basis and the test conditions, cases, test runs, and results are up to date. Once a test sequence is completed, you can use its inbuilt traceability to evaluate whether every item in the test basis has been covered by a corresponding test process.

Test objectives fulfilled?

This way, you can check which requirements have passed planned and executed tests and which requirements couldn’t be verified due to failed tests or failures that were registered during testing. You may find that some requirements have not yet been verified because the corresponding tests have not yet been performed. Information of this kind enables you to verify whether the planned coverage criteria have been fulfilled and therefore whether the test can be viewed as successfully completed.

Effective coverage criteria and traceability enable you to document your test results so that they are easily comprehensible for all the project’s stakeholders.

Other common exit criteria are discussed in section 6.3.1.