Network Protection & Automation Guide
21-12
testing may, for example, be to achieve 100% statement
coverage, in which every line of the code is executed at least
once, or to execute every possible path through the unit(s) at
least once.
21.6.3 Unit Testing Environment
Both Dynamic and Static Unit Testing are performed in the
host environment rather than the target environment.
Dynamic Unit Testing uses a
test harness
to execute the
unit(s) concerned. The test harness is designed such that it
simulates the interfaces of the unit(s) being tested - both
software-software interfaces and software-hardware interfaces
- using what are known as
stubs
. The test harness provides
the test data to those units being tested and outputs the test
results in a form understandable to a developer. There are
many commercially available testing tools to automate test
harness production and the execution of tests.
21.6.4 Software/Software Integration Testing
Software/Software Integration Testing is performed in the host
environment. It uses a test harness to simulate inputs and
outputs, hardware calls and system calls (e.g. the target
environment operating system).
21.6.5 Software/Hardware Integration Testing
Software/Hardware Integration Testing is performed in the
target environment, i.e. it uses the actual target hardware,
operating system, drivers etc. It is usually performed after
Software/Software Integration Testing. Testing the interfaces
to the hardware is an important feature of Software/Hardware
Integration Testing.
Test cases for Integration Testing are typically based on those
defined for Validation Testing. However the emphasis should
be on finding errors and problems. Performing a dry run of the
validation testing often completes Integration Testing.
21.6.6 Validation Testing
The purpose of Validation Testing (also known as Software
Acceptance Testing) is to verify that the software meets its
specified functional requirements. Validation Testing is
performed against the software requirements specification,
using the target environment. In ideal circumstances,
someone independent of the software development performs
the tests. In the case of high-integrity and/or safety critical
software, this independence is vital. Validation Testing is
‘black box’ in nature, i.e. it does not take into account the
internal structure of the software. For relays, the non-
protection functions included in the software are considered to
be as important as the protection functions, and hence tested
in the same manner.
Each validation test should have predefined evaluation criteria,
to be used to decide if the test has passed or failed. The
evaluation criteria should be explicit with no room for
interpretation or ambiguity.
21.6.7 Traceability of Validation Tests
Traceability of validation tests to software requirements is vital.
Each software requirement documented in the software
requirements specification should have at least one validation
test, and it is important to be able to prove this.
21.6.8 Software Modifications - Regression Testing
Regression Testing is not a type test in its’ own right. It is the
overall name given to the testing performed when an existing
software product is changed. The purpose of Regression
Testing is to show that unintended changes to the functionality
(i.e. errors and defects) have not been introduced.
Each change to an existing software product must be
considered in its’ own right. It is impossible to specify a
standard set of regression tests that can be applied as a
‘catch-all’ for introduced errors and defects. Each change to
the software must be analysed to determine what risk there
might be of unintentional changes to the functionality being
introduced. Those areas of highest risk will need to be
regression tested. The ultimate regression test is to perform
the complete Validation Testing programme again, updated to
take account of the changes made.
Regression Testing is extremely important. If it is not
performed, there is a high risk of errors being found in the field.
Performing it will not reduce to zero the chance of an error or
defect remaining in the software, but it will reduce it.
Determining the Regression Testing that is required is made
much easier if there is traceability from properly documented
software requirements through design (again properly
documented and up to date), coding and testing.
21.7 DYNAMIC VALIDATION TYPE TESTING
There are two possible methods of dynamically proving the
satisfactory performance of protection relays or schemes; the
first method is by actually applying faults on the power system
and the second is to carry out comprehensive testing on a
power system simulator.
The former method is extremely unlikely to be used – lead
times are lengthy and the risk of damage occurring makes the
tests very expensive. It is therefore only used on a very limited
basis and the faults applied are restricted in number and type.
Because of this, a proving period for new protection equipment
© 2011 Alstom Grid. Single copies of this document may be filed or printed for personal non-commercial use and must include this
copyright notice but may not be copied or displayed for commercial purposes without the prior written permission of Alstom Grid.