Why is Multiple Rounds of System Testing Recommended?
If you are new to system testing, you may be wondering what it is and why there is typically multiple rounds (or cycles) of testing.
Why are multiple rounds of testing needed? Because each initial round typically results in defects. Each time a new build is introduced to address the defects, a new cycle is started to verify that the inial tests, that were passed, have not been adversely affected.
Now that you know why multiple rounds of the system is required, you may be wondering what is this “Gold Build” that people talk about, the objectives of system testing, who actually executes the tests and much more.
The Theory of the “Gold Build”
The Gold build is the ideal build, that should, in theory, have little to know defects after a round of system testing. But, let me explain where this all comes from…
System Testing is needed because a system is very rarely correct on the first iteration. Typically after the expected tests are run a number of defects are detected.
If you continued where you left off after the defects have been fixed you would not know if the previous tests, that had been run and passed, have been tainted by the new code that had been implemented, are you with me?
Therefore, it is important to repeat these tests, that have previously been run, on a new iteration (cycle/round) to confirm that there have been no adverse effects to previously tested code. (Basically a form of regression).
The Gold Build itself
Usually, after a couple of rounds of testing in this manner, you can get to what is deemed as the “gold build”. The gold build is expected to give little to no defects and be the final round of testing before completing the system testing phase.
However, in reality, these things never quite go as planned. And, there can be a lot more testing iterations (cycles) than expected. Usually, based on my experience, a judgment call by the stakeholders is made, typically called a “Go-No-Go” meeting to decide if the code will be promoted.
What is system testing?
System testing focuses on the entire system as a whole. As opposed to looking at it from a functional by function perspective. At this phase, you’re looking at the entire system. It is often regarded as an end-to-end view of testing.
System testing is quite important because it is often part of a decision if the system is safe for release. In some cases, it is used for a legal or regulatory quality gate.
So, what are the objectives (Goals) of system testing?
System testing is a key part of the Software Development Lifecycle (SDLC). Because one of its main objectives is reducing risk to the production environment.
Its aim is to detect as many defects or functional issues before the system is promoted to the live environment. It is a formal method of validating that the system matches the agreed specification.
Verifying the System matches the agreed specification
Obviously, at this stage, there can be situations where the system does meet the agreed specification, but it still does not function as one would logically expect.
This could turn out to be more of a design issue rather than a system testing error (but more on this later, as to why testers should be involved early).
Finally one of the additional objectives of system testing is to provide more confidence that the system will work as expected and also no reduce the risk of defects being passed onto a later level of testing such as you 8000 etc.
What test environment is required for system testing?
Ideally, for system testing, you need a system that closely matches the true production system. However, this is not always realistic due to the cost or resource availability. Either way it is important to try and emulate as closely as you can to the real production system.
This is because you want to feel confident that when it actually goes into production it will perform in the same way that you had it working in system test.
How can automated regression help?
During system testing, as discussed earlier, there may be multiple iterations (cycles) of tests. During each of these iterations, a new build can be implemented which has defect fixes.
Speed up the regression
It is an opportunity to use automated regression to speed up these repeated tests so you can clarify that the defect build that has been implemented has not adversely affected the system. Obviously, this regression automation can add complexities as well.
Example of test Basis that can be used for system testing.
To make sure you have a reference to base your system tests against, it’s important to have a test basis. Documents that can be used for system testing are:
- System design specifications
- Functional Specification
- User stories or Epics
- User manuals
In my experience, at this stage, you’ll often be using functional specs. Especially if you are using traditional models such as the v-model. However, don’t be alarmed to be asked to test, with nothing but the system, but that’s another story.
However, user stories are just as good and quite commonplace these days, especially with agile (Click here for the advantages of the Agile model).
You may even use user manuals at this stage, depending on what you were testing.
Examples of test objects during system testing.
During system testing, you will be testing system objects. However, this can come in many forms. such as:
- Operating systems
- Firmware (For Hardware devices)
- Configuration data (e.g. white-label, off-the-shelf, software configurations)
For example, in my previous roles we have used off-the-shelf products that have already been tested and are actually commercially available software.
In these situations, we were testing the configuration of these off-the-shelf software items. This was to prove that they met the configuration specification outlined by the client.
What kind of defects can you expect to see during system testing?
As you can imagine during system testing which is typically one of the largest phases of testing, there is a wide variety of defects that can be found.
Not performing as per the spec
However, typical examples of this will be system functions not performing as per the specification.
Basic Graphical User Interface (GUI) Issues
Even basic graphical user interface issues. These could be low priority defects, such as spelling errors or incorrect labeling.
Firmware related defects (for hardware devices)
I have had experience of defects that relates to the firmware of hardware devices. Such as XML messages that are expected to cause a certain action in the hardware device not responding as outlined in the specification.
What kind of techniques are used during system testing?
A number of different testing techniques can be used during system testing. This includes tactics such as:
- Decision tables.
- Equivalence partitioning.
- Boundary value analysis (BVA)
For example, using BVA, you will be testing the boundaries of a particular input or condition in the system. So, if you were expecting values in an input box of between 1 and 3 you would have three boundaries to test:
- two negative tests (under and over the expected value
- a positive test (for the expected value).
Who does system testing?
System testing, as you can imagine, is typically done by the system testers. The reason for this is system testing is quite technical and skilled activity.
It needs to be done by an individual with good competence and understanding of the systems involved and how to capture the defects adhering to the testing standards.
Therefore, you need a skilled professional that can ensure that the system has been tested correctly.
Why should testers be involved early?
Testers should be involved as early as possible because it helps to reduce the risk of finding big errors or defects later on in the cycle.
In the SDLC, defects that are found later typically cost the company a lot more money to fix. Therefore, it is better to get these defects found as early as possible.
For example, if big defects are found at the system testing stage, that could impact a lot of resources.
You will lose time with your system testers, as well as additional development resources having to recode the issue. Also, you may even have to involve the designers or the Business Analysts as well to rethink how they can factor in this is code/design change, are you with me?
Therefore, involving testers early, obviously as well as other stakeholders, can help to eliminate these issues as early as possible before any code or testing has even started.