Author Archives: Talented Tester Support
Author Archives: Talented Tester Support
If you are interested in UAT or acceptance testing in general. You may wonder if it is used in Agile or other testing models.
Does UAT exist in Agile? Yes, UAT does exist in Agile. However, it is not reserved for the end of the project. Instead, it can be performed at the end of each sprint. There are other forms of acceptance tests performed as well (more on this later).
Now that you know that UAT is included in Agile, I will go on to explain what other types of acceptance can be involved, who typically does the testing, the benefits and much more…
Acceptance testing is similar to system testing from the perspective that you’ll be testing the entire system. However, in acceptance testing, it can form a decision if the system is viable for deployment.
It is also used as a quality gate for regulatory or contract related projects (more on this later).
There are a number of different objectives for acceptance testing.
However, the overarching objectives are to get confidence that the system has been built to the user’s expectation.
Acceptance testing is typically assumed to be at the end of the SDLC process. Therefore, the assumption by all parties is that the system has been thoroughly tested. And, it is expected that there should be minimal or no new defects found.
In reality, finding no defects is quite unrealistic. Because you will inevitably find some defects. But, what I would say is, if there is a large number of defects found at this stage it is definitely a red flag.
In most cases, if there are too many defects, it is regarded as a high project risk. And often recommended not to deploy the code.
In my experience of testing, had we have found a large number of defects during the acceptance testing stage, there would have been a lot of questions asked. We would also expect to be questioned why simple defects were not picked up during testing. Believe me, these are uncomfortable questions to answer, especially under pressure.
Now that you understand what the objectives of acceptance testing is. It is important to understand that there are various other types of acceptance testing. Each different type has different objectives and resources to perform it. They are as follows:
When acceptance testing is mentioned most people will typically assume UAT. However, it is not the only form of acceptance testing. I will agree though, it is one of the most popular, from my experience.
UAT is an opportunity for the end-user to verify and get confidence that the system performs as expected.
This usually revolves around the user requirements (Test Basis – more on this later) that were drafted from the initial stages of the project.
This stage of testing is typically performed by business users, typically end-users who will be using the system on a day-to-day basis when it goes live. These resources are usually had picked based on their knowledge of the current system (if its an existing system, that is being upgraded).
However, if it is a brand new Greenfield system, that has never been used before, then you will typically have the users who are expected to use the system in anger once it goes live into production.
Operational acceptance testing is another form of acceptance testing that is often overlooked. Essentially this involves highly technical resources, such as tech support.
The focus of this acceptance testing is different from UAT because these testers want to feel confident that they can support the system when it goes live.
For example, this could include tests like backing up and restoring the system in a simulated disaster (disaster recovery), etc.
This is quite a technical level of acceptance testing and requires experts that have had, ideally, years of experience or acquired a high level of skills and knowledge in a short space of time.
Quite a mouthful but ultimately it comes down to two separate types of testing. but there are aspects of which overlap.
Contractual acceptance testing involves testers (or representative testers) verifying that the system in question meets the agreed contract specification.
The agreed contract specification is often used to verify that the system is operating as expected. This contract and agreed behavior is usually created at the start, once a contract is agreed, at the very beginning of the project, way before testing starts.
Regulatory acceptance testing is slightly different because it involves testing to verify that a system meets certain regulatory standards.
During my experience of testing, I worked for an energy company (Gas & Electric). They were launching a Customer Management System (CMS) in a new European country.
However, the difference with this particular country is, they had a strict regulatory licensing quality standard that needed to be proven and verified before the system could go live and be awarded its license.
Therefore, I was drafted as a test manager to prove to the regulatory representatives that the system was fit for purpose to go into production in their country.
Typically for these regulatory acceptance testing, you will have a neutral third-party company that will verify what you have tested to ensure it meets their standards.
I also had experience of this, which included verifying that the test assets were created as per their specification as well as face-to-face visits to witness test what were testing.
Depending on who you are working with these tests can be quite stressful. Why? Because they usually have to be done within a restricted time frame. And, there is not usually much chances to get this right.
And, to make matters worse, the stakes are usually quite high because of the investment that has been made to get the software to pass the regulatory standards before the system can go live.
Alpha or Beta testing is a strategy used to verify that the system is fit for purpose. However, it is typically used by developers of commercial off-the-shelf software (COTS).
The objectives of this testing is to get an early understanding of the system and verify it is fit for purpose.
You may have even noticed some of these strategies used by big players such as Google. Sometimes they roll out an early stage “Beta” and get user feedback, before rolling out the final product.
The idea is to get a good dataset from the intended customers to understand if the system is working before you roll out the fully tested and production-ready version.
Like most phases of testing, test cases will typically be drawn up. However, a test basis is required to write the test cases against, are you with me? Typical documents that are used for test basis are as follows:
However, we also discussed OAT earlier. This usually requires some of the following:
As well as the test basis documents. There is obviously the actual objects that will be tested. Such as:
At this stage, defects are expected to be very little. However, as I mentioned earlier, you will expect to see some. The types of defects that usually show up during this phase are as follows:
All of these examples are important and need to be investigated. And, at this stage in the project, there is very little time to rectify these issues.
During this testing, depending on which type of acceptance testing, there can be various different resources testing it.
For UAT, you would expect the end-users to be involved or potentially some of the stakeholders.
For OAT, this is typically done by system administrators, support workers or a knowledgeable resource that has lots of working knowledge of the system in question.
Regulatory and contractual acceptance testing is typically done as a collaborative effort with the actual system testers and a representative (from the regulatory body or third party contract owner/partner).
As mentioned earlier most people assume that acceptance testing is done at the end of testing after system testing has completed.
In my experience, this is usually been the case. However, it does not necessarily have to happen at this stage. Especially when you are dealing with iterative development methodologies such as agile.
This is because each iteration is like a mini-project within itself or known as a sprint (click here for a full explanation). You may find different types of acceptance testing done at the end of each sprint, treating it like a mini-project.
If you are new to system testing, you may be wondering what it is and why there is typically multiple rounds (or cycles) of testing.
Why are multiple rounds of testing needed? Because each initial round typically results in defects. Each time a new build is introduced to address the defects, a new cycle is started to verify that the inial tests, that were passed, have not been adversely affected.
Now that you know why multiple rounds of the system is required, you may be wondering what is this “Gold Build” that people talk about, the objectives of system testing, who actually executes the tests and much more.
The Gold build is the ideal build, that should, in theory, have little to know defects after a round of system testing. But, let me explain where this all comes from…
System Testing is needed because a system is very rarely correct on the first iteration. Typically after the expected tests are run a number of defects are detected.
If you continued where you left off after the defects have been fixed you would not know if the previous tests, that had been run and passed, have been tainted by the new code that had been implemented, are you with me?
Therefore, it is important to repeat these tests, that have previously been run, on a new iteration (cycle/round) to confirm that there have been no adverse effects to previously tested code. (Basically a form of regression).
Usually, after a couple of rounds of testing in this manner, you can get to what is deemed as the “gold build”. The gold build is expected to give little to no defects and be the final round of testing before completing the system testing phase.
However, in reality, these things never quite go as planned. And, there can be a lot more testing iterations (cycles) than expected. Usually, based on my experience, a judgment call by the stakeholders is made, typically called a “Go-No-Go” meeting to decide if the code will be promoted.
System testing focuses on the entire system as a whole. As opposed to looking at it from a functional by function perspective. At this phase, you’re looking at the entire system. It is often regarded as an end-to-end view of testing.
System testing is quite important because it is often part of a decision if the system is safe for release. In some cases, it is used for a legal or regulatory quality gate.
System testing is a key part of the Software Development Lifecycle (SDLC). Because one of its main objectives is reducing risk to the production environment.
Its aim is to detect as many defects or functional issues before the system is promoted to the live environment. It is a formal method of validating that the system matches the agreed specification.
Obviously, at this stage, there can be situations where the system does meet the agreed specification, but it still does not function as one would logically expect.
This could turn out to be more of a design issue rather than a system testing error (but more on this later, as to why testers should be involved early).
Finally one of the additional objectives of system testing is to provide more confidence that the system will work as expected and also no reduce the risk of defects being passed onto a later level of testing such as you 8000 etc.
Ideally, for system testing, you need a system that closely matches the true production system. However, this is not always realistic due to the cost or resource availability. Either way it is important to try and emulate as closely as you can to the real production system.
This is because you want to feel confident that when it actually goes into production it will perform in the same way that you had it working in system test.
During system testing, as discussed earlier, there may be multiple iterations (cycles) of tests. During each of these iterations, a new build can be implemented which has defect fixes.
It is an opportunity to use automated regression to speed up these repeated tests so you can clarify that the defect build that has been implemented has not adversely affected the system. Obviously, this regression automation can add complexities as well.
To make sure you have a reference to base your system tests against, it’s important to have a test basis. Documents that can be used for system testing are:
In my experience, at this stage, you’ll often be using functional specs. Especially if you are using traditional models such as the v-model. However, don’t be alarmed to be asked to test, with nothing but the system, but that’s another story.
However, user stories are just as good and quite commonplace these days, especially with agile (Click here for the advantages of the Agile model).
You may even use user manuals at this stage, depending on what you were testing.
During system testing, you will be testing system objects. However, this can come in many forms. such as:
For example, in my previous roles we have used off-the-shelf products that have already been tested and are actually commercially available software.
In these situations, we were testing the configuration of these off-the-shelf software items. This was to prove that they met the configuration specification outlined by the client.
As you can imagine during system testing which is typically one of the largest phases of testing, there is a wide variety of defects that can be found.
However, typical examples of this will be system functions not performing as per the specification.
Even basic graphical user interface issues. These could be low priority defects, such as spelling errors or incorrect labeling.
I have had experience of defects that relates to the firmware of hardware devices. Such as XML messages that are expected to cause a certain action in the hardware device not responding as outlined in the specification.
A number of different testing techniques can be used during system testing. This includes tactics such as:
For example, using BVA, you will be testing the boundaries of a particular input or condition in the system. So, if you were expecting values in an input box of between 1 and 3 you would have three boundaries to test:
System testing, as you can imagine, is typically done by the system testers. The reason for this is system testing is quite technical and skilled activity.
It needs to be done by an individual with good competence and understanding of the systems involved and how to capture the defects adhering to the testing standards.
Therefore, you need a skilled professional that can ensure that the system has been tested correctly.
Testers should be involved as early as possible because it helps to reduce the risk of finding big errors or defects later on in the cycle.
In the SDLC, defects that are found later typically cost the company a lot more money to fix. Therefore, it is better to get these defects found as early as possible.
For example, if big defects are found at the system testing stage, that could impact a lot of resources.
You will lose time with your system testers, as well as additional development resources having to recode the issue. Also, you may even have to involve the designers or the Business Analysts as well to rethink how they can factor in this is code/design change, are you with me?
Therefore, involving testers early, obviously as well as other stakeholders, can help to eliminate these issues as early as possible before any code or testing has even started.
Integration testing is a pivotal part of testing and one of those areas that is always covered in the ISTQB syllabus. However, who actually does this and what is it? These are the questions I plan to cover in this article…
Who does integration testing? Developers and testers perform integration testing depending on the level of testing. There is a component as well as system integration testing that will be tested by their respective resources.
Now that you know who does what, let me go on to explain exactly what it is, the objectives, what types of defects you may expect to get from the process and more…
As discussed briefly, Integration testing is a term that encapsulates system as well as component-level testing. Therefore, there are different resources that perform each of these levels.
At the component level you would expect to see a developer doing the work. This is because they need to be intimate with the component and have a deep understanding of how the component works.
At the system integration level, this is typically done by testers. They also need to have a deep understanding, but not of the component, of the architecture of the system and how these interfaces should interact, are you with me?
Integration testing is an important part of the Software Development Life-cycle (SDLC). There is a number of objectives in this process and for that reason, I will give you an overview of this now.
At the integration level, we expect a level of confidence in the systems that are to be integrated. Whether that be the components or the systems themselves.
However, integration testing reduces the risks before getting into further levels of testing because there are basic integrations that may not be quite correct.
A quick example of this could be a basic XML file interface which is misinterpreted by two separate parties (but more on this later).
The benefit of finding defects early is reducing the cost further down the line. The cost of fixing defects towards the end of a project is substantially more than at the beginning or earlier stages.
Therefore it is important to capture the defects as early as possible. During integration testing is a perfect opportunity to see glaringly obvious integration issues such as what I mentioned earlier in a previous section about XML interfaces.
As discussed earlier there are different levels of integration tests at the component and the system level. But let me give you more detail on each of these levels…
At the component level, we are focused on the interaction of the individual components. This is a lower level than the system.
This is after a developer completes their component testing (unit testing). And then take it one step further to confirm that the component that they have just unit tested can interface with the next component in line.
System-level integration testing usually happens after system testing (Click here to see why multiple rounds of system testing is recommended). It is essential to confirm that the system has been tested before going into integration.
The objective here is to prove that the system integrates as defined in the specification with external systems, for example, web services.
So far we have discussed the objectives of integration testing and how it works between a component and a system level. But what will be used to confirm how the integration should work? This is where test basis items come in..
Examples of test basis documents are as follows:
During integration there are a number of different objects that can be used at both levels, as you can imagine. But, to give you an idea of these, I will list down a few of these objects to give you an idea.
Once you are in the actual integration test phase it is a good idea to find as many defects as you can, as I explained earlier. However, what kind of defects can you expect to see?
At the component level, typical defects include data issues, interface problems, or functions that may not be handled correctly. E,g, causing indefinite loops or system component errors.
At the system level, you will typically see defects around the actual system interfaces. one of the biggest issues that I personally experienced during my integration testing is XML interfaces that integrate with external systems that are hosted by third parties.
This usually comes down to ambiguous design specifications. These specs could be interpreted in two different ways. Or even data items that were not correctly defined and left open for debate.
It’s amazing the number of man-hours that can be lost just by a simple field not being defined correctly in a design specification.
However, this is the purpose of capturing these things at the integration level and not later on in the process when it cost more money.
Integration at the surface level may sound straightforward but it can get very complicated depending on how large the system is.
Also, it depends on how frequent and how far into the integration testing. Whether that be at the component or system level.
For large systems, for example, financial-based systems can be very complicated. Especially when you’re dealing with multiple third-party vendors that have to interface with different systems.
I remember dealing with a client who specialized in vehicle finance applications. During my time working for this client, I experienced a number of challenges with simple interfaces into their system.
During this process, one of the key things that helped push the project through was regular contact with the client and clearly defined specifications.
Briefly, we talked about integration testing at the system and component level but when is this usually done?
At the component-level, these tests will typically happen immediately after a component has been completed and unit tested.
However, logically speaking you need more than one component completed before the integration can be tested, obviously.
Therefore there may be a reliance on a collaboration of developers working together so that they can make integration happen.
As for the system testing, integration, as you can imagine, happens after system testing. Ideally. And again, there is a dependency on the 3rd party system being available.
Therefore, you can be at the mercy of third parties and playing the waiting game if they are not ready when you are.
You may be wondering how integration testing fits into the big scheme of things, right? Is it a black box testing activity or a white-box? The reality is, it can be white or black box testing depending on the test scenario. The best way to explain this is with an example…
Let’s start with a white box testing example. A white-box integration test could be the integration of how two web services integrate, at a component level.
An example of a black box test in this context could be, testing the integration of a vehicle finance system with an external credit referencing agencies system, in particular, making sure that the XML feed from the external agencies system integrates with your system.
So far you have learned about what integration testing is, its objective, etc. But why is it even needed? This is the question I will help to explain now…
Integration testing is essential because if dependant systems or components cannot talk to each other the system is useless, are you with me?
For example, from a system integration perspective, imagine that you have a credit checking online app, but you can log in and see your profile, but when you click the “get credit score” button you get no response because the interface to Experian is not integrated correctly, would you be happy to pay for that service? are you with me?
This is a simple but powerful example of how critical integration testing is.
In integration, there are a few different ways to actually implement it. Each one has an inherent risk associated. Let me explain what they are…
This is one of the riskiest methods. It essentially means that all integration will be tested together and fingers crossed, it will work. The problem is identifying what has caused the issue in this method. If there are so many integrations at once it is a real challenge.
As the name suggests, the testing will start at the top-level components or systems and work its way down the integration tree.
An example would be testing the login page loads the profile page of a CRM system.
As you can imagine, this is the opposite to top-down testing. The integrations at the bottom will be tested first until you reach the top. So, following on from the previous example, the login page would be tested last. This method is only really effective if the components or systems are ready at the same level.
If you are preparing for your ISTQB, you are probably keen to understand how stubs and drivers are used. For that reason, in this article, I will explain this to you…
Why are Stubs and Drivers Used in Component Testing? This is to enable a developer to unit test a code component without other dependent code modules being available. Stubs help to produce an expected output whereas a driver will send a required input to the code module.
Now that you know what they are at a high level, let me explain in more detail, with examples. Also, exactly what component testing is, the objectives and so much more.
For example, if you have an address book app and the dev wants to test the “login” code component, in the absence of a display contacts code module a stub could be used to test the login component in isolation.
In another example for a driver, if we use the address book app example again if the login page is not ready, but the “display contacts” code module is ready for testing. The developer can use a driver for the Login feature so he can test the display contacts component in isolation.
What is component testing?
This method focuses on a specific code component in isolation. It is also referred to as unit or module testing.
There is some confusion regarding unit and component testing, with other sources claiming/suggesting they are separate, but according to the latest ISTQB syllabus, they are one of the same thing. And, from my experience, they have been treated as one of the same as well.
Now that you understand what component testing is, from a high level. Let me explain how stubs and drivers are used. This will broaden your knowledge and expand on the brief explanation I covered in the earlier section.
As discussed, component testing is typically done in isolation. Meaning, to do this correctly other supporting objects will be required, such as:
Drivers are place holder (dummy) programs that call the function that you need to test. For example, earlier I used the address book analogy, this driver could be the login code module that calls the display contact component you wish to test.
These code modules take inputs from a calling function and display the required output. So, refereeing back to the address book app example, this could be used to replace the display contacts component while the developer tests the login component.
You may be wondering why do we need to use component testing. To help to make this clear I will explain the objectives of it in this section.
The main objective of component testing is to reduce the risk of the component. It is better to find defects early rather than waiting until later test levels. Why? Simple, it saves costs by detecting defects as early as possible.
It is far easier and cheaper to verify that the component meets the related specification before promoting it to integration or system testing.
I have had personal experiences in the past where code has been promoted to system test and there were glaringly obvious issues with the software that wasted many test resources time.
For example, after attempting to test the application, discovering that we are not even able to log in, are you with me?
It may not sound much to you right now. But, look at through the lens of a business. Each tester costs money per hour. If the code is not testable, the time lost could be half a day, or even worse. And, when you multiply that by the number of test resource booked to test, it gets pricey, are you with me?
Using simple techniques such as component testing at the earlier stage helps to reduce these risks.
At this component stage, another objective is verifying that the code module matches the agreed specification. This is from a functional as well as non-functional level.
If it is verified at this stage, it reduces the chance (but not eliminates) of defects going forward.
Automating regression for ever-changing components.
In some cases, when code components are changing frequently, it may be advantageous to automate some component regression tests to verify that the component(s) have not regressed since the last iteration. This is even more relevant when an Agile methodology is used.
Now that you understand what component testing is, its benefits and why we use it. It is also important to understand how it is verified. What do I mean? Well, how can you be sure it’s right? In this section, I will explain some documents that can be used to do just that. Also referred to as the “test basis”.
Here are a few documents that can be used:
In my experience, these are the most common documents used as a test basis. Typically, they clearly outline what is to be expected by the code components. Making the component testing easy to see if it is correct.
One thing to remember, these specifications can also be wrong, they are not perfect. However, that is not the objective of component testing. The concerns can be noted, but ultimately the task is to verify it meets the specification.
According to the latest ISTQB syllabus, the code and data models can also be used. To be honest, in my time I have not seen this done. But, I am not saying it’s impossible, just that I have not experienced this.
Understanding the high-level definitions of component testing is one thing, but grasping exactly, down to the code level, is another. What do I mean? Well, what type of code will be testing in this phase? Let me explain…
During component testing here are some examples of code that can be tested:
These are examples, in reality, there are more. But this is a good flavor of what can be expected.
As you can imagine, during component testing you will encounter many different issues. But, that’s a good thing, right? But, what kind of issues typically rear their ugly head? Let me explain…
During component testing here are some of the most common issues detested:
These errors are a lot easier to pick up these days. Why? Because most up to date code management software will do this automatically. But, there is always room for error.
These errors are more complex when compared to syntax errors. Why? Because they are not always obvious. The challenge with these errors is, it may look ok on the surface (passes syntax checks, etc.), but it won’t give you the expected result. This is where an experienced developer will save the business a lot of time.
Data issues can cause code errors where it is least expected. It could be a weird combination of data, that was not expected, that is actually used in some obscure scenario, that can trip up the said program.
So, now we know the type of issues we can expect. But, which resource is usually responsible for this testing? Let me explain…
Usually, the developer that actually coded the component is responsible for this testing. And from my experience, this is the case. However, I have seen many cases where the developers attempt to avoid this task and effectively promote the code directly to testers. Not ideal.
When new methodologies are being used, such as agile, with automation. There can be additional tasks thrown upon the developer to produce automated, component level, automated tests before even coding the module.
Personally, I think this is good practice. And, it saves the business wasted time and money on defective code.
You may be wondering how component integration fits into things, is it the same as component testing, or a completely different phase. In this section, I plan to explain exactly what this is.
Component integration should follow component testing, in an ideal world. At this level, it is assumed that the individual components are tested.
The objective here is to verify that the component talks to each other, as per the agreed specification. This is done to verify before moving onto the next level of testing.
Earlier we talked about component integration, and when it is expected to be done and established that it is its own unique phase. But, when is the actual component testing done? In this section, I will explain.
Component testing is typically done before component integration testing. In fact, logically, it has too. Why? Because the code components need to be verified first, before moving onto integration. Think of this as a necessary gate.
Ideally, once the code is written, competent testing will commence, to verify the code meets the detailed design or component specification.
If you are preparing for your ISTQB, or genuinely curious about the best white-box techniques prior to starting an up-coming test phase, then I have prepared some info to help you decide the best white-box testing approach.
Which white-box testing is best? The best white box testing technique is a combination of a decision based which is also run dynamically (more on this later). Because you have the potential of high test coverage with the ability to analyze how the code actually behaves.
Now that you understand which technique is best, in my opinion, let me go on to explain exactly what white-box testing is, the advantages, disadvantages, types, how it can be measured and much more.
White-box testing is effectively the opposite of black-box testing. It focuses on the internal structure of a test object, rather than just analyzing the inputs and outputs.
This comes down to analyzing the actual code of the software-based on a detailed design or equivalent specification document.
Now that you have a general idea of what this technique is, you may be wondering what the point of it is. Why should you even care about it, right? Well, let me explain some of the
This testing technique is ideal for improving the efficiency of the code, as well as spotting clear errors before getting into further phases of testing.
In my experiences of testing, developers are sometimes hesitant to do this, with the attitude that they are not testers. But, in my opinion, this is the mark of a great developer who truly checks his work before throwing it over the fence to the testers (so to speak).
Like most things, there are always set-backs with every technique. So, in this section, I will explain some of the disadvantages of this testing technique.
Firstly, whilst it’s a great way to weed out erroneous code early on in the Software Development lifecycle (SDLC). It relies on your organization having skilled developers that can perform the tests.
Usually, it the actual developer who wrote the code in the first instance that actually executes the tests, so you can guess the level of expertise required here.
The point I am getting at here is the cost. To do this correctly you need a decent budget for a skilled resource.
Impossible to catch everything
Whilst this technique helps to improve the code, prior to getting into later stages of testing, such as system testing, etc. It needs to be said, to set your expectation, that it is impossible to catch every potential defect at this level. In fact, ever, throughout any testing stage for that matter.
Don’t get me wrong, it’s worth the investment, but it is worth explaining this upfront.
In this section, I will now explain some of the actual white-box techniques that can be used. According to the ISTQB syllabus, this testing can occur at any test level. However, I will explain some of the most common techniques used.
White-box testing is typically performed by a developer at a code unit test level using one of the following techniques:
The statement based technique focuses on the individual lines of code to be delivered. The idea is to analyze the actual lines of code in the software program and have tests that verify these lines.
Decision-based techniques are slightly different. Instead of focussing on the lines of code it revolves around the conditions in the code. When I say conditions, I mean the decisions, for example, IF or case statements.
As you can imagine there are many conditions within code that will create many permutations.
Whether you use statement or decision-based techniques there is another choice to make. This is how you will actually execute these tests. For this there are two choices:
Static is a manual approach. This relies on the developer going through the code and eyeballing the code to see obvious errors. This can be done by the creator of the code.
But, it is more effective if it is done by a different developer with a similar skill set. These reviews are typically called peer reviews.
The benefit of the peer review is; it introduces fresh eyes. Often a developer will overlook obvious issues because they have simply spent too much time steering at the code. Whereas a new pair of fresh eyes will look at it from a different angle.
This is an opportunity for the reviewer to not only spot obvious errors but also suggest smarter or more efficient ways of doing things.
Dynamic is the opposite. This comes down to running the code and analyzing the outputs. This gives the developer the opportunity to compare the outputs to what’s expected from the specification.
This is a really effective method because the software can be complicated and even the most experienced developer may be surprised with output that seems to look perfect until it is dynamically run.
Understanding the concept of white-box testing is one thing, but grasping how to measure it is another. And, arguably one of the most important factors that will govern the next stage of your test object. For that reason, I will explain how this is done for statement & decision-based techniques.
For statement-based testing, the lines of code is the main focus, as discussed earlier. Therefore, for this method, the total lines of code (statements) are compared to the total tests that execute these lines.
The coverage is typically expressed as a percentage of statements executed to total statements of code.
For decision-based testing, the conditions (decisions) in the code is the main focus, as discussed earlier. Therefore, the coverage is based on this metric.
In particular, the total number of decisions executed by the tests is expressed as a percentage of the total number of decisions.
With this intricate testing, you can expect that it will require a certain level of skills to get it right. For that reason, in this section, I will explain exactly what level of skills you will expect to have.
White-box testing requires a resource that fully understands the code structure of the test object. In reality, every code object is different, so each skilled resource will develop their own unique expertise in one particular area of the system.
Either way, to begin with, you need a technical low-level coding resource that can do this task correctly. In reality, it doesn’t necessarily have to be a developer, but in my experience is nearly always is.
In this section, I will answer some questions related to white-box testing. If you have some additional questions that are not answered here please drop a comment below.
White-box focuses on the internal structure of a test object. Whereas in black-box testing, the internal structure is not important. All that matters are the inputs and outputs meet the specification.
Also, with black-box testing, you would expect testers to take on this task. However, with white-box testing, it is typically developers that do this.
This is typically measured by the total number of integrations executed by your tests as a percentage of total integrations.
Integrations are different. The expectation at this stage is that the individual component (click here to see why stubs and drivers are used in component testing) has already been tested and fit for purpose. Therefore, this stage focuses on how the outputs of component A interface with component B.
These integrations are usually in the form of a file (typically XML) with various fields and data. These data points need to match the specification to pass. Hence the need for the component level integrations. And, even more, an important way to measure its coverage.
If you are preparing for your ISTQB or just curious, you may be wondering what non-functional testing (NFT) is and how it fits into the testing life-cycle. Let me explain…
What is Non-functional Testing? It tests how well a software system behaves, based on an agreed specification. In particular, it focusses on certain aspects of the system, for example, performance, usability or security.
Now that you know what it is, let me explain a bit more detail Such as when is it used, how it can be measured if special skills are required, the types of non-functional testing and more. Keep reading…
Understanding what Non-functional testing is, is one thing. But, understanding when to use it is another. For that reason, in this section, I will explain when it is a good idea to use this. The answer may shock you because it’s not what most people assume or do.
Even though most software testers will assume that non-functional testing should be scheduled after system testing. The reality is, it should be done at all test levels. In fact, the earlier the better.
Why is this? Simply because in the event that you find a serious “show-stopping” non-functional defect. It is better to detect it sooner rather than later. The cost of finding it later is nearly always significantly more.
While planning non-functional tests you need a method to work out the test casts. For that reason, in this section, I am going to explain how you can use some known Black-box testing techniques to draw up these cases.
One of the known Black-box techniques, called boundary value analysis (BVA) can actually be used to derive non-functional test cases. IN BVA, the agreed boundaries of fields and inputs are tested to make sure that they meet the agreed specification.
Without the ability to report and quantify testing would effectively make Non-functional testing worthless. But, you may be wondering, with that being said, how can it actually be measured, right?
The coverage of non-functional tests relies on clear requirements/specifications. For each non-functional element specified, it is possible to measure the coverage based on the number of tests run per element.
This can then be used to help to find any gaps in the testing. Such gaps may be deemed as acceptable based on the risk it poses to the system, assuming a risk-based analysis is used.
Non-functional tests cover a large set of possible tests. But, you may be interested to know if any resource can run or design these tests, or does it need a special level of expertise?
In most cases, non-functional tests, for example, performance tests, typically need a specialized resource with a great level of experience. This experience could be deep knowledge of a tool, e.g. win-runner. Or specialised security-related skills to make sure the system is tested correctly.
In the last section, we touched on what type of skill set is required to do non-functional testing. However, I did not explicitly state who would be the typical resource member to do it.
Non-functional is quite a broad categorization, so there will be different roles for every different particular testing activity. For example, for performance testing, you would typically have a dedicated performance test analyst. If the testing required is quite significant, this could be a team of testers. With one of these resources being a Test lead or senior tester.
If you have security testing in the scope of your project, or you are just curious, you may be wondering what this is classified as. In this section, I will explain this.
Security testing is classed as non-functional. It is required to verify that the system in question is secure. Such as resilience to external infiltration.
To make sure that it is clear, I want to draw a distinction between functional & non-functional testing. This will help you to understand what NFT is.
Functional testing (Click here to see why Functional Testing is Required) is concerned with the functional elements of the system. Typically, this is defined in a functional spec as “features”. At this stage, we are not concerned with code-level testing. It is mainly black-box tests that focus on the inputs and expected outputs of each function. And, verifying that they meet the agreed specification.
Non-functional testing is different because it focuses on aspects of the system that are not usually addressed during functional testing. Such as how many users can use the system simultaneously (max load) before it stops responding, are you with me?
Then, to follow on with this example. Once we establish the max load, how does this compare to the documented non-functional specification? Does it support as many users as we expected, or not?
Reading about NFT is easy, but understanding it is another. Sometimes a real life example will help to put things into perspective. For that reason, in this section, I will give you an example of my first-hand experience using this method.
In one of my contracts in the past, the client was responsible for providing vehicle finance. Therefore, they have a large amount of critical data that needs to be secure and resilient. In particular, in the event of a disaster, e.g. office fire, damage to servers, etc.
I was tasked with leading an annual task they had. Which was to verify that they can recover all of their core systems following a disaster. To do this we emulated the situation by creating a “disaster day”
This testing is known as Disaster recovery. And, is a known example of non-functional testing. The objective was to prove that we can prove that each core system can be used following the disaster.
To prove this, we had systems located on another site. Their sole purpose was to provide a different backup environment that we could switch to following the disaster, are you with me?
On the recovery site we had two goals:
To prove this, we had a selection of Subject Matter Experts (SME’s). Their role was to test the recovered systems and verify that the core functionality was operational, following an agreed test plan. The time was limited (one day on site for all tests), so we had to be efficient.
The second goal, and arguably one of the most important, was to prove that the data was recovered successfully. This was done by taking a snapshot of the data before the disaster day. And then, comparing that snapshot on the recovered system.
To clarify, although this sounds complicated, it was done as simply as possible (due to time constraints. The data snapshots were as simple as taking some screenshots before the disaster and then comparing them after the data was restored on the backup systems, are you with me?
You may be wondering what is so important about these non-funtional requirements. And, you may even consider cutting corners and leaving them as a “nice to have”. But, in this section I will explain why they are required.
One of the main reasons why it is important is the performance boundaries. Once you understand these boundaries, e.g. how many users can use the system simultaneously. It can help your organization understand what they need to do to scale up. It can give you an idea of the cost of this and the impacts it may have on other projects, resources, etc.
Also, it can give an insight to see how long the system is useable in its current form. This can help with planning for future quarters, etc.
If you are preparing for your ISTQB (Click here for my prep guide), you may need to understand the nature of functional testing, and how it works. Let me explain…
Why is functional testing required? It verifies that the agreed behavior of each function works as specified in the related requirements. It is an important part of software testing and is covered throughout all levels during the software development lifecycle.
Now that you understand what functional testing is, at a high level. Let me now explain how it is measured, what skills are required, the different types and much more. Keep reading…
Functional testing is a Test type that focuses on the functions that a system has been designed to do. It can be used at all test levels and uses techniques such as Black-box testing (more on this later) to derive its tests.
The test cases that form this are based on functional requirements. These functional requirements can be across varying different levels and stages. For example, functional specifications, user stories, etc.
You may be wondering if these functional tests can cover such a broad amount of levels, how can the actual coverage be measured? In reality, this is done quite easily, assuming you have a documented test basis to work with, e.g. functional specification. Let me explain…
The coverage is measured based on the functions documented against the number of test cases created. This can be done in the form of a traceability matrix at a high level. These gaps that are defined (if found) can help to tighten up the test coverage.
In this section, I will discuss what skills are usually required when designing or executing functional tests. As you can imagine, depending on your specific requirements, it can be quite a technical process.
As I stated it is a technical process and for that reason, it requires skills or experience with the system or business process. This isn’t a deal-breaker if the individual resource does not have this, but in the absence of this, there will be a steep learning curve until it is mastered.
As you can imagine there are various different types of functional tests, as I mentioned earlier this testing is conducted at multiple levels. For that reason, I will break down some of these types to help you understand.
The following is a list of functional test types:
Component testing is a testing technique that focuses on the individual test component (or code function), for example in a calculator application, the “subtract” component (or code function) is responsible solely for subtraction.
Therefore, in component testing, you would have a test that verifies this component performs as per the specification. At this level, there is no regard for whether or not it integrates with the calculator application correctly.
This is closely related to component testing as you may have assumed. However, the key difference is how these components talk to each other.
At this stage, we assume that the component has been tested and works as expected. We only focus on how the output of component A integrates with the inputs of function B.
e.g. In the calculator App, that we referred to earlier. The substation component would be tested to see how it integrates with other components in the calculator app.
System testing ( is concerned with the system as a whole. It primarily looks at the end-to-end tasks or features that a system is expected to perform.
The main objective of this is to find system defects before they get into a live production environment. As well as verify that the system behaves as specified.
System Integration Testing (SIT – Click here to see What the difference is between SIT and UAT testing) is closely related to system testing, as you can imagine. However, the objective of this testing is to verify how the system integrates with other systems.
For example, in a parking application, there is an interface to the vehicle database that matches number plates scanned with real vehicle information.
The system integration test will verify that a number plate requested from system A will return the vehicle data successfully from System B.
This is, broadly speaking, testing that validates if the whole system is fit for purpose before going into production. It can be done by various different users of differing skill levels, for example, Operational Acceptance Testing (OAT), is typically run by very skilled support staff. Who typically have deep knowledge of the existing system (or product).
If you are wondering what an actual functional test looks like, then this section is for you. I will be explaining more about how these tests are structured and the common attributes of these tests.
Depending on what type of functional test you are executing, the actual look of the test can vary. However, there are some common attributes that you would expect. Even if the tests are very informal. These include the following:
This is the expected setup before executing the test. For example, it could be a previous test is done first, so that the data is in the correct state, etc.
This is a high-level description of the test purpose, objective, etc. It explains what this test is about.
These are the actual, step-by-step instructions to execute the test. Depending on what company you work for they can be really low-level detail (e.g. right-click on the “ok” button to submit the data”) or high-level tests that assume a level of experience (e.g. “submit the data”).
This is what we expect to see once the test has been executed. This is important because it will let us know if the test was successful or not. You may fee that this should be obvious, but, understand this, some tests could be negative, meaning the expected result could be an error.
This is the actual result of the test. Typically, this will be “Pass or Fail”. But, there can be other statuses such as “Blocked” that can be used.
If you have some interest in automation or are just thinking of ways to speed up test execution, then you may be wondering if functional tests can be automated. So, in this section, I will be explaining if this is possible and how this is typically handled at this stage.
Yes, automation can and is used during functional testing. The reality is, functional testing is quite a broad term that encapsulates many testing techniques, as you have seen already.
Therefore, there is quite a lot of scope to automate at different levels. If we take system testing, for example, this is an example of a stage where this can be used. I have used this personally at this stage. It is beneficial if you have a large set of tests that are quite similar/repetitive. This will help to speed up the process.
As we have discussed functional testing covers quite a few stages. So, you may be wondering if performance and functional are one of the same things. I will discuss this further in this section.
The short answer is no, performance and functional testing are different. Why? Because performance testing is classed as a non-functional test (Click here to see what is Non-Functional Testing).
These tests usually incorporate tests which are based on testing the performance of the system against the specification/requirements. An example of this is the number of concurrent users that a system can have at one time.
Earlier we talked about automation and it is a part of functional testing. But what about manual testing? Is that also included? This is what I will clarify in this section.
Yes, manual testing is used in functional testing. In fact, I would go as far as to say that it is used extensively. Obviously, these days there is more and more automation being used. However, manual testing is still used a lot across all test phases.
Manual testing is a method of test execution. It means that a tester will manually run test cases by supplying the necessary inputs, pre-requisites, etc. and then analyzing the results to confirm if the test passed or failed.
Once a system is promoted to live it is not the end of its updates or changes. In fact, there can be many iterations after it goes live, such as defects, new functionality, etc. But what is Change-related testing and what has that got to do with this?
What is change related testing? Change-related testing is a Test Type that has a specific purpose. It is designed to verify that a change to the system has been fixed correctly and does not have any adverse effects on the system as a consequence.
Now that you know what change related testing is let me explain what types of tests are performed during this time, how it relates to the overarching “Test Types” if some of these tests can be automated and more. Keep reading…
A test type is a group of testing activities that have specific objectives to test the system. Here is an example of other Test Types that are used:
You may be thinking this is all good, but when would you need to actually use change-related testing? In this section, I explain when and why you will use it.
This testing type is typically used following a change to a system. Usually, this is after a system is live, but this is not always the case.
These changes can be as follows:
If the system has just been upgraded due to defect fixes it is important to verify that the defect fixes have been fixed correctly and also check that their inclusion has not caused any unexpected issues to the current system (aka Regressions – more on this later).
If the system has been implemented and for whatever reason, it needs to be moved. Then this can be a risk. This risk needs to be mitigated with change-related tests to ensure that it is safe to use.
If the system has had modifications to improve or even change the behavior of the system, it needs to be tested to verify it. This will include tests to confirm that the changed functionality works as per the upgraded specification and also that it does not introduce any issues.
Similar to the section above, new functionality added will be treated much the same as changed functionality. The new functionality will be tested to verify that it works as specified, but it will also need to be checked for any regressions.
As discussed earlier change-related testing is an over-arching group of test activities. So, you may be wondering what specific techniques are used to test this. Let me explain…
This change-related test technique focuses on verifying that any new builds or code changes do not adversely affect the system. For example, a defect fix may be released to fix a calculation error in a timesheet system. The fix was designed to fix a miscalculation on weekend work rates.
After implementation the defect worked, however, the users have now discovered that their standard day rates are now being incorrectly calculated. This is an example of a system regression (Click here for Regression Testing Best Practices). And this is exactly what regression testing will deal with.
This technique confirms that the intended change meets its specification. So if we refer back to the example in the last section, the timesheet system defect fix. In this context, confirmation testing will be used to verify that the release fixed the issue with weekend rate calculations, which it did.
With all these potential tests that could be required on an ongoing basis, e.g. regression testing that we discussed earlier. You may be wondering if it would make sense to automate these tests. In this section, I will explore this for you.
When we look at the two main techniques in change-related testing the obvious one to automate is regression testing. Why? Well, confirmation testing is effectively a moving target, right?
Think about it, each time the system is changed for a defect or functionality change it could literally be anything. However, with regression you have some common tests that a run over and over again to check that the system responds in the same manner as it did pre-release, are you with me?
e.g. If we go back to our timesheet system we used earlier…. Every time an amendment is made to this system, it would be a good idea to check that the standard day-to-day calculations still work as designed.
This would be a perfect example of a group of tests that could be easily automated.
If you have heard of smoke testing before, you may be curious to see if it is used during Change-related testing as well. For that reason, I will clarify this in this section.
Smoke testing evolved from industrial engineering. When physical changes were done it was deemed as fit for purpose if no smoke was seen.
For that reason, it has evolved into a technique used in software testing. The purpose of smoke testing is to establish if the system passes some very basic tests, to confirm that the system is even remotely ready to test. This can be basic system functions that are mandatory to move forward.
Yes, smoke testing can and is used during change-related testing. Although it is not necessarily documented in exam syllabuses, in the real world it is used. I know from my own personal experience of using it in anger.
It helps to reduce wasted time and does not take much time to run the tests. In reality, these tests are so basic, one could argue that they are bordering on common sense tests that do not need a formal name or phase.
Earlier we mentioned Smoke testing and the inclusion in change-related testing. However, what about sanity testing? Could this be included also? Let me explain…
Firstly what is Sanity testing? Sanity testing is essentially a subset of regression testing. It aims to verify, with some basic tests, that the function in question is working. It does not confirm it, but it gives us an early view if something really basic is not working, which could render the regression testing a waste of time.
No, it’s similar, but not the same. Sanity testing focuses on the function under test. Whereas Smoke testing has a broad set of tests, which can be run in a short space of time.
Yes, it can and, like smoke testing, it usually is. Whether that is formally or informally. It is a great way to eliminate any time-wasting for a code fix that clearly has not addressed the problem.
In this section, I am going to answer some questions related to change-related testing. If you feel you have more questions you need answering, feel free to drop me a comment below.
Re-testing is effectively testing test cases that have previously failed. Whereas regression testing is verifying that the system has not “regressed” by re-running a subset of previously run tests.
Although they sound similar in action they are very different. Both are just as important as the other by the way, just have different objectives. Obviously, as discussed regression is one of the key activities in change-related testing.
If you are interested in an ISTQB course but have no idea what’s best for you, you are in the right place. I am going to recommend my best certifications in the ISTQB portfolio, let me explain…
What is the best Istqb course? For beginners the Certified Testing Foundation Level is ideal. For Test managers, The Advanced Level Test Managers certification is ideal. And finally, for Agile, The foundation level Agile certification is best.
Now that you know what courses I recommend, let me go on to explain how the ISTQB portfolio of courses is structured, the different streams they have, and what that even means in simple terms and more.
I remember going for a walk one lunchtime, with a tasty subway sandwich in hand. As I sat on the park bench reflecting on life and planning for my weekend, it made me appreciate life and take in a breath of fresh summertime air. However, it then dawned on me, that not everything is a walk in the park.
Same as choosing the best ISTQB course, nothing is always that straight forward, especially when you have multiple options. For that reason, let me explain now, how the courses are structured to help you.
ISTQB courses are structured based on different levels. In total there are three different levels:
Ultimately all of these levels higher levels are only possible if you have first done the foundation first.
In addition to these levels, they have another dimension which is called “Streams”.
These streams are grouped as follows:
As discussed in the previous section there are three different streams. Let me explain…
The core stream is based on the original ISTQB core fundamentals. These are applicable across all of the ISTQB certifications. Hence the reason why it is important to do the core foundation course before moving on to any other levels.
As the title of this Stream suggests it Focuses particularly on Agile. As software testing evolves, agile is becoming a critical part of software development and testing. So this Agile stream focuses primarily on agile practices.
The specialist stream focuses on specific approaches or testing activities such as mobile, automation, pharmaceutical or even medical devices based testing.
Yes, across all of these certifications. The foundation core is an absolute must and cannot be skipped.
In addition to this, any advanced level certification can only be considered if the foundation level is done first. And as you may expect, any expert level will require that the advanced is done.
In this section, I’m going to recommend what I believe to be the best ISTQB course. However, with so many different levels and streams, your best choice largely depends on what level you are at in your career.
Or, what qualifications you have. For example, if you already have a CTFL, then my recommendation for this course is obviously not applicable to you.
Therefore my recommendations are based on a selection of a few different skill levels. In particular:
The best ISTQB course for beginners is definitely the certified tester Foundation level (CTFL). This is the prerequisite for other levels if you want to progress.
This course is aimed at anyone that is actually a tester that hasn’t been formally certified. Alternatively people interested in getting into the industry that has not had any experience.
For example, this could include:
It covers a number of different core modules including the fundamentals of testing. Such as, “what is testing?”. “Why do we test”? etc.
Also covers test techniques such as:
Other areas include Software Development Life-cycle (SDLC).
The advanced level test manager certification is for testers that have reached an advanced level in testing. They are either working as a Test Manager and require a certificate for their CV or career progression.
Alternatively, they are aspiring test managers that want to get to test manager level, but have adequate experience.
It is also helpful for advanced level IT workers that wish to get a better understanding of the requirements of a test manager.
This course covers a deep understanding of test planning, estimation and it also goes into people skills. People skills, in particular, is something that some courses tend to overlook.
This includes motivating staff, how to get the best out of your team, etc.
Prerequisites for this course is a CTFL and you also require a certain amount of industry experience to be eligible for the certification.
Once you have completed this certification you can then move on to other levels such as the expert level certifications.
The foundation level agile certification is one of the newer offerings. There is often confusion about Agile. In particular, how it should be applied and used in anger during a software project.
In my experience of working with agile, I find that I have never seen Agile implemented the way across any two companies.
The agile certification is an attempt to formalize the understanding of agile and to create a solid foundation of the core concepts and practices.
It covers such topics as the fundamentals of agile. Such as what agile is, how it is handled in a software testing environment, etc. It also goes into detail about the tools that can be used within agile and more.
The prerequisites for this course is the CTFL, which is required across all of the certifications as discussed earlier.
The next levels after this particular certification are currently in development. This includes plans to release an advanced level technical agile tester certification which will be a perfect add-on to this.
How long does it take to get an ISTQB? For the CTFL exam, it will typically be 60 minutes. However, for non-English speaking countries, such as in French-speaking parts of Canada, it is extended to 75 minutes.
The course is structured into 40 multiple choice questions. Each question scoring one point. To pass the exam you need to get at least 65% right. Which means you have to get 26 out of 40 to get a passing mark.
Is the ISTQB certification useful? Yes, in my opinion. It is useful for some people, depending on what area of work you are in. If you are a tester who is just starting out, this certification is good. And will be a booster on your CV. This can help you to get more recognition and jobs in the future.
However I would say, the certification does not necessarily make you a good tester. It just proves that you understand the fundamentals of testing and what is required in the industry.
I remember turning up for football training as a young lad. I was amazed to see one of the guys on the team had the brightest, most expensive football boots on.
I knew they were expensive and he let everyone know it. After I saw him get tackled and embarrassed for the third time in a row, it reminded me that just having the gear, does not mean you have all the ideas. It’s the experience and skill that matters, not the cost of your boots.
Similar to testing, you can take the best ISTQB course. However, real life test execution, office politics, and the real world of testing is a whole new kettle of fish. This can only be mastered with experience, not by sitting an expensive exam.
If you are involved in UAT or functional testing, you may have wondered what are the key differences, or if they are one of the same things. Well, let me explain…
Is UAT functional testing? No, UAT is not functional testing. UAT is a process of checking and validating that the user requirements have been met. This is typically governed by the User Requirements document. Functional testing, on the other hand, is verification that the system meets the functional specification.
To put things into perspective, it is possible to pass functional testing, with the system meeting the functional specification, but still failing UAT. But now that you know that they are different let me explain each one in more detail, the types of testing that fall into these categories and more…
UAT is a validation process that proves the contract (user requirements) between the project team and the user has been met. This is based on User requirements agreed from the beginning of the project.
This is typically conducted by end-users (customers). Or in some cases, it is done by a product owner who is working as a proxy for the actual end-user.
UAT testing (click here to see if UAT exists in Agile) is important because it confirms that the customer’s requirements have been met. It is also possible to find issues in the system that could have been missed during functional testing.
It is also an important quality gate. Often the final gate before the system goes live.
Functional testing (Click here to see Why Functional Testing is Required) is a way of verifying that each system function meets the functional specification document.
Inputs are put into the system with the actual results (or outputs) of these functions compared against the expected results defined in the functional specification.
I have mentioned the functional specification a few times in this article. Which is a key document in the traditional v-model, used in testing? However, in the real world, from my experience, you will often find that there is no functional specification.
This is the real challenge to a tester. Because, the books ion testing, or ISTQB exam never really prepares you for this.
The main activities of functional testing use black-box testing…
Black box testing is a functional-based procedure which does not concern itself with the underlying code within each functional module.
Therefore, a tester running a black-box test does not need to know the exact code behavior within that function. Meaning they don’t have to be a developer.
They just need to know what input is required and what to expect to get out of it, as per the defined specification. Hence the reason it’s called the “black box”.
Functional testing usually tests functions such as APIs, databases, user interfaces, etc.
In this section, I’m going to explain what different types of functional testing there is.
The following forms of testing would be regarded as Functional testing:
Integration testing is a classic example of this ( Click here to see the difference between System Integration Testing and UAT testing). During integration testing, you will be verifying if each individual system module can integrate and talk to each other successfully.
For this to be successful, inputs from one module will feed into another function. Hence the reason why it is typically called Integration testing.
So, for example, in one of my previous testing projects, we were working on a large financial system. This system was responsible for checking customer debt. Depending on the debt levels, certain actions would be triggered, etc a letter to a customer.
To gather accurate data on the customers, required multiple XML feeds into a central database. It would collect data, for example, address information and credit score data from external vendors.
Each one of these XML feeds was effectively integration tests that needed to be verified using functional testing.
Verification is what most people think of when they talk about system testing.
It is a procedure to verify that the system works as per the specification. In particular, the functional specification. In basic terms, it means – is the product built correctly?
Validation is different if you are thinking of user-related testing. Then, the chances are your thinking of validation testing.
Validation in basic terms means – are we building the right product?
It Focuses on confirming if the system that has been built and verified actually delivers on the user’s expectations.
And this is validated against the User Requirements that were drafted in the very beginning of the project.
Non-functional testing is test procedures that confirm non-functional related specifications such as reliability, performance, etc.
These tests are performed based on separate non-functional requirements. They are not covered during functional testing.
An example of a non-functional test would be: How many simultaneous users can be loaded onto the system?
Typically these tests are done by specialist test resource. They focus on non-functional testing. These testers are usually very well paid because they have rare skills that are often needed by large organizations.
If if you are ever considering a highly paid contracting role for the future, this is something to consider.
Usually, this testing is done by some form of automation such as LoadRunner, QTP, etc.
The reason for this is, it is usually quite challenging replicating these test scenarios without using test automation.
An example of non-functional testing that I was involved in, as a test manager. During a previous project for a large energy company, we needed to simulate some load on the system.
We didn’t have the budget for automation tools, and time scales were tight. So, to simulate this load, we required multiple people all in the same room.
Each person was using the system at one time to verify the system could handle the load. The reality is this method is quite hard to coordinate, and there is room for error.
Also, this is quite a costly affair because you need to have everyone coordinated in the same room at the same time. And time is money. For these non-functional tests, If the project has the budget, automation is easier. because you can click a button to replicate the load.
No, white box testing focuses on the underlying code in a function. This is typically referred to as unit testing and is usually executed by an actual developer.
It is a very important activity which typically happens before the system is passed on for functional testing. Think of it as an important prerequisite before you get into black box testing (or functional testing).
End-to-end testing is a testing method to verify that the entire system application, from start to finish, performs as expected. It verifies the flow of data works as per the specification.
Q: What is compatibility Testing?
This is classed as a non-functional test. Its main job is to verify that the application is compatible with different devices, browsers, etc. For example, let’s say you were working on a web application, such as an online sweet shop.
To perform compatibility tests, you would check the web app on multiple browsers to verify that it is compatible with the major browsers. This is just one example obviously.
These requirements, governing which browsers will need to have been documented and agreed on ahaead of time so that it is not an open-ended test on whatever browser the tester feels like testing. And to also ensure that the agreed coverage is met.