Archive

Category Archives for "ISTQB"

Why is Multiple Rounds of System Testing Recommended?

If you are new to system testing, you may be wondering what it is and why there is typically multiple rounds (or cycles) of testing.

Why are multiple rounds of testing needed? Because each initial round typically results in defects. Each time a new build is introduced to address the defects, a new cycle is started to verify that the inial tests, that were passed, have not been adversely affected.

Now that you know why multiple rounds of the system is required, you may be wondering what is this “Gold Build” that people talk about, the objectives of system testing, who actually executes the tests and much more.

The Theory of the “Gold Build”

The Gold build is the ideal build, that should, in theory, have little to know defects after a round of system testing. But, let me explain where this all comes from…

System Testing is needed because a system is very rarely correct on the first iteration. Typically after the expected tests are run a number of defects are detected.

If you continued where you left off after the defects have been fixed you would not know if the previous tests, that had been run and passed, have been tainted by the new code that had been implemented, are you with me?

Regressional Testing

Therefore, it is important to repeat these tests, that have previously been run, on a new iteration (cycle/round) to confirm that there have been no adverse effects to previously tested code. (Basically a form of regression).

The Gold Build itself

Usually, after a couple of rounds of testing in this manner, you can get to what is deemed as the “gold build”. The gold build is expected to give little to no defects and be the final round of testing before completing the system testing phase.

However, in reality, these things never quite go as planned. And, there can be a lot more testing iterations (cycles) than expected. Usually, based on my experience, a judgment call by the stakeholders is made, typically called a “Go-No-Go” meeting to decide if the code will be promoted.

What is system testing?

System testing focuses on the entire system as a whole. As opposed to looking at it from a functional by function perspective.  At this phase, you’re looking at the entire system. It is often regarded as an end-to-end view of testing.

System testing is quite important because it is often part of a decision if the system is safe for release. In some cases, it is used for a legal or regulatory quality gate.

So, what are the objectives (Goals) of system testing?

objectives goals of system testing

objectives goals of system testing

System testing is a key part of the Software Development Lifecycle (SDLC). Because one of its main objectives is reducing risk to the production environment.

Detect Defects

Its aim is to detect as many defects or functional issues before the system is promoted to the live environment. It is a formal method of validating that the system matches the agreed specification.

Verifying the System matches the agreed specification

Obviously, at this stage, there can be situations where the system does meet the agreed specification, but it still does not function as one would logically expect.

This could turn out to be more of a design issue rather than a system testing error (but more on this later, as to why testers should be involved early).

Confidence

Finally one of the additional objectives of system testing is to provide more confidence that the system will work as expected and also no reduce the risk of defects being passed onto a later level of testing such as you 8000 etc.

What test environment is required for system testing?

Ideally, for system testing, you need a system that closely matches the true production system. However, this is not always realistic due to the cost or resource availability. Either way it is important to try and emulate as closely as you can to the real production system.

Why?

This is because you want to feel confident that when it actually goes into production it will perform in the same way that you had it working in system test.

How can automated regression help?

During system testing, as discussed earlier, there may be multiple iterations (cycles) of tests. During each of these iterations, a new build can be implemented which has defect fixes.

Speed up the regression

It is an opportunity to use automated regression to speed up these repeated tests so you can clarify that the defect build that has been implemented has not adversely affected the system. Obviously, this regression automation can add complexities as well.

Example of test Basis that can be used for system testing.

To make sure you have a reference to base your system tests against, it’s important to have a test basis. Documents that can be used for system testing are:

  • System design specifications
  • Functional Specification
  • User stories or Epics
  • User manuals

In my experience, at this stage, you’ll often be using functional specs. Especially if you are using traditional models such as the v-model. However, don’t be alarmed to be asked to test, with nothing but the system, but that’s another story.

However, user stories are just as good and quite commonplace these days, especially with agile (Click here for the advantages of the Agile model).

You may even use user manuals at this stage, depending on what you were testing.

Examples of test objects during system testing.

During system testing, you will be testing system objects. However, this can come in many forms. such as:

  • Applications
  • Operating systems
  • Firmware (For Hardware devices)
  • Configuration data (e.g. white-label, off-the-shelf, software configurations)

For example, in my previous roles we have used off-the-shelf products that have already been tested and are actually commercially available software.

In these situations, we were testing the configuration of these off-the-shelf software items. This was to prove that they met the configuration specification outlined by the client.

What kind of defects can you expect to see during system testing?

As you can imagine during system testing which is typically one of the largest phases of testing, there is a wide variety of defects that can be found.

Not performing as per the spec

However, typical examples of this will be system functions not performing as per the specification.

Basic Graphical User Interface (GUI) Issues

Even basic graphical user interface issues. These could be low priority defects, such as spelling errors or incorrect labeling.

Firmware related defects (for hardware devices)

I have had experience of defects that relates to the firmware of hardware devices. Such as XML messages that are expected to cause a certain action in the hardware device not responding as outlined in the specification.

What kind of techniques are used during system testing?

A number of different testing techniques can be used during system testing. This includes tactics such as:

  • Decision tables.
  • Equivalence partitioning.
  • Boundary value analysis (BVA)
  • etc…

For example, using BVA, you will be testing the boundaries of a particular input or condition in the system. So, if you were expecting values in an input box of between 1 and 3 you would have three boundaries to test:

  • two negative tests (under and over the expected value
  • a positive test (for the expected value).

Who does system testing?

System testing, as you can imagine, is typically done by the system testers. The reason for this is system testing is quite technical and skilled activity.

It needs to be done by an individual with good competence and understanding of the systems involved and how to capture the defects adhering to the testing standards.

Therefore, you need a skilled professional that can ensure that the system has been tested correctly.

Why should testers be involved early?

Testers should be involved as early as possible because it helps to reduce the risk of finding big errors or defects later on in the cycle.

In the SDLC,  defects that are found later typically cost the company a lot more money to fix. Therefore, it is better to get these defects found as early as possible.

For example, if big defects are found at the system testing stage, that could impact a lot of resources.

Why?

You will lose time with your system testers, as well as additional development resources having to recode the issue. Also, you may even have to involve the designers or the Business Analysts as well to rethink how they can factor in this is code/design change, are you with me?

Therefore, involving testers early, obviously as well as other stakeholders, can help to eliminate these issues as early as possible before any code or testing has even started.

Who Does Integration Testing? (Example/Black/White Box)

Integration testing is a pivotal part of testing and one of those areas that is always covered in the ISTQB syllabus. However, who actually does this and what is it? These are the questions I plan to cover in this article…

Who does integration testing? Developers and testers perform integration testing depending on the level of testing. There is a component as well as system integration testing that will be tested by their respective resources.

Now that you know who does what, let me go on to explain exactly what it is, the objectives, what types of defects you may expect to get from the process and more…

Why do Developers and testers do integration testing?

As discussed briefly, Integration testing is a term that encapsulates system as well as component-level testing. Therefore, there are different resources that perform each of these levels.

Component level

At the component level you would expect to see a developer doing the work.  This is because they need to be intimate with the component and have a deep understanding of how the component works.

System-level

At the system integration level, this is typically done by testers. They also need to have a deep understanding, but not of the component, of the architecture of the system and how these interfaces should interact, are you with me?

What are the objectives of integration testing?

Integration testing is an important part of the Software Development Life-cycle (SDLC). There is a number of objectives in this process and for that reason, I will give you an overview of this now.

Reducing risk

At the integration level, we expect a level of confidence in the systems that are to be integrated. Whether that be the components or the systems themselves.

However, integration testing reduces the risks before getting into further levels of testing because there are basic integrations that may not be quite correct.

A quick example of this could be a basic XML file interface which is misinterpreted by two separate parties (but more on this later).

Finding defects early

The benefit of finding defects early is reducing the cost further down the line. The cost of fixing defects towards the end of a project is substantially more than at the beginning or earlier stages.

Therefore it is important to capture the defects as early as possible. During integration testing is a perfect opportunity to see glaringly obvious integration issues such as what I mentioned earlier in a previous section about XML interfaces.

What types of integration tests are there?

As discussed earlier there are different levels of integration tests at the component and the system level. But let me give you more detail on each of these levels…

Component level integration

At the component level, we are focused on the interaction of the individual components. This is a lower level than the system.

This is after a developer completes their component testing (unit testing). And then take it one step further to confirm that the component that they have just unit tested can interface with the next component in line.

System-level

System-level integration testing usually happens after system testing (Click here to see why multiple rounds of system testing is recommended). It is essential to confirm that the system has been tested before going into integration.

The objective here is to prove that the system integrates as defined in the specification with external systems, for example, web services.

Examples of test basis items.

So far we have discussed the objectives of integration testing and how it works between a component and a system level. But what will be used to confirm how the integration should work? This is where test basis items come in..

Examples of test basis documents are as follows:

  • System (or software) design documents
  • Use cases
  • Interface definitions
  • Sequence diagrams
  • etc.

Typical objects used in integration

During integration there are a number of different objects that can be used at both levels, as you can imagine. But, to give you an idea of these, I will list down a few of these objects to give you an idea.

  • Databases
  • Interfaces
  • APIs
  • Microservices
  • etc

Common issues and defects expected in integration

Once you are in the actual integration test phase it is a good idea to find as many defects as you can, as I explained earlier. However, what kind of defects can you expect to see?

Component Level defects

At the component level, typical defects include data issues, interface problems, or functions that may not be handled correctly. E,g, causing indefinite loops or system component errors.

System-level defects.

At the system level, you will typically see defects around the actual system interfaces. one of the biggest issues that I personally experienced during my integration testing is XML interfaces that integrate with external systems that are hosted by third parties.

This usually comes down to ambiguous design specifications. These specs could be interpreted in two different ways. Or even data items that were not correctly defined and left open for debate.

It’s amazing the number of man-hours that can be lost just by a simple field not being defined correctly in a design specification.

However, this is the purpose of capturing these things at the integration level and not later on in the process when it cost more money.

The danger of complicated systems

Integration at the surface level may sound straightforward but it can get very complicated depending on how large the system is.

Also, it depends on how frequent and how far into the integration testing. Whether that be at the component or system level.

For large systems, for example, financial-based systems can be very complicated. Especially when you’re dealing with multiple third-party vendors that have to interface with different systems.

My experiences with it…

I remember dealing with a client who specialized in vehicle finance applications. During my time working for this client, I experienced a number of challenges with simple interfaces into their system.

During this process, one of the key things that helped push the project through was regular contact with the client and clearly defined specifications.

When is integration testing performed?

Briefly, we talked about integration testing at the system and component level but when is this usually done?

At the component-level, these tests will typically happen immediately after a component has been completed and unit tested.

However, logically speaking you need more than one component completed before the integration can be tested, obviously.

Therefore there may be a reliance on a collaboration of developers working together so that they can make integration happen.

System-level

As for the system testing, integration, as you can imagine, happens after system testing. Ideally. And again,  there is a dependency on the 3rd party system being available.

Therefore, you can be at the mercy of third parties and playing the waiting game if they are not ready when you are.

Is integration testing a black box testing activity?

You may be wondering how integration testing fits into the big scheme of things, right? Is it a black box testing activity or a white-box? The reality is, it can be white or black box testing depending on the test scenario. The best way to explain this is with an example…

Let’s start with a white box testing example. A white-box integration test could be the integration of how two web services integrate, at a component level.

Example of a black-box integration test…

An example of a black box test in this context could be, testing the integration of a vehicle finance system with an external credit referencing agencies system, in particular, making sure that the XML feed from the external agencies system integrates with your system.

Why is integration testing necessary?

So far you have learned about what integration testing is, its objective, etc. But why is it even needed? This is the question I will help to explain now…

Integration testing is essential because if dependant systems or components cannot talk to each other the system is useless, are you with me?

For example, from a system integration perspective, imagine that you have a credit checking online app, but you can log in and see your profile, but when you click the “get credit score” button you get no response because the interface to Experian is not integrated correctly, would you be happy to pay for that service? are you with me?

This is a simple but powerful example of how critical integration testing is.

What types of integration testing is there?

In integration, there are a few different ways to actually implement it. Each one has an inherent risk associated. Let me explain what they are…

Big-bang

This is one of the riskiest methods. It essentially means that all integration will be tested together and fingers crossed, it will work. The problem is identifying what has caused the issue in this method. If there are so many integrations at once it is a real challenge.

Top-Down

As the name suggests, the testing will start at the top-level components or systems and work its way down the integration tree.

An example would be testing the login page loads the profile page of a CRM system.

Bottom-Up

As you can imagine, this is the opposite to top-down testing. The integrations at the bottom will be tested first until you reach the top. So, following on from the previous example, the login page would be tested last.  This method is only really effective if the components or systems are ready at the same level.

 

Why are Stubs and Drivers Used in Component Testing?

If you are preparing for your ISTQB, you are probably keen to understand how stubs and drivers are used. For that reason, in this article, I will explain this to you…

Why are Stubs and Drivers Used in Component Testing? This is to enable a developer to unit test a code component without other dependent code modules being available. Stubs help to produce an expected output whereas a driver will send a required input to the code module.

Now that you know what they are at a high level, let me explain in more detail, with examples. Also, exactly what component testing is, the objectives and so much more.

Examples of Stubs & Drivers

For example, if you have an address book app and the dev wants to test the “login” code component, in the absence of a display contacts code module a stub could be used to test the login component in isolation.

In another example for a driver, if we use the address book app example again if the login page is not ready, but the “display contacts” code module is ready for testing. The developer can use a driver for the Login feature so he can test the display contacts component in isolation.
What is component testing?

why are stubs and drivers used in component testing

This method focuses on a specific code component in isolation. It is also referred to as unit or module testing.

There is some confusion regarding unit and component testing, with other sources claiming/suggesting they are separate, but according to the latest ISTQB syllabus, they are one of the same thing. And, from my experience, they have been treated as one of the same as well.

How are stubs and drivers used in component testing?

Now that you understand what component testing is, from a high level. Let me explain how stubs and drivers are used. This will broaden your knowledge and expand on the brief explanation I covered in the earlier section.

As discussed, component testing is typically done in isolation. Meaning, to do this correctly other supporting objects will be required, such as:

  • Stubs
  • Drivers
  • Test harnesses
  • etc.

Drivers

Drivers are place holder (dummy) programs that call the function that you need to test. For example, earlier I used the address book analogy, this driver could be the login code module that calls the display contact component you wish to test.

Stubs

These code modules take inputs from a calling function and display the required output. So, refereeing back to the address book app example, this could be used to replace the display contacts component while the developer tests the login component.

What are the objectives of Component Testing?

You may be wondering why do we need to use component testing. To help to make this clear I will explain the objectives of it in this section.

Reducing risk & catching defects early

The main objective of component testing is to reduce the risk of the component. It is better to find defects early rather than waiting until later test levels. Why? Simple, it saves costs by detecting defects as early as possible.

It is far easier and cheaper to verify that the component meets the related specification before promoting it to integration or system testing.

I have had personal experiences in the past where code has been promoted to system test and there were glaringly obvious issues with the software that wasted many test resources time.

For example, after attempting to test the application, discovering that we are not even able to log in, are you with me?

It may not sound much to you right now. But, look at through the lens of a business. Each tester costs money per hour. If the code is not testable, the time lost could be half a day, or even worse. And, when you multiply that by the number of test resource booked to test, it gets pricey, are you with me?

Using simple techniques such as component testing at the earlier stage helps to reduce these risks.

Functional & non-functional component verification

At this component stage, another objective is verifying that the code module matches the agreed specification. This is from a functional as well as non-functional level.

If it is verified at this stage, it reduces the chance (but not eliminates) of defects going forward.
Automating regression for ever-changing components.

In some cases, when code components are changing frequently, it may be advantageous to automate some component regression tests to verify that the component(s) have not regressed since the last iteration. This is even more relevant when an Agile methodology is used.

What documents are used for a test basis?

Now that you understand what component testing is, its benefits and why we use it. It is also important to understand how it is verified. What do I mean? Well, how can you be sure it’s right? In this section, I will explain some documents that can be used to do just that. Also referred to as the “test basis”.

Here are a few documents that can be used:

  • Detailed design
  • Component specification
  • Code
  • Data specification models

Detailed designs & component specs

In my experience, these are the most common documents used as a test basis. Typically, they clearly outline what is to be expected by the code components. Making the component testing easy to see if it is correct.

One thing to remember, these specifications can also be wrong, they are not perfect. However, that is not the objective of component testing. The concerns can be noted, but ultimately the task is to verify it meets the specification.

Code & Data Specification Models

According to the latest ISTQB syllabus, the code and data models can also be used. To be honest, in my time I have not seen this done. But, I am not saying it’s impossible, just that I have not experienced this.

Examples of test modules

Understanding the high-level definitions of component testing is one thing, but grasping exactly, down to the code level, is another. What do I mean? Well, what type of code will be testing in this phase? Let me explain…

During component testing here are some examples of code that can be tested:

  • Code modules
  • Data structures
  • Functions
  • Classes

These are examples, in reality, there are more. But this is a good flavor of what can be expected.

What kind of issues will you expect to see during component testing?

As you can imagine, during component testing you will encounter many different issues. But, that’s a good thing, right? But, what kind of issues typically rear their ugly head? Let me explain…

During component testing here are some of the most common issues detested:

  • Syntax errors in the code
  • Logical code errors
  • Data issues
  • Syntax errors

These errors are a lot easier to pick up these days. Why? Because most up to date code management software will do this automatically. But, there is always room for error.

Logical code errors

These errors are more complex when compared to syntax errors. Why? Because they are not always obvious. The challenge with these errors is, it may look ok on the surface (passes syntax checks, etc.), but it won’t give you the expected result. This is where an experienced developer will save the business a lot of time.

Data Issues

Data issues can cause code errors where it is least expected. It could be a weird combination of data, that was not expected, that is actually used in some obscure scenario, that can trip up the said program.

Who typically runs these tests?

So, now we know the type of issues we can expect. But, which resource is usually responsible for this testing? Let me explain…

Usually, the developer that actually coded the component is responsible for this testing. And from my experience, this is the case. However, I have seen many cases where the developers attempt to avoid this task and effectively promote the code directly to testers. Not ideal.

When new methodologies are being used, such as agile, with automation. There can be additional tasks thrown upon the developer to produce automated, component level, automated tests before even coding the module.

Personally, I think this is good practice. And, it saves the business wasted time and money on defective code.

What is component integration testing?

You may be wondering how component integration fits into things, is it the same as component testing, or a completely different phase. In this section, I plan to explain exactly what this is.

Component integration should follow component testing, in an ideal world. At this level, it is assumed that the individual components are tested.

The objective here is to verify that the component talks to each other, as per the agreed specification. This is done to verify before moving onto the next level of testing.

When is component testing usually executed?

Earlier we talked about component integration, and when it is expected to be done and established that it is its own unique phase. But, when is the actual component testing done? In this section, I will explain.

Component testing is typically done before component integration testing. In fact, logically, it has too. Why? Because the code components need to be verified first, before moving onto integration. Think of this as a necessary gate.

Ideally, once the code is written, competent testing will commence, to verify the code meets the detailed design or component specification.

Which White Box Testing Technique is Best? (Examples)

If you are preparing for your ISTQB, or genuinely curious about the best white-box techniques prior to starting an up-coming test phase, then I have prepared some info to help you decide the best white-box testing approach.

Which white-box testing is best? The best white box testing technique is a combination of a decision based which is also run dynamically (more on this later). Because you have the potential of high test coverage with the ability to analyze how the code actually behaves.

Now that you understand which technique is best, in my opinion, let me go on to explain exactly what white-box testing is, the advantages, disadvantages, types, how it can be measured and much more.

What is white-box testing?

what is white box testing

what is white box testing

White-box testing is effectively the opposite of black-box testing. It focuses on the internal structure of a test object, rather than just analyzing the inputs and outputs.

This comes down to analyzing the actual code of the software-based on a detailed design or equivalent specification document.

What are the advantages of White-box testing?

Now that you have a general idea of what this technique is, you may be wondering what the point of it is. Why should you even care about it, right? Well, let me explain some of the

This testing technique is ideal for improving the efficiency of the code, as well as spotting clear errors before getting into further phases of testing.

In my experiences of testing, developers are sometimes hesitant to do this, with the attitude that they are not testers. But, in my opinion, this is the mark of a great developer who truly checks his work before throwing it over the fence to the testers (so to speak).

Disadvantages of White-box testing?

Like most things, there are always set-backs with every technique. So, in this section, I will explain some of the disadvantages of this testing technique.

Firstly, whilst it’s a great way to weed out erroneous code early on in the Software Development lifecycle (SDLC). It relies on your organization having skilled developers that can perform the tests.

Usually, it the actual developer who wrote the code in the first instance that actually executes the tests, so you can guess the level of expertise required here.

The point I am getting at here is the cost. To do this correctly you need a decent budget for a skilled resource.

Impossible to catch everything

Whilst this technique helps to improve the code, prior to getting into later stages of testing, such as system testing, etc. It needs to be said, to set your expectation, that it is impossible to catch every potential defect at this level. In fact, ever, throughout any testing stage for that matter.

Don’t get me wrong, it’s worth the investment, but it is worth explaining this upfront.

Types of White-box testing

In this section, I will now explain some of the actual white-box techniques that can be used. According to the ISTQB syllabus, this testing can occur at any test level. However, I will explain some of the most common techniques used.

White-box testing is typically performed by a developer at a code unit test level using one of the following techniques:

  • Statement based
  • Decision-based

Statement based

The statement based technique focuses on the individual lines of code to be delivered. The idea is to analyze the actual lines of code in the software program and have tests that verify these lines.

Decision-based

Decision-based techniques are slightly different. Instead of focussing on the lines of code it revolves around the conditions in the code. When I say conditions, I mean the decisions, for example, IF or case statements.

As you can imagine there are many conditions within code that will create many permutations.

Static vs Dynamic

Whether you use statement or decision-based techniques there is another choice to make. This is how you will actually execute these tests. For this there are two choices:

  • Static
  • Dynamic

Static

Static is a manual approach. This relies on the developer going through the code and eyeballing the code to see obvious errors. This can be done by the creator of the code.

But, it is more effective if it is done by a different developer with a similar skill set.  These reviews are typically called peer reviews.

The benefit of the peer review is; it introduces fresh eyes. Often a developer will overlook obvious issues because they have simply spent too much time steering at the code. Whereas a new pair of fresh eyes will look at it from a different angle.

This is an opportunity for the reviewer to not only spot obvious errors but also suggest smarter or more efficient ways of doing things.

Dynamic

Dynamic is the opposite. This comes down to running the code and analyzing the outputs. This gives the developer the opportunity to compare the outputs to what’s expected from the specification.

This is a really effective method because the software can be complicated and even the most experienced developer may be surprised with output that seems to look perfect until it is dynamically run.

How can white-box testing be measured?

Understanding the concept of white-box testing is one thing, but grasping how to measure it is another. And, arguably one of the most important factors that will govern the next stage of your test object. For that reason, I will explain how this is done for statement & decision-based techniques.

Statement based measurement

For statement-based testing, the lines of code is the main focus, as discussed earlier. Therefore, for this method, the total lines of code (statements) are compared to the total tests that execute these lines.

The coverage is typically expressed as a percentage of statements executed to total statements of code.

Decision-based measurement

For decision-based testing, the conditions (decisions) in the code is the main focus, as discussed earlier. Therefore, the coverage is based on this metric.

In particular, the total number of decisions executed by the tests is expressed as a percentage of the total number of decisions.

Special skills and knowledge requirements

With this intricate testing, you can expect that it will require a certain level of skills to get it right. For that reason, in this section, I will explain exactly what level of skills you will expect to have.

White-box testing requires a resource that fully understands the code structure of the test object. In reality, every code object is different, so each skilled resource will develop their own unique expertise in one particular area of the system.

Either way, to begin with, you need a technical low-level coding resource that can do this task correctly. In reality, it doesn’t necessarily have to be a developer, but in my experience is nearly always is.

Related Questions:

In this section, I will answer some questions related to white-box testing. If you have some additional questions that are not answered here please drop a comment below.

Q: What is the difference between white and black-box testing?

White-box focuses on the internal structure of a test object. Whereas in black-box testing, the internal structure is not important. All that matters are the inputs and outputs meet the specification.

Also, with black-box testing, you would expect testers to take on this task. However, with white-box testing, it is typically developers that do this.

Q: How is coverage measured at a component integration level?

This is typically measured by the total number of integrations executed by your tests as a percentage of total integrations.

Integrations are different. The expectation at this stage is that the individual component (click here to see why stubs and drivers are used in component testing) has already been tested and fit for purpose. Therefore, this stage focuses on how the outputs of component A interface with component B.

These integrations are usually in the form of a file (typically XML) with various fields and data. These data points need to match the specification to pass. Hence the need for the component level integrations. And, even more, an important way to measure its coverage.

What is Non-Functional Testing? (2 examples)

If you are preparing for your ISTQB or just curious, you may be wondering what non-functional testing (NFT) is and how it fits into the testing life-cycle. Let me explain…

What is Non-functional Testing? It tests how well a software system behaves, based on an agreed specification. In particular, it focusses on certain aspects of the system, for example, performance, usability or security.

Now that you know what it is, let me explain a bit more detail Such as when is it used, how it can be measured if special skills are required, the types of non-functional testing and more. Keep reading…

When is non-functional testing used?

Understanding what Non-functional testing is, is one thing. But, understanding when to use it is another. For that reason, in this section, I will explain when it is a good idea to use this. The answer may shock you because it’s not what most people assume or do.

Even though most software testers will assume that non-functional testing should be scheduled after system testing. The reality is, it should be done at all test levels. In fact, the earlier the better.

Why is this? Simply because in the event that you find a serious “show-stopping” non-functional defect. It is better to detect it sooner rather than later. The cost of finding it later is nearly always significantly more.

How can Black-box testing techniques be used for this?

While planning non-functional tests you need a method to work out the test casts. For that reason, in this section, I am going to explain how you can use some known Black-box testing techniques to draw up these cases.

One of the known Black-box techniques, called boundary value analysis (BVA) can actually be used to derive non-functional test cases. IN BVA, the agreed boundaries of fields and inputs are tested to make sure that they meet the agreed specification.

How can the Non-functional tests be measured?

Without the ability to report and quantify testing would effectively make Non-functional testing worthless. But, you may be wondering, with that being said, how can it actually be measured, right?

The coverage of non-functional tests relies on clear requirements/specifications. For each non-functional element specified, it is possible to measure the coverage based on the number of tests run per element.

This can then be used to help to find any gaps in the testing. Such gaps may be deemed as acceptable based on the risk it poses to the system, assuming a risk-based analysis is used.

Are special skills required?

Non-functional tests cover a large set of possible tests. But, you may be interested to know if any resource can run or design these tests, or does it need a special level of expertise?

In most cases, non-functional tests, for example, performance tests, typically need a specialized resource with a great level of experience. This experience could be deep knowledge of a tool, e.g. win-runner. Or specialised security-related skills to make sure the system is tested correctly.

Who does Non-functional Testing?

In the last section, we touched on what type of skill set is required to do non-functional testing. However, I did not explicitly state who would be the typical resource member to do it.

Non-functional is quite a broad categorization, so there will be different roles for every different particular testing activity. For example, for performance testing, you would typically have a dedicated performance test analyst. If the testing required is quite significant, this could be a team of testers. With one of these resources being a Test lead or senior tester.

Is security testing functional or non-functional?

security testing (non-functional)

security testing (non-functional)

If you have security testing in the scope of your project, or you are just curious, you may be wondering what this is classified as. In this section, I will explain this.

Security testing is classed as non-functional. It is required to verify that the system in question is secure. Such as resilience to external infiltration.

What is the difference between functional and non-functional testing?

To make sure that it is clear, I want to draw a distinction between functional & non-functional testing. This will help you to understand what NFT is.

Functional testing (Click here to see why Functional Testing is Required) is concerned with the functional elements of the system. Typically, this is defined in a functional spec as “features”. At this stage, we are not concerned with code-level testing. It is mainly black-box tests that focus on the inputs and expected outputs of each function. And, verifying that they meet the agreed specification.

Non-functional testing is different because it focuses on aspects of the system that are not usually addressed during functional testing. Such as how many users can use the system simultaneously (max load) before it stops responding, are you with me?

Then, to follow on with this example. Once we establish the max load, how does this compare to the documented non-functional specification? Does it support as many users as we expected, or not?

My experience with Non-functional testing

Reading about NFT is easy, but understanding it is another. Sometimes a real life example will help to put things into perspective. For that reason, in this section, I will give you an example of my first-hand experience using this method.

In one of my contracts in the past, the client was responsible for providing vehicle finance. Therefore, they have a large amount of critical data that needs to be secure and resilient. In particular, in the event of a disaster, e.g. office fire, damage to servers, etc.

I was tasked with leading an annual task they had. Which was to verify that they can recover all of their core systems following a disaster. To do this we emulated the situation by creating a “disaster day”

Disaster Day (Disaster Recovery – Non-Funtional testing)

This testing is known as Disaster recovery. And, is a known example of non-functional testing. The objective was to prove that we can prove that each core system can be used following the disaster.

To prove this, we had systems located on another site. Their sole purpose was to provide a different backup environment that we could switch to following the disaster, are you with me?

The recovery Site

On the recovery site we had two goals:

  • Prove that the core systems can be used (core functionality)
  • Verify the data was restored correctly.

Proving that the core systems can be used

To prove this, we had a selection of Subject Matter Experts (SME’s). Their role was to test the recovered systems and verify that the core functionality was operational, following an agreed test plan. The time was limited (one day on site for all tests), so we had to be efficient.

Verify the data was restored correctly

The second goal, and arguably one of the most important, was to prove that the data was recovered successfully. This was done by taking a snapshot of the data before the disaster day. And then, comparing that snapshot on the recovered system.

To clarify, although this sounds complicated, it was done as simply as possible (due to time constraints. The data snapshots were as simple as taking some screenshots before the disaster and then comparing them after the data was restored on the backup systems, are you with me?

Why are non-functional requirements important?

You may be wondering what is so important about these non-funtional requirements. And, you may even consider cutting corners and leaving them as a “nice to have”. But, in this section I will explain why they are required.

Outline Performance Boundaries

One of the main reasons why it is important is the performance boundaries. Once you understand these boundaries, e.g. how many users can use the system simultaneously. It can help your organization understand what they need to do to scale up. It can give you an idea of the cost of this and the impacts it may have on other projects, resources, etc.

Also, it can give an insight to see how long the system is useable in its current form. This can help with planning for future quarters, etc.

Why Functional Testing is Required (Examples,Types)

If you are preparing for your ISTQB (Click here for my prep guide), you may need to understand the nature of functional testing, and how it works. Let me explain…

Why is functional testing required? It verifies that the agreed behavior of each function works as specified in the related requirements. It is an important part of software testing and is covered throughout all levels during the software development lifecycle.

Now that you understand what functional testing is, at a high level. Let me now explain how it is measured, what skills are required, the different types and much more. Keep reading…

What is functional testing?

Functional testing is a Test type that focuses on the functions that a system has been designed to do. It can be used at all test levels and uses techniques such as Black-box testing (more on this later) to derive its tests.

The test cases that form this are based on functional requirements. These functional requirements can be across varying different levels and stages. For example, functional specifications, user stories, etc.

How can functional testing be measured?

how to measure functional testing

how to measure functional testing

You may be wondering if these functional tests can cover such a broad amount of levels, how can the actual coverage be measured? In reality, this is done quite easily, assuming you have a documented test basis to work with, e.g. functional specification. Let me explain…

The coverage is measured based on the functions documented against the number of test cases created. This can be done in the form of a traceability matrix at a high level. These gaps that are defined (if found) can help to tighten up the test coverage.

What skills are typically required?

In this section, I will discuss what skills are usually required when designing or executing functional tests. As you can imagine, depending on your specific requirements, it can be quite a technical process.

As I stated it is a technical process and for that reason, it requires skills or experience with the system or business process. This isn’t a deal-breaker if the individual resource does not have this, but in the absence of this, there will be a steep learning curve until it is mastered.

What types of Functional testing is there?

As you can imagine there are various different types of functional tests, as I mentioned earlier this testing is conducted at multiple levels. For that reason, I will break down some of these types to help you understand.

The following is a list of functional test types:

  • Component testing
  • Component integration testing
  • System testing
  • System integration testing
  • Acceptance testing

Component testing

Component testing is a testing technique that focuses on the individual test component (or code function), for example in a calculator application, the “subtract” component (or code function) is responsible solely for subtraction.

Therefore, in component testing, you would have a test that verifies this component performs as per the specification. At this level, there is no regard for whether or not it integrates with the calculator application correctly.

Component Integration Testing

This is closely related to component testing as you may have assumed. However, the key difference is how these components talk to each other.

At this stage, we assume that the component has been tested and works as expected. We only focus on how the output of component A integrates with the inputs of function B.

e.g. In the calculator App, that we referred to earlier. The substation component would be tested to see how it integrates with other components in the calculator app.

System Testing

System testing ( is concerned with the system as a whole. It primarily looks at the end-to-end tasks or features that a system is expected to perform.

The main objective of this is to find system defects before they get into a live production environment. As well as verify that the system behaves as specified.

System Integration Testing

System Integration Testing (SIT – Click here to see What the difference is between SIT and UAT testing) is closely related to system testing, as you can imagine. However, the objective of this testing is to verify how the system integrates with other systems.

For example, in a parking application, there is an interface to the vehicle database that matches number plates scanned with real vehicle information.

The system integration test will verify that a number plate requested from system A will return the vehicle data successfully from System B.

Acceptance Testing

This is, broadly speaking, testing that validates if the whole system is fit for purpose before going into production. It can be done by various different users of differing skill levels, for example, Operational Acceptance Testing (OAT), is typically run by very skilled support staff. Who typically have deep knowledge of the existing system (or product).

How is a Functional Test Usually structured?

If you are wondering what an actual functional test looks like, then this section is for you. I will be explaining more about how these tests are structured and the common attributes of these tests.

Depending on what type of functional test you are executing, the actual look of the test can vary. However, there are some common attributes that you would expect. Even if the tests are very informal. These include the following:

  • Test pre-requisites
  • Test description.
  • Actual test steps.
  • Expected results.

Test prerequisites

This is the expected setup before executing the test. For example, it could be a previous test is done first, so that the data is in the correct state, etc.

Test Description

This is a high-level description of the test purpose, objective, etc. It explains what this test is about.

Test Steps

These are the actual, step-by-step instructions to execute the test. Depending on what company you work for they can be really low-level detail (e.g. right-click on the “ok” button to submit the data”) or high-level tests that assume a level of experience (e.g. “submit the data”).

Expected Results

This is what we expect to see once the test has been executed. This is important because it will let us know if the test was successful or not. You may fee that this should be obvious, but, understand this, some tests could be negative, meaning the expected result could be an error.

Result

This is the actual result of the test. Typically, this will be “Pass or Fail”. But, there can be other statuses such as “Blocked” that can be used.

Can Functional tests be automated?

If you have some interest in automation or are just thinking of ways to speed up test execution, then you may be wondering if functional tests can be automated. So, in this section, I will be explaining if this is possible and how this is typically handled at this stage.

Yes, automation can and is used during functional testing. The reality is, functional testing is quite a broad term that encapsulates many testing techniques, as you have seen already.

Therefore, there is quite a lot of scope to automate at different levels. If we take system testing, for example, this is an example of a stage where this can be used. I have used this personally at this stage. It is beneficial if you have a large set of tests that are quite similar/repetitive. This will help to speed up the process.

Is performance testing a functional testing activity?

As we have discussed functional testing covers quite a few stages. So, you may be wondering if performance and functional are one of the same things. I will discuss this further in this section.

The short answer is no, performance and functional testing are different. Why? Because performance testing is classed as a non-functional test (Click here to see what is Non-Functional Testing).

These tests usually incorporate tests which are based on testing the performance of the system against the specification/requirements. An example of this is the number of concurrent users that a system can have at one time.

Is functional testing and manual testing the same?

Earlier we talked about automation and it is a part of functional testing. But what about manual testing? Is that also included? This is what I will clarify in this section.

Yes, manual testing is used in functional testing. In fact, I would go as far as to say that it is used extensively. Obviously, these days there is more and more automation being used. However, manual testing is still used a lot across all test phases.

What is Manual Testing?

Manual testing is a method of test execution. It means that a tester will manually run test cases by supplying the necessary inputs, pre-requisites, etc. and then analyzing the results to confirm if the test passed or failed.

What is Change Related Testing (ISTQB Syllabus)?

Once a system is promoted to live it is not the end of its updates or changes. In fact, there can be many iterations after it goes live, such as defects, new functionality, etc. But what is Change-related testing and what has that got to do with this?

What is change related testing?  Change-related testing is a Test Type that has a specific purpose. It is designed to verify that a change to the system has been fixed correctly and does not have any adverse effects on the system as a consequence.

Now that you know what change related testing is let me explain what types of tests are performed during this time, how it relates to the overarching “Test Types” if some of these tests can be automated and more. Keep reading…

What is a Test Type?

A test type is a group of testing activities that have specific objectives to test the system. Here is an example of other Test Types that are used:

  • Functional
  • Non-functional
  • White-box testing

When would change related testing be required?

You may be thinking this is all good, but when would you need to actually use change-related testing? In this section, I explain when and why you will use it.

This testing type is typically used following a change to a system. Usually, this is after a system is live, but this is not always the case.

Types of changes

These changes can be as follows:

  • Defect Fix.
  • Change of environment.
  • Change of functionality.
  • New functionally added.

Defect Fix

defect fixes

defect fixes

If the system has just been upgraded due to defect fixes it is important to verify that the defect fixes have been fixed correctly and also check that their inclusion has not caused any unexpected issues to the current system (aka Regressions – more on this later).

Change of Environment

If the system has been implemented and for whatever reason, it needs to be moved. Then this can be a risk. This risk needs to be mitigated with change-related tests to ensure that it is safe to use.

Change of functionality

If the system has had modifications to improve or even change the behavior of the system, it needs to be tested to verify it. This will include tests to confirm that the changed functionality works as per the upgraded specification and also that it does not introduce any issues.

New Functionality added

Similar to the section above, new functionality added will be treated much the same as changed functionality. The new functionality will be tested to verify that it works as specified, but it will also need to be checked for any regressions.

Techniques Used to Test This?

As discussed earlier change-related testing is an over-arching group of test activities. So, you may be wondering what specific techniques are used to test this. Let me explain…

  • Regression Testing.
  • Confirmation testing.

Regression Testing

This change-related test technique focuses on verifying that any new builds or code changes do not adversely affect the system. For example, a defect fix may be released to fix a calculation error in a timesheet system. The fix was designed to fix a miscalculation on weekend work rates.

After implementation the defect worked, however, the users have now discovered that their standard day rates are now being incorrectly calculated. This is an example of a system regression (Click here for Regression Testing Best Practices). And this is exactly what regression testing will deal with.

Confirmation testing

This technique confirms that the intended change meets its specification. So if we refer back to the example in the last section, the timesheet system defect fix. In this context, confirmation testing will be used to verify that the release fixed the issue with weekend rate calculations, which it did.

Can you automate change-related Testing?

With all these potential tests that could be required on an ongoing basis, e.g. regression testing that we discussed earlier. You may be wondering if it would make sense to automate these tests. In this section, I will explore this for you.

When we look at the two main techniques in change-related testing the obvious one to automate is regression testing. Why? Well, confirmation testing is effectively a moving target, right?

Think about it, each time the system is changed for a defect or functionality change it could literally be anything. However, with regression you have some common tests that a run over and over again to check that the system responds in the same manner as it did pre-release, are you with me?

e.g. If we go back to our timesheet system we used earlier…. Every time an amendment is made to this system, it would be a good idea to check that the standard day-to-day calculations still work as designed.

This would be a perfect example of a group of tests that could be easily automated.

Can Smoke Testing Be Used for Change-related tests?

If you have heard of smoke testing before, you may be curious to see if it is used during Change-related testing as well. For that reason, I will clarify this in this section.

Firstly, what is Smoke Testing?

Smoke testing evolved from industrial engineering. When physical changes were done it was deemed as fit for purpose if no smoke was seen.

For that reason, it has evolved into a technique used in software testing. The purpose of smoke testing is to establish if the system passes some very basic tests, to confirm that the system is even remotely ready to test. This can be basic system functions that are mandatory to move forward.

Using Smoke Testing for Change-related testing.

Yes, smoke testing can and is used during change-related testing. Although it is not necessarily documented in exam syllabuses, in the real world it is used. I know from my own personal experience of using it in anger.

It helps to reduce wasted time and does not take much time to run the tests. In reality, these tests are so basic, one could argue that they are bordering on common sense tests that do not need a formal name or phase.

Using Sanity Testing for Change-related testing.

Earlier we mentioned Smoke testing and the inclusion in change-related testing. However, what about sanity testing? Could this be included also? Let me explain…

Firstly what is Sanity testing? Sanity testing is essentially a subset of regression testing. It aims to verify, with some basic tests, that the function in question is working. It does not confirm it, but it gives us an early view if something really basic is not working, which could render the regression testing a waste of time.

But isn’t this just the same thing as Smoke testing?

No, it’s similar, but not the same. Sanity testing focuses on the function under test. Whereas Smoke testing has a broad set of tests, which can be run in a short space of time.

Can it be used with change-related tests?

Yes, it can and, like smoke testing, it usually is. Whether that is formally or informally. It is a great way to eliminate any time-wasting for a code fix that clearly has not addressed the problem.

Related Questions:

In this section, I am going to answer some questions related to change-related testing. If you feel you have more questions you need answering, feel free to drop me a comment below.

Q: What is the difference between Re-testing and regression testing?

Re-testing is effectively testing test cases that have previously failed. Whereas regression testing is verifying that the system has not “regressed” by re-running a subset of previously run tests.

Although they sound similar in action they are very different. Both are just as important as the other by the way, just have different objectives. Obviously, as discussed regression is one of the key activities in change-related testing.

3 Best ISTQB Courses for Testers

best Istqb course

If you are interested in an ISTQB course but have no idea what’s best for you, you are in the right place. I am going to recommend my best certifications in the ISTQB portfolio, let me explain…

What is the best Istqb course? For beginners the Certified Testing Foundation Level is ideal. For Test managers, The Advanced Level Test Managers certification is ideal. And finally, for Agile, The foundation level Agile certification is best.

Now that you know what courses I recommend, let me go on to explain how the ISTQB portfolio of courses is structured, the different streams they have, and what that even means in simple terms and more.

I remember going for a walk one lunchtime, with a tasty subway sandwich in hand. As I sat on the park bench reflecting on life and planning for my weekend, it made me appreciate life and take in a breath of fresh summertime air. However, it then dawned on me, that not everything is a walk in the park.

Same as choosing the best ISTQB course, nothing is always that straight forward, especially when you have multiple options. For that reason, let me explain now, how the courses are structured to help you.

How Are ISTQB Courses Structured?

ISTQB courses are structured based on different levels. In total there are three different levels:

  • Foundation
  • Advanced
  • Expert

Ultimately all of these levels higher levels are only possible if you have first done the foundation first.

In addition to these levels, they have another dimension which is called “Streams”.

These streams are grouped as follows:

  • Agile
  • Specialist
  • Core

What does each of these streams actually mean?

As discussed in the previous section there are three different streams. Let me explain…

Core

The core stream is based on the original ISTQB core fundamentals. These are applicable across all of the ISTQB certifications. Hence the reason why it is important to do the core foundation course before moving on to any other levels.

Agile

As the title of this Stream suggests it Focuses particularly on Agile. As software testing evolves, agile is becoming a critical part of software development and testing. So this Agile stream focuses primarily on agile practices.

Specialist

The specialist stream focuses on specific approaches or testing activities such as mobile, automation, pharmaceutical or even medical devices based testing.

Are there any prerequisites required for these courses?

Yes, across all of these certifications. The foundation core is an absolute must and cannot be skipped.

In addition to this, any advanced level certification can only be considered if the foundation level is done first. And as you may expect, any expert level will require that the advanced is done.

So what is the best ISTQB course?

In this section, I’m going to recommend what I believe to be the best ISTQB course. However, with so many different levels and streams, your best choice largely depends on what level you are at in your career.

Or, what qualifications you have. For example, if you already have a CTFL, then my recommendation for this course is obviously not applicable to you.

Therefore my recommendations are based on a selection of a few different skill levels. In particular:

  • Beginners – with little to no experience in testing.
  • Test managers
  • Agile

Best ISTQB course for beginners

The best ISTQB course for beginners is definitely the certified tester Foundation level (CTFL). This is the prerequisite for other levels if you want to progress.

This course is aimed at anyone that is actually a tester that hasn’t been formally certified. Alternatively people interested in getting into the industry that has not had any experience.

For example, this could include:

  • Test analysts
  • UAT testers
  • Project Managers with an interest in testing
  • The list goes on…

It covers a number of different core modules including the fundamentals of testing. Such as, “what is testing?”. “Why do we test”? etc.

Also covers test techniques such as:

  • Black box testing
  • Whitebox testing

Other areas include Software Development Life-cycle (SDLC).

Best ISTQB course for Test Managers (or Aspiring Managers)

The advanced level test manager certification is for testers that have reached an advanced level in testing. They are either working as a Test Manager and require a certificate for their CV or career progression.

Alternatively, they are aspiring test managers that want to get to test manager level, but have adequate experience.

It is also helpful for advanced level IT workers that wish to get a better understanding of the requirements of a test manager.

This course covers a deep understanding of test planning, estimation and it also goes into people skills. People skills, in particular, is something that some courses tend to overlook.

This includes motivating staff, how to get the best out of your team, etc.

Prerequisites for this course is a CTFL and you also require a certain amount of industry experience to be eligible for the certification.

Once you have completed this certification you can then move on to other levels such as the expert level certifications.

The best ISTQB certification for agile

The foundation level agile certification is one of the newer offerings. There is often confusion about Agile. In particular, how it should be applied and used in anger during a software project.

In my experience of working with agile, I find that I have never seen Agile implemented the way across any two companies.

The agile certification is an attempt to formalize the understanding of agile and to create a solid foundation of the core concepts and practices.

What does this certification cover?

It covers such topics as the fundamentals of agile. Such as what agile is, how it is handled in a software testing environment, etc.  It also goes into detail about the tools that can be used within agile and more.

The prerequisites for this course is the CTFL, which is required across all of the certifications as discussed earlier.

The next levels after this particular certification are currently in development. This includes plans to release an advanced level technical agile tester certification which will be a perfect add-on to this.

Related questions:

How long does it take to get an ISTQB? For the CTFL exam, it will typically be 60 minutes. However, for non-English speaking countries, such as in French-speaking parts of Canada, it is extended to 75 minutes.

The course is structured into 40 multiple choice questions. Each question scoring one point. To pass the exam you need to get at least 65% right. Which means you have to get 26 out of 40 to get a passing mark.

Is the ISTQB certification useful? Yes, in my opinion. It is useful for some people, depending on what area of work you are in. If you are a tester who is just starting out, this certification is good. And will be a booster on your CV. This can help you to get more recognition and jobs in the future.

However I would say, the certification does not necessarily make you a good tester. It just proves that you understand the fundamentals of testing and what is required in the industry.

I remember turning up for football training as a young lad. I was amazed to see one of the guys on the team had the brightest, most expensive football boots on.

I knew they were expensive and he let everyone know it. After I saw him get tackled and embarrassed for the third time in a row, it reminded me that just having the gear, does not mean you have all the ideas. It’s the experience and skill that matters, not the cost of your boots.

Similar to testing, you can take the best ISTQB course.  However, real life test execution, office politics, and the real world of testing is a whole new kettle of fish. This can only be mastered with experience, not by sitting an expensive exam.

Is UAT functional testing?

If you are involved in UAT or functional testing, you may have wondered what are the key differences, or if they are one of the same thing. Well, let me explain…

Is UAT functional testing? No, UAT is not functional testing. UAT is a process of checking and validating that the user requirements have been met. This is typically governed by the User Requirements document. Functional testing, on the other hand, is verification that the system meets the functional specification. 

To put things into perspective, it is possible to pass functional testing, with the system meeting the functional specification, but still failing UAT. But now that you know that they are different let me explain each one in more detail, the types of testing that fall into these categories and more…

What is UAT?

UAT is a validation process which proves the contract (user requirements) between the project team and the user has been met.  This is based on User requirements agreed from the beginning of the project.

This is typically conducted by end users (customers). Or in some cases, it is done by a product owner who is working as a proxy for the actual end-user.

Why is UAT testing important?

UAT testing is important because it confirms that the customer’s requirements have been met. It is also possible to find issues in the system that could have been missed during functional testing.

It is also an important quality gate. Often the final gate before the system goes live.

What is functional testing?

Functional testing (Click here to see Why Functional Testing is Required) is a way of verifying that each system function meets the functional specification document.

Inputs are put into the system with the actual results (or outputs) of these functions compared against the expected results defined in the functional specification.

I have mentioned the functional specification a few times in this article. Which is a key document in the traditional v-model, used in testing? However, in the real world, from my experience, you will often find that there is no functional specification.

This is the real challenge to a tester. Because, the books ion testing, or ISTQB exam never really prepare you for this.

The main activities of functional testing use black box testing…

Black box testing is a functional based procedure which does not concern itself with the underlying code within each functional module.

Therefore, a tester running black box tests does not need to know the exact code behavior within that function. Meaning they don’t have to be a developer.

They just need to know what input is required and what to expect to get out of it, as per the defined specification. Hence the reason it’s called the “black box”.

Functional testing usually tests functions such as APIs, databases, user interfaces, etc.

What different types of functional testing is there?

In this section, I’m going to explain what different types of functional testing there is.

The following forms of testing would be regarded as Functional testing:

  • Regression testing
  • Smoke testing
  • System integration testing

Integration testing is a classic example of this ( Click here to see the difference between System Integration Testing and UAT testing). During integration testing, you will be verifying if each individual system module can integrate and talk to each other successfully.

For this to be successful, inputs from one module will feed into another function. Hence the reason why it is typically called Integration testing.

Working Example from my experience…

So, for example, in one of my previous testing projects, we were working on a large financial system. This system was responsible for checking customer debt. Depending on the debt levels, certain actions would be triggered, etc a letter to a customer.

To gather accurate data on the customers, required multiple XML feeds into a central database. It would collect data, for example, address information and credit score data from external vendors.

Each one of these XML feeds was effectively integration tests that needed to be verified using functional testing.

Related questions:

Q: What is the difference between verification and validation?

Verification is what most people think of when they talk about system testing.

It is a procedure to verify that the system works as per the specification. In particular, the functional specification. In basic terms, it means – is the product built correctly?

Validation

Validation is different if you are thinking of user-related testing. Then, the chances are your thinking of validation testing.

Validation in basic terms means – are we building the right product?

It Focuses on confirming if the system that has been built and verified actually delivers on the user’s expectations.

And this is validated against the User Requirements that were drafted in the very beginning of the project.

Q: What is non-functional testing?

Non-functional testing is test procedures that confirm non-functional related specifications such as reliability, performance, etc.

These tests are performed based on separate non-functional requirements. They are not covered during functional testing.

An example

An example of a non-functional test would be: How many simultaneous users can be loaded onto the system?

Typically these tests are done by specialist test resource. They focus on non-functional testing. These testers are usually very well paid because they have rare skills that are often needed by large organizations.

Highly Paid

If if you are ever considering a highly paid contracting role for the future, this is something to consider.

Usually, this testing is done by some form of automation such as LoadRunner, QTP,  etc.

The reason for this is, it is usually quite challenging replicating these test scenarios without using test automation.

My example of a real-life manual load test…

An example of non-functional testing that I was involved in, as a test manager. During a previous project for a large energy company, we needed to simulate some load on the system.

We didn’t have the budget for automation tools, and time scales were tight. So, to simulate this load, we required multiple people all in the same room.

Each person was using the system at one time to verify the system could handle the load. The reality is this method is quite hard to coordinate, and there is room for error.

Also, this is quite a costly affair because you need to have everyone coordinated in the same room at the same time. And time is money. For these non-functional tests, If the project has the budget, automation is easier. because you can click a button to replicate the load.

Q: Is white box testing functional testing?

No, white box testing focuses on the underlying code in a function. This is typically referred to as unit testing and is usually executed by an actual developer.

It is a very important activity which typically happens before the system is passed on for functional testing. Think of it as an important prerequisite before you get into black box testing (or functional testing).

Q: What is end-to-end testing?

End-to-end testing is a testing method to verify that the entire system application, from start to finish, performs as expected. It verifies the flow of data works as per the specification.

Q: What is compatibility Testing?

This is classed as a non-functional test. Its main job is to verify that the application is compatible with different devices, browsers, etc. For example, let’s say you were working on a web application, such as an online sweet shop.

To perform compatibility tests, you would check the web app on multiple browsers to verify that it is compatible with the major browsers. This is just one example obviously.

These requirements, governing which browsers will need to have been documented and agreed on ahaead of time so that it is not an open-ended test on whatever browser the tester feels like testing. And to also ensure that the agreed coverage is met.

What is difference between SIT and UAT testing?

If you have ever worked in system testing, you must have seen when a system does not integrate correctly or just doesn’t work as expected, even after the system testing has passed. This is usually due to the lack of SIT (System integration testing), but how is this different from UAT? Let me explain…

What is the difference between SIT and UAT testing? SIT (System Integration Testing) focusses on proving that the system functions integrate successfully. With each input/output message invoking the next function successfully. Whereas UAT proves that the system meets the User requirements.

Now you know the difference. Let me explain who works on these phases, how they fit into the software development life cycle, the different types that are available and more.

What is UAT testing?

UAT (User Acceptance Testing) testing is a method to verify that the system application meets the user requirements that were laid out before the software development began.

It verifies from a user perspective that they are happy to accept the system.

What is SIT?

SIT stands for Software Integration Testing. It is basically a way to verify that the overall system integrates with the subsystem modules as expected.

Essentially, it makes sure that the end-to-end system hangs together as expected after system testing.

SIT typically follows system testing and is based on verifying that the system meets the functional specification. This is the process typically used with the V-model.

When is UAT usually done?

UAT usually happens towards the end of a project test phase. Usually, the last step before the system is accepted to go into production.

This is after its been proven in system testing, SIT and sometimes other phases such as OAT (Operational Acceptance Testing).

The reason it is usually at the end of the project is that at this point in time the system should be almost clear of all system serious system defects.

In fact, if you are following the correct procedure it will have had to have passed the exit criteria (click here to see what QA sign off is) of the other phases before it gets to UAT. So, by that time there should only be non-show stopper defects if any.

Who usually carries out this UAT testing?

UAT testing is typically done by a designated product owner. When I say product owner, these are users that typically use the system day in day out and are deemed as product experts or SMEs (Subject matter experts).

When I say experts they are not necessarily technical people. They are usually very skilled in using the application on a day-to-day basis, so they know all the nuances or expected behaviors, are you with me?

As well as verifying the requirements have been met, they are ideal for catching ad-hoc issues that could ordinarily get missed by a less experienced individual.

Their input is valuable because there are a lot of things that they know about the system that maybe even the technical testers may overlook.

Who usually does SIT?

SIT is usually done by very technical and skilled testers. These are usually dedicated resource who focus on testing in their job role.

They usually have very good technical skills and understanding of the modules that will be integrated. For example, these skills may include advanced SQL query skills.

What types of UAT testing are there?

There are two types of UAT testing which are as follows:

  • Alpha testing
  • Beta testing

In all honesty, these distinct types of UAT testing are not always implemented by every company. In my years of experience working on software projects, which includes some large blue-chip companies, it is very rarely split up into these two distinct phases.

But, for your benefit, I will break these two distinct approaches.

Alpha testing

Alpha UAT is typically done on the development site by a product owner or user accepted acceptance representative. This user will be able to test the system at the control of the development or project team.

Therefore, the project team is still largely in control of the environment and user access, etc.

Beta testing

Beta testing, on the other hand, is typically done on the clients actual site. This is using their own environment outside of the project teams control.

The benefit is, it mimics the real-life environment more. There still may be a representative from the project team on site to lend assistance, but not always.

Let me share an example of this from my experience of testing. Whilst working for an energy software consultancy. We were tasked with providing some software for them.

After SIT testing, was a phase of UAT on the client site using their environment. Any defects were entered into our defect management system for triage.

So what types of SIT is there?

SIT can also be broken down into two major approaches:

  • Top-down
  • Bottom-up

Let Me Explain…

Top-down approach

The top-down approach is focusing on the main parent function. The child modules can or will be replaced by stubs.

What is a stub?….

A stub is a test harness that will return an expected result to a parent function. This is used to test an application with missing (or incomplete) functions.

For example…  Cash machine…

Let’s assume we have a cash machine application and we want to test the interface that the customer uses (the parent module). In this example we want to test that the interface links to two child functions:

  • Withdraw cash page
  • View Statement

In this example, we could replace the whole withdrawal cash page with a stub. This could be written to return an expected value when the user selects the option from the main interface page.

This is a way of testing to see how the parent function will perform in the absence of the parent function.

Bottom up approach

The bottom up approach is essentially the complete opposite of top-down. You start with the lower level modules and use drivers instead of stubs.

A driver is the opposite of a stub. It will actually provide the input that you expect into a bottom level function to prove that the child function works.

So if we revert back to a cash machine application example…

If we want to prove that after viewing your statement you are returned to the parent menu. A driver could be used to provide a value to confirm that it performed as expected.

The importance of a UAT Kick-off meeting.

An information session or UAT kick-off session is a basic session to transfer information to the UAT testers before starting the UAT phase.

In this session, you will give important information such as

  • login information
  • System availability times
  • The expected duration of the UAT phase
  • Who is the point of call for any issues
  • Defect detail expectations
  • If screenshots are required and detail required

So, in a nutshell, it’s a way of establishing the expectation for UAT after following system integration testing.

In my experience of UAT testing, this is always been a very useful session. Because, if you do not have this initial startup information meeting you will find that you’ll get a lot of these questions asked by multiple people during the test process. Often, you will end up duplicating information again and again.

Related questions:

What is the difference between SIT and System Testing? SIT, as discussed earlier, is mainly focused on how the individual system application modules integrate together.

System Testing focuses on the behavior of the system functions. Hence the reason why it is often called functional testing. The goal of System testing is to prove the system meets the functional specification.

Why do you even need system integration testing? It proves the dependencies between system modules. Sometimes it is possible to get carried away focusing on which tests pass at a functional level.

But, not focusing on how they will hang together once all the functions are proven.

1 2 3 4
Skip to toolbar