What is Sanity Testing and How to Do It Right?

What is sanity testing and why is it necessary?

Sanity testing is a type of software testing that is used to verify that a small change or a bug fix in the software is working as intended and has not caused any unexpected side effects or broken any existing functionality. Interestingly, according to a HackerRank report, the most common bugs found in production were a result of deploying untested or broken code. As a result, sanity testing is crucial to ensure all aspects of the software are checked accordingly.

The main goal of sanity testing is to validate the changes made to one or few specific parts of the application and not detailed testing. It is a quick and superficial check to ensure that the system is still functioning as expected after a change has been made. Additionally, sanity testing is often used as a preliminary step before conducting more extensive testing, such as regression testing. It is an important part of the software development process because it helps to ensure that changes to the system do not negatively impact its overall stability and functionality. Usually sanity tests are derived from existing regression tests, but, unlike regression testing, sanity testing typically is unscripted.

Sometimes the term 'sanity testing' may be used interchangeably with 'smoke testing'. They are both similar in that they determine whether it is possible and reasonable to continue testing further, however, there are a couple of key differences between the two.

Difference between sanity testing and smoke testing

As the name implies and as we explained above, sanity testing aims at checking the obvious—whether the intended result of a code change works correctly. Smoke testing, on the other hand, checks if nothing else important was broken in the process.

Some believe that the term 'smoke testing' originated in electronic hardware testing. In his book “Lessons Learned in Software Testing”, Bret Pettichord provides the origin of the term:

"The phrase smoke test comes from electronic hardware testing. You plug in a new board and turn on the power. If you see smoke coming from the board, turn off the power. You don't have to do any more testing."

Thankfully, in software testing there isn’t any actual smoke in the event of failure.

Smoke testing is a type of testing that is used to determine if the software is stable enough to proceed with further testing. It is usually done early in the testing process and is designed to ensure that the most important functionality of the software is working correctly. For people new to software testing it can be difficult to differentiate between smoke and sanity testing, as they are similar. Both are used to verify the basic functionality of an application, but there are three main differences:

• Scope
  Sanity testing is typically more focused in scope than smoke testing. Sanity testing checks specific functionality, for example, if a particular bug fix is working as expected. Smoke testing, in contrast, has a broader and more thorough test scope that checks the most crucial functionality of the application.
• Depth
  Sanity testing is a superficial check for one or few specific parts of the application. It is a quick and lightweight testing process that is not designed to uncover deep-seated issues or errors in the system. In comparison, smoke testing is a more thorough test that can uncover major issues or defects in the system, as it tests the critical functionalities independent of the new feature, change or fix.
• Purpose
  The primary purpose of sanity testing is to confirm that a small change to the system has not caused any unintended side effects or broken any existing functionality. Smoke testing, on the other hand, is used to verify the overall functionality of the system and ensure that it is stable and ready for further testing.

To sum up, the main difference between sanity and smoke testing is that sanity testing checks one or few specific parts of the application, while smoke testing checks the critical functionalities independent of the new feature, change or fix. Now that the main characteristics of sanity and smoke testing are clear, there is another question—which should be done first, sanity or smoke testing?

When should sanity testing be done?

Let’s look at the software development process for a mobile application as an example. After a new mobile app build is released, it usually requires smoke testing. If smoke testing is successful, then sanity testing is conducted. And if this is also successful, then more extensive testing—functional and/or regression testing—is done. Sanity testing is usually performed after smoke testing because smoke testing ensures that the build is stable. See the flowchart below.

If smoke testing fails, it usually indicates that there are serious problems with the software. Further testing should not be done until the issues are resolved. If sanity testing fails, it usually indicates that there is a problem with a specific feature or functionality of the software. In both cases the development team should be notified and the issues that caused the failure should be addressed. If smoke tests passed but sanity tests failed after which a new version of the mobile app is released, it would be a good idea to perform smoke testing before repeating the sanity testing.

Performing smoke testing before repeating sanity testing helps ensure that the new version of the mobile app is stable and that it is safe to proceed with further testing. This helps to prevent further problems from occurring and helps to ensure that the software is of high quality. Once smoke testing has been completed successfully, sanity testing can be repeated to verify that the specific feature or functionality that failed previously is now working as expected.

These repeated testing efforts may seem like a lot, but without sanity testing, if a full testing cycle is started, then testing efforts most likely would go into vain if a new release is required. The pros of sanity testing outweigh the cons.

Pros and cons of sanity testing

Pros of sanity testing:

• Quick. Sanity testing helps to avoid wasting time and effort by quickly determining whether an application is too flawed to merit more rigorous testing.
• Efficient. Sanity testing is an efficient way to verify that a small change or modification to the system has not caused any unintended side effects or broken any existing functionality.
• Focused scope. Because it is a more focused test, sanity testing allows testers to focus on one or few specific areas of the system and confirm that the changes there have not caused any unintended side effects.
• Cost-effective. Sanity testing is typically a fast and lightweight testing process that does not require a lot of resources or time to complete. As a result, it can be a cost-effective way to ensure that the system is still functioning as expected after a change has been made.

Cons of sanity testing

• Limited scope. The limited scope of sanity testing means that it may not uncover deep-seated issues or defects in the system.
• May not detect all issues. Because it is a superficial check, sanity testing may not detect all issues or errors that may be present in the system.
• May not be sufficient. In some cases, the changes made to the system may be more significant than a simple bug fix or code change, in which case a more thorough testing process, such as regression testing, may be needed to ensure the stability and functionality of the system.

Now that we know what sanity testing is, why and when it should be done as well as the advantages and disadvantages of it, only one question remains.

How to do sanity testing right?

Here are three best practices for conducting effective sanity testing:

1. Define the scope. Clearly define the scope of the sanity testing process, establish what exactly needs to be tested. This should include the specific areas of the system that will be tested, as well as any specific requirements or constraints that need to be considered. Sanity testing usually is unscripted, which means that there are no test steps to follow, however, defining the scope (for example, as a checklist of things that need to be tested) would definitely be useful so that nothing important is missed.
   
2. Use automated tools. If possible, it is a good idea to automate tests that are run often. Consider using automation testing tools to help streamline the sanity testing process and reduce the time and effort required to complete it. Automating tests speeds up checking and comparing the expected result with the actual one and allows tests to run in parallel. It would also decrease the reliance on regression testing since sanity tests are based on regression tests and if the sanity tests are automated that means that troubleshooting can be continuous.
   
3. Review the results. The results of the sanity testing will help to identify any areas where additional testing may be needed. This may include running additional test cases or conducting more thorough testing, such as regression testing.

These best practices may as well be used for other types of testing, like smoke or regression testing, because they are more or less universal. However, sanity testing is relatively easy to do compared to other types of testing that, of course, have their own benefits and purpose but require much more effort. By following these best practices, the sanity testing process will be effective and help to confirm the stability and functionality of the application after small changes or modifications have been made.

Key takeaways

Sanity testing is used to validate the changes made to one or few specific parts of the application in order to make sure that the software is still functioning as expected after a small change or a bug fix. It is often used as a preliminary step before conducting more extensive testing, such as regression testing. In comparison to sanity testing, smoke testing is a broader and more thorough test that checks the most crucial functionality of the application but sanity testing checks only specific functionality after some changes have been made to it.

To ensure the best results from sanity testing, make sure to: clearly define the testing scope, use automated tools if possible and review the results in order to identify areas where additional testing should be done.

Do you have an application or software solution that could use some testing? Check out our different software testing services and contact us for more information on how we can help you.