In software, the term smoke testing describes the process of validating code changes before the changes are checked into the product’s source tree. After code reviews, smoke testing is the most cost effective method for identifying and fixing defects in software. Smoke tests are designed to confirm that changes in the code function as expected and do not destabilize an entire build.
 Note |
|---|
| The term smoke testing originated in the hardware industry. The term derived from this practice: After a piece of hardware or a hardware component was changed or repaired, the equipment was simply powered up. If there was no smoke, the component passed the test. |
The following guidelines describe best practices for smoke testing. The effects of following the guidelines will vary widely, ranging from improving communication among team members to developing specific ways to use testing and debugging tools.
Work with the Developer
Because smoke testing focuses on changed code, you must work with the developer who wrote the code. You will have to understand:
-
What changed in the code. To understand the change, you will also have to understand the technology used; the developer can help explain it.
-
How the change affects the functionality.
-
How the change affects the interdependencies of various components.
Conduct a Code Review Before Smoke Testing
Before you run a smoke test, conduct a code review that focuses on any changes in the code. Code reviews are the most effective and efficient method to validate code quality and ensure against code defects and faults of commission. Smoke tests ensure that the primary critical or weak area identified either by code review or risk assessment is primarily validated, because if it fails the testing cannot continue.
Install Private Binaries on a Clean Debug Build
Because a smoke test must focus on validating only the functional changes in updated binaries, the test must run on a clean test environment by using the debug binaries for the files being tested.
| Note |
|---|
| Testing with mismatched binaries is a common mistake in smoke testing. To avoid this mistake, when there is a dependency between two or more updated binaries, include all the updated binaries in the test build. Otherwise, the results of the test might not be valid. |
Create Daily Builds
Daily builds require the team members to work together and encourage the developers to stay in sync. If iterations of new builds are delayed, the delay can easily cause products with multiple dependencies to get out of sync. Following a process of building daily and smoke testing any changed or new binaries ensures high quality.
Make a quality daily build the team’s top priority. If the build is broken because a code check-in has not been smoke-tested, require the developer and tester to stop all other work until the problem is resolved. The penalty for breaking the build should not be harsh, but the penalty should emphasize that a proper daily build is the team’s top priority. |
You need not perform exhaustive tests. The purpose of smoke testing is not to ensure that the binary is 100% error-free. This would require too much time. You perform smoke tests to validate the build at a high level. You want to ensure that changes in a binary do not destabilize the general build or cause catastrophic errors in functionality.
Web and Load Testing
When you build your Web tests and load tests, it is a good practice to run a smoke test before running any long, heavy test. In Web and in load testing, smoke testing is short, light test. You use a smoke test to validate that everything is correctly configured and running as expected before running your tests for performance or stress testing.
You can use build types in Team Foundation Build to define the settings under which a team project is built. Before running a build on a build type, you must first create one, for more information,
Running a Build on Build Type
All files located in the Team Build Types folder for a build are downloaded to the build computer (build directory) folder. Ensure that no unnecessary confidential information is stored in this folder.
-
In Team Explorer, select the appropriate team project.
-
On the Build menu, click Build {Team Project Name} to display the Build dialog box.
-
Under the Build type drop-down list, select the build type that you want to run.
-
Under Build Location, retain the default computer configured to use the build type or select a Build machine from the drop-down list.
-
Under Build directory, retain the default directory configured for the build type or change the default directory that the build uses.
Click Build to start the build.
Daily Build and Smoke Test
If you want to create a simple computer program consisting of only one file, you merely need to compile and link that one file. On a typical team project involving dozens, hundreds, or even thousands of files, however, the process of creating an executable program becomes more complicated and time consuming. You must "build" the program from its various components.
A common practice at Microsoft and some other shrink-wrap software companies is the "daily build and smoke test" process. Every file is compiled, linked, and combined into an executable program every day, and the program is then put through a "smoke test," a relatively simple check to see whether the product "smokes" when it runs.
BENEFITS. This simple process produces several significant benefits.
It minimizes integration risk. One of the greatest risks that a team project faces is that, when the different team members combine or "integrate" the code they have been working on separately, the resulting composite code does not work well. Depending on how late in the project the incompatibility is discovered, debugging might take longer than it would have if integration had occurred earlier, program interfaces might have to be changed, or major parts of the system might have to be redesigned and reimplemented. In extreme cases, integration errors have caused projects to be cancelled. The daily build and smoke test process keeps integration errors small and manageable, and it prevents runaway integration problems.
It reduces the risk of low quality. Related to the risk of unsuccessful or problematic integration is the risk of low quality. By minimally smoke-testing all the code daily, quality problems are prevented from taking control of the project. You bring the system to a known, good state, and then you keep it there. You simply don't allow it to deteriorate to the point where time-consuming quality problems can occur.
It supports easier defect diagnosis. When the product is built and tested every day, it's easy to pinpoint why the product is broken on any given day. If the product worked on Day 17 and is broken on Day 18, something that happened between the two builds broke the product.
It improves morale. Seeing a product work provides an incredible boost to morale. It almost doesn't matter what the product does. Developers can be excited just to see it display a rectangle! With daily builds, a bit more of the product works every day, and that keeps morale high.
USING THE DAILY BUILD AND SMOKE TEST. The idea behind this process is simply to build the product and test it every day. Here are some of the ins and outs of this simple idea.
Build daily. The most fundamental part of the daily build is the "daily" part. As Jim McCarthy says (Dynamics of Software Development, Microsoft Press, 1995), treat the daily build as the heartbeat of the project. If there's no heartbeat, the project is dead. A little less metaphorically, Michael Cusumano and Richard W. Selby describe the daily build as the sync pulse of a project (Microsoft Secrets, The Free Press, 1995). Different developers' code is allowed to get a little out of sync between these pulses, but every time there's a sync pulse, the code has to come back into alignment. When you insist on keeping the pulses close together, you prevent developers from getting out of sync entirely.
Some organizations build every week, rather than every day. The problem with this is that if the build is broken one week, you might go for several weeks before the next good build. When that happens, you lose virtually all of the benefit of frequent builds.
Check for broken builds. For the daily-build process to work, the software that's built has to work. If the software isn't usable, the build is considered to be broken and fixing it becomes top priority.
Each project sets its own standard for what constitutes "breaking the build." The standard needs to set a quality level that's strict enough to keep showstopper defects out but lenient enough to dis-regard trivial defects, an undue attention to which could paralyze progress.
At a minimum, a "good" build should
- compile all files, libraries, and other components successfully;
- link all files, libraries, and other components successfully;
- not contain any showstopper bugs that prevent the program from being launched or that make it hazardous to operate; and
- pass the smoke test.
Smoke test daily. The smoke test should exercise the entire system from end to end. It does not have to be exhaustive, but it should be capable of exposing major problems. The smoke test should be thorough enough that if the build passes, you can assume that it is stable enough to be tested more thoroughly.
The daily build has little value without the smoke test. The smoke test is the sentry that guards against deteriorating product quality and creeping integration problems. Without it, the daily build becomes just a time-wasting exercise in ensuring that you have a clean compile every day.
The smoke test must evolve as the system evolves. At first, the smoke test will probably test something simple, such as whether the system can say, "Hello, World." As the system develops, the smoke test will become more thorough. The first test might take a matter of seconds to run; as the system grows, the smoke test can grow to 30 minutes, an hour, or more.
Establish a build group. On most projects, tending the daily build and keeping the smoke test up to date becomes a big enough task to be an explicit part of someone's job. On large projects, it can become a full-time job for more than one person. On Windows NT 3.0, for example, there were four full-time people in the build group (Pascal Zachary, Showstopper!, The Free Press, 1994).
Add revisions to the build only when it makes sense to do so. Individual developers usually don't write code quickly enough to add meaningful increments to the system on a daily basis. They should work on a chunk of code and then integrate it when they have a collection of code in a consistent state-usually once every few days.
Create a penalty for breaking the build. Most groups that use daily builds create a penalty for breaking the build. Make it clear from the beginning that keeping the build healthy is the project's top priority. A broken build should be the exception, not the rule. Insist that developers who have broken the build stop all other work until they've fixed it. If the build is broken too often, it's hard to take seriously the job of not breaking the build.
A light-hearted penalty can help to emphasize this priority. Some groups give out lollipops to each "sucker" who breaks the build. This developer then has to tape the sucker to his office door until he fixes the problem. Other groups have guilty developers wear goat horns or contribute $5 to a morale fund.
Some projects establish a penalty with more bite. Microsoft developers on high-profile projects such as Windows NT, Windows 95, and Excel have taken to wearing beepers in the late stages of their projects. If they break the build, they get called in to fix it even if their defect is discovered at 3 a.m.
No comments:
Post a Comment