Five Fundamentals of a Successful Application Test Strategy

This research note addresses the need to develop application test strategy as an integral first step of an application testing process. Specifically, this note:

• Defines an application test strategy.
• Explains the goals of an application test strategy.
• Describes the steps required to craft and maintain a test strategy.

A test strategy is required to achieve thorough and successful application testing. Consider testing scope, schedule, resources, defect tracking methods, and functional metrics when developing an application test strategy.

Implementation Point

Many IT managers face a similar struggle when it comes to application test planning. On one hand, they understand that vigorous testing is necessary to validate system performance and functionality. On the other hand, the push to meet strict development deadlines often leaves little, if any, time to complete testing.

Developing an application test strategy is an integral first step of an application testing process. If created early in the development lifecycle, a test strategy will pave the way to effective test execution. Defining a detailed test strategy requires input from testing, development, and business analyst resources. The result of this effort is a detailed document that outlines the scope, schedule, resources, defect tracking methods, and functional metrics of the testing effort.

Key Considerations

An application test strategy is a critical input into the application testing phase of the system development lifecycle. A test strategy is a document that defines the scope, approach, and schedule of the planned testing activities. It identifies, at a high level, the items that require testing along with the resources who will execute the testing.

A test strategy should be developed early in the development cycle when the IT manager can obtain consensus and commitment from the entire project team including development, testing, operations, and project stakeholders. A test strategy is a development aid and should be promoted as such; shaping the test strategy prior to beginning application development allows the project team time to digest the strategy and incorporate its finite details into related test plans.

Why Create a Test Strategy?

The IT manager should note there are several advantages to defining an application test strategy. These include the following:

• A test strategy eliminates confusion among test team members. It defines the processes and procedures for application testing in addition to who is testing specific features and functionality.
• A test strategy outlines all test items deemed in scope, reducing the risk of forgetting to test certain aspects of the application.
• A test strategy established early in the development lifecycle includes a schedule that coordinates with the overall project plan. This allows for flexibility to modify the project schedule to accommodate the test strategy if necessary.
• A test strategy ensures consistency throughout the testing effort. A test strategy will clearly indicate how testing should be designed, executed, and reported.
• Employ the ITA Premium “Application Test Strategy Template” to document a comprehensive testing plan in the early stages of the application development lifecycle.

Implementation & Integration

Enterprises must adopt a formal strategy for creating and maintaining an application test strategy. If crafted well, a test strategy can be used for repeated cycles of application development. Follow Info-Tech’s approach depicted in Figure 1 to develop an application test strategy.

Figure 1. Test Strategy Development Steps
Source: Info-Tech Research Group

Step 1: Define Testing Scope, Objectives and Approach

Defining the testing scope and objectives requires the inclusion of several pieces of information:

• Objectives. Test strategy objectives give a summary of the products to be tested, schedule overview including major milestones, and resource overview.
• Scope. The test strategy scope defines the high-level items that will be tested and also details items excluded from testing. The scope should include applications and systems that are in scope and out of scope. If major functions or process (e.g. accounts receivable, contact management, service requests) are known, those should also be included.
• Approach. The testing approach specifies the testing phases that will be integrated in the overall testing effort and estimates the required effort and duration for each. Each type of testing requires its own test plan. Refer to the ITA Premium research note, “Set Your Sights on the Six Software Testing Targets” to understand the six types of testing recommended to achieve a high quality end product: Unit Testing, Functional Testing, System Testing, Regression Testing, System Integration Testing, and Acceptance Testing. The testing approach should also include a list of all deliverables required for each type of testing. The deliverable list will include items such as test plans, test scenarios, test cases, and test signoff procedures.

Step 2: Establish Test Schedule

The test schedule outlines the dates during which each phase of testing is planned to take place. Based on the testing scope and approach the test manager should estimate the planned start and duration required for each test phase. Specifically, the test schedule should include:

• An overall start and end date. These dates denote the first day that the first phase of testing will commence and the last phase of testing will be completed. All phases of testing must be planned between these two dates.
• Start and end date of each test phase. These dates denote the start and end dates of the subtesting phases that are executed within the overall test strategy.
• Test entry criteria. Test entry criteria indicate what is required to initiate a specific testing phase. For example, to initiate unit testing, entry criteria may state that each module be coded to the specifications outlined in the detailed design document.
• Test exit criteria. Test exit criteria indicate what is required to declare a testing phase complete. For example, to complete functional testing, the exit criteria may indicate that each piece of functionality pass through a test cycle without raising any high severity defects.
• Test milestones. The test schedule should include any significant milestones that will be reached throughout testing efforts.

While planning the schedule, the test manager is often asked to accommodate dates provided by the project management team; the test manager will be given a date when testing must be complete. In some situations the dates are unachievable based on the amount of testing required to ensure quality. It is the responsibility of the test manager to alert project management when this occurs. To rectify this situation, the project management team should look for alternate solutions such as revising the testing end date, removing application functionality or reducing the testing effort.

Step 3: Assign Testing Resources

Testing resources requirements are based on the scope of the testing effort and the testing schedule. Resources should be planned to complete the testing described in the scope within the time allotted in the schedule.

The test strategy should include a matrix that states which testing resources are assigned to particular testing assignments. This information should be available to the testing team at the onset of the project. Providing this information early on will allow the testing team members to form a working relationship with the developers of the code.

The testers should be aware of the developers who coded the areas they are responsible for testing, allowing them to raise potential questions and concerns with them. Table 2 shows a sample high-level matrix that assigns resources to specific testing areas.

Table 2. Sample Testing Resource Assignments
Source: Info-Tech Research Group

Step 4: Define Test Environment Requirements

Test environment requirements should be specified in the application test strategy:

• Hardware. Identify the hardware required to build a testing environment. Computers, networks, printers, and mainframe access are examples of physical machinery that may be required to conduct application testing.
• Software. Identify the software required to complete testing activities. This includes software in addition to what is being tested. Test management software, automated testing software, and databases are examples of software that may be required to complete testing.
• Location. Identify where the application testers will be located. Determine what office space, seating assignments, and office furniture are required to suit the testing effort. Consider if the testers require a special environment to perform testing such as a “clean room,” a disaster recovery site, a testing lab or a remote-site.
• Publications. Identify all documents, manuals, and user guides that are required to support the testing team. The publications may include use cases, application requirements, out-of-the-box application guides, and application design specifications.

Step 5: Ascertain Defect Tracking Procedures

The test strategy should define a procedure to raise, track, and resolve defects found throughout testing. The procedure can be as involved as using defect tracking software or can be as simple as maintaining a list of defects in a spreadsheet shared among the testers and developers. Regardless, it is imperative that the following information be documented:

• Date Raised. Specify the date the defect was raised.
• Raised By. Specify the resource who raised the defect.
• Issue Description. A description of the defect. Where possible, include detailed steps on how to reproduce the issue.
• Testing Phase. Specify the phase in which the defect was raised, i.e. functional testing, system testing, etc.
• Assigned To. Specify the developer to whom the defect is assigned for resolution.
• Sign-Off Date. Specify the date the developer fixes and signs-off on the defect resolution back to the tester.
• Sign-Off Description. The developer provides a description of what was done to fix the issue.
• Severity. Assign a severity to the defect. There are several options available to help define defect severity. The following scale provides a manageable range of severity ranking and is based on information found on www.stickyminds.com.
  • High: A major issue where a large piece of functionality or major system component is completely broken. There is no workaround and testing must halt.
  • Medium: A significant issue where a large piece of functionality or major system component is not working properly. There is a workaround, however, and testing can proceed.
  • Low: A minor issue that imposes some loss of functionality, but for which there is an acceptable and easily reproducible workaround. Testing can proceed without interruption.
• Priority. Assign a priority to the defect. A defect priority level can be used along with severity to help determine the urgency in fixing the defect. The following three point priority scale can be used in common testing practices. The levels are:
  • High: This has a major impact on the customer. This must be fixed immediately.
  • Medium: This has a major impact on the customer. The problem should be fixed before release of the current version in development, or a patch must be issued if possible.
  • Low: This has a minor impact on the customer. The flaw should be fixed if there is time, but it can be deferred until the next release

Step 6: Define Constructive Testing Metrics

Test metrics provide a vehicle to communicate testing progress to the project team. Metrics detail what has been tested, when and how often it has been tested, and whether or not the testing was successful. The test strategy should include all metrics required by the IT Manager needs to report on progress and make informed decisions about the direction of the project. Some common metrics included in test strategies are:

• Functional Metrics. Number of requirements verified (may be broken down by testing phase, component, testing resource or all of the above).
• Code Metrics. Percent of code executed (may be broken down by test and/or component).
• Problem Metrics. Problems found per day, problems found per component.
• Schedule Metrics. Percent of tests completed.
• Estimated Days to Completion.
• Time to Complete Testing by Component.

Testing metrics can be used to generate charts and graphs to help report progress to project management. Although useful, creating visual aids can be time consuming and might never used to make business decisions. Involve project management when producing reporting metrics to help determine which data will be most valuable.

Bottom Line

A test strategy is required to achieve thorough and successful application testing. Consider testing scope, schedule, resources, defect tracking methods, and functional metrics when developing an application test strategy.