ROI Example

Dynamic Analysis as an Investment

For the purpose of this return on investment example, we will consider two classifications for software bugs: bugs found during the development lifecycle (by the developer or an independent tester) and bugs found during operations (i.e., on-orbit).

The process for finding software defects during the lifecycle involves the tester researching and reporting the failure, the programmer identifying and fixing the fault, the release engineer producing a new release, the system administration team installing the new release in the test environment, and the tester retesting the new release to confirm the fix and to check for regression.

The costs of failure during operations are higher than those associated with defects found during testing. In these cases, not only does the same process described above occur, but there are additional costs for releasing a fix in operations (a more expensive process) rather than releasing to the test lab. Not to mention the additional costs of the operations team having to troubleshoot the issue when it is identified and costs associated with downtime or lost functionality due to the defect.

Two observations lay the foundation for viewing dynamic analysis as an investment. First, like any cost equation in business, it is desirable to reduce the cost of quality. Second, while it is often cheaper to prevent problems than to repair them, early detected failures cost less than failures found during operations.

Hypothetical Case Study

Assumptions: On average, each release contains 100 runtime bugs that are identified and repaired over the life of the release. Issues found during testing cost $1,000 to fix and issues detected during operations cost $10,000 to fix (a factor of 10 for on-orbit issues).

For the purpose of this example, consider two scenarios.

1. No independent testing is performed. Seventy-five (75) issues are found during developer testing, leaving 25 defects to be found during operations.
2. Independent testing is performed. Seventy-five (75) issues are found during developer testing, 15 issues are found during independent testing, and 10 are found during operations.

These are hypothetical numbers and the return on investment is directly affected by the cost to fix issues in operations. However, the return on investment is proportional to the cost difference between finding issues during development/independent testing as compared to finding them during operations as shown in the graph below.