Why adding more test phases may not reduce bugs – The Peltzman Effect

After a previous release was so buggy, the management team added an additional test phase, expecting quality to go up. However, when it was unchanged, everyone was puzzled and demanded to know why adding more testing had not improved quality.

We test to reduce bugs. We are often tempted to add another test phase or quality process, in the belief that we will somehow reduce the number of bugs in the finished product. But what if adding phases made absolutely no difference to quality?

In this session, we learn of a risk compensation phenomenon known as the Peltzman Effect. We learn why the benefits expected from adding quality processes rarely materialise, but are instead consumed by behavioural changes elsewhere that leave the bug level unchanged.
Using examples from traffic safety, dangerous sports, and safety equipment, we show how an increase in safety frequently leads to a compensatory increase in risk taking behaviour.

We demonstrate how this Peltzman Effect applies in software development. We show how improving one area invariably leads to compensatory behaviour elsewhere in the development process, with the net result that quality remains unchanged, even if progress towards the overarching project goal is improved.

We explore the circumstances under which the Peltzman Effect becomes important in software development. We learn when adding quality processes improve quality and when adding them makes no difference.

We also learn how we can reduce the Peltzman Effect, plus how we can improve quality above its present value in an organisation, not by introducing additional quality phases but by shifting the relative importance of quality as a project goal.

Key Takeaways

  • Risk improvements in one area are frequently compensated for by changes in another area. This is the Peltzman Effect.
  • A quality improvement may result in benefits other than a simple bug reduction. For example, teams may choose to take the benefit as an earlier delivery.
  • Teams have control over how they choose to consume a quality or other benefit.