The Thursday Quote - Phillip G. Armour

---

"While the business of software has evolved from actually punishing people for finding defects, we can still be pretty ambivalent about them."

Testing: Failing to Succeed by Phillip G. Armour, from Communications of the ACM October 2011

---

So, explain it to me again: Why can't we punish people for finding bugs?

...oh, yeah, ethics, morality, legality, economics, common sense, all that stuff. But you have to admit, it's a common desire to hear only what we want to hear.

Anyway, here's the wider context for that quote, an excerpt from Philip Armour's recent article on software testing:

---

There are two situations in software testing that scare testers: when they see "too many" defects and when they do not see "enough." Too many defects may indicate the system contains a lot of problems even though it has moved into the testing phase. Too few discovered defects might imply the product is of high quality but it usually means the testing process itself is not working.

This is symptomatic of the paradox of testing: We want to find defects in the system under test, but we do not really want to find them. While the business of software has evolved from actually punishing people for finding defects, we can still be pretty ambivalent about them. Experienced testers have a good feel for the balance between finding too many defects and not finding enough2 but most organizations and most customers would still prefer to hear "we haven't found many defects" than "we've found huge numbers of defects."

If we view testing as a knowledge acquisition activity rather than simply a post hoc quality assurance process we get a different view. We also get a different view when we line testing up against the Five Orders of Ignorance. Zero Order Ignorance (0OI) is lack of ignorance; it is proven knowledge, knowledge that we have acquired and against which some trial has made that certifies it as "correct." Second Order Ignorance (2OI) is lack of awareness of ignorance; it occurs when we don't know something and we are unaware that we don't know it. That is, we have ignorance about our lack of knowledge. We test systems primarily for these two of the Five Orders of Ignorance and their focus is quite different.

---

He wrote about the The Five Orders of Ignorance back in October 2000; here's a summary:

• 0th Order Ignorance (0OI) — Lack of Ignorance: "I know this."
  

• 1st Order Ignorance (1OI) — Lack of Knowledge: "I know that I don't know something."
  

• 2nd Order Ignorance (2OI) — Lack of Awareness: "I don't know what it is that I don't know.
  

• 3rd Order Ignorance (3OI) — Lack of Process: "I don't know how to figure out what it is that I don't know."
  

• 4th Order Ignorance (4OI) — Meta Ignorance: "I don't know about the five orders of ignorance."
  

At first reading the five orders seem deep and significant... but are they useful? After all, it's impossible to hang on to 4th Order Ignorance, it's gone by the time you read about it!

---

Next week: Moshe Y. Vardi

---