A Range of Errors

Ah… data values within a range. So many possibilities for fun. Ran into one of those today while reviewing a potential work item being proposed. As presented:

Add "Length of Incarceration" range

Select from a list
* Less than 1 year
* 1-3 years
* 3-5 years
* 5-10 years
* 10-20 years
* More than 20 years

Spot the problem? What happens if the length of incarceration was 1, 3, 5, or 10 years? Which of the two ranges that include these values should be chosen?

Presumably we’re grouping this data into ranges for reporting purposes.

At what point does it become statistically significant that given a situation where an offender spent 5 years incarcerated, Elizabeth always chooses 3-5 years, Ross always chooses 5-10 years, and Eduardo sometimes chooses one range and sometimes chooses another?

In any system with user-entered data, GIGO can apply, but let’s help the users by designing a system which makes it harder to input garbage.


Testing Like an Airline Pilot with James Bach at STARWest

Yesterday James Bach presented one of the day-long tutorials preceding the STARWest conference. I tweeted a bunch during his talk, but I also wanted to revisit my notes and identify some key points and thoughts. And because I’m all generous and shit, I’m sharing that here 🙂

Critical Thinking for Software Testers

The talk was about critical thinking as a tester. The natural opening to the talk was identifying what is meant by testing and being a tester.

What is Testing (vs. Checking)?

Checking is the process of noting whether something worked exactly as expected off of a checklist of repeatable procedures. This is where most so-called “automated testing” falls… it’s not really testing… it’s checking that the software behaved as expected based on a pre-written list of specific expectations.

Testing is the broader process and cannot be simply reduced to a set of instructions. Testing requires a human brain to analyze things and make decisions on the fly, reacting and adapting. In this respect, testing is much like being a doctor, lawyer, or airline pilot… while there are parts of these professions that require following a checklist for the expected scenario, it’s also required that a professional have the ability to think on their feet, adapting to the unexpected.

Testers are agents for the user

Basic testing is straightforward and uses reflex thinking – faster and looser. Excellent testing requires slower, surer thinking. Excellent testing involves a difficult social and psychological process in addition to the technical aspect. It requires mixing technical observation with context in order to understand what’s important.


A signfiicant portion of the afternoon discussed modeling and observation; models are fundamental to critical thinking. The ones and zeros of a software program do not exist on their own. The application and elements within the application usually model other things.

What you see is not all there is.

Models link observation and inference… and a good tester must be able to distinguish the two.

Test Cases and Observations

If someone asks how many test cases are needed to test a program or function, well, that’s a really crappy question that will lead to a meaningless answer. A better question would be how will you test this? The number of test cases doesn’t mean much… much like we don’t measure program complexity or productivity by lines of code, we shouldn’t much care about the number of test cases.

Documents and statements are stories, not reality

Huh? Really? So?

Three easy questions when testing.

Huh? – gets at the meaning… implies that you may not understand what’s really going on

Really? – gets at the facts… perhaps what you understand may not be true

So? – gets at the risks… the truth may not matter, or it may matter more than you think

The overall theme of the day fit in line with Bach’s philosophies on context-driven and exploratory testing and analysis. Check out Lessons Learned in Software Testing: A Context-Driven Approach for more.