Testing Like an Airline Pilot with James Bach at STARWest

Yesterday James Bach presented one of the day-long tutorials preceding the STARWest conference. I tweeted a bunch during his talk, but I also wanted to revisit my notes and identify some key points and thoughts. And because I’m all generous and shit, I’m sharing that here 🙂

Critical Thinking for Software Testers

The talk was about critical thinking as a tester. The natural opening to the talk was identifying what is meant by testing and being a tester.

What is Testing (vs. Checking)?

Checking is the process of noting whether something worked exactly as expected off of a checklist of repeatable procedures. This is where most so-called “automated testing” falls… it’s not really testing… it’s checking that the software behaved as expected based on a pre-written list of specific expectations.

Testing is the broader process and cannot be simply reduced to a set of instructions. Testing requires a human brain to analyze things and make decisions on the fly, reacting and adapting. In this respect, testing is much like being a doctor, lawyer, or airline pilot… while there are parts of these professions that require following a checklist for the expected scenario, it’s also required that a professional have the ability to think on their feet, adapting to the unexpected.

Testers are agents for the user

Basic testing is straightforward and uses reflex thinking – faster and looser. Excellent testing requires slower, surer thinking. Excellent testing involves a difficult social and psychological process in addition to the technical aspect. It requires mixing technical observation with context in order to understand what’s important.


A signfiicant portion of the afternoon discussed modeling and observation; models are fundamental to critical thinking. The ones and zeros of a software program do not exist on their own. The application and elements within the application usually model other things.

What you see is not all there is.

Models link observation and inference… and a good tester must be able to distinguish the two.

Test Cases and Observations

If someone asks how many test cases are needed to test a program or function, well, that’s a really crappy question that will lead to a meaningless answer. A better question would be how will you test this? The number of test cases doesn’t mean much… much like we don’t measure program complexity or productivity by lines of code, we shouldn’t much care about the number of test cases.

Documents and statements are stories, not reality

Huh? Really? So?

Three easy questions when testing.

Huh? – gets at the meaning… implies that you may not understand what’s really going on

Really? – gets at the facts… perhaps what you understand may not be true

So? – gets at the risks… the truth may not matter, or it may matter more than you think

The overall theme of the day fit in line with Bach’s philosophies on context-driven and exploratory testing and analysis. Check out Lessons Learned in Software Testing: A Context-Driven Approach for more.


2 thoughts on “Testing Like an Airline Pilot with James Bach at STARWest

  1. Pingback: Five Blogs – 4 October 2013 | 5blogs

  2. Pingback: Great Expectations | Kwality Rules

Chime in with your thoughts

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s