Wednesday, February 8, 2012

Weekend Testers Americas #24 - Black Box Software Testing: Practice 3

Today I participated in Weekend Testers Americas Session #24.  The mission for this session was fairly open ended.

* In today's mission we would like you to frame your exploration on your 
* We are about to give you a 'black box machine'. 
* You will have 60 minutes. 
* At the end of the timebox present your notes (no more than page long) to the group chat.
* What to report and in what format - we leave it up to your decision. 
* We hope everybody will bring a unique bit of experience we all can learn from.

Ultimately it boiled down to an exploratory testing session on a practice  flash applet.  Because of the way the mission was framed, each individual tester, or testing pair defined the mission for themselves. Since the general feel of the mission was so open, and my instincts were to work the practice as an exploratory session, I felt there was a lot of room to try things, perhaps things I wouldn't normally have done.   While some of the testers paired, I elected to go it alone this time though.

Before I get into my synopsis of the exercise and my conclusions of how that session went, I'd like to give credit to James Lyndsay who I am told is the one who produced this particular practice over at his site  There are a number of practice exercises that could be used for some general and basic practice of exploratory testing skills.  From what I've read he also has some more in depth training as well, which could be a useful if exploratory testing is an area you feel could help you in your testing efforts.  I'll leave it to my readers to decide that for themselves.    Now, if you'd like to try the exercise before reading my report, head on over to and explore it to your hearts content for up to an hour.  Otherwise, what follows would be considered a 'Spoiler'.   So when you are ready to continue reading, please click 'Read More' below.

Before I begin my observations, I'm going to take a moment to be a bit critical of my attempt at this session.  Admittedly, it's been a while since I've done focused exploratory testing.  I have done some exploratory testing in the past, but my current tasks does not have exploration as a key component as I see it.  So there is no established 'routine' for exploration that I might normally have fell back into using.  This could have been a plus or a minus.  Additionally, in some ways  I was less than prepared.  While I did use Shmuel Gershon's RapidReporter for note taking, at one point in, I realized I didn't have a notepad and pen handy, (something that normally I always have handy, but for some reason didn't think to grab before this session.)  Since I didn't want to lose time to go hunt for a notepad in my other room, I proceeded with out it.  In hindsight, if I were doing this again, I think I could have perhaps better organized my effort if I had taken time to use a notepad to organize my thoughts after the first few minutes of the session.

What follows is my report as submitted to the session, plus later, notes as recorded into rapid reporter.


Mission: Explore functionality.

Page Load testing:

Chrome: First Try Failed to Load - about:blank;  Second Try: spun out but never loaded.
Firefox: Inconclusive, wasn't setup for flash (odd thought it was) (and tried to download the swf)
IE 9: Loads no problems.  Will use IE 9 for testing

Box is Win 7 64 Bit.

First thought was what areas did the mouse change, or appear clickable.  The screen has four buttons in blue and yellow in a 2x4 pattern on the left, a dial gauge that is circular, with a needle, green until like the last twenty percent.  A label in the lower left hand corner that says Puzzle 3.  A red and green LED below the dial, and a pinkish red image that I'll call a spool in the bottom right.  It appears the 4 blue and 4 yellow buttons are clickable, as well as the red spool and ? mark in the bottom right.

Hovering over the Spool also revealed the Test "work room productions; Test Strategy quality assurance' text. The spool surprised me a bit, because it pops up a message from the swf's creator, but only as long as you hold the mouse button down.  Same with the ? mark which gives some hints.  Oops noone told me it would give me hints, Spoilers!

Observations, the text has a transparent background for the spool, and is a bit difficult to read due to this (for me anyways)  It's not entirely illegible but someone with poorer eyesight might not be able to read it.

Next try clicking buttons, initialy thought was to do a zig zag pattern from Blue 1 down.  (wondering if order matters Permutations vs Combinations that is.)  Tried a sequence going from left to right then down the left to right then down the left to see what happens when all buttons pressed.  Noticed that at 7 or 6 buttons the red light may come on, but not sure again if order matters here or not.  I tried a number of different combinations here, and then thought if trying some key combinations.  A drag of finger across keyboard revealed that the buttons would get 'inset' if a key was pressed, seems that's the U key that causes it.  IS this cosmetic or does it affect behavior would require a more detailed plan I think to figure out.

Stepping back, noting to self that I forgot to have a notebook handy to write things on quickly between clicks.  Would really like to to do a truth table for this thing if I had more time I would do so.

Ooo Pressing S and T seems to cause a target to show up, and I notice that it seems to be clickable with data entered into it. (only numerics though) Interesting.

Eventually I got the needle to move, stop, decrease, increase, halt with no lights on, green light on, both green and red, etc.  So I wonder what the lights acutally indicate since they don't indicate which direction it's moving, which would have been my thought.

Conclusion: If I had a note pad handy, a truth table would have been better to get what combinations of buttons did what.  Thus more testing would be needed if it were just me to obtain that.)

That's the short synopsis of the testing we did for about sixty minutes.  You can see the reports of other testers  at It is always interesting to look at the approaches others took, and what they discovered.  In some cases, testers focused on that combination, and what combinations caused the needle to move, or the lights to come on.  Some tried to visualize it as a control system for some system.  While my report didn't reflect it, I was imagining it as an air gauge, or a switching system with a Volt or Amp Gauge where the possibility could be that going to far into the red would cause something 'bad' to happen.  Nothing seemed to happen when we got there that I noticed though.

Some of the testers tried to define what the states might mean for the buttons.  Is a button up, considered on or off? I didn't really spend much time wondering what those buttons met.  If I had access to the customer, which another tester did note, I'd probably have asked for some background for the control system, and that likely would have had a profound impact on how I viewed and proceeded with my testing.  One tester, Pradeep Soundararajan, thought he'd seen the exercise before, and rather than just testing the device, spent time devising the possible repercussions of testing it.  He spent time identifying useful heuristics that could be needed to properly test something like this.  I actually found this refreshing, as he identified some potential biases, and problems you might have to answer when testing something like this.

I'll let my readers go to the Experience Report over at  to draw their own conclusions, but I definitely benefited from seeing how people in different backgrounds saw testing of this kind.   Before I close this blog, I'd like to post my notes from rapid reporter, so you can see my thought process as I went about the testing.

Session Report | Powered by Rapid Reporter

 Show autogenerated rows
TimeReporterTypeContentScreenshot RTF Note 
2/4/2012 1:01:14 PMTim Western(Rapid Reporter version)
2/4/2012 1:01:14 PMTim WesternSession ReporterTim Western
2/4/2012 1:01:14 PMTim WesternSession CharterWeekend Testing Americas #24 Mission: TBD
2/4/2012 1:01:21 PMTim WesternSetupJoin Skype Converation
2/4/2012 1:01:26 PMTim WesternSetupSay Hi and Check in.
2/4/2012 1:03:26 PMTim WesternSetupIntroductions
2/4/2012 1:11:36 PMTim WesternSetupMission statement:
2/4/2012 1:11:53 PMTim WesternSetupIn today's mission we would like you to frame your exploration on your own. We
2/4/2012 1:12:04 PMTim WesternSetupare about to give you a 'black box machine'.
2/4/2012 1:12:11 PMTim WesternSetupYou will have 60 minutes. At the end of the timebox present your notes (no more
2/4/2012 1:12:20 PMTim WesternSetupthan page long) to the group chat.
2/4/2012 1:12:28 PMTim WesternSetupWhat to report and in what format - we leave it up to your decision. We hope
2/4/2012 1:12:34 PMTim WesternSetupeverybody will bring a unique bit of experience we all can learn from.
2/4/2012 1:12:59 PMTim WesternSetup[1:11:44 PM] Weekend Testers Americas:
2/4/2012 1:14:55 PMTim WesternSetupChrome test opening:
2/4/2012 1:15:03 PMTim WesternTestOpen in Chrome results in blank screen? odd
2/4/2012 1:15:48 PMTim WesternTestclosed Chrome; try again... now instead of about:blank its spinning.
2/4/2012 1:19:04 PMTim WesternTestTry Firefox; but it wants to download (may not have plugin installed
2/4/2012 1:19:08 PMTim WesternTestTry IE 9... and it loads
2/4/2012 1:19:50 PMTim WesternNoteSee 8 buttons 2x4 half blue half gold 2 read out lights left red; right green; and a meter. Label at bottom says puzzle 3
2/4/2012 1:20:20 PMTim WesternNoteSince only IE 9 will load for me (will explore why later) I'll use IE 9 for testing
2/4/2012 1:20:50 PMTim WesternCheckhover over flash to see what is clickable; or where mouse over changes.
2/4/2012 1:21:40 PMTim WesternNoteHover over image in lower right reveals ''Work room productions; Test Strategy Quality Assurance text
2/4/2012 1:21:53 PMTim WesternCheckwhat happens when you click on the icon in lower right; now being called the spool
2/4/2012 1:22:52 PMTim WesternCheckClick reveals some text; partiall transparent. Seems to be a message from the author.
2/4/2012 1:22:59 PMTim WesternCheckCan't read entire text and type at t same time.
2/4/2012 1:23:27 PMTim WesternChecktext covers most of the app; but it bleeds through making reading a bit tricky.
2/4/2012 1:23:51 PMTim WesternCheckClick the ? above the spool; reveals another message.
2/4/2012 1:25:03 PMTim WesternChecknothing in the blue area appears clickable.
2/4/2012 1:25:26 PMTim WesternCheckSo now trying the blue and Gold Buttons; will do a zig zag starting in upper left.
2/4/2012 1:26:05 PMTim WesternCheckSeems that four buttons pressed causes green light to come up
2/4/2012 1:26:21 PMTim WesternCheckTry all blue: (if any yellow are pressed beyond four it has both lit up
2/4/2012 1:26:48 PMTim WesternCheckGreen light lights up with B1; Y1; B2.
2/4/2012 1:26:56 PMTim WesternCheckStays on with Y2
2/4/2012 1:27:02 PMTim WesternCheckStays on with B3
2/4/2012 1:27:07 PMTim WesternCheckstays on with Y3
2/4/2012 1:27:13 PMTim WesternCheckstays on with b4
2/4/2012 1:27:20 PMTim WesternCheckGoes red with Y4
2/4/2012 1:27:23 PMTim WesternCheckun click all
2/4/2012 1:28:05 PMTim WesternCheckSeen combinations of 7 and 6 buttons that will stay green. Y4; B1; Y2; B2; Y2; B3..
2/4/2012 1:29:10 PMTim WesternCheckNow check if any single button causes the red
2/4/2012 1:30:04 PMTim WesternCheckS+T Exposes Timer
2/4/2012 1:30:49 PMTim WesternCheckIf I click in the target that shows up with the target/timer thing I can type into it
2/4/2012 1:30:59 PMTim WesternCheckOnly seems to except numerics
2/4/2012 1:32:00 PMTim WesternCheckTrying finger drag across keys to see if any other special commands.. noticed that a box can go around the buttons; how did I do that?
2/4/2012 1:32:24 PMTim WesternCheckU puts a box around the buttons
2/4/2012 1:33:37 PMTim WesternCheckthe lower right yellow button now seems to cause a red condition when certain pairs of the blue are pressed along with it.
2/4/2012 1:33:48 PMTim WesternCheckB1 B2 -> Green
2/4/2012 1:33:56 PMTim WesternCheckB1B3 -> Green
2/4/2012 1:34:03 PMTim WesternCheckB1-B4 -> Red
2/4/2012 1:34:17 PMTim WesternCheckB2;3 -> No indicator
2/4/2012 1:34:33 PMTim WesternCheckB3;4 with Y4 -> both red and green
2/4/2012 1:34:54 PMTim WesternCheckIf both Y3;Y4 are prssed; either B3;4 causes red
2/4/2012 1:35:23 PMTim WesternCheckNow checking pairs of blue with Y3
2/4/2012 1:35:35 PMTim WesternCheckB1;2->Green
2/4/2012 1:35:47 PMTim WesternCheckB1;3 -> none
2/4/2012 1:35:54 PMTim WesternCheckB1; $ -> none
2/4/2012 1:36:13 PMTim WesternCheckB2;3 -> Nothing
2/4/2012 1:36:22 PMTim WesternCheckB3;4 -> Green
2/4/2012 1:36:26 PMTim WesternCheckInteresting is that the only one?
2/4/2012 1:36:36 PMTim WesternCheckAll but B3 -> Green
2/4/2012 1:36:43 PMTim WesternCheckAll but B4 -> Green
2/4/2012 1:36:55 PMTim WesternCheckIf I had more time I'd setup a truth table for this chart and see what happens.
2/4/2012 1:37:25 PMTim WesternNext TimeSet up a truth table on a sheet of paper and note what the combinations are. I'm betting there are some that are DOn't Care some that are alway on. could be wrong.
2/4/2012 1:38:20 PMTim WesternNext TimeAha; i finally got the dial to move by rapid clicking
2/4/2012 1:38:36 PMTim WesternCheckB1; B2 Y2; B4; Y 4 un check B2 and it rolls
2/4/2012 1:38:47 PMTim WesternCheckunchecking B4 causes the dial to go down
2/4/2012 1:39:21 PMTim WesternChecknow what happens when it gets in the red of the dial?
2/4/2012 1:39:41 PMTim WesternCheckdial can be pegged
2/4/2012 1:39:44 PMTim WesternCheckdoesn't move
2/4/2012 1:39:49 PMTim WesternChecknow trying to unclick B4
2/4/2012 1:40:03 PMTim WesternCheckred light went off still pegged in red
2/4/2012 1:40:13 PMTim WesternCheckB2 and dial is going down again
2/4/2012 1:40:35 PMTim WesternCheckhit combinatoin b1 B2 Y3 B4 Y 4 to stop at 12 oclock position.
2/4/2012 1:45:24 PMTim WesternCheckeach button... do they do the same thing each time clicked click each button on and off five times to see
2/4/2012 1:46:02 PMTim WesternChecknot with individual buttons
2/4/2012 1:46:12 PMTim WesternCheckhowever; I've noticed that sometimes the needle goes up when its not green.
2/4/2012 1:46:39 PMTim WesternCheckB2;3;4 Y2;3 causes needle go down and green
2/4/2012 1:47:04 PMTim WesternCheckalso if you press B2;3; Y2;3 -> Y1 goes down.
2/4/2012 1:51:23 PMTim WesternSession End. Duration00:50:10

What follos is a very a brief conclusion after I looked over the report from RapidReporter before I began to compile my notes to share with the other session participants:

Conclusion:  There seems to be some sequence or groupings that determine if any, one, or all lights go on, and in sequence one button toggled may affect the dials movement up or down, or freezing.  I didn't feel I got accurate information as I was not taking as concise a notes in this regard as I would have liked too.  I could see doing a combinatoric check on this thing for combinations up to 7 or 8 buttons before having a full idea of how its logic would flow.

Lastly, as far as the Weekend Testing session went, I think it went well.   I'm glad I attended and participated.  I have felt stuck in a rut when I have been testing of late, or focusing on logic and business rules rather than what interfaces may actually be doing.  My testing environment at home was actually not a distraction at all, but I'm wishing now I'd taken even five minutes to get a note pad to do a truth table.  It would have saved me some effort I think, and perhaps given me more confidence in the session over all.    In truth, at present my tasks have been more coding than testing centered for the last few months, so exercises like this were important to keep some of my testing knowledge fresh.   If you've never had the chance to attend a Weekend Testing session, or if you are interested in participating in our next Weekend Testing Americas Session, it will likely fall on the first Saturday of March (March 3rd, 2012), and typically starts at 2 PM Eastern time for me.

Acknowledgements:  The exercise was produced by James Lyndsay who has a number of such exercises for practicing exploratory testing.  Thanks to James for allowing us to use this in our session today.  His site can be found at:  and this exercise is one of the public Black Box Testing Exercises at:  I give major kudos to James for having a practice applet like this on his site and would recommend anyone interested in learning more, to please contact him to find out how you can learn more about Exploratory Testing.

No comments:

Post a Comment