Mission:* In today's mission we would like you to frame your exploration on yourown.* We are about to give you a 'black box machine'.* You will have 60 minutes.* At the end of the timebox present your notes (no more than page long) to the group chat.* What to report and in what format - we leave it up to your decision.* We hope everybody will bring a unique bit of experience we all can learn from.
Ultimately it boiled down to an exploratory testing session on a practice flash applet. Because of the way the mission was framed, each individual tester, or testing pair defined the mission for themselves. Since the general feel of the mission was so open, and my instincts were to work the practice as an exploratory session, I felt there was a lot of room to try things, perhaps things I wouldn't normally have done. While some of the testers paired, I elected to go it alone this time though.
Before I get into my synopsis of the exercise and my conclusions of how that session went, I'd like to give credit to James Lyndsay who I am told is the one who produced this particular practice over at his site http://www.workroom-productions.com/. There are a number of practice exercises that could be used for some general and basic practice of exploratory testing skills. From what I've read he also has some more in depth training as well, which could be a useful if exploratory testing is an area you feel could help you in your testing efforts. I'll leave it to my readers to decide that for themselves. Now, if you'd like to try the exercise before reading my report, head on over to http://www.workroom-productions.com/papers/puzzle_03.swf and explore it to your hearts content for up to an hour. Otherwise, what follows would be considered a 'Spoiler'. So when you are ready to continue reading, please click 'Read More' below.
Before I begin my observations, I'm going to take a moment to be a bit critical of my attempt at this session. Admittedly, it's been a while since I've done focused exploratory testing. I have done some exploratory testing in the past, but my current tasks does not have exploration as a key component as I see it. So there is no established 'routine' for exploration that I might normally have fell back into using. This could have been a plus or a minus. Additionally, in some ways I was less than prepared. While I did use Shmuel Gershon's RapidReporter for note taking, at one point in, I realized I didn't have a notepad and pen handy, (something that normally I always have handy, but for some reason didn't think to grab before this session.) Since I didn't want to lose time to go hunt for a notepad in my other room, I proceeded with out it. In hindsight, if I were doing this again, I think I could have perhaps better organized my effort if I had taken time to use a notepad to organize my thoughts after the first few minutes of the session.
What follows is my report as submitted to the session, plus later, notes as recorded into rapid reporter.
Page: http://www.workroom-productions.com/papers/puzzle_03.swf
Mission: Explore functionality.
Page Load testing:
Chrome: First Try Failed to Load - about:blank; Second Try: spun out but never loaded.
Firefox: Inconclusive, wasn't setup for flash (odd thought it was) (and tried to download the swf)
IE 9: Loads no problems. Will use IE 9 for testing
Box is Win 7 64 Bit.
First thought was what areas did the mouse change, or appear clickable. The screen has four buttons in blue and yellow in a 2x4 pattern on the left, a dial gauge that is circular, with a needle, green until like the last twenty percent. A label in the lower left hand corner that says Puzzle 3. A red and green LED below the dial, and a pinkish red image that I'll call a spool in the bottom right. It appears the 4 blue and 4 yellow buttons are clickable, as well as the red spool and ? mark in the bottom right.
Hovering over the Spool also revealed the Test "work room productions; Test Strategy quality assurance' text. The spool surprised me a bit, because it pops up a message from the swf's creator, but only as long as you hold the mouse button down. Same with the ? mark which gives some hints. Oops noone told me it would give me hints, Spoilers!
Observations, the text has a transparent background for the spool, and is a bit difficult to read due to this (for me anyways) It's not entirely illegible but someone with poorer eyesight might not be able to read it.
Next try clicking buttons, initialy thought was to do a zig zag pattern from Blue 1 down. (wondering if order matters Permutations vs Combinations that is.) Tried a sequence going from left to right then down the left to right then down the left to see what happens when all buttons pressed. Noticed that at 7 or 6 buttons the red light may come on, but not sure again if order matters here or not. I tried a number of different combinations here, and then thought if trying some key combinations. A drag of finger across keyboard revealed that the buttons would get 'inset' if a key was pressed, seems that's the U key that causes it. IS this cosmetic or does it affect behavior would require a more detailed plan I think to figure out.
Stepping back, noting to self that I forgot to have a notebook handy to write things on quickly between clicks. Would really like to to do a truth table for this thing if I had more time I would do so.
Ooo Pressing S and T seems to cause a target to show up, and I notice that it seems to be clickable with data entered into it. (only numerics though) Interesting.
Eventually I got the needle to move, stop, decrease, increase, halt with no lights on, green light on, both green and red, etc. So I wonder what the lights acutally indicate since they don't indicate which direction it's moving, which would have been my thought.
Conclusion: If I had a note pad handy, a truth table would have been better to get what combinations of buttons did what. Thus more testing would be needed if it were just me to obtain that.)
That's the short synopsis of the testing we did for about sixty minutes. You can see the reports of other testers at http://weekendtesting.com/archives/2393 It is always interesting to look at the approaches others took, and what they discovered. In some cases, testers focused on that combination, and what combinations caused the needle to move, or the lights to come on. Some tried to visualize it as a control system for some system. While my report didn't reflect it, I was imagining it as an air gauge, or a switching system with a Volt or Amp Gauge where the possibility could be that going to far into the red would cause something 'bad' to happen. Nothing seemed to happen when we got there that I noticed though.
Some of the testers tried to define what the states might mean for the buttons. Is a button up, considered on or off? I didn't really spend much time wondering what those buttons met. If I had access to the customer, which another tester did note, I'd probably have asked for some background for the control system, and that likely would have had a profound impact on how I viewed and proceeded with my testing. One tester, Pradeep Soundararajan, thought he'd seen the exercise before, and rather than just testing the device, spent time devising the possible repercussions of testing it. He spent time identifying useful heuristics that could be needed to properly test something like this. I actually found this refreshing, as he identified some potential biases, and problems you might have to answer when testing something like this.
I'll let my readers go to the Experience Report over at http://weekendtesting.com/archives/2393 to draw their own conclusions, but I definitely benefited from seeing how people in different backgrounds saw testing of this kind. Before I close this blog, I'd like to post my notes from rapid reporter, so you can see my thought process as I went about the testing.
Session Report | Powered by Rapid Reporter
Show autogenerated rows
| Time | Reporter | Type | Content | Screenshot | RTF Note |
|---|---|---|---|---|---|
| 2/4/2012 1:01:14 PM | Tim Western | (Rapid Reporter version) | 1.12.01.06 | ||
| 2/4/2012 1:01:14 PM | Tim Western | Session Reporter | Tim Western | ||
| 2/4/2012 1:01:14 PM | Tim Western | Session Charter | Weekend Testing Americas #24 Mission: TBD | ||
| 2/4/2012 1:01:21 PM | Tim Western | Setup | Join Skype Converation | ||
| 2/4/2012 1:01:26 PM | Tim Western | Setup | Say Hi and Check in. | ||
| 2/4/2012 1:03:26 PM | Tim Western | Setup | Introductions | ||
| 2/4/2012 1:11:36 PM | Tim Western | Setup | Mission statement: | ||
| 2/4/2012 1:11:53 PM | Tim Western | Setup | In today's mission we would like you to frame your exploration on your own. We | ||
| 2/4/2012 1:12:04 PM | Tim Western | Setup | are about to give you a 'black box machine'. | ||
| 2/4/2012 1:12:11 PM | Tim Western | Setup | You will have 60 minutes. At the end of the timebox present your notes (no more | ||
| 2/4/2012 1:12:20 PM | Tim Western | Setup | than page long) to the group chat. | ||
| 2/4/2012 1:12:28 PM | Tim Western | Setup | What to report and in what format - we leave it up to your decision. We hope | ||
| 2/4/2012 1:12:34 PM | Tim Western | Setup | everybody will bring a unique bit of experience we all can learn from. | ||
| 2/4/2012 1:12:59 PM | Tim Western | Setup | [1:11:44 PM] Weekend Testers Americas: http://www.workroom-productions.com/papers/puzzle_03.swf | ||
| 2/4/2012 1:14:55 PM | Tim Western | Setup | Chrome test opening: http://www.workroom-productions.com/papers/puzzle_03.swf | ||
| 2/4/2012 1:15:03 PM | Tim Western | Test | Open in Chrome results in blank screen? odd | ||
| 2/4/2012 1:15:48 PM | Tim Western | Test | closed Chrome; try again... now instead of about:blank its spinning. | ||
| 2/4/2012 1:19:04 PM | Tim Western | Test | Try Firefox; but it wants to download (may not have plugin installed | ||
| 2/4/2012 1:19:08 PM | Tim Western | Test | Try IE 9... and it loads | ||
| 2/4/2012 1:19:50 PM | Tim Western | Note | See 8 buttons 2x4 half blue half gold 2 read out lights left red; right green; and a meter. Label at bottom says puzzle 3 | ||
| 2/4/2012 1:20:20 PM | Tim Western | Note | Since only IE 9 will load for me (will explore why later) I'll use IE 9 for testing | ||
| 2/4/2012 1:20:50 PM | Tim Western | Check | hover over flash to see what is clickable; or where mouse over changes. | ||
| 2/4/2012 1:21:40 PM | Tim Western | Note | Hover over image in lower right reveals ''Work room productions; Test Strategy Quality Assurance text | ||
| 2/4/2012 1:21:53 PM | Tim Western | Check | what happens when you click on the icon in lower right; now being called the spool | ||
| 2/4/2012 1:22:52 PM | Tim Western | Check | Click reveals some text; partiall transparent. Seems to be a message from the author. | ||
| 2/4/2012 1:22:59 PM | Tim Western | Check | Can't read entire text and type at t same time. | ||
| 2/4/2012 1:23:27 PM | Tim Western | Check | text covers most of the app; but it bleeds through making reading a bit tricky. | ||
| 2/4/2012 1:23:51 PM | Tim Western | Check | Click the ? above the spool; reveals another message. | ||
| 2/4/2012 1:25:03 PM | Tim Western | Check | nothing in the blue area appears clickable. | ||
| 2/4/2012 1:25:26 PM | Tim Western | Check | So now trying the blue and Gold Buttons; will do a zig zag starting in upper left. | ||
| 2/4/2012 1:26:05 PM | Tim Western | Check | Seems that four buttons pressed causes green light to come up | ||
| 2/4/2012 1:26:21 PM | Tim Western | Check | Try all blue: (if any yellow are pressed beyond four it has both lit up | ||
| 2/4/2012 1:26:48 PM | Tim Western | Check | Green light lights up with B1; Y1; B2. | ||
| 2/4/2012 1:26:56 PM | Tim Western | Check | Stays on with Y2 | ||
| 2/4/2012 1:27:02 PM | Tim Western | Check | Stays on with B3 | ||
| 2/4/2012 1:27:07 PM | Tim Western | Check | stays on with Y3 | ||
| 2/4/2012 1:27:13 PM | Tim Western | Check | stays on with b4 | ||
| 2/4/2012 1:27:20 PM | Tim Western | Check | Goes red with Y4 | ||
| 2/4/2012 1:27:23 PM | Tim Western | Check | un click all | ||
| 2/4/2012 1:28:05 PM | Tim Western | Check | Seen combinations of 7 and 6 buttons that will stay green. Y4; B1; Y2; B2; Y2; B3.. | ||
| 2/4/2012 1:29:10 PM | Tim Western | Check | Now check if any single button causes the red | ||
| 2/4/2012 1:30:04 PM | Tim Western | Check | S+T Exposes Timer | ||
| 2/4/2012 1:30:49 PM | Tim Western | Check | If I click in the target that shows up with the target/timer thing I can type into it | ||
| 2/4/2012 1:30:59 PM | Tim Western | Check | Only seems to except numerics | ||
| 2/4/2012 1:32:00 PM | Tim Western | Check | Trying finger drag across keys to see if any other special commands.. noticed that a box can go around the buttons; how did I do that? | ||
| 2/4/2012 1:32:24 PM | Tim Western | Check | U puts a box around the buttons | ||
| 2/4/2012 1:33:37 PM | Tim Western | Check | the lower right yellow button now seems to cause a red condition when certain pairs of the blue are pressed along with it. | ||
| 2/4/2012 1:33:48 PM | Tim Western | Check | B1 B2 -> Green | ||
| 2/4/2012 1:33:56 PM | Tim Western | Check | B1B3 -> Green | ||
| 2/4/2012 1:34:03 PM | Tim Western | Check | B1-B4 -> Red | ||
| 2/4/2012 1:34:17 PM | Tim Western | Check | B2;3 -> No indicator | ||
| 2/4/2012 1:34:33 PM | Tim Western | Check | B3;4 with Y4 -> both red and green | ||
| 2/4/2012 1:34:54 PM | Tim Western | Check | If both Y3;Y4 are prssed; either B3;4 causes red | ||
| 2/4/2012 1:35:23 PM | Tim Western | Check | Now checking pairs of blue with Y3 | ||
| 2/4/2012 1:35:35 PM | Tim Western | Check | B1;2->Green | ||
| 2/4/2012 1:35:47 PM | Tim Western | Check | B1;3 -> none | ||
| 2/4/2012 1:35:54 PM | Tim Western | Check | B1; $ -> none | ||
| 2/4/2012 1:36:13 PM | Tim Western | Check | B2;3 -> Nothing | ||
| 2/4/2012 1:36:22 PM | Tim Western | Check | B3;4 -> Green | ||
| 2/4/2012 1:36:26 PM | Tim Western | Check | Interesting is that the only one? | ||
| 2/4/2012 1:36:36 PM | Tim Western | Check | All but B3 -> Green | ||
| 2/4/2012 1:36:43 PM | Tim Western | Check | All but B4 -> Green | ||
| 2/4/2012 1:36:55 PM | Tim Western | Check | If I had more time I'd setup a truth table for this chart and see what happens. | ||
| 2/4/2012 1:37:25 PM | Tim Western | Next Time | Set up a truth table on a sheet of paper and note what the combinations are. I'm betting there are some that are DOn't Care some that are alway on. could be wrong. | ||
| 2/4/2012 1:38:20 PM | Tim Western | Next Time | Aha; i finally got the dial to move by rapid clicking | ||
| 2/4/2012 1:38:36 PM | Tim Western | Check | B1; B2 Y2; B4; Y 4 un check B2 and it rolls | ||
| 2/4/2012 1:38:47 PM | Tim Western | Check | unchecking B4 causes the dial to go down | ||
| 2/4/2012 1:39:21 PM | Tim Western | Check | now what happens when it gets in the red of the dial? | ||
| 2/4/2012 1:39:41 PM | Tim Western | Check | dial can be pegged | ||
| 2/4/2012 1:39:44 PM | Tim Western | Check | doesn't move | ||
| 2/4/2012 1:39:49 PM | Tim Western | Check | now trying to unclick B4 | ||
| 2/4/2012 1:40:03 PM | Tim Western | Check | red light went off still pegged in red | ||
| 2/4/2012 1:40:13 PM | Tim Western | Check | B2 and dial is going down again | ||
| 2/4/2012 1:40:35 PM | Tim Western | Check | hit combinatoin b1 B2 Y3 B4 Y 4 to stop at 12 oclock position. | ||
| 2/4/2012 1:45:24 PM | Tim Western | Check | each button... do they do the same thing each time clicked click each button on and off five times to see | ||
| 2/4/2012 1:46:02 PM | Tim Western | Check | not with individual buttons | ||
| 2/4/2012 1:46:12 PM | Tim Western | Check | however; I've noticed that sometimes the needle goes up when its not green. | ||
| 2/4/2012 1:46:39 PM | Tim Western | Check | B2;3;4 Y2;3 causes needle go down and green | ||
| 2/4/2012 1:47:04 PM | Tim Western | Check | also if you press B2;3; Y2;3 -> Y1 goes down. | ||
| 2/4/2012 1:51:23 PM | Tim Western | Session End. Duration | 00:50:10 |
What follos is a very a brief conclusion after I looked over the report from RapidReporter before I began to compile my notes to share with the other session participants:
Conclusion: There seems to be some sequence or groupings that determine if any, one, or all lights go on, and in sequence one button toggled may affect the dials movement up or down, or freezing. I didn't feel I got accurate information as I was not taking as concise a notes in this regard as I would have liked too. I could see doing a combinatoric check on this thing for combinations up to 7 or 8 buttons before having a full idea of how its logic would flow.
Lastly, as far as the Weekend Testing session went, I think it went well. I'm glad I attended and participated. I have felt stuck in a rut when I have been testing of late, or focusing on logic and business rules rather than what interfaces may actually be doing. My testing environment at home was actually not a distraction at all, but I'm wishing now I'd taken even five minutes to get a note pad to do a truth table. It would have saved me some effort I think, and perhaps given me more confidence in the session over all. In truth, at present my tasks have been more coding than testing centered for the last few months, so exercises like this were important to keep some of my testing knowledge fresh. If you've never had the chance to attend a Weekend Testing session, or if you are interested in participating in our next Weekend Testing Americas Session, it will likely fall on the first Saturday of March (March 3rd, 2012), and typically starts at 2 PM Eastern time for me.
Acknowledgements: The exercise was produced by James Lyndsay who has a number of such exercises for practicing exploratory testing. Thanks to James for allowing us to use this in our session today. His site can be found at: http://www.workroom-productions.com/ and this exercise is one of the public Black Box Testing Exercises at: http://www.workroom-productions.com/black_box_machines.html I give major kudos to James for having a practice applet like this on his site and would recommend anyone interested in learning more, to please contact him to find out how you can learn more about Exploratory Testing.
No comments:
Post a Comment