You can find the first part of this post at Ben Minson’s blog, Gryphon Mountain Journals.
In the first part of this post, I talked about how our biggest challenge in adopting Steve Krug’s method of do-it-yourself usability testing was that our team was testing the documentation on co-workers. There were some other challenges, too.
Incredibly Specific Scenarios
People with a wide range of job titles use our software. They work in the same facility, but their domain knowledge, education level, influence, and familiarity with the system differ wildly. Wide variations between facilities can determine what types of job titles and tasks that our users have,too. I created a draft table of user roles, at one point, and I was looking at nine potential personas.
This meant that our time was limited for exploring scenarios specific to each user type. We had three users for 20-30 minutes each once a month. If the scenarios were not well-vetted and specific, who knows who our proxy users were pretending to be when we tested them?
At times, testers said things that made me realize that they were approaching the testing tasks not as if they were a customer, but an internal employee using the documentation. Sloppy scenarios could mean incorrect conclusions. As I improved scenarios, I started adding instructions like, “Pretend you are a system administrator in hospital laboratory.” I did this for each scenario to prevent testers from slipping “out of character.”
If I were continuing this project, my next step would be to identify the most critical types/personas and focus on scenarios pertinent to those. I hadn’t gotten that systematic, yet, but to a certain extent, I did use some educated guesses to focus on important users.
As I discovered the complexities of writing scenarios that didn’t flop, often what we used on test day was what I had discovered the most about during the course of preparing for the session, rather than scenarios that focused on the most important things to test in the content.
Recommendations, Meet Lunch Hour
I mentioned that we had about an hour and a half for testing each month. After the testing session, we had an hour for discussion over lunch, which Krug refers to as the debriefing. Our schedule was a modification of Krug’s recommendation, which is an hour per participant. As it was, people found it difficult to commit to 2 1/2 hours, and many skipped the noontime debriefing, even though we had the option to take a late lunch afterward.
The approach that my manager preferred was to make the testing sessions optional, but to have the outcomes of the discussions be binding. I thought this fit well with the idea of letting the benefits speak for themselves. It also meant that people we needed for some discussions were not present, putting a slight crimp in creating action items. That can be an issue in many meetings, though.
A Fuzzy To-do List
Ah, the outcomes of the debriefing, the post-testing discussion. Krug’s process is to go around the room and ask each observer to name the most severe issues he or she observed. From that list, then the group chooses the most severe issues and discussed solutions right then. The focus is on solutions that can be implemented that month.
This worked for us the first time, sort of. In our world, a month is a short, short time. It was hard to list anything that could definitely be done within the month, except for things like, “Add it to the department meeting agenda,” and, “Contact Jim to get more information.”
So, a good proportion of our next steps involved getting more information or prioritizing the issue on the to-do list where it already lived. The testing served as future ammunition, which was important, but not very exciting.
Testing the Content vs. Testing the Application
In our first testing session, we tested the application that our co-workers use to enter release notes, as well as the help system for that application. The next session tested our new help format and navigation. But for the most part, our sessions were to focus on testing the content of the help systems—do the instructions work?
So, when we set about testing the content of our release notes, I didn’t think there was need to give much a heads up to the writer on our team who helps develop that application. We were strictly looking at release notes written by various members of the team.
Except, people still got bogged down in the application. My scenarios started them too early in the process, where they had to use the application to find the notes. I thought I would pick up some peripheral information that way, but that’s where people got stuck giving feedback.
I think it’s easy to get distracted by giving feedback on a tangible window in an application, especially for our co-workers who help develop software. To test the content, I would need to skip the part where they have to deal with the interface, and just have them use the instructions I’m testing to perform tasks in the system.
Unexpected Benefits: Communication, Audience Analysis, Bartering
I’ve mentioned that we didn’t always get kick-ass to-do lists out of our testing sessions. We did get support for our ongoing improvements that we were trying to get sponsored, and we also got something else: communication with other teams.
We invited product people to watch the release note testing, and that gave us an opportunity to ask them more questions about our audience. It gave us a chance to demonstrate why sometimes we make the requests we make for them to improve the notes they write.
While writing the scenarios, I had to ask many people from other departments to vet them. “Would a customer do this? Would they know that?” I picked up quite bit of peripheral information about our audience that way.
Finally, the testing can be a way to offer benefits to other teams. Do you want upgraded software for the knowledge base? Let us test it so you can take metrics to upper management. Do you need to train a couple of your employees on the release note application? Have them come and watch how people use it. It’s one more way for the technical communications department to add value.