Tag Archives: usability testing

Research Lab vs. Wild: which do you prefer?

Are you a UX designer or researcher who has the luxury to do some research in a lab? Not every organization has an on-site facility where things like empathy interviews and usability tests can take place. But for those that do, let’s explore the pros and cons of the lab and working in the wild.

I have had the ability to work in both a professional usability lab and perform ethnographic research in a person’s natural environment. I have my opinions about both, but I will reserve those until the end. First, let’s cover a bit about the difference between working with users in these environments.

Working in a Lab

A lab is a great set up. The tables and chairs are all set up for you. It’s quiet and protected from outside noises a person will usually encounter in a public place. The lab can be technically fitted with luxuries like video and audio recording capabilities. They usually have power sources, so users can use in-house and already loaded equipment like computers and mobile devices. When an organization has a lab, they often have a budget for fancy recording devices like a sled for recording mobile actions or eye tracking software. A lab is a great place to get controlled, scripted and qualified studies and act as a calm testing environment.

But a lab is not real. It’s not a person’s real home or office. It’s not the actual place a person would be performing an action like shopping online or texting with friends. A lab is a fake set up requiring a person to suspend belief and pretend they are in their natural environment. And this can be done to a point. But if people surf the web with their feet up on the table, with a tablet on their lap and a mobile device in their hand, using pre-described lab equipment will not realistically record a person’s natural behavior. Plus, just being put in a lab setting puts a person on edge and makes them feel uncomfortable. We all feel strange when we know we are being recorded. We probably even act differently and are not even aware of it. Yes, knowing these factors, we as researchers should try to combat these fears and make the environment feel as homey as possible. This just cannot be completely done since it truly is a false environment as far as the participant is concerned.

Despite a lab having many technical and environmental luxuries, you can see why many people opt to view people in their natural environment. Not to mention that most companies cannot afford to have an on site lab or pay to use external lab facilities. So for those who have no lab, it’s time to talk about jumping in to the wild and observing people in their natural habitat

Working in the wild

Observing participants in their natural environment provides many advantages. First, it is cost prohibitive. We won’t have to pay for lab facilities to see a person work in their home or office. Second, people feel more comfortable if they are on their own turf. It’s their home, so they know where to sit, how they work and where they normally do things. Also, the beauty of performing a contextual inquiry, or observing someone in their natural environment, is that you can take note of the items around them. How do they sit? What is their set up? What items do they have around them? What items do they use while interacting online? What is around them that is distracting? Who are they interacting with? And so much more. When you are in a person’s place, you can see the special notes and shortcuts they keep handy. Do they have their password written on a Post It note right on the computer monitor? All of these little environmental clues are golden nuggets that cannot be copied in a lab environment. Finally, one more benefit of people working in their own space is that you are getting a much more accurate picture of who they are and how they fit in to their space. You can never gain this kind of insight when they are sitting in a foreign lab using foreign equipment and getting instruction on how they can do things rather than exploring an interface on their own.

Despite the many advantages of entering a person’s real world and viewing how they really work in this context, there are also issues in working within these surroundings. Going to someone’s home many not be the safest situation because you don’t know exactly what you are getting in to. Make sure you travel with at least one “note taking” buddy to ensure your safety. Also you are very likely to encounter obstacles like noise, distractions, space limitations, and inaccessibility that can make researching in an environment tricky at best. Plus, when observing people in their surroundings, they can fall victim to The Hawthorne Affect. Basically, that is when people modify their normal behavior when they are being observed. Just keep in mind that this could happen with any, if not all participants so plan accordingly. Keep these factors in mind, and cater your research to accommodate such potential pitfalls. However, don’t let these deter you from going in to the wild. From my experience, I have gathered much richer insight by seeing a person reacting to their environment in real time, as opposed to the comfortable, staged lab setting.

What do I choose? Lab or Environment?

As is the classic answer to so many questions. It depends…. Obviously if you don’t have a lab, then that decision is easy: go out in to the wild. But if you do have a lab, I would encourage you to still go out of the building and see people in their natural space. It clearly provides a context that you will never get in a lab.

Usability review: talk to your users ASAP

As professional in the User Experience field, we’ve all heard it time and time again:

  • Test early and test often.
  • The only way to find out if it really works is to test it.
  • We do not see things as they are; we see things as we are.
  • The sooner you start to code, the longer the program will take.
  • Just because nobody complains doesn’t mean all parachutes are perfect
  • Don’t guess. Test.
Jennifer Blatz UX Design usability review and testing
If your product has a visual representation, and has not been built, go ahead and get user feedback from the visual mocks.

To get early feedback is easier said than done. I know that it takes time to create the visual mocks. Then things are moving so quickly (2 week sprints!) and it feels like we don’t have time to run our ideas by users. Finally, we feel like we know what it best for the product. We have done some user research. Our ideas have evolved as the project has progressed. We have made some decisions, that we think are for the better.

The earlier you test your site designs, the sooner you can find any problems and fix them.

But it’s this type of thought process that you need to take pause and stop in your tracks. Just when you think you are doing the right thing, that is the moment that your assumptions and biases are likely to creep in. So, this is a great point to take a break and get user feedback before you proceed down the rabbit hole any further.

To start with, we sent the wireframes to the visual designer to mock up our concept with more visual reality. We made sure that the flow we wanted to show the users was visually represented enough through our mocks.

Once the mocks were in a state we felt we could share with the user, we needed to figure out what type of information we wanted to gain from our usability review. Next, I came up with a brief list of topics and content we wanted to focus on.

Jennifer Blatz UX Design usability review
Have a script close at hand whenever you are conducting an interview so that you don’t forget key questions and concerns.

I wanted to first focus on the overall presentation and design, as well as have them look at the navigation and taxonomy that we were presenting. Then after getting their feedback on the overall design, I wanted to take them through a specific work flow, and see if it made sense to the user.

I think that a common error in usability testing is giving the person too much information as soon as they see your prototype. Don’t jump the gun!! Take this opportunity to get initial reaction feedback. Let the person participating in the usability review take a moment to take in the design, layout and wording. Let them get acquainted with the page and get that “first few seconds” feedback. Please don’t miss the opportunity.

Jennifer Blatz UX design usability test
Even if you only have wireframes, and not accurate visual mocks, share whatever design you have with your users to gather that valuable feedback on what you have started.

Then after taking a few minutes to talk to the user about the overall design, you can then take a deeper dive in to a work flow. I found that in this instance, the first half of the conversation focused on the website’s design, terminology and assessing if the user understood how to get started on some general tasks. In other words, I wanted to see if the navigation choices made sense. Also, I wanted to see if they had any cognitive dissonance with any of the terms we had used. Next, we asked the users how they would perform a specific task. In this case, it was how they would start to place an order. Along their usability journey, they brought up a lot of good questions and educated us on how they do this process now. It was all very insightful, and helpful to find out what we had done right, and what we needed to improve. That, my dear friends, is the whole goal of performing this usability review.

Want to watch the entire usability review live in the flesh? Please feel free to watch the video. If you have specific feedback on my approach or any aspect of my session, please share your thoughts in the comments. Just like designing a website or software, my research methods are always in a state of improvement and iteration. Please help me make my product, in this case ME, even better.