Category Archives: testing

Usability lab on a shoestring budget

Posted on by

Usability testing should be an important part of the development of any user interface. Ensuring that the interface is intuitive and easy to use is critical for its success. However, running usability sessions with real users often strikes fear into project teams. They assume that it will be a costly and time consuming process and will confuse as much as it clarifies the design process.  This article aims to demonstrate how easy it is to set up an effective usability lab on a shoestring budget.

Background

The USeD project aims to improve the interface of a data download website which provides   spatial data to the education sector in the UK.  User testing is an integral part of the USeD project and carrying out iterative assessment exercises will drive the development of the interface.  However, the project budget is quite modest and most of it is assigned for designing and coding the interface.

A discussion with our usability expert on the usefulness of various techniques suggested that most issues with an interface could be identified using quite simple techniques such as task-based exercises. Eye tracking allows testing to focus on very specific problems and it was better to identify general issues first before considering advanced techniques.

User Task Based Testing

Task based testing centers around setting users a series of small, distinct tasks that have been designed to test the functionality of an interface.  The initial tasks should be quite straight forward but later ones can be more involved allowing sessions to explore more advanced aspects of the interface.  Tasks should give the user a clear understanding of what they want to achieve but should allow them the flexibility to explore the interface. This flexibility can reveal how users discover functionality in the interface.  In these testing sessions we have 6 tasks and each session will last up to 45 minutes. Any longer than this and it is probably that the user will tire and loose focus.

So, how can you set up an effective user testing lab in your own office using pretty much “stuff” that you find lying around or “borrow”, temporarily?  The recipe below describes how we went about the task.

Ingredients:

  • 2 rooms, close together or preferably next to each other
  • 2 computers
  • 3 screens
  • 1 web cam
  • 1 mic
  • 1 set of baby monitor
  • A sprinkle of free software
  • 1 really helpful systems support person

First of all, having two rooms is a huge benefit as it means that the only the candidate and the facilitator (person running the test) need to be in the test room. This reduces the stress on the user during the test so that it feels less like a test. A nervous or flustered user will not interact with the interface in a naturally which may affect the results of the tasks.  Having the rooms next together makes things much easier as you can run cables between them.

Test lab

Test Room

  • Set up a computer that is typical of the ones you expect users to access the interface through in normal use. If users are likely to use a laptop or a 15 inch monitor, it would be unfair to run the test on a 21 inch monitor.
  • Set up a web cam that shows the user and the facilitator. This should be set up in an unobtrusive way and is to monitor general body language rather than detailed facial expressions or eye movements.
  • Position the transmitting part of the baby monitor so that it will pick up the conversation
  • Place a microphone dictaphone to capture the conversation between the candidate and the facilitator. This is really just a back up in case parts of the conversation get missed.
  • Make sure you provide some water for the candidates and a bit of chocolate never hurts.

Observation room

The observation lab can be set up in various ways but if you have access to two monitors then this makes things easier.

  • Set up the computer with a “Yâ€� splitter to two monitors. Monitor 1 will show the users screen and monitor 2 will display the webcam feed.  Set the monitors up about 1.5m away from the observers.  This will give them room to make notes and setting the back a bit means that they can easily scan both monitors at the same time without the “watching tennis” effect.
  • The receiving part of the baby monitor will provide the live audio from the other room.
  • Remember some water and chocolate or sugary sweets to keep the observers alert

 

Observation room


Porting the display

To display the users screen, we used some free software called “Zonescreen�. This has to be installed on both computers. Once installed, start ZoneScreen on the machine in the user lab, set this to as the HOST. Make a note of the i.p address. On the computer in the observation room, start ZoneScreen and set the session to REMOTE and enter the i.p address of the computer in the other room. You should now be able to see everything that happens on the user computer.

Webcam

The webcam feed is a little bit trickier. We experimented with broadcasting this across our network, but there was often a lag of up to 20-30seconds which made it very difficult to follow what was actually going on. As we had the luxury of having two rooms next to each other, we were able to connect the webcam to the computer in the observation lab. To do this you need a powered USB extension. The 10m extension we used occasionally failed, possibly as the power attenuated along its length. Replacing this with a 5m cable solved the problem.

Results

This set up worked really well.  The observers were able to see the candidates screen, hear everything that was said.  The webcam was useful to give everything context.  You could tell when the candidate had turned to speak to the facilitator and you could monitor their general body language.  There was only the slightest delay on the screen display feed, but this did not cause a problem. The baby monitors might seem very low tech but they are reliable and effective.

So, what did all this cost?  All the software was free and well we scavinged everything except the 5m powered usb cable and the baby monitors.  The total cost of this equipment was £40.  A huge thanks to Nik, EDINA’s small system support officer, who managed to find the software and put the lab together.

Version 3 User Testing

Posted on by

Repairs

This round of testing concentrates on Version 3 of the New Data Downloader User Interface. The two previous interfaces have undergone an “expert review” and testing with EDINA staff.  Many issues have been identified and solutions have been implemented.  This version of the interface will be tested with actual users.

Finding Users

Finding actual user who could test the interface meant returning to the Digimap user log.  We identified staff and students who had used the current downloader and who were affiliated with an institution in the Edinburgh/Glasgow area. Candidates were divided into three categories:

  1. those that had used the current downloader 5 times or more
  2. those that had used the current downloader less than 5 times
  3. those that had used other digimap services but had not used the current downloader.

We stuck to roughly the same format as the previous user testing session, a series of 5 set tasks that would explore much of the interface and site functionality. Each candidate would have a maximum of 45 minutes to work through the tasks leaving 15 minutes for discussion between the facilitator and the observer. We intended to have 6 candidates starting at 10am giving adequate time, or so we thought, to check the system was working on the day of the test.

We tweaked the tasks slightly, making changes to the way we presented information to the candidates.  This was in response to feedback from the first round of testing with internal EDINA staff.  It is amazing what candidates will extract from your handout that you have not even noticed and sometimes small pieces of information bias or mislead a test. This highlights how important a dry run is before organising sessions with external users.

Lessons

So what did we learn from this session?  Well this can be separated into things that would improve how we ran tests and things to improve the user interface.

About the test:

  1. Set up the lab and test that everything works on the day of the test. Do not assume that just because it worked yesterday it will work today
  2. run through the actual tasks during your test as if you were a candidate. (i tested the new interface on my computer and it was fine, but the first candidate struggled and “things” just didn’t seem right.  A bit of panicking later we discovered that the UI didn’t run quite as intended in Firefox 8. The 15 minutes between candidates gave me time to download Chrome and test that everything was working)
  3. Try not to run a session on the same day as a fire alarm test. (yup, 5 minutes into the first candidates session the fire alarm went off and stayed on for over a minute.  This was a scheduled test and i had completely forgotten about it.  Live and learn.)
  4. Keep calm and carry on – even when everything seems to be going wrong you can still get something out of a session.  If you discover a bug, just get the candidate to move on.  If the interface becomes unusable, or the candidate gets flustered and disengages, just move onto discussing the interface and the process. Ask some questions that dont require them to use the interface such as  “how would they like to interact to get data” or “what similar interfaces have they used, in this case it might be google maps or bing maps. This may allow you to ease them back into the tests.
  5. Dont worry if the candidate seems shy and isnt saying much. Remember to ask them to explain what they are doing and why and they will most probably relax into the situation. A slow, quiet user who takes time to thing can provide insightful feedback, you just have to coax them into thinking out loud.
  6.  “how would they like to interact to get

About the User Interface:

  1. Some users found it difficult to see what button to press on the Basket popup, they were not sure if the appearance of this window indicated that their order had been placed, or if they still had to “do” something to place the order.(closer examination of this issue reveals that some of the confusion may be related to two buttons that were added to the My Basket window between version 2 and version 3. They are the same size and colour as the Place Order button and may dilute the importance of the Order button.)
  2. The “Add to Basket” button was still not prominent enough, users often did not spot it. (we had already tweaked this and in this version, the button was initially grey, then flashed red when items were selected from the product list, and was then blue like the other function buttons.)
  3. All pop-up windows must close when an action button is pressed.  User often left thinking they still have something to do in the pop-up.
  4. Toggle between pan and draw rectangle still not absolutely clear.  Moving the search function out of the select area has helped but more thought needed on how to make this toggle clearer to the user.
  5. My Account section is confusing to users.  Not sure why there are two grids displayed.  Need to think how to make this section clearer to users when it appears but retain the functionality of re-ordering an order or part of an order.
  6. Selecting data through the Bounding Box not clear to all users. Some struggled to interpret the Upper Right X/Y and Lower Left X/Y diagram. (not clear if users struggled with this because they were not initially sure what a bounding box was, or what X/Y were. However, we hope that the interface will be learnable so that novice users will be able to learn how to select data using things like the bounding box through the information presented to them in the UI. The language and terms used in the UI are industry standard terms which are useful to know if you work with spatial data.)
  7. Add a text input box to sit alongside the search button.  A couple of users didn’t initially use the search function and commented that they hadn’t spotted it and were instinctively looking for a text input box where they could add search terms.

This is just a summary of the main points that we extracted from the session. You will find the complete list in the Version 3 User Testing Report (LINK).

Summing Up

Overall, the testing was a success and we have a number of development tasks that we can focus on.  Previous testing had identified issues with the process of selecting an area, selecting data and adding it to the basket. This seems to have been largely resolved and we have seen a migration of the main issues to the Basket and the My Account sections.  This is encouraging and suggests that the initial steps are now more intuitive.

However, some of the changes we implemented after Version 2 seem to have created as many issues as they have solved.  This is particularly clear in the case of the Basket.  Adding two extra buttons (clear basket and add more data) appears to have diluted the importance of the Place Order button.  This is unfortunate as the most important button on the Basket pop-up is the Place Order button.

 

Results of UI testing on Version 2

Posted on by

So, you think you have a good, usable project which clearly sets out what the user has to do to get what they want…….. and then you do some user testing.  The UI testing on Version two of the downloader was extremely useful, it pointed out many things that we had missed and now seem just so obvious.  This post will outline the main points that emerged from the testing and will describe how we ran the tests them self. But before we start, it is important to remember that the test revealed many positive things about the interface and users thought it was an improvement over the current system.   This post will now concentrate on the negatives but we shouldn’t be too depressed.

Setup

We decided to run this UI testing in a different configuration than we intend to run the tests with external students.  We wanted to allow our usability expert to be able to guide us through the test so that we would conduct the test using best practice.  Viv was to be the “facilitator” and Addy was the “Observer”.  David was observing everything and would provide feedback between test.

We had 5 candidates who would each run through 5 tasks during a 40-50minute period. We left 30 minutes between each test to allow us time to get feedback from David and to discuss the tests.  As it turned out, the day was quite draining and I wouldn’t recommend trying to do more than 6 candidates in a day.  Your brain will be mush by the end of it and you might not get the most out of the final sessions.

Results

The tests went well and we improved as the day went on thanks to feed back from the usability expert David Hamill.  It was certainly useful to have David facilitate a session so that we could observe him in action.

The participants all said that they thought the interface was easy to use and quite straight forward. However, it was clear that most users struggled with the process of

  1. selecting an area of interest
  2. selecting data products
  3. adding these products to the basket
  4. submitting the order

As the primary role of the interface is to allow users to order data this seems to be an area that will need significant investigation before the next iteration.  Other issues that arose during the sessions include:

  • The “Search and Select An Area” still seemed to confuse users.  Some struggled to see that they had to actually select an area in addition to just navigate to the area using the map
  • Basket Button looks busy and is not prominent enough.
  • Download limits not obvious to the user
  • Users often couldn’t recover from minor mistakes and “looked for a reset button” (technically you don’t need a reset button but the users didn’t know this so this needs addressed)
  • Preview Area in the Basket was not all that useful, the popup covered the map which showed the selection. In addition to previewing the geographical extent selected, this should also preview the data product selected.
  • Make the info buttons easier to browse through
  • Add more information to the “Use Tile Name” section, perhaps investigate how we can integrate this with the view grid function on the right of the map window.
  • Add a clear all button to the basket area.

A detailed report of the main issues that emerged during the user testing can be found in the Version 2 Testing Report(pdf).

The testing session was a success on two levels.  Viv and I learnt a great deal about conducting UI tests by having the usability expert present and we identified some key areas of the interface that were causing users problems.  Most of these are glaringly obvious once they have been pointed out to you, but then that is the point of UI testing i suppose!

Recommendations from UI Testing 1

Posted on by

Based on the first round of UI testing and the report prepared by our usability expert, a number of suggested improvements were put forward for the data downloader.  Actually, the report prepared by David was very useful.  The report did a number of things that made it easy to discuss issues with the development team and identify the potential solution.  In particular, a UI testing report should:

  • Rank the reported issues  High, Medium and Low, you can decide to rank within these categories if you wish
  • provide an annotated screenshot
  • add supporting text description
  • suggest a solution, perhaps with a mock-up or link to another site that demonstrates the solution
  • report “bugs” separately
  • dont go overboard – listing 100 faults is not helpful. (note – hopefully you shouldn’t have 100+ faults if you have understood the who, what, why of the design brief)

The USeD development team were able to sit around a table and come up with a course of action on each one of the issues raised in the UI Expert report.  How long did this take? 60 minutes. That’s all, honest.  A well presented document that described each issue clearly meant the developer could concentrate on the technical solution and judge whether a solution could be implemented in the development time available. Only a couple of issues got “parked”.  This was generally to do with the capabilities of the software and hardware and implications of server load.

A list of the changes that we will make to the interface is given at the end of this post.  So what have we learnt from this initial review?

  1. the usability expert liked the new site so that was good.
  2. He pointed out several things that are now blindingly obvious to the USeD team.
  3. We now appreciate the benefits of presenting UI results clearly, they shape the development of the UI.
  4. We know that we can make the UI better and easier to interact with.
  5. We know what we have to focus on for version 2

We now have about 2 weeks of development time before we roll out Version 2 for some more UI testing.

Points that will be investigated for Version 2

  1. Remove “do one of the following”
  2. Isolate pan as it is a separate function to draw or select
  3. On load, products list should be collapsed
  4. Add better headers to products list such as “Product” “Allowance”
  5. Make it more obvious when an allowance is exceeded, perhaps red text or strike thru text.
  6. Add a modal box if a user selects a product for which the allowance has been exceeded
  7. Remove lock/unlock icon
  8. Make info button change on hover
  9. Add info about which data product is being viewed in the map window.
  10. Remove Scale-bar
  11. Define area buttons need cleaned up – perhaps 1 define button that opens a pop-up
  12. Add more descriptive help to functions like define Square in the popups
  13. Popup and modal boxes should close when “GO” is pressed
  14. In basket, change preview and trash can to change colour on hover
  15. Give the “Add to Basket” button more prominence.
  16. Greyed out buttons too subtle (add to basket/define circle(not functioning at present))
  17. Are arrows the best symbol to use for clickable function submit buttons?
  18. Possible to get lost in map, have a reset view (planned but functionality issues prevented it going out in Version 1)

Aspects that we could not take forward with an explination of why.

  • Change the product names to be more meaningful and descriptive – the OS are insistent that their products should described by the correct products name. We also agree that this is sensible as many products are quite similar in some ways, but different in others. Conveying this in 2-50 characters would be difficult.  Further, using the official product name means that Digimap users are using the correct terms that are used in the public and private sectors.
  • When a user selects a product, update the map window to show the product. This would make a stronger visual link between what users see and what they eventually get. The preview windows are too small to be very useful. – This would be possible for some products but initiating this to say preview Mastermap or other large scale mapping products.  The load on the server to facilitate this could potentially be considerable and impact on the speed of the service. In addition, maps would not render in a readable format on screen if users tried to view large areas of large scale maps.

Link to the usability report prepared by Usability Expert

Testing Unlock 3: new API, new features, soon even documentation

Posted on by

This week we are public testing version 3 of Unlock – a fairly deep rewrite including a new simpler API and some more geometrical query functions (searching inside shapes, searching using a buffer). New data – providing a search across Natural Earth Data, returning shapes for countries, regions, etc worldwide. So at last we can use Natural Earth for search, and link it up to geonames point data for countries. We also have an upgraded version of the Edinburgh Geoparser so have date and event information as well as place-name text mining, in Unlock Text.

The new search work is now on our replicated server at Appleton Tower and in a week or two we’ll switch the main unlock.edina.ac.uk over to the new version (keeping the old API supported indefinitely too). Here are notes/links from Joe Vernon. If you do any testing or experimentation with this we’d be very interested to hear how you got on. Note you can add ‘format=json‘ to any of these links to get javascript-useful results, ‘format=txt‘ to get a csv, etc.

‘GENERIC’ SEARCHING

http://geoxwalk-at.edina.ac.uk/ws/search?name=sheffield

http://geoxwalk-at.edina.ac.uk/ws/search?name=wales&featureType=european

http://geoxwalk-at.edina.ac.uk/ws/search?featureType=hotel&name=Marriott&minx=-79&maxx=-78&miny=36&maxy=37&operator=within

NATURAL EARTH GAZETTEER

http://geoxwalk-at.edina.ac.uk/ws/search?name=lake&gazetteer=naturalearth&country=canada

DISTANCE BETWEEN TWO FEATURES

Distance between Edinburgh and Glasgow (by feature ID):

http://geoxwalk-at.edina.ac.uk/ws/distanceBetween?idA=14131223&idB=11153386

SEARCHING WITHIN A FEATURE – ‘SPATIAL MASK’

United Kingdom’s feature ID is: 14127855

Searching for ‘Washington’s within the United Kingdom…

http://geoxwalk-at.edina.ac.uk/ws/search?name=Washington&spatialMask=14127855

Also, note the difference between searching for within the bounding box of the UK, or adding the ‘realSpatial‘ parameter, which uses the polygon of the feature concerned.

http://geoxwalk-at.edina.ac.uk/ws/search?name=Washington&spatialMask=14127855format=txt&maxRows=100&realSpatial=no

http://geoxwalk-at.edina.ac.uk/ws/search?name=Washington&spatialMask=14127855&format=txt&maxRows=100&realSpatial=yes

In this case, it picks up entries in Ireland if using the bounding box rather than the UK’s footprint.

SPATIAL SEARCHING WITH A BUFFER

8 hotels around the Royal Mile
http://geoxwalk-at.edina.ac.uk/ws/search?featureType=hotel&minx=-3.2&maxx=-3.19&miny=55.94&maxy=55.95&operator=within

75 within 2km
http://geoxwalk-at.edina.ac.uk/ws/search?featureType=hotel&minx=-3.2&maxx=-3.19&miny=55.94&maxy=55.95&operator=within&buffer=2000

FOOTPRINTS & POSTCODES

…should still be there:
http://geoxwalk-at.edina.ac.uk/ws/footprintLookup?identifier=14131223
http://geoxwalk-at.edina.ac.uk/ws/postCodeSearch?postCode=eh91pr

IMPLICIT COUNTRY SEARCHING

http://geoxwalk-at.edina.ac.uk/ws/search?format=txt&gazetteer=geonames&featureType=populated place&name=louth
vs
http://geoxwalk-at.edina.ac.uk/ws/search?format=txt&gazetteer=geonames&featureType=populated place&name=louth, uk

TIME BOUNDED SEARCH (still in development)

http://geoxwalk-at.edina.ac.uk/ws/search?name=edinburgh&startYear=2000&endYear=2009

http://geoxwalk-at.edina.ac.uk/ws/search?name=edinburgh&startYear=2000&endYear=2010

Very happy with all this, bringing the Unlock service up to offering something usefully distinctive again, trying to restrain myself from saying (“if X was so easy why don’t we do Y?”)

User Participation

Posted on by

User Participation

Dave and Chris have previously blogged about the test-run with the Historical Society, which was largely unsuccessful due to a series of problems from the outset. Despite not being able to fully access the application (but having seen several demo’s on Chris’s computer), both Robin and Andrew seemed very enthusiastic about the potential for the application, which was really encouraging. Both seemed to grasp the concept and methods of navigation well, despite admitting to not being particularly computer literate or owning a mobile phone!

Shortly after (following some trouble-shooting) were the test runs with staff and Diploma students from college. The first was with Vicky, from the conservation unit. Having never used an iPhone before, she found the application fairly straightforward to navigate, although concern was expressed at the size of the buttons along the bottom of the main screen – there were a few occassions where the wrong menu was selected, and due to signal problems it often took a long time to return to the main screen.

01


A quick run-through outside Old College

Generally, this round of testing went without too many hitches. Remembering to turn 3G on makes a big difference. I’m not particularly iPhone savvy, so had to be instructed by Peter over the phone on how to switch it on – I’m not sure if it’s safe to assume that other iPhone users would know to (or how to) do this?

02

On the move….and walking through time!

Figuring it would be a popular route for future users (sightseers, students, historians etc.), we walked from Chamber Street to the bottom of the High Street. Although the maps loaded pretty quickly, Vicky found the naming and filing to be quite confusing and frustrating, and suggested that there should be an option to list all of the maps chronologically, rather than just by scale or name (i.e. ‘Edinburghshire’).

03

Vicky in the 18th century…

At the time of testing, the conservation students had a project site in Aitchison Close, so we paid a visit. It was really great to see Vicky get excited by selecting and tagging the different maps! She was really positive about the benefit the application could have on projects and site visits. Her only suggestion at this point was that perhaps it would be worthwhile considering photographs of former streetscapes and buildings (either through markers or as a seperate option in a chronological list similar to the maps). We discussed the concept of the markers a bit more – I thought perhaps this was where the answer could lie, but Vicky felt that it would be better to have the images already available rather than hoping that someone else had uploaded them (or having to track them down and upload them personally).

04


We walked back up the High Street towards Chamber Street, and hit a few minor glitches along the way – despite a full signal, on several occassions Vicky’s screen went white then sent her to the start-up screen to log in again. Thankfully this seemed to be shortlived! Before returning the iPhones to Peter and Petra we both created routes using the markers we had created. Again, this was something that Vicky was enthusiastic about – like the team, she felt that pre-existing routes laden with information (as well as the users ability to create their own) would be a valuable and interesting learning tool.

Later in the same week I went out and about with Feng, Klas (both students) and Ian. We walked from Chamber Street to the Castle Esplanade, then through some of the nearby closes where Ian knew there had been several radical changes.

05


Klas, Feng and Ian discussing their finds on the Edinburgh Castle Esplanade

Again, the application ran smoothly, with the only draw back being the length of time it took to load some of the maps. This seemed to be largely due to the poor (or non-existent) signal we got in the closes and courtyards. It’s annoying that this hinders it so much, but even more annoying is the fact that it’s outwith our control.

Everyone was really excited about the application, especially Ian – we got an amazing commentary from him as we walked around, which gave a great insight into how adding all this information (via markers, routes etc) could greatly enrich the users experience of and interaction with the city.

Having already discussed the previous feedback from Vicky, Robin and Andrew, the group had little else to add, other than comments about the occassionally disappointing 3G signal. From the esplanade we walked back towards Nicolson Square before heading to Chamber Street (with no signal problems!). Klas (an M.Phil. student) is interested in the relationship between people and the city of Edinburgh, and was encouraged by the potential for the application as a research/design tool.

All in all, the application got a huge thumbs up from all who tested it, which was a great result for us.

Thanks to everybody for volunteering – your feedback has been much appreciated and greatly valued!

07