User study test plan
Protocol: testing customizable glasses
Introduction/scope of the evaluation
The purpose of this study is to test the usability of the product, based on the following 4 aspects: effectiveness, efficiency, perceived satisfaction and qualitative perspective. Thus, the study will consist of the two test studies which will be performed under monitored and unmonitored conditions to gain both qualitative and quantitative data which can be used to formulate the final design requirements.
In general, this paper starts with introducing usability testing methods and the necessary data that needs to be collected for this testing. This is done to identify challenges and common pitfalls associated with the use of the product, and then based on the findings and insights of this study, the possible design improvements will be formulated.
Literature study and co-design activities of the project suggest that the main issue the co-designer faces is with the lack of products on the market that tackle several complex symptomatic problems at once. Thus the idea was to design one pair of glasses that can be customised and adjusted to help with different symptoms and for different circumstances. Since many solutions exist on the market, our product focuses more on comfort and convenience and thus, this is what we will test for.
Looking back at the product goals that were defined at the beginning of the project they were used as a basis for the use evaluation test. In the case of effectiveness, it was defined that the user should be able to attach and detach the additional lenses, headband and the eyecap.
There are going to be 2 tests, the results of which will be used to evaluate the product and formulate the design recommendations.
In order to test the effectiveness and efficiency of the product, an in person usability testing can be conducted for which both quantitative and qualitative data will be collected. The user will be asked to do a set of tasks. During the process, the observations of how the participant approaches the product will be collected. It is necessary for us to gain data regarding:
- The ease with which the full range of target users completes the tasks.
- The functionality of the product: whether the designed mechanism for the attaching and detaching of lenses and eyecap is good enough to make it possible for the user to complete the tasks within a given range of time.
- Whether it makes it possible for the user to complete a set of tasks in certain contexts
- Affordances in the product is clear and can be easily identified; whether user can intuitively make the connection on how to use the product and its features:
- Appropriateness recognisability
- Learnability
- Operability
- Possible use error
- User interface aesthetics
The second test covers the qualitative perspective and the level of satisfaction of the users. For this, unmonitored remote usability methods will be utilized. This is a test for the more developed prototype to test comfort and emotional aspects of the product (less alienation). User completes certain actions and qualitative results are documented by the user in a questionnaire/ journal form to be reviewed at a later time. In this way we do not interfere with/ have influence on the users’ experience with the product.
Conducting user study test plan with visually impaired people
Because the participants of the user study are visually impaired, the methodology will alter. The tasks will be read out loud firstly.
Test 1: Assessing the effectiveness and the efficiency of the product in the controlled experiment setup.
The following variables will be measured and manipulated during the study:
Table 3: the description of the variables in the study.
Hypothesis 1 (effectiveness):
It is expected that the users will be able to complete each task under one minute.
Completion of the tasks within the specified time frame signifies that the components are functioning successfully and are not negatively affecting the effectiveness of the product.
Hypothesis 2 (efficiency):
The efficiency is dependent on how well the participant understood the functionality of the product through the design clues. It is expected that the participant will be able to successfully complete the tasks without provided instructions.
Method:
The data will be collected using measurements and observations. Because the participants involve people with visual impairment, it is important to read each task out loud to make the process more efficient.
Setting of the experiment:
- Prepare the table first:
a. The table should have the product and the 3 parts of the product: eyecap, headband and extra lens frame.
b. The table should have the description of the study on it. - Read the instructions for the study out loud.
- State the part that has to be attached and detached and ask the participants to proceed with the task.
- Start the stopwatch when the participant starts to do the task.
- Stop the stopwatch when the participant completes the task.
- Note the result in the following table:
- Prepare the table first:
Table 4, time taken for the participants to complete the tasks related to different components of the product.
7. Repeat steps 3-5 with the other parts of the product.
Tasks for participants to perform:
- Attach the eyecap to the glass frame.
- Attach the extra lenses to the glass frame.
- Attach the headband to the glasses.
Note: The tasks are not specific and do not provide complete instructions for the participants. This is done to check whether the affordances communicate the design correctly. In case when the participants will not know how to approach the product, provide explanations and note the number of times the participants were not able to understand the design affordances.
Test 2: Assessing the perceived satisfaction and qualitative perspective of the user in unmonitored settings of the experiment.
Test 2 will be unmonitored and will focus on the desirability of the product. To conduct the study, the user will be offered a product and asked to use it over a week (7 days) daily. After this period of time, the user will be asked a set of questions, the answers of which will reflect the desirability of the product.
Table 5
Method:
After the period of 7 days, the interview with the co-designer will be held.
The list of the following questions will be asked:
Table 6
Results from co-designer evaluation
An in-person usability testing of our mid-fidelity has been conducted with our codesigner with the goal of identifying any usability issues.
Test for effectiveness & efficiency
Effectiveness and efficiency of the product are tested based on how easy and intuitive the design is for users to understand and how much the product’s design matches its purpose. For this summative evaluation, the length of time to complete a task and errors made while completing a task are recorded whilst using a “Think-Aloud Protocol,” (Baxter et al., 2015b).
To start, all the separate components were spread out on the table to observe how the user interacts with the product as she thinks aloud. Then, a test for user friendliness is conducted by letting her attach all the parts using her intuition without any assistance or further information.
‘Errors’ she made:
- She placed the foam in front of the frame & after a few seconds she understood that the eyecap must go through the legs of the glasses as there are holes designed into it. This error is likely due to the foam material as the hole is very small and not too noticeable. In the actual design there will be bands on the side of the eyecap so the use intent is more clear.
- It took her a moment to figure out whether the lens would be placed in the ‘slot’/ gap behind the main frame. Again, this is due to the foam material as it does not cover the top of the glasses completely thus she mistook this as a slot. In the main design the silicone is flexible and the rubbery material will grip on to the frames better.
- Attaching the buttons to the lens the wrong way around. However, she immediately noticed that this was not the way it works and fixed it.
Table 7
Comments on security of the product/ headband:
- She says that it is a perfect fit and very secure when she is bending down.
- She doesn’t love how it looks but she knows that it is the ideal solution for comfort and security.
Comments on comfort:
- She is happy with the comfort that the foam provides and considers using this material as a solution for her eyecap after the operation. This is because her eyes will be extra sensitive and swollen after an operation thus maximum comfort is a priority for her.
Comments on function:
- She really likes the clipping of the lens frames to each other as it allows for less room between the lenses and so the double vision is minimised.
Test for satisfaction
It was explained that the prototype is used to test the function only and that the mechanism of the clipping and the material for the eyecap will vary in the real thing.
Comments on aesthetics of the prototype:
- The shape and style does not bother her (to wear in public).
- Ideally she would prefer if the eye caps were clear and made out of a more durable material as she mentioned that she likes to clean her glasses 20-30 times a day thus silicone is a better option which will be the case for our final product.
- She stated that the design is subtle and elegant with the use of the buttons and the option to clip the flexible component to the side when not in use. She said that “if people don’t know they won’t see it.”
Following this images of the final concept sketches were presented to her so she gets an idea of what the actual product will look like and how it will work.
Comments on the final concept sketches:
- She is very happy with the final sketch, “Wonderfull, really!”
- She loves the fit for the comfort of looking in the optimal spot on the lens
- By the lens being close to each other, less light will go between the lenses so less glare with the mechanism.
- She appreciates the basic/ subtle design and that it does not look to different from normal glasses
- She is very happy with the functionality , which is the most important thing for her.
- She knows that she can do a lot of things in her daily life using this product i.e the cord is useful for running, the silicone cup and the shape of it is comfortable
- She said that it is important for her to be able to wear it all day without it bothering her
- The husband asked if the silicone part can be 3d printed and the costs for this
- We explained that it is possible and a very customisable option with the 3D modelling
Comments on convenience:
- She was very pleased with the case idea because currently she uses all the separate glasses cases or a big square box to store everything in one place.
Interview questions:
Following this we conducted an interview by asking her more specific questions with the aim of finding improvements for the product so it fits better for her and a broader target audience:
- Does the prototype do what it’s supposed to?
-Yes, it is the perfect solution/ alternative to wearing double glasses over each other as she does currently. The gap between the lenses are minimised which improves. - Do users think the product’s design matches its purpose?
-Yes, it has al the functions it needs to have. - What’s the first thing users would want to do on this product? Can they do that?
-The first component that she interacted with was the foam eyecap - When they explore the product, do they become confused at any point?
-The holes for the eyecap in the foam prototype part was not immediately noticeable - Does anything distract them or get in their way?
-No, it is very basic, which is good. Because it does everything it needs to do. - Are there any features they completely ignore?
–No, she can’t mis any of the features - Does the information architecture and navigation make sense? (Can users find what they’re looking for?)
-Yes, they can - Does your target market feel like this product was designed for them?
–Yes and she showed great enthusiasm towards sharing this design to her community of acquaintances who have problems with eyesight - What, if anything, would make your users want to use this product frequently?
-The comfort and that it functions exactly as she needs - How likely or unlikely would they be to recommend the finished product to a friend?
–Very likely. She wishes to share this with her community of friends with similar issues and also her optician. - How would they describe this product using their own words?
Subtle, elegant, basic, very functional and comfortable
Summation of design recommendations
In general, the design intent was very well received by the user and the learning curve for the correct use of the product was very quick and intuitive. The main error made was due to the properties of the foam material where the holes were not visible.
From the use testing, the following design recommendations were obtained:
Functional
- Foam-like material for eyecap for high level comfort post-surgery
- More clear design of use intent for the sliding mechanism of the eyecap and signifying the correct side of the extra frames to attach to the button
Aesthetic
- More subtle headband design
- Clear material for eyecaps
Prioritisation of findings
Figure 45
In the above diagram (Figure 45), design recommendations are placed under four quadrants; luxuries, strategic, targeted and high value (Baxter et al., 2015b). The requirement of having a foam-like eyecap material is placed under high value as it is a high impact due to the improved comfort it provides (which is a high priority for the codesigner) with low effort to implement as the only change required is the processing of the silicone material. Recommendations in this quadrant should be implemented first.
The recommendation of a more subtle headband design is placed under “targeted” because it has low impact on the functional aspect of the product. However, it is low effort and cost to implement thus it is good to tackle recommendations from this quadrant along with the “high-value”.
Under “luxuries” are the low-impact issues that require more effort to implement. The recommendation of the see-through material is contradicting the foam-like structure and since comfort is of greater priority to the codesigner, the issue of aesthetics is placed under luxury. Additionally, the recommendation of a more obvious eyecap attachment mechanism is not of high impact as this issue came from the material choice of the prototype and this is not the case in the concept sketches that we presented. This quadrant provides the lowest return on investment and thus, may or may not be addressed only after the recommendations in the other three quadrants.
What's next
Involvement of stakeholders
After conducting further user research activity and testing with a much larger sample size of participants, results will be relayed to the stakeholder. The groups’ responses and reactions should be documented in order to make more concrete decisions on how design recommendations should impact the product.
Requirements for medical device industry
When dealing with the medical device industry, there is huge emphasis on compliance to existing international standards of regulations to avoid/ minimise use errors, technology abandonment and potential hazards in some cases.
According to Baxter et al. (2015b) at least 15 participants from each representative user group (codesigner and broader target group) are required for evaluation and validation testing with as close as possible to the fully functioning product to optimise medical device design. Another requirement are records showing that the usability engineering work has been conducted.