Contents
- TL;DR
- Summary
- Observations
- Participants
1. TL;DR
- We’re going to move forward with Variant B, as expected usage of the feature is low.
- Most participants mentioned that they would not use the feature often, that’s why we keep changes to the existing version minimal (see T306709 for action items). We keep the way shortcuts are ordered (see production version of the app).
- Similar to the debates we had in the team, there is no clear winner out of the 2 variants. The preference depends on what people are used to in the services/apps they use.
- We evaluate that the advantage of having space for another item in the toolbar in variant B outweighs the slightly better discoverability and the significant engineering effort of A.
- Answering the research questions from the protocol
- Do the updated tooltip designs with ‘GOT IT’ option help participants find the new ‘Customize toolbar’ feature?
- Yes, they do – we will keep the ‘GOT IT’ option.
- Do the changes in information architecture for both version A and B help users in finding ‘Customize toolbar’?
- Yes, moving the menu item out of ‘Theme’ improved discoverability. We’re balancing this with the expected usage of the feature (see 6) and think the tooltip will serve its purpose.
- Do the additional default tooltips on the details help users in understanding the interface and what to do now?
- Yes, almost all participants were able to drag and drop the items. However, we need to make sure they are properly localized.
- Which of the variants performs better in regards to users understanding how items in the toolbar connect to items in the overflow menu?
- There’s a tendency in the test (6/10 participants), that variant A does a better job at connecting toolbar and overflow menu. The finding will be balanced out with the expected usage of the feature (see 6) and people that are already used to the existing way to change their toolbar icons.
- Is the updated language (shortcuts instead of ‘actions and destinations’) more understandable to users?
- Yes, we will move forward with ‘shortcuts’ language.
- Gauge how participants expect the customize toolbar feature to work and how often they would want to use it.
- Most participants would not use the feature often.
2. Summary and recommendations
- 🅰️ 6 out 10 participants prefer variant A, for the following reasons:
- Overflow menu at the top reminds him of an internet page, would not go to look for it at the top if it wasn’t indicated, the bottom variant is less confusing, would be ok with B as well.
- “It’s awesome, it was easier to understand and find.”
- Likes this version better as it consolidate options in one place at the bottom.
- The options are closer together, it works very well, people would benefit from the feature, maybe grouping could be helpful?.
- It’s easy to understand how to find the customize option, is confused by variant B.
- Expects the customization of the toolbar where the toolbar is, realizes that you loose 1 item in the toolbar in this version, but “you really don’t need more than 4 in the toolbar”.
- 🅱️ 4 out 10 participants prefer variant B, for the following reasons:
- Literally suggested variant B during his test, prefers 1 more item in the toolbar, gives more flexibility.
- One more link the bottom toolbar, a lot of people have a habit to click at the top menu to look for things.
- It’s similar to Google Chrome.
- Most participants would not use the feature often. Here are reasons:
- Would probably customize his toolbar just once or twice, and wouldn’t regularly use it again even though it’s a really good feature.
- Would use the feature 2-3 times and thinks it would improve the experience.
- Would use it once.
- Thinks that not everyone would use the feature since a lot of people would use the default settings (20% of people would use it).
- Would not change her links very often, maybe change it 1 or 2 time thing. (every few months), will likely need time to find out which are her favorites.
- Would not use the feature that often, e.g. once a month or more, is confused by variant B.
- Wouldn’t use it a lot, just uses it to switch the theme, only uses the ‘Language’ and ‘Find in page’ feature.
- Wouldn’t mess with it too much but would use it until she figured out her preferences.
- Fix these translation issues:
- Default tooltips in the ‘Customize toolbar’ detail screen are not translated in most languages.
- Translate ‘Categories’ and ‘More’ into Arabic.
- Hindi:
- ‘Customize your toolbar’, ‘Edit history’ and ‘Categories’ are not translated.
- Optimize ‘GOT IT’ translation in Hindi, participants had difficulties understand.
- Japanese:
- One participant mentions that the Japanese translation for ‘GOT IT’ means ‘ROGER THAT’. There might be a better translation (JA Male A #2)
- Edit history is not translated.
- Arabic: ‘Customize toolbar’ entry point in the menu (overflow menu at the top or bottom sheet) are inverted in RTL languages. Needs to be fixed.
- People understand how to move things around in the ‘Customize toolbar’ detail screen, the default helper tooltips are working.