GoSpotCheck Image Recognition Training System

GSC - IR Training System.png

Behind the scenes of GoSpotCheck’s image recognition (IR) technology, there are advanced algorithms and statistical models that detect patterns to improve data accuracy. With the help of human auditing in the IR training system, the data outputs evolve and become more intelligent over time.

The problem

GoSpotCheck’s image recognition feature is focused on identifying products in coolers and shelves at convenience stores. However, there is a high demand for image recognition within the beer, wine, and spirits space, specifically alcohol menus. Beer, wine, and spirits suppliers and distributors want to use image recognition to identify their product names, price, and positions on menus. This means we needed to train a model and build a new training system that allows for audits of menus.

 

Understanding who uses this tool

The target users of the menu training system are hourly contractors (taggers), who work remotely in the United States and Nepal. They choose their schedule, but are required to complete a specified number of audits per week. Users from Nepal know some English, but it is not their first language. 

Additionally, the training system has several power users that approve or reject the changes made by taggers. These people are a little more skilled and knowledgeable about the products that need to be identified. They are currently all located in the United States.

 

The team was comprised of six key people

  • Product Designer (me)

  • Product Manager

  • Data Manager

  • Software Engineer 

  • Computer Vision and Machine Learning Engineer

  • QA Engineer

I coordinated and led all facets of design, including information architecture, user flows, UI design, interaction design, and InVision prototyping. I also conducted research through user and customer interviews, and by reviewing analytics on the existing system.

 

To get started, the team needed to do some discovery

This phase helped us better understand the goals and customer needs. To do this, the Product Manager and I interviewed existing customers who have an interest in gathering beer, wine, and spirits menu data via image recognition. 

Next, we spoke to users (taggers) of the training system and reviewed Pendo analytics to identify some current areas where people may be getting stuck or experiences that work specifically well.

After that, I hosted a brainstorming session with the entire team to work out how the machine learning technology needed to function and identify any technical constraints the team was aware of.

 

We identified a number a challenges

  • Every menu is different. We have no idea how a menu will look or how they will be structured.

  • What customers consider a menu varies. Just to name a few, a menu could be a piece of paper, a handwritten chalkboard, a table tent, a tablet… and so on.

  • Machine learning is complicated. There is a process to train the model and many technical constraints that impact the user experience.

  • Customers don’t understand how image recognition and machine learning works. It seems like magic, but it’s not. It takes time to perfect output accuracy.

  • A lot of trust is put into taggers. Taggers are expected to audit images in the training system quickly without a lot of instruction. Many taggers are also located outside of the U.S. and are not familiar with the products they need to identify.

 

Defining the flows was a huge team effort

This was a huge process the whole team was involved in. The flow needed to be broken up into many steps to ensure each step was quick and intuitive, the right data was collected from the menu image, and the model continuously improved.

After defining the flow, I drew some wireframes and reviewed them with the team

 

Coming up with a UI that made sense for taggers

Once the flow and structure was created, I designed high fidelity mockups, reviewing each step with the team along the way.  During this process, I was very careful to remember the tagger’s needs. Specifically, keeping text to minimum and using mostly visual queues and colors to make associations.

Because no menu is the same and the background color of menu images can’t be predicted, choosing colors that are accessible is not easy. During the design process, I found a variety of menus and tested contrast ratios to find colors that were still in the GoSpotCheck brand family, but were also visible on most menus.

 

Users of this tool need quick interactions and visual queues

A lot of thought was put into the interaction design, as users must be able to make modifications to tags, segments, size and more. To accomplish this, I created a variety of user-friendly hot keys so taggers do not need to rely on their mouse for every action. I also designed custom cursors to match the selected tool. Throughout the entire interaction design process, I worked very closely with engineering to ensure the user experience was intuitive.

 

Conducting usability testing

While the training system is an internal tool, ease of use is critical to GoSpotCheck’s machine learning model. The product learns patterns and improves outputs based on the taggers’ inputs that could have long lasting effects on its accuracy. For that reason, it was important to get it right. 

The training system has many micro-interactions and tools, so it was easiest to conduct user testing while the product was in development. 

As soon as a feature in the tool was ready for testing, the Product Manager and I met with one of our internal taggers to gather feedback. Based on each session, we then identified how we can make adjustments to the system in the future. 

 

Planning for improvements

Before the application was released, users were trained on how to use it. The trainings were recorded so the team could hear the feedback and questions to help make decisions on how to improve the usability of the product and onboarding in the future.

Additionally, Pendo was installed on the product to gather metrics and track user interactions.