One of the world’s leading food processors was losing millions of dollars annually because of an overly manual and error-prone process that resulted in a higher-than-tolerable instance of misidentified and lost inventory on the factory floor. Leaders knew the existing process was woefully inefficient and riddled with repeated patterns of human error. Nearly all agreed that significant change was needed. Namely, a much more efficient system that reduced the load on its factory staff while increasing the accuracy and reliability of the labeling process.
We’re trying to apply the most cutting-edge technology in order to derive new insights so that we can enable new ways of working.
— VP, Emerging Technology & Analytics
Accurately tracking products as they move through production facilities across the country is critical; the consequences of misidentified or mislabeled products—typically a result of human error—are immediate, resulting in spoiled products and a lot of wasted money. Still, the identification and labeling processes used at most of the plant locations across the country were too dependent on human accuracy.
Typically, tracking inventory through this particular production facility is a largely manual process that relied heavily on human judgement before critical data was then entered into a system. What’s more, that process wasn’t always accurate. In fact, more often than not, it was wrong, which led to a number of unwanted outcomes, including both under and over-processing products. Each of these outcomes cost the company millions of dollars.
For all involved, the challenge was difficult: How do we design an entirely new, restaurant training tool when: (1) the majority of the restaurant team members aren’t allowed to use their own mobile devices on the job, (2) there are only a few, shared training devices for each restaurant, and (3) Leaders across the country reported a mostly positive experience with printable materials?
Any new solution would first need to pass an initial test
Together, we quickly zeroed in on a number of potential solutions that might help to mitigate the product identification and processing errors. Many of which utilized machine learning (ML) and computer vision (CV) to increase the accuracy and efficiency of the identification process. Still, each of these solutions were untested and unproven. So, we set out to quickly prototype and test in a real-world environment in order to prove or disprove the efficacy of such a solution before investing more fully. The goal was simple: to test and deploy the ML and CV models in a small, controlled fashion before a larger planned rollout across other processing facilities.
There’s rarely existing training data for most bespoke enterprise machine learning initiatives; this was the case in this instance, too. Most importantly, for this effort, there wasn’t an existing way to get an overhead view of the packed product inside the plant. Instead, we had to devise a way to capture that data ourselves. So, in short, we found a way to associate the timestamps from the weight scale with the timestamps from overhead streaming video to generate a base training data set. Lastly, we took the necessary step to validate every single image (since they were known to be imperfect) and balanced the training set so that we represented the various images equitably to ensure quality results.
We simply didn’t know how staff (those working inside the production facilities) would actually “take” to a new, tech-driven solution. What’s more, there’s not yet any well-defined rules or best practices to follow here. So, in order to give ourselves the best chance at success, we worked closely with staff to create a more efficient experience without the complexity one might imagine from a machine learning solution. Doing so both created a better solution from the start, but instantly formed the next round of training in order to “dial in” the model even further.
It’s common practice to deploy transactional user software in the cloud, but it wasn’t that simple in this case. There wasn’t a straightforward method to deploy an intricate computer vision system—complete with live video streams, machine learning, and user interaction dialog—within a production facility. All involved ultimately evaluated a number of options, which, when coupled with some of the strategic aims of the effort, helped inform the construction of a truly modern combination of services that were used to bring the solution to life.
The driving goal of this effort—and our number one priority—was to find a way to correct the existing inventory loss issues. We knew there was some promise in using tech like computer vision and machine learning, coupled with the most modern infrastructure, to devise a solution that worked well within a really complex environment. Ultimately, the fusion of design, machine learning engineering, DevOps and production operations gave us a fighting chance.
— Colin Shaw, Director of Machine Learning, RevUnit
First and foremost, we had to understand the existing processes in workflows in each production facility. So, we set out for several “ride along” visits, carefully watching how different facilities each handled the same tasks. Additionally, we spent time with those who were actually doing the work, observing and asking questions along the way. These field visits and interviews, which were but a small part of our initial research, gave us enough of a starting point to quickly and confidently move forward.
Next, we made quick strides to develop a machine learning model so that we would be able to develop a working prototype that consumed real-world training data. First, we began collecting critical training data from some of the cameras that had been stationed at various locations inside of the product facilities. Second, we worked quickly to create multiple training sets while simultaneously validating their accuracy with several trained sets of eyes. Finally, through a bit of repeated trial and error, we were able to get to a reliable data model for prototyping purposes. After just three weeks, we had a functional model working at 90% accuracy.
Lastly, we created a front-end application that was then connected to the training model so that all involved could showcase technical feasibility and accuracy to senior leadership. In effect, the front-end UI allowed us to show how the product identification process could be vastly improved using a combination of machine learning and computer vision, allowing plant workers to easily verify accuracy while focusing on higher-level tasks. Finally, as part of the handoff of the model and prototype, we spent several weeks working hand-in-hand with a variety of internal teams in order to train each on the various tools and skills needed in order to maintain and scale the solution to other production facilities.
We’re trying to apply technology everywhere we can as a company to drive down costs and drive up efficiency and business value.
— VP, Emerging Technology & Analytics
Together, we quickly developed a machine learning model and accompanying front-end UI that allows production-plant staff to more efficiently and accurately label massive quantities of products so that they can be processed correctly. The functioning model produced a 90% accuracy rate after just three weeks of development, testing, and validation, which represents an improvement of more than 20% when compared with the existing, manual process.
In short, the model is able to very quickly identify a specific product type, the associated stock-keeping unit (SKU), and a variety of other associated data inputs, all of which are absolutely critical for accurate inventory processing. An automated scale then records the weight of the product before an operator simply verifies the correct weight and SKU number. The system is also able to more accurately detect flaws or impurities in either product or processing equipment, which has led to elevated food safety measures.
Introduced machine learning into an overly manual and outdated process
Utilized existing infrastructure and hardware to drastically improve efficiency
Created near real-time access to product inventory data, more accurate reporting
Increased product identification accuracy to 90% (a massive improvement)
Allowed plant employees to focus their efforts on on higher-level tasks
Internal teams acquired new skills to allow for ongoing maintenance and upkeep
Increased confidence among senior leadership teams, specifically operations leaders
Modeled how to work quickly and efficiently, shipping a prototype at scale
The process and results gave senior leadership more confidence; specifically, they now had another “proof point” that objectively showed how more novel emerging technologies could be used to potentially solve a variety of critical operational challenges at scale. What’s more, the success of the prototype helped to unlock additional investment for similar initiatives.