Waymo Label Requesting Redesign
Redesigning how ML engineers request human-labeled data at Waymo — collapsing a sprawling, tribal-knowledge process into a guided multi-step form that doesn't need a labeling engineer on call.
A standalone raw HTML/CSS/JS prototype of the handoff
Note: These screens are my own redesigns. Copy, labels, and advanced settings were rewritten or invented to protect Waymo's confidential information — some deliberately unrelated to the real tool.
The problem
A hand-held process that stopped scaling as Waymo grew.
Human-labeled data powers self-driving: it’s how what the car sees becomes something a model can learn from. Labels range from 2D bounding boxes to timestamps to open questions about a run — and every new label type starts with an ML engineer explaining to a group of labelers exactly what they need and how to do it.
Historically, the labeling team hand-held every ML engineer through most of their labeling requests. New labeling requests often required new detailed policies that were explained through layers of documentation, new protos for new attributes, and a back-and-forth with the labeling team to understand what needed to be defined and why.
That worked when the company was small. As Waymo grew, it became a bottleneck with a labeling engineer required every time someone needed something new.
Challenge
Can ML engineers request labels directly from Labeling Ops without requiring multiple bugs, discussions, and help from labeling engineers?
Research
Cognitive walkthroughs, a heuristic evaluation, and interviews.
I worked on several different forms of research to understand the current process and its pain points:
- Cognitive walkthroughs: I simulated a brand-new ML engineer and an experienced user each creating a new labeling request with custom fields, and jotted down every point of confusion, every question, and every place where the user had to leave the flow to look up information.
- Heuristic evaluation of the existing workflow against Nielsen’s 10 usability heuristics.
- User interviews with both experienced and newer members of ML engineering teams to understand their common workflows, pain points, and where they needed help.
Key insights
Too many ways in, an internal vocabulary, and advanced settings even for the simplest requests.
Charting the current process revealed the first problem before any interview did. Documentation pointed to several possible starting points, and each of those led to a different path through the request process. Multiple entry points, redundant documentation, internal vocabulary, and required steps that only applied to advanced users made the process painful for new users.
Multiple entry points, redundant documentation, internal vocabulary, and required steps that only applied to advanced users made the process painful for new users.
From there I inventoried every piece of information a request actually needs, grouped the related settings, and separated what users change constantly from the advanced settings they touch infrequently.
I pulled documentation out of several sources and added clearer guidance within the interface itself, so users could learn the vocabulary and understand the options without leaving the flow.
The redesign
One hub, one wizard, and documentation moved into the interface.
I redesigned the request flow as a wizard. I grouped related settings into steps, promoted documentation from tooltips into the interface itself, and tucked advanced settings behind toggles.
Components of the redesign:
- A central hub for creating labeling requests and tracking their progress
- A guided wizard with clear in-context instruction and hidden advanced settings
- Streamlined continuous and one-time data importing
The new wizard breaks up a long process into saveable, manageable steps, and hides advanced features unless needed.
The redesigned quality-control step uses plain-language documentation for each method.
A new section configures labeling questions right in the labeling request flow, and shows a preview of what the labeler will see.
A review step provides feedback before saving.
The new dashboard for tracking labeling requests, instead of using the overloaded bug tracker.
What the work established
- A central place for creating and tracking requests.
- Documentation as interface. Explanations moved out of tooltips and out of documentation and into the moments where users make decisions, so the form teaches the vocabulary it uses.
- Progressive disclosure. Gathered research data about what was commonly used and what wasn’t, and tucked advanced settings behind toggles so users only see what they need.
The intended effect: ML engineers who can make and track their own requests, and labeling engineers who get their time back.