AI Agent Observability.
Duration
May 2025 - Present
Location
Palo Alto, CA
Role
UX Engineer
Task
AI Agent Observability
Dashboard
Human-in-the-loop
Making 20K+ monthly AI agent conversations observable, auditable and actionable
Verc's voice agent answers and places collection calls for agencies at scale. As call volume grew, ops teams couldn't tell why a call failed, which calls needed a human, or whether every call met compliance requirements. I designed the observability layer between the AI agent and the humans responsible for it.
Key Contributions:
– Designed agent observability dashboards and audit logs covering 20K+ monthly customer conversations.
– Cut issue diagnosis time by 31.5% by surfacing failure patterns and human-in-the-loop handoff trends.
– Turned every call into a reviewable record: AI summary, speaker-separated transcript, recording and a compliance check.
IMPACT & VALUE
From "what happened on that call?" to a one-click answer
Before, answering "why did this call fail?" meant pulling logs and listening to recordings. Now the answer is one click away, and recurring problems show up as trends before anyone has to go looking.
Highlight:
– 20K+ conversations per month made observable and auditable
– 31.5% faster issue diagnosis
– Human handoffs (callbacks, reviews) surfaced as a daily queue instead of buried in logsTEST
MAIN CHALLENGE
An AI agent that talks to consumers about debt must be trustworthy to ops, compliance and the agency's clients at once.
Every call carries regulatory risk: recording disclosure, Mini-Miranda, TCPA consent. When the agent fails, for example when identity can't be verified, the cost isn't a bad UX moment; it's a missed payment or a compliance gap.
GOAL
Let a small ops team supervise thousands of AI conversations without listening to them.
My role was to decide where humans should look, and what they should see first.
Design Progress.
FEATURE 1
Home: an Action Queue for human-in-the-loop work
The home screen answers "what needs me today": callbacks to make, calls to review, overdue items, plus collection totals and AI-generated insights such as a past-due plan or journeys with no eligible accounts.
Tabs split the work by type (Callbacks, Need review, Inbox, Journeys, Portal requests, Payments), and every row opens the call details directly: one click from task to context.
FEATURE 2
Voice performance: the in-call funnel
Greeting → Identity confirmed → Authenticated → Payment link.
Showing every stage as a funnel turns "the agent isn't converting" into "authentication drops here; look there." KPI cards compare against the prior period, and trend charts separate known vs. unknown callers and authentication vs. conversion over time.
FEATURE 3
Call details: the audit record
Each call opens to account context, outcome, callback schedule and a compliance check (recording disclosure, Mini-Miranda, TCPA opt-in), followed by an AI summary, a speaker-separated recording timeline, and a transcript with one-click translation.
Outcomes and flags are editable inline, so a reviewer can correct the AI's judgment on the spot.
FEATURE 4
Failure patterns, not single failures
Calls are grouped by outcome and flag, so recurring failure modes (for example identity mismatch → callback) appear as trends instead of one-off tickets. This is where the 31.5% reduction in diagnosis time came from.
Evaluation.
HOW I BUILT IT
Every call becomes a reviewable record
I prototyped each view directly in the production codebase with Claude Code and Figma agents, validated it on staging with real call data, and shipped through pull requests: frontend-only changes I merged myself, anything touching call data went through engineer review.
ITERATION
From a queue of calls to a home for decisions
The first version was a list of calls called "Queue." Watching ops use it showed the real job wasn't browsing calls; it was clearing decisions. I rebuilt it as Home: one Action Queue with Callbacks first, rows that open call details instead of the account page, and per-tab filters.
ONGOING IMPACT
The story does not end here...
The same observability layer now covers every voice tenant.
The same observability layer now covers every voice tenant, giving each agency a shared view of how the AI performs on their accounts.
Reflection.
REFLECTION 1
1. Observability is a design problem
The data already existed. The design work was deciding what a human should see first, and what they should never have to dig for.
REFLECTION 2
2. Human-in-the-loop needs a home
Handoffs only work when they arrive as a queue someone owns. A flag in a log isn't a handoff; a callback on someone's Home screen is.