Design Document

Document Status: Design Phase | Version: 1.0 | Last Updated: November 17, 2025

Overview

This document describes a queue-based call annotation and feedback system for Weave. The system allows users to create queues, manually add calls to them in batches, then systematically annotate or review those calls for quality assurance and feedback collection.

The system supports multiple annotators working from a shared pool where any annotator can claim and complete work, preventing duplicate effort while enabling team collaboration.

What's In Scope (MVP)

The initial release enables users to:

• Create queues with name, description, and associated scorers
• Manually curate calls by adding them to queues in batches
• Track annotation progress (pending, in-progress, completed, skipped)
• Support review workflows where calls with existing feedback can be re-reviewed

What's NOT In Scope (MVP)

<aside> ⚠️

Automatic Call Population — Queues do NOT automatically populate based on filter criteria. Users must manually select and add calls to queues.

Call Removal — Once a call is added to a queue, it cannot be removed. Users should carefully curate which calls to add.

Filter-based Queue Creation — No support for creating queues with automatic filters that populate calls.

</aside>

System Architecture

Core Entities

The system consists of five main entities with clear relationships:

┌──────────────────────────┐
│   annotation_queues      │  ← Queue definition (name, description, scorers)
└────────────┬─────────────┘
             │ 1:N
             ▼
┌──────────────────────────┐
│ annotation_queue_items   │  ← Call membership (which calls are in queue)
└────────────┬─────────────┘
             │ 1:1 (shared pool) or 1:N (multi-annotator)
             ▼
┌──────────────────────────┐
│ annotator_queue_items_   │  ← Per-annotator workflow state
│ progress                 │     (pending/claimed/completed/skipped)
└────────────┬─────────────┘
             │ N:1
             ▼
┌──────────────────────────┐
│      calls_merged        │  ← The actual traced call
└────────────┬─────────────┘
             │ 1:N
             ▼
┌──────────────────────────┐
│      feedback            │  ← Annotations created by annotators
└──────────────────────────┘