Technical annotation layers and QA validation.
We deliver structured metadata maps alongside raw video streams. We partition our labeling work into seven explicit abstraction levels.
Metadata Ontology
Seven levels of abstraction.
RoboWorkData structures task demonstrations beyond simple video formats to enable structured learning.
Level 1: Episode Metadata
- Episode ID reference
- Task family definition
- Natural language instructions
- Anonymized contributor alias
- Environment mockup ID
- Camera specifications & sync offsets
- Duration & frame counts
- Clearance status
Level 2: Temporal Actions
- Start and end timestamps (ms)
- Action verb classification
- Active object identifiers
- Active hand labels (left/right/bimanual)
- Subtask outcome (success/failure)
- Annotation confidence score
{
"episode_id": "RWD-SORT-000042",
"start_time_s": 4.20,
"end_time_s": 6.85,
"action": "pick_up",
"active_object": "plastic_bottle",
"active_hand": "right",
"outcome": "success"
}Level 3: Hand-Object Interaction
- Approach phases
- Contact start & release frames
- Grasp type tags (pinch, power, wrap)
- Manipulation force direction tags
- Object state transitions (e.g. open/closed)
Level 4: Computer Vision Labels (Optional)
- 2D/3D hand bounding boxes
- Object bounding box tracks
- Segmentation mask sequences
- 2D hand keypoint coordinates
- Occlusion status checks
Level 5: Spatial Labels (Optional)
- 3D hand pose sequences
- 3D object trajectories
- Static camera pose matrices
- Depth coordinate arrays
- SLAM coordinate alignments
Level 6: Language & Reasoning (Optional)
- Natural language step guides
- Procedural plan layouts
- Subtask logic decompositions
- Failure visual explanation Q&As
- Occlusion reasoning descriptions
Level 7: Outcome & Recovery Logs
- Task success markers
- Partial success indicators
- Failure mode classifications
- Object drops or slip logs
- Recovery trajectory segments
Core Schema
Our standard delivery schema fields.
Every custom dataset pilot ships with the following core dictionary columns by default.
| Field Name | Data Type | Description |
|---|---|---|
| episode_id | string | Unique episode identifier within the project collection. |
| task_instruction | string | The natural-language action instruction given to the contributor. |
| action_label | string | Verbal classification of active segment (e.g. wipe, grasp, sort). |
| start_timestamp_ms | int | Millisecond offset representing start of action segment. |
| end_timestamp_ms | int | Millisecond offset representing end of action segment. |
| active_object | string | Classification name of physical object being manipulated. |
| active_hand | enum | The hand used: left, right, or bimanual. |
| contact_state | boolean | True if physical contact with active object is detected. |
| outcome | enum | Success, failure, or incomplete task status. |
| qa_grade | string | Assigned quality grade: A, B, or C. |
Quality Control
The nine-point human QA checklist.
Before a recorded clip is cleared for inclusion in a licensed dataset, our reviewers execute a complete manual audit.
Dataset release grades.
Fully cleared for commercial pretraining. Flawless lighting, crisp sharpness, complete task execution, and verified temporal labeling. Zero privacy or calibration concerns.
Minor lighting fluctuations or brief minor occlusions. Highly usable for broad training datasets, benchmarking, and representation learning.
Noticeable quality fluctuations, partial camera views, or significant hand occlusion. Usable only for target procedural understanding, task planning, or subtask decomposition pretraining.
Fails basic protocol guidelines, features unredacted privacy fields, lacks visible start/end states, or has damaged frames. Permanently excluded from delivery packages.
Interested in custom label mapping?
We work with perception teams to integrate specialized bounding boxes, keypoints, or custom segment parameters into the delivery card.