Episode Data Labeling SOP

General Intelligence Labs

Each egocentric clip requires clip-level metadata, post-capture narration, and a transcript aligned to the visible actions. Keep these labels separate from the recordings; this workflow excludes bounding boxes, frame-level annotations, and burned-in overlays.

1. Match the source files

Use the raw MCAP recording, preview video, and capture/session records. Confirm the preview plays, its duration and source range match the capture records, and all identifiers point to the same capture. Preserve each clip’s relationship to its episode and source recording.

Use the supplied manifest schema, category taxonomy, narration examples, and delivery conventions. Resolve missing files, mismatched identifiers, or missing specifications with the project contact before processing the affected clip.

2. Complete the clip metadata

Create one metadata record per clip. Populate complete.json and session.json according to the supplied schema, preserving its field names, types, units, and file structure.

Include the following information in the designated fields:

• Task: supplied machine-readable name, such as assemble_scooter_mirrors_and_windscreen.
• Category: code from the supplied taxonomy.
• Industry: applicable L1 classification.
• Visual action: concise description of the activity shown.
• Capture slice: clip duration and range within the source capture.
• Capture metadata: device class, app version, frame rate, location name and GPS, capturer, operator ID, and environment ID.
• Sensor manifest: associated camera, IMU, and audio channels.

Copy factual metadata from capture records. Refer missing values or unclear classifications to the project contact; do not estimate coordinates, invent identifiers, or infer the meanings of unknown codes.

3. Record the narration

Watch the footage after capture and record a continuous first-person voiceover in chronological order.

Open with one short sentence in present tense describing the location and relevant surroundings:

“I am at my desk with a camera and a cleaning cloth in front of me.”

State that context once. Narrate subsequent actions and location changes in simple past tense:

• Cover walking, looking, picking up objects, manipulating them, and putting them down.
• Include meaningful small actions, such as opening a lid, pressing a release button, rotating an object, or folding a cloth.
• Identify objects consistently by type and any useful visible feature: color, pattern, readable label, or part name.
• Specify where objects were placed, including the shelf, compartment, surface, or relative position.
• Name the hand or hands when visible and useful for understanding the manipulation.

For example:

“I picked up the camera with both hands. I opened the battery lid with my right hand and pressed the battery release button. I removed the battery and placed it on the desk beside the cleaning cloth.”

Describe repetitive activity naturally, calling out changes in object, hand use, direction, placement, or action. Avoid repeating identical sentences at fixed intervals.

Use direct descriptions without opinions or commentary about the recording. Do not speak timestamps. Flag unclear intervals rather than guessing obscured actions, unreadable labels, or object identities.

4. Align the transcript to the video

Transcribe the narration and correct the text against the audio. Locate each described action in the video and assign timestamps using the supplied transcript format.

Use time relative to the beginning of the delivered clip, and identify that clip in the transcript. Retain its source capture range in the metadata.

The time a sentence was spoken may differ from the time its action occurred. Check alignment against the footage, especially after pauses, replay, or changes in playback speed.

If narration is inaccurate or omits an action, correct the audio and update the transcript and alignment together.

5. Check and deliver

Replay the footage alongside the aligned transcript and verify:

• The narration covers the setting and visible sequence, including meaningful changes during repetitive activity.
• Object descriptions, manipulations, hand use, and placements match the footage.
• Transcript text matches the narration, and timestamps locate the described actions.
• Both JSON files parse and conform to the supplied schema; metadata matches capture records and taxonomy.
• Recordings and previews have no added annotations or overlays.
• Every required file is readable and unambiguously linked to its clip and episode.

Record errors by clip and field or time range, correct them, and recheck the affected outputs. Resolve source-data or interpretation questions with the project contact. Identify any unresolved issue and affected deliverable explicitly; do not mark that package complete.

Deliver the raw MCAP capture, preview video, complete.json, session.json, narration audio, and video-aligned timestamped transcript using the supplied naming and directory conventions.