Using MALPredictionImport to upload video object tracks where an object temporarily disappears

Python SDK
1

Hi there! I’m interested in uploading an existing set of video object annotations with MALPredictionImport, namely in the situation where an object disappears from view, then reappears. For example, suppose an object exists in frames 10-13 and 19-25, but NOT in frames 14-18. Right now, I have this payload:

# Confidence scores are not supported for frame specific bounding box annotations and VideoObjectAnnotation class
bbox_dm = {
  "top":617,
  "left":1371,
  "height":419,
  "width":505
}

# Python Annotation
bbox_annotation = [
  lb_types.VideoObjectAnnotation(
    name = "bbox_video",
    keyframe=True,
    frame=10,
    segment_index=0,
    value = lb_types.Rectangle(
          start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]), # x = left, y = top
          end=lb_types.Point(x=bbox_dm["left"] + bbox_dm["width"], y=bbox_dm["top"] + bbox_dm["height"]), # x= left + width , y = top + height
      )
  ),
  lb_types.VideoObjectAnnotation(
    name = "bbox_video",
    keyframe=True,
    frame=13,
    segment_index=0,
    value = lb_types.Rectangle(
          start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]), # x = left, y = top
          end=lb_types.Point(x=bbox_dm["left"] + bbox_dm["width"], y=bbox_dm["top"] + bbox_dm["height"]), # x= left + width , y = top + height
      )
  ),
  lb_types.VideoObjectAnnotation(
    name = "bbox_video",
    keyframe=True,
    frame=19,
    segment_index=0,
    value = lb_types.Rectangle(
          start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]),
          end=lb_types.Point(x=bbox_dm["left"] + bbox_dm["width"], y=bbox_dm["top"] + bbox_dm["height"]),
      )
  ),
  lb_types.VideoObjectAnnotation(
    name = "bbox_video",
    keyframe=True,
    frame=25,
    segment_index=0,
    value = lb_types.Rectangle(
          start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]),
          end=lb_types.Point(x=bbox_dm["left"] + bbox_dm["width"], y=bbox_dm["top"] + bbox_dm["height"]),
      )
  )
]

However, when uploaded to Labelbox, this will cause the box to be interpolated for frames 14-18, whereas I want to indicate that there should be NO box for those frames. Is there a way to indicate this in the payload, while preserving the fact that the boxes in frames 10-13 and 19-25 correspond to the same object?

I appreciate any tips that anyone has for this situation. Thank you!

2

Hello @rszeto

Segment_index is what will create the “separation” between your annotations.
Example python annotation → 

bbox_dm = {"top": 100, "left": 200, "height": 50, "width": 100}

bbox_annotation = [
    # Segment 0: Object appears from frame 1 to 10
    lb_types.VideoObjectAnnotation(
        name="bbox_video",
        keyframe=True,
        frame=1,
        segment_index=0,  # First continuous appearance
        value=lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]),
            end=lb_types.Point(
                x=bbox_dm["left"] + bbox_dm["width"],
                y=bbox_dm["top"] + bbox_dm["height"],
            ),
        ),
    ),
    lb_types.VideoObjectAnnotation(
        name="bbox_video",
        keyframe=True,
        frame=10,
        segment_index=0,  # Still segment 0 - ends the first appearance
        value=lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]),
            end=lb_types.Point(
                x=bbox_dm["left"] + bbox_dm["width"],
                y=bbox_dm["top"] + bbox_dm["height"],
            ),
        ),
    ),
    # Segment 1: Object reappears from frame 14 to 20
    lb_types.VideoObjectAnnotation(
        name="bbox_video",
        keyframe=True,
        frame=14,
        segment_index=1,  # Second continuous appearance (NEW segment)
        value=lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]),
            end=lb_types.Point(
                x=bbox_dm["left"] + bbox_dm["width"],
                y=bbox_dm["top"] + bbox_dm["height"],
            ),
        ),
    ),
    lb_types.VideoObjectAnnotation(
        name="bbox_video",
        keyframe=True,
        frame=20,
        segment_index=1,  # Still segment 1 - ends the second appearance
        value=lb_types.Rectangle(
            start=lb_types.Point(x=bbox_dm["left"], y=bbox_dm["top"]),
            end=lb_types.Point(
                x=bbox_dm["left"] + bbox_dm["width"],
                y=bbox_dm["top"] + bbox_dm["height"],
            ),
        ),
    ),
]

Segment_index 0 covers frame 1 to 10
No annotation in frane 11 to 13
Segment_index 1 covers frame 14 to 20

image
image1644×380 9.05 KB

3

Hi @PT , thanks for the reply and for clarifying the purpose of segment_index! Now I’m wondering how this might apply for multiple instances of the bbox_video class, e.g., importing the equivalent of the annotation below:

image
image1540×352 23 KB

Naïvely, I tried creating two sets of VideoObjectAnnotation with name=”bbox_video 1” and name=”bbox_video 2”, but that didn’t seem to work. Let me know if you have any suggestions here.

4

for several features, create an additional list with the annotation payload and then loop the output like:

label = []
annotations_list = [
bbox_annotation_1,
bbox_annotation_2
]

for annotation in annotations_list:
label.append(
lb_types.Label(data={"global_key": global_key}, annotations=annotation))
5

Got it. I was able to adapt this to my code, thanks!

1 Like