Wrong export with MAL import on video segmentation masks

[pio.fina]

Hi there!

I’m using MAL to import mask annotations for video assets following this guide. After preparing the MAL payload, I run MALPredictionImport.create_from_objects, which executes successfully with no error messages (only warnings about pydantic masks serialization).

However, I’m experiencing 2 kinds of problems:

1. Visualization: uploaded annotations are not visible in the timeline but are uploaded correctly, as I can see them in the main window overlayed on the video’s frames.
   

   

2. Export: worse than above, when I export the project annotations, there are only a random subset of frames with MAL annotations and others are missing in projects.<project_id>.labels[0].annotations.frames json’s object:

...
  "projects": {
    "cm8y6bikn0fp007274622fmxl": {
      "name": "mal-video-segmentation-masks-issue",
      "labels": [
        {
          "label_kind": "Video",
          "version": "1.0.0",
          "id": "cm8y6ipa90cv207f6gsgh8og4",
          "annotations": {
            "frames": {
              "9": { ... }
               
              "15": {...}
             }
...

Instead, the content of projects.<project_id>.labels[0].annotations.segments and projects.<project_id>.labels[0].annotations.key_frame_feature_map is coherent with the imported data, ie, each uploaded instance mask has an associated feature ID with correct keyframes and segments (I’m uploaded annotations for a small video segment, from frame 1 to 19 only for odd frame indices):

...
          "segments": {
            "cm8y6iqwc0fit07fs6mqk2adx":[[7,7],[17,17],[11,11],[3,3],[19,19],[5,5],[13,13],[1,1],[9,9],[15,15]],
            "cm8y6iqwc0fiu07fs1m5ch0mr":[[7,7],[17,17],[11,11],[3,3],[19,19],[5,5],[13,13],[1,1],[9,9],[15,15]],
            "cm8y6iqwc0fiv07fs989z9vdb":[[7,7],[17,17],[11,11],[3,3],[19,19],[5,5],[13,13],[1,1],[9,9],[15,15]],
            "cm8y6iqwc0fiw07fs659s7wrp":[[7,7],[17,17],[11,11],[3,3],[19,19],[5,5],[13,13],[1,1],[9,9],[15,15]]
          },
          "key_frame_feature_map": {
            "cm8y6iqwc0fit07fs6mqk2adx": [1,3,5,7,9,11,13,15,17,19],
            "cm8y6iqwc0fiu07fs1m5ch0mr": [1,3,5,7,9,11,13,15,17,19],
            "cm8y6iqwc0fiv07fs989z9vdb": [1,3,5,7,9,11,13,15,17,19],
            "cm8y6iqwc0fiw07fs659s7wrp": [1,3,5,7,9,11,13,15,17,19]
          },
...

This is not the expected behaviour and makes MAL unusable for our use cases since, even if visible in the UI, annotations can’t be properly exported for further processing

More info that could be useful:

• Exported frames are different from the one visualized in the timeline.
• Reproducing the issue several times leads to a different subset of visualized/exported frames.
• I tried to change the number of frames with MAL annotations (1, 2, 3, 4, 5, …, 19). The problem occurs with n. frames > 3 and only works as expected if I upload 1 or at max 2 annotated frames.
• The composite and binary masks for the exported frames are coherent with the imported data and visualization in the UI.
• I tried to replace labelbox.types classes with raw ndjson payload, but nothing changed!
• I tried to export with parameters {interpolated_frames:True}, but nothing changed!
• I tried both exports from UI and SKD, and still the same problem!
• An MRE is available here
• This issue is not present for VideoObjectAnnotation and VideoObjectClassification, everything work as expected
• Full exported annotations are available here with relative project ids and other internals.

Can you confirm whether this is a known/unknown issue? Are there any suggested workarounds for using MAL with video segmentation masks?