MALPredictionImport.create_from_objects failing without producing error codes aside from AnnotationImportState.Failed

Python SDK
,
1

I am making a script using the SDK to upload external pseudo labels as predictions to a Labelbox project. I am using this code to upload mask labels.

    upload_job = MALPredictionImport.create_from_objects(
        client=client,
        project_id="XXXXXXXXXXX",
        name="mal_job" + str(uuid.uuid4()),
        predictions=Labels,
    )
    upload_job.wait_till_done(show_progress=True)
    print("Errors:", upload_job.errors)

Labels is a List of <class ‘labelbox.data.annotation_types.label.Label’>

The Label objects are created using

lb_types.Label(data={"uid": uid}, annotations=annotations)

Annotations is a list of objects generated from this code:

color = (4) # 1 - 4 depending on class
    mask_data_4 = lb_types.MaskData.from_2D_arr(composite_mask)
    mask_annotation_4 = lb_types.ObjectAnnotation(
        name="XXXXXX", value=lb_types.Mask(mask=mask_data_4, color=color)
    )

Running my script with logging.DEBUG results in the following before the error:

DEBUG:labelbox.client:Query: query getModelAssistedLabelingPredictionImportPyApi($projectId : ID!, $name: String!) {
            modelAssistedLabelingPredictionImport(
                where: {projectId: $projectId, name: $name}){
                    errorFileUrl inputFileUrl name progress state statusFileUrl id project{autoAuditNumberOfLabels autoAuditPercentage createdAt dataRowCount description editorTaskType isBenchmarkEnabled isConsensusEnabled lastActivityTime allowedMediaType modelSetupComplete name queueMode setupComplete id updatedAt uploadType}
                }}, params: {'projectId': 'XXXXXXXXXX', 'name': 'XXXXXX'}, data None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.labelbox.com:443
DEBUG:urllib3.connectionpool:https://api.labelbox.com:443 "POST /graphql HTTP/1.1" 200 None
DEBUG:labelbox.client:Response: {"data":{"modelAssistedLabelingPredictionImport":{"errorFileUrl":null,"inputFileUrl":"XXXXXXXXXXXXXX","name":"XXXXXXXXX","progress":"100%","state":"FAILED","statusFileUrl":null,"id":"XXXXXXXXXXXXX","project":{"autoAuditNumberOfLabels":3,"autoAuditPercentage":0,"createdAt":"XXXXXXXX","dataRowCount":1214,"description":"XXXXXXXX","editorTaskType":null,"isBenchmarkEnabled":true,"isConsensusEnabled":true,"lastActivityTime":"XXXXXXXXX","allowedMediaType":"IMAGE","modelSetupComplete":null,"name":XXXX,"queueMode":"CATALOG","setupComplete":null,"id":"XXXXXXXXX","updatedAt":"XXXXXXX","uploadType":"MANUAL"}}}}

Script terminates while trying to print upload_job.errors

 "XXXX/labelbox/schema/annotation_import.py", line 64, in errors
    return self._fetch_remote_ndjson(self.error_file_url)
 "XXXX/labelbox/schema/annotation_import.py", line 141, in _fetch_remote_ndjson
    raise ValueError("Import failed.")

Due to self.error_file_url being None I have hit a roadblock. Has anyone encountered this error before? Any help would be greatly appreciated.

Note: The job is listed as failed in the project notifications tab.

1 Like
2

Hey @ian,
I checked the mask data and seems there might be an issue with those, are you converting those to byte?
Also, the NDJSON as png in the mask dictionary instead of imBytes and the color is missing.
This is what I would expect from your import:

{'name': 'mask',
  'classifications': [],
  'mask': {'imBytes': 'iVBORw0KGgoAAAANSUhEUgAACgAAAAarCAYAAADJL/CBAAAAAXNSR0IArs4c6QAAIABJREFUeF7s3VtuJcUSQNFuJsHQGCNDYxKALNGShfw4x46syl217rc7T9SK+EJbuj9/+B8BAgQIECBAgAABAgQIECBAgAABAgQIECBAgAAB [...] pNWAvPlAAAAABJRU5ErkJggg==',
   'colorRGB': (41, 161, 42)}}

If you have sample I can test it.
There is an example of mask import: labelbox-python/examples/annotation_import/image.ipynb at main · Labelbox/labelbox-python · GitHub

Many thanks,
PT

3

@ptancre Thanks for the help!
Labelbox is not receiving the masks but the job is succeeding (with errors).

I changed the way I was making the labels. Instead of using:

lb_types.MaskData.from_2D_arr(composite_mask)

Now I am using:

comp_mask_bytes = np_to_bytes(comp_mask)
mask_data_1 = lb_types.MaskData(im_bytes=comp_mask_bytes)

This resulted in the masks looking like this:

{"uuid": "XXXXXXXXXXXXX",
 "dataRow": {"id": "XXXXXXXXXXXXX"},
 "name": "Sky",
 "classifications": [],
 "mask": {"png": "iVBO[...]gg=="}}

I traced it very deep into labelbox/serialization/ndjson/objects.py and found that the NDMask was using a _PNGMask (probably since I wasn’t using a url). The rest of the object seems to encode it correctly so instead I just manually changed the masks in validation so they read:

{"uuid": "XXXXXXXXXXXXX",
 "dataRow": {"id": "XXXXXXXXXXXXX"},
 "name": "Sky",
 "classifications": [],
 "mask": {"imBytes": "iVBO[...]gg==", "colorsRGB": [0, 0, 255]}}

At this point the ndjson file is getting to labelbox but the Import is succeeding with errors.

Specifically every prediction fails. The error file reads this for all predictions:

{"uuid": "XXXXXXXXXX",
 "dataRow": {"id": "XXXXXXXX", "globalKey": null},
 "status": "FAILURE",
 "errors": [{"name": "InvalidAnnotation", "message": "Found empty mask", "additionalInfo": null}]}

The ndjson file seems complete. I can access the input_file_url from my MALPredictionImport object and it downloads a ndjson that includes the encoded mask data.

Can’t seem to make sense of why it doesn’t work despite being in the correct format.

1 Like
4

Is there any way I could see why labelbox is not finding the masks? I have tried pretty much everything I can think of but nothing is working.

Does anyone have an example of making MaskData from a local png and importing is with MALPredictionImport that can be confirmed to work?

5

I meant to update you, I was testing some of your encoded image via Base64 to Image Decoder / Converter and seems they are completely opaque black.
Again If you have a PNG that you can share I could help, otherwise the reference that we have in our notebook would be the way to go.

cp_mask_url = "https://storage.googleapis.com/labelbox-datasets/image_sample_data/composite_mask.png"
colors = extract_rgb_colors_from_url(cp_mask_url)
response = requests.get(cp_mask_url)

mask_data = lb.types.MaskData(im_bytes=response.content) # You can also use "url" instead of img_bytes to pass the PNG mask url.
rgb_colors_for_mask_with_text_subclass_tool = [(73, 39, 85), (111, 87, 176), (23, 169, 254)]

cp_mask = []
for color in colors:
  # We are assigning the color related to the mask_with_text_subclass tool by identifying the unique RGB colors
  if color in rgb_colors_for_mask_with_text_subclass_tool:
    cp_mask.append(
      lb_types.ObjectAnnotation(
        name = "mask_with_text_subclass", # must match your ontology feature"s name
        value=lb_types.Mask(
          mask=mask_data,
          color=color),
        classifications=[
          lb_types.ClassificationAnnotation(
            name="sub_free_text",
            value=lb_types.Text(answer="free text answer sample")
          )]
      )
  )
  else:
     # Create ObjectAnnotation for other masks
    cp_mask.append(
      lb_types.ObjectAnnotation(
        name="mask",
        value=lb_types.Mask(
            mask=mask_data,
            color=color
        )
    )
  )
6

I ended up using the NDJSON method for creating the annotations manually.

There were two issues with my script:

  1. My labels were 256x256 but my project needed 1280x720
  2. When generating the imBytes, I was encoding an RGB image for the annotations. When making the annotations in NDJSON imBytes needed to be of a PNG with the ‘1’ format (binary). Adding img.convert(‘1’) fixed this.

Thank you ptancre for helping me with this.

1 Like