Confused to setup ground truth

Python SDK
1

Hi, i have big problem to setup my ground truth, first i used my labelbox data to build( train ) mask rcnn model that detect human body muscle, we know that each muscle have complex shape and when i want to use this trained model for new dataset to setup ground truth project, i dont know how set my mask parameter.

in the ground truth documentو Shown in a line that we have to attach our annotation like:

annotations = {
    'externalId1': [
        {
            'bbox': [10, 10, 50, 50],  # x,y,h,w
            "class": 'dog'
        },
    ],
}

i also know the mask annotation type is like:

ObjectAnnotation(
  name = "mask",
  value = Mask(mask=MaskData(arr=np_mask), color=color),
)

but how can i use this when i have multi complex shape like stomache,shoulder…? maybee i have 8-20 object in one input image.

also i used mask rcnn of coco data, how can i assign my stimated object for mask labelbox type? my model output parameters are:

image_meta, gt_class_id, gt_bbox, gt_mask

here is my output of stimated mask:

3
32956×1527 205 KB

i just need to upload my mask annotation but coco gt_mask is boolian array like what i attached here…what can i do and how can i run ground truth for this objects and this model?

2

Hi, thanks for reaching out. Even though your model encodes the predictions in a single mask, technically each class is a separate annotation. When uploading annotations you can share this one mask and use the color to identify which pixels belong to the class. E.g.

np_mask = ...
mask_data = MaskData(arr = np_mask.astype(np.uint8))
annotations = [
        ObjectAnnotation(name = "stomach", value = Mask(mask = mask_data, color = [128,128,0])),
        ObjectAnnotation(name = "shoulder", value = Mask(mask =mask_data, color = [128,0,0])),
]

In the above example all x,y pixels with the value of [128,128,0] belong to the stomach class. All pixels with a value of [128,0,0] belong to the shoulder class.

You can also encode multiple instances into the same mask by assigning a different color to each instance.

annotations = [
        ObjectAnnotation(name = "stomach", value = Mask(mask = mask_data, color = [128,128,0])),
        ObjectAnnotation(name = "shoulder", value = Mask(mask =mask_data, color = [128,0,0])),
        ObjectAnnotation(name = "shoulder", value = Mask(mask =mask_data, color = [128,0,128])),
]

Please let me know if you have any other questions.

3

Dear msokoloff,
Thanks to your answer, so base on your answer first i need to convert my gt_mask (boolian array) to uint8 and my ground truth will be like this:

from labelbox import Client, OntologyBuilder
from labelbox.data.annotation_types import (
    LabelList,
    ImageData,
    Rectangle,
    ObjectAnnotation,
    Label
)
from labelbox.data.serialization import NDJsonConverter

client = Client()
API_KEY = 'xxxxx'
client = Client(api_key=API_KEY)
PROJECT_ID = "cl73cnx5bcx5y081i61d3ebed"
project = client.get_project(PROJECT_ID)

data_rows_iterator = dataset.data_rows()
data_row = next(data_rows_iterator)
data_row
for data_row in dataset.data_rows():
  data_rows_iterator = dataset.data_rows()
  data_row = next(data_rows_iterator)
  print(data_row)
# upload for queued data rows
queued_data_rows = project.export_queued_data_rows()

np_mask = my gt_mask (boolian array)
mask_data = MaskData(arr = np_mask.astype(np.uint8))
annotations = [
        ObjectAnnotation(name = "stomach", value = Mask(mask = mask_data, color = [128,128,0])),
        ObjectAnnotation(name = "shoulder", value = Mask(mask =mask_data, color = [128,0,0])),
]

labels = LabelList()

for datarow in queued_data_rows:
    uploads = []
    for annot in annotations[datarow.external_id]:
        bbox = annot['bbox']
        uploads.append(
            ObjectAnnotation(
                name=annot['name'],
                value=Rectangle.from_xyhw(bbox[0], bbox[1], bbox[3], bbox[2]),
            )
        )

    labels.append(Label(data=ImageData(uid=datarow['id']), annotations=uploads))

# Convert names to feature schema ids
labels.assign_feature_schema_ids(OntologyBuilder.from_project(project))

labels_ndjson = list(NDJsonConverter.serialize(labels))

does it right?
also how can i set automatically assign color for objects? i need to create different color manually for each object?

4

That looks mostly right, although you are creating bbox annotations instead of segmentation masks. What is the output of your model? How are you encoding the instance? You also can provide separate mask data for each instance. E.g.

ObjectAnnotation(name = "stomach", value = Mask(mask = MaskData(arr = np_mask_1.as_type(np.uint8)), color = [128,128,0]))
ObjectAnnotation(name = "stomach", value = Mask(mask = MaskData(arr = np_mask_2.as_type(np.uint8)), color = [128,128,0]))

In this case you aren’t sharing the the segmentation masks between the objects. They each have their own mask.

5

for example when my input image have 17 mask object, my gt_mask (boolian array) shape is:
(512, 512, 17)
but when i convert it by your way, mean use this:

mask_data = MaskData(arr = gt_mask.astype(np.uint8))

my shape will be empty (), why?
i know each of third column of gt_mask related to each object mask and if i choose of one of that like :

np_mask1=gt_mask[:,:,3]

but this was not work!, I upload my colab project here:

https://www.mediafire.com/file/n01lnn4pfubka1x/Ground_Truth_Labelbox.ipynb/file

You can check all these parameter in that…actually i confiused to how can i use my model output prediction to my mask data, can you check my file?

As shown in my attached file, my outputs are as follows:

ScreenShot_20220907221916
ScreenShot_20220907221916960×417 14.9 KB

and when run this block:
ScreenShot_20220907221944
ScreenShot_20220907221944872×610 61.7 KB

also for creating annotation like below:

for datarow in queued_data_rows:
    uploads = []
    for annot in annotations[datarow.external_id]:
        bbox = annot['bbox']
        uploads.append(
            ObjectAnnotation(
                name=annot['name'],
                value=Rectangle.from_xyhw(bbox[0], bbox[1], bbox[3], bbox[2]),
            )
        )

how can i create it for segmentation mask? sry for my these question, thanks for your support.

6

Just please help, i really need help…