Change class of bbox annotations for PDF documents

Hello,

I am trying to pre-annotate various elements (paragraphs, headers, …) of PDF documents with predictions from a ML model, upload these annotations to Labelbox and correct them online.

The prediction and upload processes work (minimal example below) fine but unlike for entities, it does not seem possible to change the class of bbox annotations. The only alternative I see is deleting and re-creating a new one of a different class by hand, which is really impractical. And conversely, I cannot find a way to upload entity annotations via the bboxes.

I also tried creating a radio classification inside a single bbox class but visually this is much less interesting (no colors for different classes) and the upload failed (see below).

Any hints?

import os
import uuid
import labelbox as lb
import labelbox.types as lb_types
from labelbox.schema.queue_mode import QueueMode


client = lb.Client(api_key=os.environ["LB_API_KEY"])

classes = [
      "Text",
      "Section-header",
      "Caption",
      "Page-footer",
      "Page-header",
      "Picture",
]

tools = [
    lb.Tool( tool=lb.Tool.Type.BBOX,name=name) 
    for name in classes
  ]  # DOES NOT WORK WITH NER INSTEAD OF BBOX

ontology_builder = lb.OntologyBuilder(tools=tools)
ontology = client.create_ontology("MyOntology",
                                  ontology_builder.asdict(),
                                  media_type=lb.MediaType.Document)


# # Create a Labelbox project
project = client.create_project(name="MyProject",
                                    queue_mode=QueueMode.Batch,
                                    media_type=lb.MediaType.Document)
project.setup_editor(ontology)

dataset = client.get_dataset(dataset_uid)
data_row = next(dataset.data_rows())

predictions = [{"bbox": [1138.596, 5.894, 1140.978, 16.84], "block_type": "Page-header"}, "page_no": 0}]

annotations = [lb_types.ObjectAnnotation(
            name=prediction["block_type"],
            value=lb_types.DocumentRectangle(
                start=lb_types.Point(x=prediction["bbox"][0], y=prediction["bbox"][1]),
                end=lb_types.Point(x=prediction["bbox"][2], y=prediction["bbox"][3]),
                page=prediction["page_no"],
                unit=lb_types.RectangleUnit.POINTS
            )
        ) for prediction in predicitons
]

labels = [
    lb_types.Label(
        data=lb_types.DocumentData(uid=data_row.uid),
        annotations=annotations,
    )
]

upload_job = lb.MALPredictionImport.create_from_objects(
    client = client,
    project_id = project.uid,
    name="pdf_annotation_upload_" + str(uuid.uuid4()),
    predictions=labels)

upload_job.wait_until_done()
# Errors will appear for annotation uploads that failed.
print("Errors:", upload_job.errors)
print("Status of uploads: ", upload_job.statuses)

I tried the following for a single bbox class with nested classifications:

# tools = [lb.Tool( tool=lb.Tool.Type.BBOX,
#           name="PDF object",
#           classifications=[
#             lb.Classification(
#                 class_type=lb.Classification.Type.RADIO,
#                 name="Type",
#                 options=[lb.Option(name) for name in classes]
#             )
#           ]
# )]

[...]

# annotations = [lb_types.ObjectAnnotation(
#             name="PDF object",
#             value=lb_types.DocumentRectangle(
#                 start=lb_types.Point(x=prediction["bbox"][0], y=prediction["bbox"][1]),
#                 end=lb_types.Point(x=prediction["bbox"][2], y=prediction["bbox"][3]),
#                 page=page_no,
#                 unit=lb_types.RectangleUnit.POINTS
#             ),
#             classifications=[lb_types.ClassificationAnnotation(
#                                 name="Type",
#                                 value=lb_types.Radio(answer=lb_types.ClassificationAnswer(name=prediction["block_type"]))
#             )]
#         ) for page_no, items in enumerate(predictions) for prediction in items 
# ]

Hey @palito

Are you referring to the missing change class option for bbox :

whereas NER :

Labelbox 🔊 2024-03-12 12-40-42(1)

let me know!

Many thanks,
PT

Exactly, that is what is missing! (or an easy-to-use alternative)

Thanks for confirming @palito we are checking internally to add this option.
I will keep you updated.

Many thanks,
PT

Hi @ptancre , any news?

Thanks!