Skipping the initial review task in a consensus project

Using Labelbox Annotate
1

Is there a way we can skip the initial review task but still get the consensus label in a consensus type of project? When a data row goes through review, we can get the consensus label using the selected_label_id. But if we want to skip review because it is too time-consuming, how can we get the consensus label?

1 Like
2

Can you explain the use case of why you are using Consensus projects? The intent for the selected_label_id is to show the consensus label that is the winner amongst the consensus labels. So if we dont have the review being done in the review task, could you help with why you are looking for a particular label?

Also have you tried selecting all data rows in an “In review” status and use the Bulk Action of “Move to Step” and select “Done”?
Attaching an example here.

image
image3040Ă—1250 391 KB

3

Thanks for your answer!

We have a use case where we need the consensus label to ensure sufficient data quality. To ensure high velocity, ideally, we would like that all rows where a consensus is reached automatically pass review and move to Done whereas all other rows (no consensus reached) go to Rework. If this is not possible, we would happily just export only rows with a consensus and add a new batch in the queue with all others (so a more manual Rework process).

Yes, we have tried the bulk action to move all rows into “Done” and can confirm that the selected_label_id is NOT generated in the output in that case. Not sure if this is a desired behavior

4

@pchaudhuri any more ideas on how to get the selected_label_id when the review is skipped?

5

A couple of approaches come to mind here - sorry just getting back here:

  • Option 1:

    • Create 2 tasks - 1 as a holding task; 1 as a review task for low consensus data rows

    • The holding task is where everything after the labeling will end up

    • Once data rows are in the holding task, go to Data Rows, filter for data rows that don’t have consensus above a certain threshold and send them to the Low consensus review task

    • Select the remaining data rows that have consensus above a certain threshold and send them to Done

  • Option 2 (upcoming):

    • Have a consensus score filter in Review tasks in Workflows - this is being worked on right now and will be available soon
6

Thanks for getting back to me. Option 1 is not really a solution as for the subset moving directly from Holding to Done, we still won’t have the winning label (or select_label_id) readily available…

Any timeline for Option 2? Also, it seems that the selected_label_id is an output specific to a review task, so it is not clear whether Option 2 would help with automatically getting the winning label when the consensus score is above a threshold…

7

Are you able to determine low versus high consenses at the catalog level? If so, you could classify your dataset into two groups, one for low-likely, one for high-likely, and then send the high-likely off to it’s own annotate project that is setup as a benchmark instead of a consensus project. Without knowing more about your project/use case it’s hard for me to really offer a good solution.