Avoid reviewing own labels

Using Labelbox Annotate
1

Hello,

I have a project with multiple labelers/reviewers, and I would like to have a cross-review of the labels, meaning review of the labels to be done by a person other than the one who made the labels. So far I didn’t find a way to avoid review of self-labeled images. Is there any option that I missed?

1 Like
2

Hey there!

Hope you are well today. So, how many labelers do you have working on a project? Let’s assume, for now, it is 2 or 3.

The easiest way to do this is to go to workflows and create a workflow for Labeler X and Labeler Y.

You can assign each workflow to “Is labeled by”, and have that for both labelers to review the other persons labels.

Let me know if that helps!

3

Thank you so much for the quick answer!

I tried the workflows, and it kind of works… for 2 labelers.
As soon as I add a 3rd one (and this will be the case of my project) the labels of the 3rd labeler go directly to the Done state, without going through any of the review tasks… Let me detail how I set all up, maybe I made a mistake somewhere.

I have 3 labelers: L1, L2 and L3. I created 3 Review tasks (screenshot below), “L1”, “L2” and “L3”, and set the “Is labeled by” to :

  • L2 & L3 for task L1
  • L1 & L3 for task L2
  • L1 & L2 for task L3

However, when L1 and L2 annotate, the images go to the respective L2 and L1 review tasks, but none of the images goes to the L3 tasks. Moreover, when L3 annotates, the images go neither in the L1 nor L2 tasks, but directly to Done.
I was expecting to see the images annotated by L3 to go to both L1 and L2 tasks, and images annotated by both L1 and L2 to go to the L3 task.

What did I miss?

Additional clarification: the aim is to have one image reviewed once - for instance, image labelled by L1 sent to tasks L2 and L3 and sent to Done if L2 or L3 do the review, not L2 and L3.

(and thanks again for your quick answer! highly appreciated!)

Workflows
Workflows1482×639 15.7 KB

4

I think your use case would work if the filters were AND, in Workflow the logic is OR so overlapping tasks may not work the way it is set up.
Adding individual labeler can solve your issue here :

L1 → L1 (labeled by)
L2 → L2 (labeled by)
L3 → L3 (labeled by)

Once approved they should go straight to Done.

1 Like
5

Thank you! I think I’ll do that.

1 Like