How to interact with the Labeling Review Queue?

Python SDK
,
1

Hello,

I am not sure how exactly the Review queue works, but a nice feature would be able to have the ability to control which labels are reviewed by humans. Is there a way to change the review state of a data row via SDK or GraphQL query such that it is removed from the review queue?

For instance, if we have 100% coverage on Review Step and then could utilize the ‘label-created webhook’ to kick off a model-execution for each label to compute some sort of compliance score against the human label. If this compliance score is above some threshold we would remove it from the review queue.
If not, this label would be ‘flagged’ in a sense, and remain in the review queue for human eyes.

Automating the picking of what is manually reviewed vs. not by Labelbox labelers could be helpful to expensive labeling tasks requiring lots of manual oversight.

Thanks

2

Hello Chris!

There is a way to create reviews through both the SDK and GraphQL, I will be listing both below. When you review a label, that removes the label from the review queue and would not need a human to review that label since it is not queued up for any human reviewer to look at.

SDK: (The recommended method for using in a script or pipeline)
Label.create_review()

GraphQL: (Good to use for one-off reviews or if you would like to create reviews from the GQL explorer tab)

mutation createReview {
  createReview(data: {
    label: {
      connect: {
        id: "INSERT LABEL ID"
      }
    },
    project: {
      connect: {
        id: "INSERT PROJECT ID"
      }
    },
    score: -1
  }){
    id
    score
  }
  
}

I hope you found this helpful!

1 Like
3

Thank you, this is exactly what I was looking for!