I am not sure how exactly the Review queue works, but a nice feature would be able to have the ability to control which labels are reviewed by humans. Is there a way to change the
review state of a data row via SDK or GraphQL query such that it is removed from the review queue?
For instance, if we have 100% coverage on Review Step and then could utilize the ‘label-created webhook’ to kick off a model-execution for each label to compute some sort of compliance score against the human label. If this compliance score is above some threshold we would remove it from the review queue.
If not, this label would be ‘flagged’ in a sense, and remain in the review queue for human eyes.
Automating the picking of what is manually reviewed vs. not by Labelbox labelers could be helpful to expensive labeling tasks requiring lots of manual oversight.