I have a set of conversations and need to assign all the data rows to both of my labelers so I can compare the annotations from both labelers. It is currently doing first come first serve and only allowing one labeler to label a data row. I want both to be able to annotate all of the data rows and be able to compare annotations for all data rows.
Hi @carlos, please see our documentation on consensus, which is a feature used for quality analysis.
Using consensus, you can set all data rows to be labeled whatever number of times you desire (in your case, it would match the number of labelers you have working on the project).
1 Like
Hi @Zeke I tried consensus and selected the number of labelers I was going to use. Did it with the Python Sdk and once one of the labelers started annotating, the number of data rows to label for the second labeled started to decrease.
@carlos Can you please share with me the project ID of the project in question?
Did you ensure that you selected 100% coverage for the batch in question and matched the number of labels to the number of people who would be creating labels in the project?
1 Like