Hello Labelbox Community! 
Let’s dive into the fun stuff and figure out whether Consensus or Benchmark is the right choice for your project’s quality settings 
Labelbox offers two robust tools designed to evaluate annotation quality. Both aim to provide insights into the consistency and accuracy of annotations across projects, but they operate differently and serve distinct purposes.
Benchmarks 
- Purpose: Benchmarks are designed to establish a “gold standard” label, which are compared with other labels. They are particularly useful for identifying high-quality annotations and understanding the variability in annotation practices among labelers.
- Calculation: The benchmark score is calculated by averaging the scores for each annotation class (e.g., object-type and classification-type) to create an overall score for the asset. Each annotation class is weighted equally. This score serves as an initial indicator of label quality, the clarity of your ontology, and/or the clarity of your labeling instructions.
- Usage: Benchmarks are beneficial for setting quality standards and comparing the performance of labelers against established criteria. However, they require a designated “benchmark” label to calculate agreement scores effectively.
Consensus 
- Purpose: Consensus aims to identify the most agreed-upon annotations among all labelers working on a dataset. It’s about finding common ground and reducing variability in annotations.
- Calculation: Similar to benchmarks, consensus scores are also calculated by averaging the scores for each annotation class. However, consensus focuses on the agreement among all annotations.
- Usage: Consensus is useful for real-time quality analysis and immediate corrective actions towards improving training data and model performance. It provides a dynamic measure of annotation quality that adjusts as new annotations are added or existing ones are modified.
Key Differences
- Gold Standard vs. Agreement: Benchmarks rely on a predefined gold standard, whereas consensus seeks to find the most commonly agreed upon annotations among all labelers.
- Flexibility: Consensus operates in real-time and adjusts as new annotations are made, offering a more fluid approach to quality assessment. Benchmarks, on the other hand, are static once set and do not adjust with new annotations.
- Application: Benchmarks are ideal for establishing baseline quality standards and evaluating labeler performance over time. Consensus is better suited for ongoing quality monitoring and immediate feedback on annotation practices.
Both features play crucial roles in managing annotation quality. Benchmarks provide a fixed point of comparison, while consensus offers a dynamic measure of agreement among labelers. Depending on the specific needs of your project—whether it’s setting quality standards or continuously monitoring annotation quality—you may find one method more suitable than the other or even use them in combination for comprehensive quality assurance. 