Checkbox JSON output format

[btalberg]

We’ve created a video job with a single-option checklist used to identify work periods (the person in our video is either “working” or not working). When a work period is NOT occurring, the labeler unchecks or deselects the “working” checkbox. So essentially, we’ve created a simple checkbox toggle labeling task.

From the frontend viewer, marking and viewing work periods is incredibly intuitive.

When I look at the export_v2 JSON output though, it isn’t easy to differentiate selected/checked periods from deselected/unchecked periods. Each state is shown with a perfectly identical data signature.

Am I missing something? Shouldn’t there be a clear way of differentiating a checkbox annotation that’s in a checked/selected state from a deselected state?

Here’s a snapshot of how the toggle checklist appears to the labeler. Below that I’ve included the JSON output from the Python SDK’s export_v2 method. Refer to the JSON’s annotations section to see how the output is perfectly identical between checked and not checked checkbox annotations:

SCREENSHOT

JSON:

[
  {
    "data_row": {
      "id": "cljx1k5ok2as3074kaygcclev",
      "external_id": "3cbbcfec-b008-4963-ab81-0a635809608b",
      "global_key": "3cbbcfec-b008-4963-ab81-0a635809608b",
      "row_data": "HIDDEN",
      "details": {
        "dataset_id": "cljt1eve5058007xmcpn11ajk",
        "created_at": "2023-07-10T15:51:54.000+00:00",
        "updated_at": "2023-07-10T15:51:54.000+00:00",
        "last_activity_at": "2023-07-10T16:35:14.000+00:00",
        "created_by": "HIDDEN"
      }
    },
    "media_attributes": {
      "height": 972,
      "width": 1296,
      "mime_type": "video/mp4"
    },
    "attachments": [],
    "metadata_fields": ["HIDDEN"],
    "projects": {
      "cljx29jh8023y07y6amb08lf0": {
        "name": "Identify work periods (testing)",
        "labels": [
          {
            "label_kind": "Video",
            "version": "1.0.0",
            "id": "cljx2aqwk06gi07x1440efvb2",
            "label_details": {
              "created_at": "2023-07-10T16:12:49.000+00:00",
              "updated_at": "2023-07-10T16:35:14.000+00:00",
              "created_by": "HIDDEN",
              "reviews": []
            },
            "performance_details": {
              "seconds_to_create": 10,
              "seconds_to_review": 77,
              "skipped": False
            },
            "annotations": {
              "frames": {
                "34": {
                  "objects": {},
                  "classifications": [
                    {
                      "feature_id": "cljx2avh000013b6ljtot65ru",
                      "feature_schema_id": "cljx28zjk06es07x140tp0bfi",
                      "name": "Work period (checkbox)",
                      "checklist_answers": [
                        {
                          "feature_id": "cljx2avh000023b6lc7xj6prb",
                          "feature_schema_id": "cljx28zjk06et07x13hx96xei",
                          "name": "Work period",
                          "classifications": []
                        }
                      ]
                    }
                  ]
                },
                "249": {
                  "objects": {},
                  "classifications": [
                    {
                      "feature_id": "cljx2avh000013b6ljtot65ru",
                      "feature_schema_id": "cljx28zjk06es07x140tp0bfi",
                      "name": "Work period (checkbox)",
                      "checklist_answers": [
                        {
                          "feature_id": "cljx2avh000023b6lc7xj6prb",
                          "feature_schema_id": "cljx28zjk06et07x13hx96xei",
                          "name": "Work period",
                          "classifications": []
                        }
                      ]
                    }
                  ]
                },
                "259": {
                  "objects": {},
                  "classifications": [
                    {
                      "feature_id": "cljx2avh000013b6ljtot65ru",
                      "feature_schema_id": "cljx28zjk06es07x140tp0bfi",
                      "name": "Work period (checkbox)",
                      "checklist_answers": [
                        {
                          "feature_id": "cljx2avh000023b6lc7xj6prb",
                          "feature_schema_id": "cljx28zjk06et07x13hx96xei",
                          "name": "Work period",
                          "classifications": []
                        }
                      ]
                    }
                  ]
                },
                "454": {
                  "objects": {},
                  "classifications": [
                    {
                      "feature_id": "cljx2avh000013b6ljtot65ru",
                      "feature_schema_id": "cljx28zjk06es07x140tp0bfi",
                      "name": "Work period (checkbox)",
                      "checklist_answers": [
                        {
                          "feature_id": "cljx2avh000023b6lc7xj6prb",
                          "feature_schema_id": "cljx28zjk06et07x13hx96xei",
                          "name": "Work period",
                          "classifications": []
                        }
                      ]
                    }
                  ]
                }
              },
              "segments": {
                "cljx2avh000013b6ljtot65ru": [
                  [
                    34,
                    34
                  ],
                  [
                    249,
                    249
                  ],
                  [
                    259,
                    259
                  ],
                  [
                    454,
                    454
                  ]
                ]
              },
              "key_frame_feature_map": {
                "cljx2avh000013b6ljtot65ru": [
                  34,
                  249,
                  259,
                  454
                ]
              },
              "classifications": []
            }
          }
        ],
        "project_details": {
          "ontology_id": "cljx28zik06er07x16cfcdbg1",
          "task_id": "e1586c90-02d1-4e70-98f7-a7508fa82b0d",
          "task_name": "Initial review task",
          "batch_id": "daf96b40-1f39-11ee-97a5-1b81de52e19b",
          "priority": 1,
          "consensus_expected_label_count": 1,
          "workflow_history": [
            {
              "action": "Move",
              "created_at": "2023-07-10T16:12:49.094+00:00",
              "created_by": "HIDDEN",
              "previous_task_name": "Initial labeling task",
              "previous_task_id": "b98d4a78-4956-04dc-bae8-3f4438b8d2c3",
              "next_task_name": "Initial review task",
              "next_task_id": "e1586c90-02d1-4e70-98f7-a7508fa82b0d"
            },
            {
              "action": "Move",
              "created_at": "2023-07-10T16:12:49.070+00:00",
              "created_by": "HIDDEN",
              "next_task_name": "Initial labeling task",
              "next_task_id": "b98d4a78-4956-04dc-bae8-3f4438b8d2c3"
            }
          ]
        }
      }
    }
  }
]

[Zeke]

Hi @btalberg, in the export, you see just the keyframes (in other words, where an annotation was modified). Therefore, using your current setup and the above export, you can devise the start and end point of when the checkbox was selected. For example, frames 34-249 and 259-454 are the segments in which the checkbox is selected for the video in question.

Part of the challenge here, though, is due to the use of a checkbox classification rather than a radio classification. Checkboxes are designed for multi-choice classification tasks (multiple selections at any given moment), whereas radios are explicitly designed for single-choice classification (yes/no, in this case). I would recommend that you explore using a radio classification for this labeling task that asks the question of if work is occurring and has yes and no options. This should involve the same amount of clicks and will produce much clearer data in the export.

Please note that if you have already labeled some data rows with the checklist classification, you must create a different classification rather than converting the existing one to a radio.

[btalberg]

Got it. Thanks for the clarity, @Zeke. I actually started with a single-option radio classification before I tried the checkbox option. Because it was single-option though, the output also lacked the information I was looking for. I’m pushing on this single-option approach because my goal was to avoid having our labelers explicitly mark “no”. From our testing, asking our labelers to only mark the “yes” moments (yes = work occurring) simplified the task. But for the sake of clarity in the data output, I suppose I’ll move back to an explicit yes/no radio classification as you suggest.