Uploading local files JSONDecodeError

Using Python SDK, dataset.create_data_rows(image_rows) works so long as I have less than 20 or so rows, but for any more than that I consistently get the following error:

Traceback (most recent call last):                                                                                                                 
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/requests/models.py", line 971, in json                         
    return complexjson.loads(self.text, **kwargs)                                                                                                  
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads                                                                                  
    return _default_decoder.decode(s)                                                                                                              
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode                                                                                  
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())                                                                                              
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode                                                                              
    raise JSONDecodeError("Expecting value", s, err.value) from None                                                                               
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)                                                                            
                                                                                                                                                   
During handling of the above exception, another exception occurred:                                                                                
                                                                                                                                                   
Traceback (most recent call last):                                                                                                                 
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/labelbox/client.py", line 391, in upload_data                  
    file_data = response.json().get("data", None)                                                                                                  
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/requests/models.py", line 975, in json                         
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)                                                                                             
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)                                                                     
                                                                                                                                                   
During handling of the above exception, another exception occurred:                                                                                
                                                                                                                                                   
Traceback (most recent call last):                                                                                                                 
  File "/home/coral/coralnet-collector/import-labelbox.py", line 178, in <module>                                                        
    main()                                                                                                                                         
  File "/home/coral/coralnet-collector/import-labelbox.py", line 170, in main                                                            
    upload_images(image_points_dict, args.dataset, ignore_names, token)                                                                            
  File "/home/coral/coralnet-collector/import-labelbox.py", line 86, in upload_images                                                    
    task = dataset.create_data_rows(image_rows)                                                                                                    
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/labelbox/schema/dataset.py", line 205, in create_data_rows     
    descriptor_url = self._create_descriptor_file(items)                                                                                           
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/labelbox/schema/dataset.py", line 433, in _create_descriptor_fi
le                                                                                                                                                 
    items = [future.result() for future in as_completed(futures)]
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/labelbox/schema/dataset.py", line 433, in <listcomp>
    items = [future.result() for future in as_completed(futures)]
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()                                                                                                                     
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/labelbox/schema/dataset.py", line 418, in convert_item
    item = upload_if_necessary(item) 
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/labelbox/schema/dataset.py", line 293, in upload_if_necessary
    item_url = self.client.upload_file(row_data)
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/labelbox/client.py", line 331, in upload_file
    return self.upload_data(content=f.read(),
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
    return retry_target(
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
  File "/home/coral/coralnet-collector/.venv/lib/python3.10/site-packages/labelbox/client.py", line 393, in upload_data
    raise labelbox.exceptions.LabelboxError(
labelbox.exceptions.LabelboxError: Failed to upload, unknown cause('Failed to upload, unknown cause', JSONDecodeError('Expecting value: line 1 colu
mn 1 (char 0)'))

hey Nathan! this function should work on chunks of 150,000 data rows.
do you mind sharing the image_rows payload you are using?
i suspect it has some formatting issue

Sure, I’ve tried two types of payloads. I’ve done local path only:

[
    "/data/coralnet/3412/images/2691056.jpg",
    "/data/coralnet/3412/images/2691070.jpg",
    "/data/coralnet/3412/images/2691078.jpg",
    "/data/coralnet/3412/images/2691080.jpg",
    "/data/coralnet/3412/images/2691082.jpg",
    "/data/coralnet/3412/images/2691091.jpg",
    "/data/coralnet/3412/images/2691096.jpg",
    "/data/coralnet/3412/images/2691099.jpg",
    "/data/coralnet/3412/images/2691101.jpg",
    "/data/coralnet/3412/images/2691104.jpg",
    "/data/coralnet/3412/images/2691116.jpg",
    "/data/coralnet/3412/images/2691123.jpg"
]

I’ve also done objects with metadata:

[
    {
        "row_data": "/data/coralnet/3412/images/2691056.jpg",
        "global_key": "2691056",
        "media_type": "IMAGE"
    },
    {
        "row_data": "/data/coralnet/3412/images/2691070.jpg",
        "global_key": "2691070",
        "media_type": "IMAGE"
    },
    {
        "row_data": "/data/coralnet/3412/images/2691078.jpg",
        "global_key": "2691078",
        "media_type": "IMAGE"
    },
    {
        "row_data": "/data/coralnet/3412/images/2691080.jpg",
        "global_key": "2691080",
        "media_type": "IMAGE"
    },
    {
        "row_data": "/data/coralnet/3412/images/2691082.jpg",
        "global_key": "2691082",
        "media_type": "IMAGE"
    },
    {
        "row_data": "/data/coralnet/3412/images/2691091.jpg",
        "global_key": "2691091",
        "media_type": "IMAGE"
    }
]

Both work with small numbers, but fail with larger numbers of rows.