Uploading Files to Bisque via Python

In this blog post, we discuss how to quickly upload a dataset to Bisque using python. The next blog post will then talk about how to convert binary masks (made in matlab) to annotations usable by Bisque for validation or modification.

We use Bisque to share and annotate slides with our clinicians for two main reasons:

  1. Its usage of bio-formats and openslide allows it to function with all of the file formats we use, (e.g., svs, big-tiff, tif, png, jpg, etc.) in a single interface. This means a user need only learn one tool, as opposed to multiple proprietary tools.
  2. Its near real-time rendering of large (3GB+) files, obviates the need for the transfer of the large underlying files. This was typically done by either physically moving external tera-byte hard drives, or having a clinician come on-site to review/perform annotations. Bisque loads only pixels necessary for display on screen, and compresses them highly before transferring, such that the end user, regardless of physical location, can browse through 100k x 100k files seamlessly.

With that said, we’d like to upload files to Bisque and assign them to a dataset for easy viewing. Using the popular python library requests, which handles REST and form data brilliantly, this process becomes rather seamless.

There are a 3 steps that we take, which we will discuss in some detail:

  1. Upload the image using a POST on BASE/import/transfer, where BASE is the URL of the server.
  2. Extract URI of resource from response text of import, and save for later
  3. Create a Dataset using a POST and the xml formed with the URIs from Step 2.

 

Step 1: Uploading

To upload the file, we first create a dictionary which contains filename and we open a file descriptor to the file itself for uploading.

  1. files={'file':(fname,open(fname, "rb"))}

Afterwards, we use the requests library to produce a POST on import/transfer. Note we also need to authorize ourselves with our username and password:

  1. response=requests.post('%s/import/transfer'%BASE_URL,files=files,auth=(USER,PASSWD))
Step 2: URI Extraction

At this point the file is now uploaded. The response text contains the URI of the file, which we need later for assigning it to a dataset. We use a regular expression to grab the URI and add the file to the list of files to add to the dataset.

  1. uri_pattern = re.compile(ur'uri="(?P\S+)"')
  2. file_uri=re.findall(uri_pattern, response.text)
  3. dataset_files.append(file_uri[0])

For example, this is at typical response for uploading a file called 8951_01401.tif

  1. <resource uri="/ds/image?name=8951_01401.tif&amp;wpublic=None&amp;format=None&amp;tag_query=&amp;limit=1000000&amp;offset=0&amp;tag_order=&amp;view=short">
  2. <image created="2015-02-25T06:58:26" name="8951_01401.tif" owner="http://hawking.ebme.CWRU.Edu:8080/data_service/00-zFzVh6LJFDGHbEvkDyNEiS" permission="private" resource_uniq="00-zRJsRZfHvhdjao4XtseDVG" ts="2015-02-25T06:58:26" uri="http://hawking.ebme.CWRU.Edu:8080/data_service/00-zRJsRZfHvhdjao4XtseDVG" value="file://admin/8951_01401.tif"/>
  3. </resource>

And our code extracts the uri:

http://hawking.ebme.CWRU.Edu:8080/data_service/00-zRJsRZfHvhdjao4XtseDVG

Step 3: Dataset Creation

Now that we have all of the files uploaded, and their associated URIs, we need to submit properly formatted xml to the server to create the dataset. We do this by setting the beginning tag and then iterating over each file, adding its filename to the xml, finishing with the closing dataset tag:

  1. dataset_xml=''%DATASET_NAME
  2.  
  3. for i in xrange(0,len(dataset_files)):
  4.         dataset_xml+='%s' %(i,dataset_files[i])
  5.  
  6. dataset_xml+=''

Afterwards, we tell the server that the data its going to be seeing is XML by setting the content type. And then we run the post request:

  1. headers = {'content-type': 'text/xml'}
  2. response=requests.post('%s/data_service/dataset'%BASE_URL,auth=(USER,PASSWD),data=dataset_xml,headers=headers)

That’s all there is too it!
Here is our dataset which contains the images bundled:


dataset

And here are the images:


images
The actual script I use is a bit more full-features as it has the ability to set parameters via the command line (url, username, password, etc), the full source for that is available here

2 thoughts on “Uploading Files to Bisque via Python”

  1. Interesting post about how to access BisQue from a python script. How does one access an existing dataset on bisque? For example if I wanted to add annotations to an existing bisque dataset.

    Thanks

Leave a Reply

Your email address will not be published. Required fields are marked *