APIs - Introduction to Dataverse APIs

Introduction to Dataverse APIs

Who is this guide designed for?

This guide is intended for researchers using the Texas Data Repository (TDR) to search for, access, or publish data. It can also benefit librarians and other administrative staff at TDR member institutions who are interested in learning about the use of Dataverse APIs. The guide is structured to provide an introduction to the use of scripted approaches for automating TDR data management workflows.

What is Dataverse?

Dataverse is an open source research data repository software that runs institutional data repositories such as the Texas Data Repository and is developed by an active community of developers at institutions around the world. For the purposes of this guide, we will typically be using Dataverse to refer to the open source software project and its APIs and use Dataverse collection to refer to a collection of datasets within our TDR instance of Dataverse.

How can data be organized in a Dataverse instance (like the Texas Data Repository)?

Open

Understanding how data is organized within TDR is important for successful use of the Dataverse APIs for accessing and managing data in the Texas Data Repository. 

A Dataverse collection is a container for datasets (research data, code, documentation, and metadata) and other nested Dataverse collections, which can be set up for individual researchers, departments, journals, and organizations.

Each Dataverse collection contains datasets. Each dataset should include data files and metadata that describes the data files. Each dataset may also include code associated with the data files and additional documentation describing the data files and project from which they derive. 

For more information, see the TDR User Guide. 

What is an API?

In very simple terms, an API is a messenger that lets two programs talk to each other (i.e., you ask for something, and it gets/does it for you). More specifically, an API, or an Application Programming Interface, defines how an end user can write code to interact with an external software application. APIs are routinely used behind the scenes in everyday settings, such as allowing third-party log-in to a website (e.g., using your SSO or Google account).  

The Dataverse software that TDR runs on has several defined APIs that can be utilized to send instructions to our Dataverse instance for operations such as uploading files or requesting data. These Dataverse APIs are REST APIs that allow instructions to be sent programmatically using HTTP (hypertext transfer protocol) methods and properly formatted URLs that define the end points for Dataverse resources.The API endpoints for the Dataverse APIs that can be used to interact with TDR are defined in the Dataverse API Documentation. These API endpoints can be used in URLs that you utilize in a web browser, but are more typically called by scripted processes, like Python scripts, that you might create to automate certain tasks.

Why is using an API helpful?

Using APIs and scripted processes to carry out operations can provide several benefits, compared to relying on a graphical user interface, like the TDR website. Scripted processes that use API calls can be more efficient for executing a large number of operations, can ensure consistency, and can be easily modified then re-run if adjustments need to be made. They can also circumvent potential limitations of a web interface, which might have more stringent download/upload thresholds, for example. 

Obtaining an API token

APIs can be public, semi-public, or private; allowing all users or only some users to use the API. This can be done to protect access to sensitive data or processes that are accessible via an API, or to prevent users from overwhelming a system through overuse of an API. The Dataverse instance on which TDR runs requires use of an API token (also sometimes referred to as an API key) for many operations so that the system can identify the user that is using the API to submit a request. You can create your API token by logging in to TDR, clicking on your account name at the top right of your TDR window, selecting "API Token" from the drop down menu that appears, and then clicking "Create Token". 

Open

Open

This process will create a new Dataverse API token that is specific to your account in the Texas Data Repository. Your token should look like a random string of letters and numbers like c6527048-5bdc-48b0-a1d5-ed1b62c8113b that you can copy and paste into Python scripts that make use of Dataverse APIs, or you can utilize your API token with certain tools which rely on the Dataverse APIs (like the Python DVUploader tool). 

As you start using your TDR API token it is important to consider the following:

• Your token is sensitive because anyone who knows the token can utilize Dataverse API calls that access your TDR account and data so you should treat your token like a password and not share it with others. 

• Take care to avoid accidentally including your API token in scripts that you share on GitHub. (GitHub is a cloud-based platform that acts as a central repository for storing, managing, and sharing code projects).

• Dataverse API tokens automatically expire after 1 year from the date of generation; so remember to annually create a new token.

• If you notice your older scripts do not seem to be working, make sure that the API token being used has not expired. 

• You can also revoke your existing token and generate a new one at any point if it is accidentally shared or getting close to expiration.

Scripted and command line approaches for working with a Dataverse installation (Python and cURL)

Using your API token, you can now access Dataverse API endpoints that allow data management operations to be carried out in TDR. An API endpoint is a specific URL where an application running on a server receives requests and sends responses. It serves as the point of interaction between two programs (like a Python script running on your computer and Dataverse software on the TDR server), allowing them to communicate and exchange data. In order to know which endpoints are available (and what operations you can carry out using Dataverse APIs you will need to read the Dataverse API documentation at https://guides.dataverse.org/en/latest/api/intro.html#list-of-dataverse-apis . You can send instructions using these API endpoints through the command line interface on your computer using cURL or by running scripts that are written in a language like Python.

cURL (which stands for client URL) is a command-line tool and library used for transferring data to or from a server using a variety of protocols like HTTP. It is commonly used by developers and system administrators for tasks such as making API requests, downloading files, and automating web-related functions. The "curl" command in a terminal is a client that uses the libcurl library to perform these data transfers without user interaction, unlike a web browser. 

If you are using the command line interface, like the Terminal on a Mac or Command Prompt on a Windows computer, you will use the cURL command line tool, as in the example below, which can be used to list the files that are in a published dataset. While cURL might be useful in certain scenarios if a user is already used to working with command line tools, in most cases, users might find it more helpful to make API calls using a scripted process written in Python. You will find that example commands for Dataverse API endpoints are typically provided in cURL command format in the Dataverse user guide, and these commands will need to be adjusted to be called within a Python script. You can use a tool like https://curlconverter.com/ to help you with the conversion process or you can manually convert the cURL command to Python code if you are comfortable making that adjustment (see example of how code varies below).

cURL command example from the Dataverse documentation:

curl "https://demo.dataverse.org/api/datasets/24/versions/1.0/files"

Python command equivalent:

import requests

response = requests.get('https://demo.dataverse.org/api/datasets/24/versions/1.0/files') 

Moving forward on this guide page, we will focus on the use of Python, since that is the more common approach for working with the Dataverse APIs. The API endpoints for Dataverse are grouped into different Dataverse APIs depending on their functions. While the endpoints are grouped under different API names like Search, Data Access, and Metrics, all of the API endpoints can be used in very similar ways. These groupings can make it easy for you to search for the particular endpoint that you are interested in based on the type of task you are looking to automate. For example, if you would like to use a scripted process for finding published data in TDR you would want to look at the API endpoints under the Dataverse Search API. 

It is important to note that not all API endpoints require an API token. For example, API endpoints that provide publicly available data do not require an API token. However, endpoints under certain Dataverse APIs, such as the Data Access API, do typically require a token in your script. Below is an example of an endpoint that does require an API token. This example consists of Python code that can be used to compare versions of a published dataset:

import requests

response = requests.get('https://dataverse.tdl.org/api/dataverses/utexas/contents')

print(response.text)

You can run this code by creating a new Python script on your computer and running it locally, or, if you do not have Python set up yet on your machine, you could simply run this code in the Google Colab environment. To try running the code in Colab, first sign in with your Google account and then visit https://colab.research.google.com/ , click "+ new notebook", paste the code above into the code cell at the top of the new notebook where it says "start coding…", and then click the play/run button next to the code block. When the code runs you should see a list of all of the contents in the UT Austin (utexas) Dataverse collection. 

Open

To see how easily the code can be adapted, you could try replacing "utexas" in the codeblock above with "baylor" to instead retrieve a list of contents from the Baylor University Dataverse collection.

If you are trying to utilize an API endpoint that requires authentication it is important to make sure that you include your TDR API token. The example code below can also be run in Google Colab, but in order for this code to work you will need to replace the 'xxxxxxxx-xxxx-xxxxxxx-xxxxxxx' place holder text with your actual TDR API token. Make sure that you leave the quotation marks around your API token when you paste it in to ensure that it is read by the Python interpreter as a string object. If done successfully, this code will show you a list of all content in TDR that includes mention of "trees". If you do not enter your API token successfully in the code snippet below, when you run the Python script it will produce the following error: {"status":"ERROR","message":"Bad API key"}.

import requests

headers = {'X-Dataverse-key': 'xxxxxxxx-xxxx-xxxxxxx-xxxxxxx'}

response = requests.get('https://dataverse.tdl.org/api/search?q=trees', headers=headers)

print(response.text)

Next steps for utilizing the Dataverse APIs

Now that you understand the basics of utilizing the Dataverse APIs to interact with the Texas Data Repository, you can explore using these APIs in your workflows. You can consider creating Python scripts locally using IDE software like VSCode on your computer instead of utilizing Google Colab to run your Python processes and automate TDR workflows. 

You can also explore other potentially useful Dataverse API endpoints like those to help automate:

• File uploads to a dataset

• File downloads from a dataset

• Use of metrics API for tracking dataset downloads (see https://guides.dataverse.org/en/latest/api/metrics.html for ideas)