DPLA Harvest Process
- Elliot Williams (Unlicensed)
- Courtney C. Mumma
TDL sends metadata from our members to DPLA quarterly, i.e. every three months. Currently, our harvests are scheduled for January, April, July, and October.
Please note that TDL DPLA onboards new members to TXHub, adds new collections to your institutional harvest, and makes mapping changes during only two of the four quarterly harvest periods – April and October.
The process of sharing metadata from your repository to DPLA involves several steps:
1. Harvesting - TDL downloads metadata from your repository through a process called “harvesting”. In most cases, we do this using OAI-PMH, a standard method for sharing metadata in XML format.
Most common digital repository systems include OAI-PMH harvesting, including DSpace, CONTENTdm, Islandora, and others. TDL staff can help you determine if your system supports OAI-PMH.
After you set up an OAI-PMH feed in your repository and give TDL the information about how to access it, we can download your metadata automatically every quarter.
TDL will only harvest metadata for collections you want to share with DPLA. You can share all of your materials, or only selected collections. Before each harvest, we will contact you and ask for any new collections you would like to include.
Each quarter, TDL re-harvests every collection shared by your institution. That way, we get any changes to the collection, such as updated metadata and added or deleted items.
If you expect a major change to your metadata or repository system (e.g. migrating systems, changes to metadata schema, a new OAI-PMH endpoint), please let TDL know at least two months before the next quarterly harvest (but ideally as soon as possible).
2. Mapping - Because it collects metadata from a wide range of institutions, DPLA specifies a particular metadata structure and format. After harvesting, TDL transforms your metadata (or “maps” it) to the format that DPLA expects. Standardizing metadata to the DPLA format means that your resources will be more discoverable within the DPLA portal.
When you join the TDL DPLA aggregation service, we will work with you to establish a metadata mapping that fits your existing metadata, based on the TDL’s metadata guidelines. Together, we will look at your metadata elements and how you use them, and determine how they fit into the DPLA metadata application profile.
We try to use the same mapping for all collections from an individual institution. However, we understand that some collections have unique metadata needs, and we can account for those within your institution’s mapping.
Once it is created, we will use the same metadata mapping every quarter to transform your metadata before sending it to DPLA. However, if your metadata structure changes, or new collections require tweaks to the mapping, we can update it as necessary.
For more information about metadata requirements and mappings, see the TDL DPLA Metadata Guidelines.
3. Ingest - TDL delivers the mapped metadata from all of our contributing members to DPLA in one package. DPLA then collects that metadata, runs some processes to validate and enrich the metadata, and ingests it into their systems. After those processes are complete, and the search index is updated, your records are available in the DPLA portal.