Specifying a default Store:
The storage option to use in Dataverse v.4.18.1 is configured through Glassfish jvm options. The primary choice is made through specifying the storage-driver-id, which can currently be file, s3, or swift, e.g.
-Ddataverse.files.storage-driver-id=s3
In the multistore version of Dataverse, multiple stores can be active at the same time. However, one still needs to be designated as the default. To do this, it uses the same jvm option, but the values are now allowed to specify the id of any configured store (versus just the three values above which only define the store ‘type’), e.g.
-Ddataverse.files.storage-driver-id=s3tacc
Configuring a specific Store:
In Dataverse 4.18.1, the three types of stores are configured through several type-specific jvm options. The simplest of these is a file store that is defined by the path where files should be stored, e.g.
-Ddataverse.files.file.directory=/usr/local/glassfish4/glassfish/domains/domain1/files
In the multistore version, all stores must have an id, type, and label, as well as type-specific information. Thus a file store can be configured using:
-Ddataverse.files.file.type=file
-Ddataverse.files.file.label=LocalFileSystem
-Ddataverse.files.file.directory=/usr/local/glassfish4/glassfish/domains/domain1/files
where the id ('file' in this case) is implicit in the names of the jvm options. A second file store with the id ‘testing’ could be set up as:
-Ddataverse.files.testing.type=file
-Ddataverse.files.testing.label=TestingOnly
-Ddataverse.files.testing.directory=/tmp
Other types of stores are similarly configured. As with file store configuration, the existing jvm option choices are replaced with similar ones that specify the id of a specific store. For example, the s3 option to configure the bucket was:
-Ddataverse.files.s3-bucket-name=dataverse-dev-tacc-s3
It now includes the id, e.g.
-Ddataverse.files.s3tacc.bucket-name=dataverse-dev-tacc-s3
A complete s3 configuration would be:
-Ddataverse.files.s3.type=s3
-Ddataverse.files.s3.profile=s3
-Ddataverse.files.s3.label=TDL
-Ddataverse.files.s3.bucket-name=dataverse-dev-s3
A more complex configuration would be:
-Ddataverse.files.s3tacc.type=s3
-Ddataverse.files.s3tacc.label=TACC
-Ddataverse.files.s3tacc.custom-endpoint-url=http://129.114.52.102:9006
-Ddataverse.files.s3tacc.path-style-access=true
-Ddataverse.files.s3tacc.profile=s3tacc
-Ddataverse.files.s3tacc.upload-redirect=true
-Ddataverse.files.s3tacc.proxy-url=https://18.211.108.182:8888/s3tacc
In general, configurations for s3 and swift stores can include all of the existing options described in the Dataverse manuals, with the analogous modification to those shown to include a store specific identifier in the option.
The complex example above includes 3 new options implemented as part of the TDL large data project:
-Ddataverse.files.<id>.profile - this option specifies an aws profile to use with the specific store. AWS configuration files allow multiple profiles to be defined, each with it’s own username/password. This option links a Dataverse S3 store with a specific AWS profile.
-Ddataverse.files.<id>.upload-redirect - specifying true configures an S3 store to support direct upload of files to the store, versus streaming files through the Dataverse (Glassfish) server to a temporary location before sending it to the specified store. As described elsewhere, this option affects both upload via the Dataverse API and the Dataverse web interface.
-Ddataverse.files.s3tacc.proxy-url - s3 stores can be configured to use a proxy (such as nginx) for direct file uploads when the s3 endpoint itself is firewalled. This option was created to support testing, but could be used in production if desired (potentially slowing uploads).