DuraSync(web interface tool) DuraSync(command-line tool) - The launch command to get it started changes from a simple java -jar duracloudsync-6.1.0.jar to a more complicated one that requires arguments:
- java -jar duracloudsync-6.1.0.jar -h dcloud.tdl.org -r 443 -u USER -i 1 -s SPACE_NAME -c DIRECTORIES_TO_MONITOR
- the -i 1 specifies to push to the #1 storage provider (in our
- case S3)... you can use -i 0 for S3 or -i 2 for Glacier
- the -s SPACE_NAME needs to be a space that you have access to on the
- S3 storage provider (and not "SPACE_NAME")
- the -u USER needs to be your duracloud username (and not "USER")
- the -c DIRECTORIES_TO_MONITOR is a list (separated by a space) of directories (ex. -c directory1 directory2) to have the sync tool monitor for changes
- The tool comes with a "help" usage menu when passed the --help argument (where I got the previous definitions of parameters from) in case you need a more custom configuration
- java -jar duracloudsync-6.1.0.jar -h dcloud.tdl.org -r 443 -u USER -i 1 -s SPACE_NAME -c DIRECTORIES_TO_MONITOR
- The DuraSync web interface that runs on http://localhost:8888/sync will not be accessible
- If the tool was previously configured using the command-line arguments, it will store its configuration file by default to $HOME/duracloud-sync-work in a file called synctool.config
- java -jar duracloudsync-6.1.0.jar -g $HOME/duracloud-sync-work/synctool.config
- If this file is to be re-used to launch the tool from the command line again, the -g option can be given at runtime to launch the tool and not have to pass it all the arguments in the command line again:
java -jar duracloudsync-6.1.0.jar -g $HOME/duracloud-sync-work/synctool.config
- Once launched, it will ask for your DuraCloud password and being synchronizing your local directories to the SPACE_NAME and storage provider specified with -i #
DuraCloud Retrieval Tool(command-line tool) - URL for downloading the java tool from us: https://s3.amazonaws.com/tdl-duracloud.utilities/retrievaltool-6.1.0-driver.jar
-
- The launch command to get it started is requires arguments:
java -jar retrievaltool-6.1.0-driver.jar -h dcloud.tdl.org -r 443 -i 1 -s SPACE_NAME -u USER -c LOCAL_CONTENT_DIR -w WORK_DIR -t NUM_THREADS- the -i 1 specifies to pull from the #1 storage provider (in our case
- S3)... you can use -i 0 for S3 or -i 2 for Glacier
- the -s SPACE_NAME needs to be a space that you have access to on the
- S3 storage provider (and not "SPACE_NAME")
- the -u USER needs to be your duracloud username (and not "USER")
- the -c LOCAL_CONTENT_DIR is the local directory where the retrieved content will be stored (and not "LOCAL_CONTENT_DIR")
- the -w WORK_DIR is a directory for the tool to use to store its log files and status files (and not "WORK_DIR")
- the -t NUM_THREADS is the number of concurrent download threads the retrieval tool should use (defaults to 3 threads if not specified as an argument)
- Testing with multiple threads with iRods has a caveat:
- Try not to press Control-C to cancel the retrieval... it will cancel the client-side only and the server will continue to pull the existing files out of iRods.
I did this like 5 times with 9 threads each... meaning there were 45 connections going on on the server side to the iRods server and it compounded lag to the point that iRods was pretty much getting hammered by Amazon and my retrieval stalled. I had to restart the DuraCloud server to get it to come back.- The tool comes with a "help" usage menu when passed the --help argument (where I got the previous definitions of parameters from) in case you need a more custom configuration
- java -jar retrievaltool-6.1.0-driver.jar -h dcloud.tdl.org -r 443 -i 1 -s SPACE_NAME -u USER -c LOCAL_CONTENT_DIR -w WORK_DIR -t NUM_THREADS
- Once launched, it will ask for your DuraCloud password and begin synchronizing your remote SPACE_NAME from the storage provider specified with -i # down to the LOCAL_CONTENT_DIR specified in the parameters
|