Synchronizing data between campus and XSEDE

Some research projects—particularly those with team members at multiple campuses—need to maintain copies of the project's data on a campus system and on an XSEDE system. (This need was recognized and documented by XSEDE's Campus Bridging initiative.) Keeping these copies synchronized is important, but it shouldn't be time-consuming for researchers. Cloud synchronization services like Dropbox and Google Drive aren't always practical for research data at supercomputing scale. A new feature of XSEDE's Globus data management service, "Timers," makes synchronizing research data easy. Globus first described this feature in a blog post last November.

XSEDE's description of this feature and instructions for use with XSEDE are on the Data Management page in the XSEDE User Portal. (Look under "Data Synchronization" at the end of the page for details.) In summary, you can now create a timer task in Globus to run a synchronization transfer (synchronization transfers only transfers files that are newer or different on the source) at a frequency you specify. You can set up unidirectional or bidirectional synchronization, and you can configure an end date or a maximum number of synchronization runs. With this new feature, you can keep copies of your project's data--on XSEDE and one or more campuses--up-to-date as your team adds to or updates the data.