1.5. Remote Indexed Archives for dataset storage and backup

If DataLad datasets should be backed-up, made available for collaborations with others, or stored or managed in a central location, remote indexed archive (RIA) stores, dataset storage locations that allow for access to and collaboration on DataLad datasets, may be a suitable solution. They are flat, flexible, file-system based repository representations of any number of datasets, and they can exist on all standard computing infrastructure, be it personal computers, servers or compute clusters, or even super computing infrastructure – even on machines that do not have DataLad installed.

Note

Setting up and interacting with RIA stores requires DataLad version 0.13.0 or higher. In order to understand this section, some knowledge on Git-internals and overcoming any fear of how checksums and UUIDs look can be helpful.

1.5.1. Technical details

RIA stores can be created or extended with a single command from within any dataset. DataLad datasets can subsequently be published into the datastore as a means of backing up a dataset or creating a dataset sibling to collaborate on with others. Alternatively, datasets can be cloned and updated from a RIA store just as from any other dataset location. The subsection RIA store workflows a few paragraphs down will demonstrate RIA-store related functionality. But prior to introducing the user-facing commands, this section starts by explaining the layout and general concept of a RIA store.

1.5.1.1. Layout

RIA stores store DataLad datasets. Both the layout of the RIA store and the layout of the datasets in the RIA store are different from typical dataset layouts, though. If one were to take a look inside of a RIA store as it is set up by default, one would see a directory that contains a flat subdirectory tree with datasets represented as bare Git repositories and an annex. Usually, looking inside of RIA stores is not necessary for RIA-related workflows, but it can help to grasp the concept of these stores.

The first level of subdirectories in this RIA store tree consists of the first three characters of the dataset IDs of the datasets that lie in the store, and the second level of subdatasets contains the remaining characters of the dataset IDs. Thus, the first two levels of subdirectories in the tree are split dataset IDs of the datasets that are stored in them1. The code block below illustrates how a single DataLad dataset looks like in a RIA store, and the dataset ID of the dataset (946e8cac-432b-11ea-aac8-f0d5bf7b5561) is highlighted:

 /path/to/my_riastore
 ├── 946
 │   └── e8cac-432b-11ea-aac8-f0d5bf7b5561
 │       ├── annex
 │       │   └── objects
 │       │       ├── 6q
 │       │       │   └── mZ
 │       │       │       └── MD5E-s93567133--7c93fc5d0b5f197ae8a02e5a89954bc8.nii.gz
 │       │       │           └── MD5E-s93567133--7c93fc5d0b5f197ae8a02e5a89954bc8.nii.gz
 │       │       ├── 6v
 │       │       │   └── zK
 │       │       │       └── MD5E-s2043924480--47718be3b53037499a325cf1d402b2be.nii.gz
 │       │       │           └── MD5E-s2043924480--47718be3b53037499a325cf1d402b2be.nii.gz
 │       │       ├── [...]
 │       │       └── [...]
 │       ├── archives
 │       │   └── archive.7z
 │       ├── branches
 │       ├── config
 │       ├── description
 │       ├── HEAD
 │       ├── hooks
 │       │   ├── applypatch-msg.sample
 │       │   ├── [...]
 │       │   └── update.sample
 │       ├── info
 │       │   └── exclude
 │       ├── objects
 │       │   ├── 05
 │       │   │   └── 3d25959223e8173497fa7f747442b72c31671c
 │       │   ├── 0b
 │       │   │   └── 8d0edbf8b042998dfeb185fa2236d25dd80cf9
 │       │   ├── [...]
 │       │   │   └── [...]
 │       │   ├── info
 │       │   └── pack
 │       ├── refs
 │       │   ├── heads
 │       │   │   ├── git-annex
 │       │   │   └── master
 │       │   └── tags
 │       ├── ria-layout-version
 │       └── ria-remote-ebce196a-b057-4c96-81dc-7656ea876234
 │           └── transfer
 ├── error_logs
 └── ria-layout-version

If a second dataset gets published to the RIA store, it will be represented in a similar tree structure underneath its individual dataset ID. If subdatasets of a dataset are published into a RIA store, they are not represented underneath their superdataset, but are stored on the same hierarchy level as any other dataset. Thus, the dataset representation in a RIA store is completely flat2. With this hierarchy-free setup, the location of a particular dataset in the RIA store is only dependent on its dataset ID. As the dataset ID is universally unique, gets assigned to a dataset at the time of creation, and does not change across the life time of a dataset, no two different datasets could have the same location in a RIA store.

The directory underneath the two dataset-ID-based subdirectories contains a bare git repository (highlighted above as well) that is a clone of the dataset.

What is a bare Git repository?

A bare Git repository is a repository that contains the contents of the .git directory of regular DataLad datasets or Git repositories, but no worktree or checkout. This has advantages: The repository is leaner, it is easier for administrators to perform garbage collections, and it is required if you want to push to it at all times. You can find out more on what bare repositories are and how to use them here.

Note that bare Git repositories can be cloned, and the clone of a bare Git repository will have a checkout and a worktree, thus resuming the shape that you are familiar with.

Inside of the bare Git repository, the annex directory – just as in any standard dataset or repository – contains the dataset’s keystore (object tree) under annex/objects3. In conjunction, keystore and bare Git repository are the original dataset – just differently represented, with no working tree, i.e., directory hierarchy that exists in the original dataset, and without the name it was created under, but stored under its dataset ID instead.

If necessary, the keystores (annex) can be (compressed) 7zipped archives (archives/), either for compression gains, or for use on HPC-systems with inode limitations4. Despite being 7zipped, those archives can be indexed and support relatively fast random read access. Thus, the entire key store can be put into an archive, re-using the exact same directory structure, and remains fully accessible while only using a handful of inodes, regardless of file number and size. If the dataset contains only annexed files, a complete dataset can be represented in about 25 inodes.

Taking all of the above information together, on an infrastructural level, a RIA store is fully self-contained, and is a plain file system storage, not a database. Everything inside of a RIA store is either a file, a directory, or a zipped archive. It can thus be set up on any infrastructure that has a file system with directory and file representation, and has barely any additional software requirements (see below). Access to datasets in the store can be managed by using file system permissions. With these attributes, a RIA store is a suitable solution for a number of usecases (back-up, single or multi-user dataset storage, central point for collaborative workflows, …), be that on private workstations, web servers, compute clusters, or other IT infrastructure.

Software Requirements

On the RIA store hosting infrastructure, only 7z is to be installed, if the archive feature is desired. Specifically, no Git, no git-annex, and no otherwise running daemons are necessary. If the RIA store is set up remotely, the server needs to be SSH-accessible.

On the client side, you need DataLad version 0.13.0 or later. Starting with this version, DataLad has the create-sibling-ria command and the git-annex ora-remote special remote that is required to get annexed dataset contents into a RIA store.

1.5.1.2. git-annex ORA-remote special remotes

On a technical level, beyond being a directory tree of datasets, a RIA store is by default a git-annex ORA-remote (optional remote access) special remote of a dataset. This allows to not only store the history of a dataset, but also all annexed contents.

What is a special remote?

A special-remote is an extension to Git’s concept of remotes, and can enable git-annex to transfer data to and from places that are not Git repositories (e.g., cloud services or external machines such as an HPC system). Don’t envision a special-remote as a physical place or location – a special-remote is just a protocol that defines the underlying transport of your files to and from a specific location.

The git-annex ora-remote special remote is referred to as a “storage sibling” of the original dataset. It is similar to git-annex’s built-in directory special remote (but works remotely and uses the hashdir_mixed2 keystore layout). Thanks to the git-annex ora-remote, RIA stores can have regular git-annex key storage and retrieval of keys from (compressed) 7z archives in the RIA store works. Put simple, annexed contents of datasets can only be pushed into RIA stores if they have a git-annex ora-remote.

Certain applications will not require special remote features. The usecase Scaling up: Managing 80TB and 15 million files from the HCP release shows an example where git-annex key storage is explicitly not wanted. For most storage or back-up scenarios, special remote capabilities are useful, though, and thus the default5. The datalad create-sibling-ria command will automatically create a dataset representation in a RIA store (and set up the RIA store, if it does not exist), and configure a sibling to allow publishing to the RIA store and updating from it. With special remote capabilities enabled, the command will automatically create the special remote as a storage-sibling and link it to the RIA-sibling. With the sibling and special remote set up, upon an invocation of datalad push --to <sibling>, the complete dataset contents, including annexed contents, will be published to the RIA store, with no further setup or configuration required6.

1.5.1.3. Advantages of RIA stores

Storing datasets in RIA stores has a number of advantages that align well with the demands of central dataset management on shared compute infrastructure, but are also well suited for most back-up and storage applications. In a RIA store layout, the first two levels of subdirectories can host any number of keystores and bare repositories. As datasets are identified via ID and stored next to each other underneath the top-level RIA store directory, the store is completely flexible and extendable, and regardless of the number or nature of datasets inside of the store, a RIA store keeps a homogeneous directory structure. This aids the handling of large numbers of repositories, because unique locations are derived from dataset/repository properties (their ID) rather than a dataset name or a location in a complex dataset hierarchy. Because the dataset representation in the RIA store is a bare repository, “house-keeping” as well as query tasks can be automated or performed by data management personnel with no domain-specific knowledge about dataset contents. Short maintenance scripts can be used to automate basically any task that is of interest and possible in a dataset, but across the full RIA store. A few examples are:

  • Copy or move annex objects into a 7z archive.

  • Find dataset dependencies across all stored datasets by returning the dataset IDs of subdatasets recorded in each dataset.

  • Automatically return the number of commits in each repository.

  • Automatically return the author and time of the last dataset update.

  • Find all datasets associated with specific authors.

  • Clean up unnecessary files and minimize a (or all) repository with Gits garbage collection (gc) command.

The usecase Building a scalable data storage for scientific computing demonstrates the advantages of this in a large scientific institute with central data management. Due to the git-annex ora-remote special remote, datasets can be exported and stored as archives to save disk space.

Todo

link to ukb chapter as example

1.5.2. RIA store workflows

The user facing commands for interactions with a RIA store are barely different from standard DataLad workflows. The paragraphs below detail how to create and populate a RIA store, how to clone datasets and retrieve data from it, and also how to handle permissions or hide technicalities.

1.5.2.1. Creating or publishing to RIA stores

A dataset can be added into an existing or not yet existing RIA store by running the datalad create-sibling-ria command (datalad-create-sibling-ria manual), and subsequently published into the store using datalad push. Just like the datalad siblings add command, for datalad create-sibling-ria, an arbitrary sibling name (with the -s/--name option) and a URL to the location of the store (as a positional argument) need to be specified. In the case of RIA stores, the URL takes the form of a ria+ URL, and the looks of this URL are dependent on where the RIA store (should) exists, or rather, which file transfer protocol (SSH or file) is used:

  • A URL to an SSH-accessible server has a ria+ssh:// prefix, followed by user and hostname specification and an absolute path: ria+ssh://[user@]hostname:/absolute/path/to/ria-store

  • A URL to a store on a local file system has a ria+file:// prefix, followed by an absolute path: ria+file:///absolute/path/to/ria-store

Note that it is always required to specify an absolute path in the URL!

Note

The upcoming demonstration of RIA stores uses the DataLad-101 dataset the was created throughout the Basics of this handbook. If you want to execute these code snippets on a DataLad-101 dataset you created, there is one modification that needs to be done first:

If necessary, adjust the submodule path!

Back in Subdataset publishing, in order to appropriately reference and link subdatasets on hostings sites such as GitHub, we adjusted the submodule path of the subdataset in .gitmodules to point to a published subdataset on GitHub:

# in DataLad-101
$ cat .gitmodules
[submodule "recordings/longnow"]
	path = recordings/longnow
	url = https://github.com/datalad-datasets/longnow-podcasts.git
	datalad-id = b3ca2718-8901-11e8-99aa-a0369f7c647e
[submodule "midterm_project"]
	path = midterm_project
	url = https://github.com/adswa/midtermproject
	datalad-id = 229a787c-906d-11ea-96fe-833ef2c0cd3b

Later in this demonstration we would like to publish the subdataset to a RIA store and retrieve it automatically from this store – retrieval is only attempted from a store, however, if no other working source is known. Therefore, we will remove the reference to the published dataset prior to this demonstration and replace it with the path it was originally referenced under.

# in DataLad-101
$ datalad subdatasets --contains midterm_project --set-property url ./midterm_project
subdataset(ok): midterm_project (dataset)

To demonstrate the basic process, we will create a RIA store on a local file system to publish the DataLad-101 dataset from the handbook’s “Basics” section to. In the example below, the RIA sibling gets the name ria-backup. The URL uses the file protocol and points with an absolute path to the not yet existing directory myriastore.

# inside of the dataset DataLad-101
$ datalad create-sibling-ria -s ria-backup ria+file:///home/me/myriastore
[INFO] Start checking pre-existing sibling configuration Dataset(/home/me/dl-101/DataLad-101) 
[INFO] Discovered sibling here in dataset at /home/me/dl-101/DataLad-101 
[INFO] Discovered sibling gin in dataset at /home/me/dl-101/DataLad-101 
[INFO] Discovered sibling roommate in dataset at /home/me/dl-101/DataLad-101 
[INFO] Finished checking pre-existing sibling configuration Dataset(/home/me/dl-101/DataLad-101) 
[INFO] create siblings 'ria-backup' and 'ria-backup-storage' ... 
[INFO] Fetching updates for Dataset(/home/me/dl-101/DataLad-101) 
[INFO] Configure additional publication dependency on "ria-backup-storage" 
create-sibling-ria(ok): /home/me/dl-101/DataLad-101 (dataset)

Afterwards, the dataset has two additional siblings: ria-backup, and ria-backup-storage.

$ datalad siblings
.: here(+) [git]
.: roommate(+) [../mock_user/DataLad-101 (git)]
.: ria-backup(-) [/home/me/myriastore/07c/74330-9069-11ea-96fe-833ef2c0cd3b (git)]
.: gin(+) [/home/me/pushes/DataLad-101 (git)]
.: ria-backup-storage(+) [ora]

The storage sibling is the git-annex ora-remote and is set up automatically unless create-sibling-ria is run with the --no-storage-sibling flag. By default, it has the name of the RIA sibling, suffixed with -storage, but alternative names can be supplied with the --storage-name option.

Take a look into the store

Right after running this command, a RIA store has been created in the specified location:

$ tree /home/me/myriastore
/home/me/myriastore
├── 07c
│   └── 74330-9069-11ea-96fe-833ef2c0cd3b
│       ├── annex
│       │   └── objects
│       ├── archives
│       ├── branches
│       ├── config
│       ├── description
│       ├── HEAD
│       ├── hooks
│       │   ├── applypatch-msg.sample
│       │   ├── commit-msg.sample
│       │   ├── fsmonitor-watchman.sample
│       │   ├── post-update.sample
│       │   ├── pre-applypatch.sample
│       │   ├── pre-commit.sample
│       │   ├── prepare-commit-msg.sample
│       │   ├── pre-push.sample
│       │   ├── pre-rebase.sample
│       │   ├── pre-receive.sample
│       │   └── update.sample
│       ├── info
│       │   └── exclude
│       ├── objects
│       │   ├── info
│       │   └── pack
│       ├── refs
│       │   ├── heads
│       │   └── tags
│       └── ria-layout-version
├── error_logs
└── ria-layout-version

15 directories, 17 files

Note that there is one dataset represented in the RIA store. The two-directory structure it is represented under corresponds to the dataset ID of DataLad-101:

# The dataset ID is stored in .datalad/config
$ cat .datalad/config
[datalad "dataset"]
	id = 07c74330-9069-11ea-96fe-833ef2c0cd3b

In order to publish the dataset’s history and all its contents into the RIA store, a single datalad push to the RIA sibling suffices:

$ datalad push --to ria-backup
[INFO] Start enumerating objects 
[INFO] Start counting objects 
[INFO] Start compressing objects 
[INFO] Start writing objects 
[INFO] Start resolving deltas 
[INFO] Start enumerating objects 
[INFO] Start counting objects 
[INFO] Start compressing objects 
[INFO] Start writing objects 
[INFO] Start resolving deltas 
copy(ok): books/TLCL.pdf (file) [to ria-backup-storage...]
copy(ok): books/bash_guide.pdf (file) [to ria-backup-storage...]
copy(ok): books/byte-of-python.pdf (file) [to ria-backup-storage...]
copy(ok): books/progit.pdf (file) [to ria-backup-storage...]
publish(ok): . (dataset) [refs/heads/master->ria-backup:refs/heads/master [new branch]]
publish(ok): . (dataset) [refs/heads/git-annex->ria-backup:refs/heads/git-annex [new branch]]

Take another look into the store

Now that dataset contents have been pushed to the RIA store, the bare repository contains them, although their representation is not human-readable. But worry not – this representation only exists in the RIA store. When cloning this dataset from the RIA store, the clone will be in its standard human-readable format.

$ tree /home/me/myriastore
/home/me/myriastore
├── 07c
│   └── 74330-9069-11ea-96fe-833ef2c0cd3b
│       ├── annex
│       │   └── objects
│       │       ├── F1
│       │       │   └── Wz
│       │       │       └── MD5E-s4242644--f4e1c8ebfb5c89a69ff6d268eb2e63e3.pdf
│       │       │           └── MD5E-s4242644--f4e1c8ebfb5c89a69ff6d268eb2e63e3.pdf
│       │       ├── G6
│       │       │   └── Gj
│       │       │       └── MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf
│       │       │           └── MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf
│       │       ├── jf
│       │       │   └── 3M
│       │       │       └── MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf
│       │       │           └── MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf
│       │       └── WF
│       │           └── Gq
│       │               └── MD5E-s1198170--0ab2c121bcf68d7278af266f6a399c5f.pdf
│       │                   └── MD5E-s1198170--0ab2c121bcf68d7278af266f6a399c5f.pdf
│       ├── archives
│       ├── branches
│       ├── config
│       ├── description
│       ├── HEAD
│       ├── hooks
│       │   ├── applypatch-msg.sample
│       │   ├── commit-msg.sample
│       │   ├── fsmonitor-watchman.sample
│       │   ├── post-update.sample
│       │   ├── pre-applypatch.sample
│       │   ├── pre-commit.sample
│       │   ├── prepare-commit-msg.sample
│       │   ├── pre-push.sample
│       │   ├── pre-rebase.sample
│       │   ├── pre-receive.sample
│       │   └── update.sample
│       ├── info
│       │   └── exclude
│       ├── objects
│       │   ├── info
│       │   └── pack
│       │       ├── pack-409dd939dfa96072f004f939868d0fcbc41dd767.idx
│       │       ├── pack-409dd939dfa96072f004f939868d0fcbc41dd767.pack
│       │       ├── pack-b0874468fd957dcd4a602a730aabaceeaef42afa.idx
│       │       └── pack-b0874468fd957dcd4a602a730aabaceeaef42afa.pack
│       ├── ora-remote-33f4615c-dddd-4f49-9e81-2c357e427f63
│       │   └── transfer
│       ├── refs
│       │   ├── heads
│       │   │   ├── git-annex
│       │   │   └── master
│       │   └── tags
│       └── ria-layout-version
├── error_logs
└── ria-layout-version

29 directories, 27 files

A second dataset can be added and published to the store in the very same way. As a demonstration, we’ll do it for the midterm_project subdataset:

$ cd midterm_project
$ datalad create-sibling-ria -s ria-backup ria+file:///home/me/myriastore
[INFO] Start checking pre-existing sibling configuration Dataset(/home/me/dl-101/DataLad-101/midterm_project) 
[INFO] Discovered sibling here in dataset at /home/me/dl-101/DataLad-101/midterm_project 
[INFO] Discovered sibling github in dataset at /home/me/dl-101/DataLad-101/midterm_project 
[INFO] Finished checking pre-existing sibling configuration Dataset(/home/me/dl-101/DataLad-101/midterm_project) 
[INFO] create siblings 'ria-backup' and 'ria-backup-storage' ... 
[INFO] Fetching updates for Dataset(/home/me/dl-101/DataLad-101/midterm_project) 
[INFO] Configure additional publication dependency on "ria-backup-storage" 
create-sibling-ria(ok): /home/me/dl-101/DataLad-101/midterm_project (dataset)
$ datalad push --to ria-backup
[INFO] Start enumerating objects 
[INFO] Start counting objects 
[INFO] Start compressing objects 
[INFO] Start writing objects 
[INFO] Start enumerating objects 
[INFO] Start counting objects 
[INFO] Start compressing objects 
[INFO] Start writing objects 
copy(ok): .datalad/environments/midterm-software/image (file) [to ria-backup-storage...]
copy(ok): pairwise_relationships.png (file) [to ria-backup-storage...]
copy(ok): prediction_report.csv (file) [to ria-backup-storage...]
publish(ok): . (dataset) [refs/heads/master->ria-backup:refs/heads/master [new branch]]
publish(ok): . (dataset) [refs/heads/git-annex->ria-backup:refs/heads/git-annex [new branch]]

Take a look into the RIA store after a second dataset has been added

With creating a RIA sibling to the RIA store and publishing the contents of the midterm_project subdataset to the store, a second dataset has been added to the datastore. Note how it is represented on the same hierarchy level as the previous dataset, underneath its dataset ID:

$ cat .datalad/config
[datalad "dataset"]
	id = 49864bd6-9069-11ea-96fe-833ef2c0cd3b
[datalad "containers.midterm-software"]
	updateurl = shub://adswa/resources:1
	image = .datalad/environments/midterm-software/image
	cmdexec = singularity exec {img} {cmd}
$ tree /home/me/myriastore
/home/me/myriastore
├── 07c
│   └── 74330-9069-11ea-96fe-833ef2c0cd3b
│       ├── annex
│       │   └── objects
│       │       ├── F1
│       │       │   └── Wz
│       │       │       └── MD5E-s4242644--f4e1c8ebfb5c89a69ff6d268eb2e63e3.pdf
│       │       │           └── MD5E-s4242644--f4e1c8ebfb5c89a69ff6d268eb2e63e3.pdf
│       │       ├── G6
│       │       │   └── Gj
│       │       │       └── MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf
│       │       │           └── MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf
│       │       ├── jf
│       │       │   └── 3M
│       │       │       └── MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf
│       │       │           └── MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf
│       │       └── WF
│       │           └── Gq
│       │               └── MD5E-s1198170--0ab2c121bcf68d7278af266f6a399c5f.pdf
│       │                   └── MD5E-s1198170--0ab2c121bcf68d7278af266f6a399c5f.pdf
│       ├── archives
│       ├── branches
│       ├── config
│       ├── description
│       ├── HEAD
│       ├── hooks
│       │   ├── applypatch-msg.sample
│       │   ├── commit-msg.sample
│       │   ├── fsmonitor-watchman.sample
│       │   ├── post-update.sample
│       │   ├── pre-applypatch.sample
│       │   ├── pre-commit.sample
│       │   ├── prepare-commit-msg.sample
│       │   ├── pre-push.sample
│       │   ├── pre-rebase.sample
│       │   ├── pre-receive.sample
│       │   └── update.sample
│       ├── info
│       │   └── exclude
│       ├── objects
│       │   ├── info
│       │   └── pack
│       │       ├── pack-409dd939dfa96072f004f939868d0fcbc41dd767.idx
│       │       ├── pack-409dd939dfa96072f004f939868d0fcbc41dd767.pack
│       │       ├── pack-b0874468fd957dcd4a602a730aabaceeaef42afa.idx
│       │       └── pack-b0874468fd957dcd4a602a730aabaceeaef42afa.pack
│       ├── ora-remote-33f4615c-dddd-4f49-9e81-2c357e427f63
│       │   └── transfer
│       ├── refs
│       │   ├── heads
│       │   │   ├── git-annex
│       │   │   └── master
│       │   └── tags
│       └── ria-layout-version
├── 498
│   └── 64bd6-9069-11ea-96fe-833ef2c0cd3b
│       ├── annex
│       │   └── objects
│       │       ├── VF
│       │       │   └── 27
│       │       │       └── MD5E-s347--7d984f53676358222aa7aa55980f205b.csv
│       │       │           └── MD5E-s347--7d984f53676358222aa7aa55980f205b.csv
│       │       ├── xZ
│       │       │   └── pk
│       │       │       └── MD5E-s175447--1f7f416d7c317c1f6208a940aa50c700.png
│       │       │           └── MD5E-s175447--1f7f416d7c317c1f6208a940aa50c700.png
│       │       └── zJ
│       │           └── 8f
│       │               └── MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10
│       │                   └── MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10
│       ├── archives
│       ├── branches
│       ├── config
│       ├── description
│       ├── HEAD
│       ├── hooks
│       │   ├── applypatch-msg.sample
│       │   ├── commit-msg.sample
│       │   ├── fsmonitor-watchman.sample
│       │   ├── post-update.sample
│       │   ├── pre-applypatch.sample
│       │   ├── pre-commit.sample
│       │   ├── prepare-commit-msg.sample
│       │   ├── pre-push.sample
│       │   ├── pre-rebase.sample
│       │   ├── pre-receive.sample
│       │   └── update.sample
│       ├── info
│       │   └── exclude
│       ├── objects
│       │   ├── 01
│       │   │   └── 6a67d14e520cbd435f178f3647e92d65cc41e6
│       │   ├── 03
│       │   │   └── 9aaa442e10f3d941861bf415c232ef79623ff8
│       │   ├── 07
│       │   │   └── 9d97c367520b1291fe2b4c6abeb862371b02ab
│       │   ├── 09
│       │   │   └── f265aa5fb7f8335e71eca38ea3765c55422ade
│       │   ├── 0b
│       │   │   └── 9c1e2db0a572806c59e331eb9659520f9b84b8
│       │   ├── 11
│       │   │   └── 9ff044791a6e5fbe8d98bab341001d97e9feb7
│       │   ├── 16
│       │   │   └── 24268838a9df78c6b45067d3e51a54cd24265c
│       │   ├── 17
│       │   │   └── 79c7a04c92b5063882079ac7c939b16737cf8f
│       │   ├── 1b
│       │   │   ├── 884a75d32efbcf9e470c79aad44b36d5212fcd
│       │   │   └── e7f989909d57adc3cbf391a003eb180b9201d4
│       │   ├── 1e
│       │   │   └── d9d2f6b504b19c0f01816178233c68835f758c
│       │   ├── 20
│       │   │   └── b7722d57db1eb57a1f23c2a2195a6924991d76
│       │   ├── 25
│       │   │   ├── e586ecfb4e7f5c4e721a27aa89a91328fbb0b3
│       │   │   └── fce1e390322c8c6c5730abf664dcc1fa4a9482
│       │   ├── 26
│       │   │   └── 058d30d26912dc073d04e4d941a7aa58f56a8e
│       │   ├── 28
│       │   │   ├── 5b49c93e70d2f424a1587c4bee0839aef04ce0
│       │   │   └── b0b472bd0803c463b92835a5354eb6dd9ed87d
│       │   ├── 2d
│       │   │   ├── a25fc76e159a7282fb15ddccfd283bb8209adc
│       │   │   └── aafe9f4c2197f3a71e6ba57ccb38537a3ffee1
│       │   ├── 2e
│       │   │   ├── 8b77a098853cb47ef3aa8ff9bb9434f3b490a9
│       │   │   └── dc2849d6d80079e0dd4654ea2d2be588535b5d
│       │   ├── 30
│       │   │   └── ddc631b6be71192d54c59a0332f50e9712e039
│       │   ├── 34
│       │   │   ├── 328e2631ce2083fab97fe14a74f323436e9b10
│       │   │   └── adffd2533a202c50ffddc4c704735cbd85ac7e
│       │   ├── 35
│       │   │   └── 26135630732215ca9ac3e6685de7749d08194b
│       │   ├── 36
│       │   │   └── 65051e51d9cf1a394810796e3f8860d307eb4e
│       │   ├── 41
│       │   │   └── e377d4a16c9d9d0a0ccd759e4a97b9fe1c842c
│       │   ├── 42
│       │   │   └── d194b1b34e12bf285dffeb4805bc1d69f537a8
│       │   ├── 43
│       │   │   └── 4a2c841e0afb5889a56c9d681f1e368e1cba95
│       │   ├── 44
│       │   │   └── ae8d678b8457faaa9f61d48dfdf1ed352726ec
│       │   ├── 45
│       │   │   └── 61c9262bb471ee71d29e2d7bdc293984f49968
│       │   ├── 46
│       │   │   └── aa9052a302b33fed9f4b8b7cb6c42a7cbb468b
│       │   ├── 49
│       │   │   └── 0c2273d9e84484a2975168f26cf74fb6f67e3e
│       │   ├── 4b
│       │   │   └── 825dc642cb6eb9a060e54bf8d69288fbee4904
│       │   ├── 4d
│       │   │   └── f64f42f48b1a4967db997220bbf0f0dd8cfa87
│       │   ├── 54
│       │   │   └── 582be6c832e9fd930093888449c7402672b4f2
│       │   ├── 5b
│       │   │   └── 2146b6a5b2cd63e18d88e2a89842d302513fcd
│       │   ├── 5e
│       │   │   └── 19ac4ced001dd4dcd6ce35402a4d6faf2063b4
│       │   ├── 5f
│       │   │   └── 760a2bffeec5a70bb57556b317a56260171ee5
│       │   ├── 67
│       │   │   ├── 01d6df9a2ff757d18a527221bcc0495ab23871
│       │   │   └── 9b57f861f0bdda966c050cfa255c4ef845169f
│       │   ├── 6c
│       │   │   └── ad10336e597e4a116b08238ecd14e9736601b1
│       │   ├── 6d
│       │   │   └── aecfb63901745da4092a65d6845595aaba159d
│       │   ├── 6e
│       │   │   └── 06b1855a8f46cb0e75e463d494d4925e7d7ee8
│       │   ├── 75
│       │   │   └── ed15d4e993800cd3941834b320307473aeec2f
│       │   ├── 76
│       │   │   └── 5aa87cb2195464eaaf6111d6fc05a9858088b3
│       │   ├── 7a
│       │   │   └── 8e02e5fe51410a762028bb580d312c726acc04
│       │   ├── 7b
│       │   │   └── 30b1622e25b5caec3aa14b3f134963bef5c303
│       │   ├── 80
│       │   │   └── 0282a43f30b90c404a5013a66176e42660f7dd
│       │   ├── 84
│       │   │   └── a246e40bee01cb7874214aec54f2647fe9ae4e
│       │   ├── 86
│       │   │   ├── 25fb149e1a9f75078aa07ff99731bf53a28e39
│       │   │   ├── 5f587d62f77a67bc668ac19f0dbde14a9ba5a3
│       │   │   └── fcfb86393bb514a81fa8f2e566c258b5ba3c30
│       │   ├── 91
│       │   │   └── 492d48edfb8fca8c69cf2d954fc513bf127048
│       │   ├── 92
│       │   │   └── b29f29f48ae0de816b1b482c5e25af6b050c9a
│       │   ├── 94
│       │   │   └── b570eac65b1d11c79fed128d3e95ffeff6ef92
│       │   ├── 95
│       │   │   └── 9a71b5b3546cb2218aa592bf597f72d12b3341
│       │   ├── 96
│       │   │   ├── 4e52c11eab853d85eeb253fe8770bf654a6c6d
│       │   │   └── aab3752c464da88a1a9288dd3c02285ee0b882
│       │   ├── 97
│       │   │   └── 55c04f77cbf8be067a6885522bac2a90d906b7
│       │   ├── a3
│       │   │   └── 0b54dacb69f02c8235e577d50030fa3a7ff817
│       │   ├── a9
│       │   │   └── 1de06229ed56b3dd48b1e28723594c50b61c6f
│       │   ├── aa
│       │   │   └── 4dd0adc688b3bca67067faa8fd37121f786acb
│       │   ├── ab
│       │   │   ├── 192239fd08666c72e4e97a4563d5c4b749e3e1
│       │   │   └── a0db2ba59d95c681d7024c512d902c5ff58359
│       │   ├── b1
│       │   │   └── bfb8febf9f78beb91b898e67f2431e0c8e2299
│       │   ├── b4
│       │   │   └── 6a2d5a30bbe8e012cab29849675a7633aa5afc
│       │   ├── b5
│       │   │   └── 40820107ca5718dc0f196a08edf3729239bfef
│       │   ├── b9
│       │   │   ├── 73796f0bf3393e4ae3744d958f971a99764568
│       │   │   └── e8811d4f0c5d9231eec689bbd54f110d901ec9
│       │   ├── bb
│       │   │   └── 587281e08622cec88d67a9dcb97763ecc48535
│       │   ├── bd
│       │   │   └── 519d3db8032b5e41601791510a384d22039c2f
│       │   ├── bf
│       │   │   └── c49c8259c178003cfc12620587daa77382dab1
│       │   ├── c3
│       │   │   └── aaefef9a2470b31ba9213350046ff7cde75737
│       │   ├── c6
│       │   │   └── 2daca519a2e572150adb1af280de9e974fba89
│       │   ├── c7
│       │   │   └── 005ee38bdb77d377914330261eb7ee5e67c9f1
│       │   ├── ca
│       │   │   ├── 5f02fbeebb08100a0249854417092427c9809e
│       │   │   └── 7219c669044afccbf4a8315dfe9172c4c68867
│       │   ├── cc
│       │   │   └── 974ed68198ef0208477ad155de9d5ad8bb3c71
│       │   ├── d1
│       │   │   └── 5e9004b0bb3398fcdd2a8499f878428bb0599a
│       │   ├── d4
│       │   │   └── 2ab9b01c97fead21e3fac008af6c3e10c7993f
│       │   ├── d8
│       │   │   └── ccf5e3c852fa4550147f8bd81f2b1a500f10ec
│       │   ├── db
│       │   │   ├── 8309b345037bd6a185f47c3b22d31d429a8d6e
│       │   │   ├── b20986e68cf44f5798e4eea295a82d440e4097
│       │   │   └── daef1da3d90b63b0e8d3dcc5670ff53908e3d0
│       │   ├── dc
│       │   │   └── cb06641e0bd0799f8cdaf9d5eb592b218afb75
│       │   ├── e4
│       │   │   └── 1d09b048b806266b3b6f575bf42f4d1f90ff4c
│       │   ├── e6
│       │   │   └── 9de29bb2d1d6434b8b29ae775ad8c2e48c5391
│       │   ├── e8
│       │   │   └── e4cb494b5b94e27f6c8784ffd26eb9aefff120
│       │   ├── ed
│       │   │   └── 56908ba436877436cd3538026cad42e551799d
│       │   ├── f0
│       │   │   ├── 0c32701c59d74813aade92baa6c6471c5ecf64
│       │   │   └── eb8b04fd93d0d26e2d1b1b166b883804766656
│       │   ├── f4
│       │   │   └── 65c12605f751c160a4de805c675cba32f7ff52
│       │   ├── fc
│       │   │   └── f71e65253fed692b965d6f508624c5916429e6
│       │   ├── fe
│       │   │   └── 2f7b920392320c2f3930c659f5095a4c0d0ea9
│       │   ├── info
│       │   └── pack
│       ├── ora-remote-f5b5c7a3-7e00-4b1b-92a0-eb7687831c82
│       │   └── transfer
│       ├── refs
│       │   ├── heads
│       │   │   ├── git-annex
│       │   │   └── master
│       │   └── tags
│       └── ria-layout-version
├── error_logs
└── ria-layout-version

133 directories, 143 files

Thus, in order to create and populate RIA stores, only the commands datalad create-sibling-ria and datalad push are required.

1.5.2.2. Cloning and updating from RIA stores

Cloning from RIA stores is done via datalad clone from a ria+ URL, suffixed with a dataset identifier. Depending on the protocol being used, the URLs are composed similarly to during sibling creation:

  • A URL to a RIA store on an SSH-accessible server takes the same format as before: ria+ssh://[user@]hostname:/absolute/path/to/ria-store

  • A URL to a RIA store on a local file system also looks like during sibling creation: ria+file:///absolute/path/to/ria-store

  • A URL for read (without annex) access to a store via http (e.g., to a RIA store like store.datalad.org, through which the HCP dataset is published) looks like this: ria+http://store.datalad.org:/absolute/path/to/ria-store

The appropriate ria+ URL needs to be suffixed with a # sign and a dataset identifier. One way this can be done is via the dataset ID. Here is how to clone the DataLad-101 dataset from the RIA store using its dataset ID:

$ datalad clone ria+file:///home/me/myriastore#07c74330-9069-11ea-96fe-833ef2c0cd3b myclone
[INFO] Cloning dataset to Dataset(/home/me/dl-101/myclone) 
[INFO] Attempting to clone from file:///home/me/myriastore/07c/74330-9069-11ea-96fe-833ef2c0cd3b to /home/me/dl-101/myclone 
[INFO] Completed clone attempts for Dataset(/home/me/dl-101/myclone) 
[INFO] Configure additional publication dependency on "ria-backup-storage" 
configure-sibling(ok): . (sibling)
install(ok): /home/me/dl-101/myclone (dataset)
action summary:
  configure-sibling (ok: 1)
  install (ok: 1)

There are two downsides to this method: For one, it is hard to type, remember, and know the dataset ID of a desired dataset. Secondly, if no additional path is given to datalad clone, the resulting dataset clone would be named after its ID. An alternative, therefore, is to use an alias for the dataset. This is an alternative dataset identifier that a dataset in a RIA store can be configured with.

Configure an alias for a dataset

In order to define an alias for an individual dataset in a store, one needs to create an alias/ directory in the root of the datastore and place a symlink of the desired name to the dataset inside of it. Here is how it is done, for the midterm project dataset:

First, create an alias/ directory in the store:

$ mkdir /home/me/myriastore/alias

Afterwards, place a symlink with a name of your choice to the dataset inside of it. Here, we create a symlink called midterm_project:

$ ln -s /home/me/myriastore/498/64bd6-9069-11ea-96fe-833ef2c0cd3b /home/me/myriastore/alias/midterm_project

Here is how it looks like inside of this directory:

$ tree /home/me/myriastore/alias
/home/me/myriastore/alias
└── midterm_project -> /home/me/myriastore/498/64bd6-9069-11ea-96fe-833ef2c0cd3b

1 directory, 0 files

Afterwards, the alias name, prefixed with a ~, can be used as a dataset identifier:

datalad clone ria+file:///home/me/myriastore#~midterm_project
[INFO] Cloning dataset to Dataset(/home/me/dl-101/midterm_project) 
[INFO] Attempting to clone from file:///home/me/myriastore/alias/midterm_project to /home/me/dl-101/midterm_project 
[INFO] Completed clone attempts for Dataset(/home/me/dl-101/midterm_project) 
[INFO] Configure additional publication dependency on "ria-backup-storage" 
configure-sibling(ok): . (sibling)
install(ok): /home/me/dl-101/midterm_project (dataset)
action summary:
  configure-sibling (ok: 1)
  install (ok: 1)

This makes it easier for others to clone the dataset and will provide a sensible default name for the clone if no additional path is provided in the command.

The dataset clone is just like any other dataset clone. Contents stored in Git are present right after cloning, while the contents of annexed files is not yet retrieved from the store and can be obtained with a datalad get.

$ cd myclone
$ tree
.
├── books
│   ├── bash_guide.pdf -> ../.git/annex/objects/WF/Gq/MD5E-s1198170--0ab2c121bcf68d7278af266f6a399c5f.pdf/MD5E-s1198170--0ab2c121bcf68d7278af266f6a399c5f.pdf
│   ├── byte-of-python.pdf -> ../.git/annex/objects/F1/Wz/MD5E-s4242644--f4e1c8ebfb5c89a69ff6d268eb2e63e3.pdf/MD5E-s4242644--f4e1c8ebfb5c89a69ff6d268eb2e63e3.pdf
│   ├── progit.pdf -> ../.git/annex/objects/G6/Gj/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf
│   └── TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf
├── code
│   ├── list_titles.sh
│   └── nested_repos.sh
├── Gitjoke2.txt
├── midterm_project
├── notes.txt
└── recordings
    ├── interval_logo_small.jpg
    ├── longnow
    ├── podcasts.tsv
    └── salt_logo_small.jpg

5 directories, 11 files

To demonstrate file retrieval from the store, let’s get an annexed file:

$ datalad get books/progit.pdf
get(ok): books/progit.pdf (file) [from ria-backup-storage...]

What about creating RIA stores and cloning from RIA stores with different protocols

Consider setting up and populating a RIA store on a server via the file protocol, but cloning a dataset from that store to a local computer via SSH protocol. Will this be a problem for file content retrieval? No, in all standard situations, DataLad will adapt to this. Upon cloning the dataset with a different URL than it was created under, enabling the special remote will initially fail, but DataLad will adaptive try out other URLs (including changes in hostname, path, or protocol) to enable the ora-remote and retrieve file contents.

Just as expected, the subdatasets are not pre-installed. How will subdataset installation work for datasets that exist in a RIA store as well, like midterm_project? Just as with any other subdataset! DataLad cleverly handles subdataset installations from RIA stores in the background: The location of the subdataset in the RIA store is discovered and used automatically:

$ datalad get -n midterm_project
[INFO] Cloning dataset to Dataset(/home/me/dl-101/myclone/midterm_project) 
[INFO] Attempting to clone from /home/me/myriastore/07c/74330-9069-11ea-96fe-833ef2c0cd3b/midterm_project to /home/me/dl-101/myclone/midterm_project 
[INFO] Attempting to clone from file:///home/me/myriastore/498/64bd6-9069-11ea-96fe-833ef2c0cd3b to /home/me/dl-101/myclone/midterm_project 
[INFO] Completed clone attempts for Dataset(/home/me/dl-101/myclone/midterm_project) 
[INFO] Configure additional publication dependency on "ria-backup-storage" 
install(ok): /home/me/dl-101/myclone/midterm_project (dataset) [Installed subdataset in order to get /home/me/dl-101/myclone/midterm_project]

More technical insights into the automatic ria+ URL generation are outlined in the findoutmore below:

On cloning datasets with subdatasets from RIA stores

The usecase Scaling up: Managing 80TB and 15 million files from the HCP release details a RIA-store based publication of a large dataset, split into a nested dataset hierarchy with about 4500 subdatasets in total. But how can links to subdatasets work, if datasets in a RIA store are stored in a flat hierarchy, with no nesting?

The key to this lies in flexibly regenerating subdataset’s URLs based on their ID and a path to the RIA store. The datalad get command is capable of generating RIA URLs to subdatasets on its own, if the higher level dataset contains a datalad get configuration on subdataset-source-candidate-origin that points to the RIA store the subdataset is published in. Here is how the .datalad/config configuration looks like for the top-level dataset of the HCP dataset:

[datalad "get"]
    subdataset-source-candidate-origin = "ria+http://store.datalad.org#{id}"

With this configuration, a datalad get can use the URL and insert the dataset ID in question into the {id} placeholder to clone directly from the RIA store.

This configuration is automatically added to a dataset that is cloned from a RIA store, but it can also be done by hand with a git config command7.

Beyond straightforward access to datasets, RIA stores also allow very fine-grained cloning operations: Datasets in RIA stores can be cloned in specific versions.

Cloning specific dataset versions

Optionally, datasets can be cloned in a specific version, such as a tag or branch by appending @<version-identifier> after the dataset ID or the dataset alias. Here is how to clone the BIDS version of the structural preprocessed subset of the HCP dataset that exists on the branch bids of this dataset:

$ datalad clone ria+http://store.datalad.org#[email protected]

If you are interested in finding out how this dataset came into existence, checkout the use case Scaling up: Managing 80TB and 15 million files from the HCP release.

Updating datasets works with the datalad update and datalad update --merge commands introduced in chapter Collaboration. And because a RIA store hosts bare Git repositories, collaborating becomes easy. Anyone with access can clone the dataset from the store, add changes, and push them back – this is the same workflow as for datasets hosted on sites such as GitHub, GitLab, or Gin.

1.5.2.3. Permission management

In order to limit access or give access to datasets in datastores, permissions can be set at the time of RIA sibling creation with the --shared option. If it is given, this option configures the permissions in the RIA store for multi-users access. Possible values for this option are identical to those of git init --shared and are described in its documentation. In order for the dataset to be accessible to everyone, for example, --shared all could be specified. If access should be limited to a particular Unix group (--shared group), the group name needs to be specified with the --group option.

1.5.2.4. Configurations and tricks to hide technical layers

In setups with a central, DataLad-centric data management, in order to spare users knowing about RIA stores, custom configurations can be distributed via DataLad’s run-procedures to simplify workflows further and hide the technical layers of the RIA setup. For example, custom procedures provided at dataset creation could automatically perform a sibling setup in a RIA store, and also create an associated GitLab repository with a publication dependency to the RIA store to ease publishing data or cloning the dataset. The usecase Building a scalable data storage for scientific computing details the setup of RIA stores in a scientific institute and demonstrates this example.

To simplify repository access beyond using aliases, the datasets stored in a RIA store can be installed under human-readable names in a single superdataset. Cloning the superdataset exposes the underlying datasets under a non-dataset-ID name. Users can thus get data from datasets hosted in a datastore without any knowledge about the dataset IDs or the need to construct ria+ URLs, just as it was done in the usecases Scaling up: Managing 80TB and 15 million files from the HCP release and Building a scalable data storage for scientific computing. From a user’s perspective, the RIA store would thus stay completely hidden.

Standard maintenance tasks by data stewards with knowledge about RIA stores and access to it can be performed easily or even in an automated fashion. The usecase Building a scalable data storage for scientific computing showcases some examples of those operations.

1.5.3. Summary

RIA stores are useful, lean, and undemanding storage locations for DataLad datasets. Their properties make them suitable solutions to back-up, central data management, or collaboration use cases. They can be set up with minimal effort, and the few technical details a user may face such as cloning from dataset IDs can be hidden with minimal configurations of the store like aliases or custom procedures.

Footnotes

1

The two-level structure (3 ID characters as one subdirectory, the remaining ID characters as the next subdirectory) exists to avoid exhausting file system limits on the number of files/folders within a directory.

2(1,2)

Beyond datasets, the RIA store only contains the directory error_logs for error logging and the file ria-layout-version for a specification of the dataset tree layout in the store (last two lines in the code block above). The ria-layout-version is important because it identifies whether the keystore uses git-annex’s hashdirlower (git-annex’s default for bare repositories) or hashdirmixed layout (which is necessary to allow symlinked annexes, relevant for ephemeral clones). To read more about hashing in the key store, take a look at the docs.

3

To re-read about how git-annex’s object tree works, check out section Data integrity, and pay close attention to the hidden section. Additionally, you can find a lot of background information in git-annex’s documentation.

4

The usecase

Todo

Link UKBiobank on supercomputer usecase once ready

shows how this feature can come in handy.

5

Special remote capabilities of a RIA store can be disabled at the time of RIA store creation by passing the option --no-storage-sibling to the datalad create-sibling-ria command.

6

To re-read about publication dependencies and why this is relevant to annexed contents in the dataset, checkout section Beyond shared infrastructure.

7

To re-read on configuring datasets with the git config, go back to sections DIY configurations and More on DIY configurations.