9.2. Miscellaneous file system operations

With all of the information about symlinks and object trees, you might be reluctant to perform usual file system managing operations, such as copying, moving, renaming or deleting files or directories with annexed content.

If I renamed one of those books, would the symlink that points to the file content still be correct? What happens if I’d copy an annexed file? If I moved the whole books/ directory? What if I moved all of DataLad-101 into a different place on my computer? What if renamed the whole superdataset? And how do I remove a file, or directory, or subdataset?

Therefore, there is an extra tutorial today, and you attend. There is no better way of learning than doing. Here, in the safe space of the DataLad-101 course, you can try out all of the things you would be unsure about or reluctant to try on the dataset that contains your own, valuable data.

Below you will find common questions about file system management operations, and each question outlines caveats and solutions with code examples you can paste into your own terminal. Because these code snippets will add many commits to your dataset, we are cleaning up within each segment with common Git operations that manipulate the datasets history – be sure to execute these commands as well (and be sure to be in the correct dataset).

9.2.1. Renaming files

Let’s try it. In Unix, renaming a file is exactly the same as moving a file, and uses the mv command.

$ cd books/
$ mv TLCL.pdf The_Linux_Command_Line.pdf
$ ls -lh The_Linux_Command_Line.pdf
total 16K
lrwxrwxrwx 1 elena elena 131 2019-06-18 16:13 bash_guide.pdf -> ../.git/annex/objects/WF/Gq/✂/MD5E-s1198170--0ab2c121✂MD5.pdf
lrwxrwxrwx 1 elena elena 131 2019-06-18 16:13 byte-of-python.pdf -> ../.git/annex/objects/xF/42/✂/MD5E-s4161086--c832fc13✂MD5.pdf
lrwxrwxrwx 1 elena elena 133 2019-06-18 16:13 progit.pdf -> ../.git/annex/objects/G6/Gj/✂/MD5E-s12465653--05cd7ed5✂MD5.pdf
lrwxrwxrwx 1 elena elena 131 2019-06-18 16:13 The_Linux_Command_Line.pdf -> ../.git/annex/objects/jf/3M/✂/MD5E-s2120211--06d1efcb✂MD5.pdf

Try to open the renamed file, e.g., with evince The_Linux_Command_Line.pdf. This works!

But let’s see what changed in the dataset with this operation:

$ datalad status
untracked: /home/me/dl-101/DataLad-101/books/The_Linux_Command_Line.pdf (symlink)
  deleted: /home/me/dl-101/DataLad-101/books/TLCL.pdf (symlink)

We can see that the old file is marked as deleted, and simultaneously, an untracked file appears: the renamed PDF.

While this might appear messy, a datalad save will clean all of this up. Therefore, do not panic if you rename a file, and see a dirty dataset status with deleted and untracked files – datalad save handles these and other cases really well under the hood.

$ datalad save -m "rename the book"
delete(ok): books/TLCL.pdf (symlink)
add(ok): books/The_Linux_Command_Line.pdf (file)
save(ok): . (dataset)

The datalad save (manual) command will identify that a file was renamed, and will summarize this nicely in the resulting commit:

$ git log -n 1 -p
commit d05e0ded✂SHA1
Author: Elena Piscopia <elena@example.net>
Date:   Tue Jun 18 16:13:00 2019 +0000

    rename the book

diff --git a/books/TLCL.pdf b/books/The_Linux_Command_Line.pdf
similarity index 100%
rename from books/TLCL.pdf
rename to books/The_Linux_Command_Line.pdf

Note that datalad save commits all modifications when it’s called without a path specification, so any other changes will be saved in the same commit as the rename. If there are unsaved modifications you do not want to commit together with the file name change, you could give both the new and the deleted file as a path specification to datalad save, even if it feels unintuitive to save a change that is marked as a deletion in a datalad status (manual):

$ datalad save -m "rename file" oldname newname

Alternatively, there is also a way to save the name change only using Git tools only, outlined in the Find-out-more on faster renaming. If you are a Git user, you will be very familiar with it.

Faster renaming with Git tools

Git has built-in commands that provide a solution in two steps.

If you have followed along with the previous datalad save, let’s revert the renaming of the files:

$ git reset --hard HEAD~1
$ datalad status
HEAD is now at 39e55c8 add container and execute analysis within container
nothing to save, working tree clean

Now we are checking out how to rename files and commit this operation using only Git: A Git-specific way to rename files is the git mv command:

$ git mv TLCL.pdf The_Linux_Command_Line.pdf
$ datalad status
    added: /home/me/dl-101/DataLad-101/books/The_Linux_Command_Line.pdf (symlink)
  deleted: /home/me/dl-101/DataLad-101/books/TLCL.pdf (symlink)

We can see that the old file is still seen as “deleted”, but the “new”, renamed file is “added”. A git status displays the change in the dataset a bit more accurately:

$ git status
On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	renamed:    TLCL.pdf -> The_Linux_Command_Line.pdf

Because the git mv (manual) places the change directly into the staging area (the index) of Git[1], a subsequent git commit -m "rename book" will write the renaming – and only the renaming – to the dataset’s history, even if other (unstaged) modifications are present.

$ git commit -m "rename book"
[main 515fcc2] rename book
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename books/{TLCL.pdf => The_Linux_Command_Line.pdf} (100%)

Especially when renaming directories with many files, this can be much faster than a mv followed by datalad save,

To summarize, renaming files is easy and worry-free. Do not be intimidated by a file marked as deleted – a datalad save will rectify this. Be mindful of other modifications in your dataset, though, and either supply appropriate paths to datalad save, or use Git tools to exclusively save the name change and nothing else.

Let’s revert this now, to have a clean history.

$ git reset --hard HEAD~1
$ datalad status
HEAD is now at 39e55c8 add container and execute analysis within container
nothing to save, working tree clean

9.2.2. Moving files from or into subdirectories

Let’s move an annexed file from within books/ into the root of the superdataset:

$ mv TLCL.pdf ../TLCL.pdf
$ datalad status
untracked: /home/me/dl-101/DataLad-101/TLCL.pdf (symlink)
  deleted: /home/me/dl-101/DataLad-101/books/TLCL.pdf (symlink)

In general, this looks exactly like renaming or moving a file in the same directory. There is a subtle difference though: Currently, the symlink of the annexed file is broken. There are two ways to demonstrate this. One is trying to open the file – this will currently fail. The second way is to look at the symlink:

$ cd ../
$ ls -l TLCL.pdf
lrwxrwxrwx 1 elena elena 131 2019-06-18 16:13 TLCL.pdf -> ../.git/annex/objects/jf/3M/✂/MD5E-s2120211--06d1efcb✂MD5.pdf

The first part of the symlink should point into the .git/ directory, but currently, it does not – the symlink still looks like TLCL.pdf would be within books/. Instead of pointing into .git, it currently points to ../.git, which is non-existent, and even outside of the superdataset. This is why the file cannot be opened: When any program tries to follow the symlink, it will not resolve, and an error such as “no file or directory” will be returned. But do not panic! A datalad save will rectify this as well:

$ datalad save -m "moved book into root"
$ ls -l TLCL.pdf
delete(ok): books/TLCL.pdf (symlink)
add(ok): TLCL.pdf (file)
save(ok): . (dataset)
lrwxrwxrwx 1 elena elena 128 2019-06-18 16:13 TLCL.pdf -> .git/annex/objects/jf/3M/✂/MD5E-s2120211--06d1efcb✂MD5.pdf

After a datalad save, the symlink is fixed again. Therefore, in general, whenever moving or renaming a file, especially between directories, a datalad save is the best option to turn to.

Why a move between directories is actually a content change

Let’s see how this shows up in the dataset history:

$ git log -n 1 -p
commit b315f5c4✂SHA1
Author: Elena Piscopia <elena@example.net>
Date:   Tue Jun 18 16:13:00 2019 +0000

    moved book into root

diff --git a/TLCL.pdf b/TLCL.pdf
new file mode 120000
index 0000000..34328e2
--- /dev/null
+++ b/TLCL.pdf
@@ -0,0 +1 @@
+.git/annex/objects/jf/3M/✂/MD5E-s2120211--06d1efcb✂MD5.pdf
\ No newline at end of file
diff --git a/books/TLCL.pdf b/books/TLCL.pdf
deleted file mode 120000
index 4c84b61..0000000
--- a/books/TLCL.pdf
+++ /dev/null
@@ -1 +0,0 @@
-../.git/annex/objects/jf/3M/✂/MD5E-s2120211--06d1efcb✂MD5.pdf
\ No newline at end of file

As you can see, this action does not show up as a move, but instead a deletion and addition of a new file. Why? Because the content that is tracked is the actual symlink, and due to the change in relative location, the symlink needed to change. Hence, what looks and feels like a move on the file system for you is actually a move plus a content change for Git.

‘datalad save’ internals: ‘git annex fix’

A datalad save command internally uses a git commit (manual) to save changes to a dataset. git commit in turn triggers a git annex fix (manual) command. This git-annex command fixes up links that have become broken to again point to annexed content, and is responsible for cleaning up what needs to be cleaned up. Thanks, git-annex!

Finally, let’s clean up:

$ git reset --hard HEAD~1
HEAD is now at 39e55c8 add container and execute analysis within container

9.2.3. Moving files across dataset boundaries

Generally speaking, moving files across dataset hierarchies is not advised. While DataLad blurs the dataset boundaries to ease working in nested dataset, the dataset boundaries do still exist. If you move a file from one subdataset into another, or up or down a dataset hierarchy, you will move it out of the version control it was in (i.e., from one .git directory into a different one). From the perspective of the first subdataset, the file will be deleted, and from the perspective of the receiving dataset, the file will be added to the dataset, but straight out of nowhere, with none of its potential history from its original dataset attached to it. Before moving a file, consider whether copying it (outlined in the next but one paragraph) might be a more suitable alternative.

If you are willing to sacrifice[2] the file’s history and move it to a different dataset, the procedure differs between annexed files, and files stored in Git.

For files that Git manages, moving and saving is simple: Move the file, and save the resulting changes in both affected datasets (this can be done with a recursive datalad save from a top-level dataset, though).

$ mv notes.txt midterm_project/notes.txt
$ datalad status -r
 modified: midterm_project (dataset)
untracked: midterm_project/notes.txt (file)
  deleted: notes.txt (file)
$ datalad save -r -m "moved notes.txt from root of top-ds to midterm subds"
add(ok): notes.txt (file)
save(ok): midterm_project (dataset)
delete(ok): notes.txt (file)
add(ok): midterm_project (dataset)
add(ok): .gitmodules (file)
save(ok): . (dataset)
action summary:
  add (ok: 3)
  delete (ok: 1)
  save (notneeded: 2, ok: 2)

Note how the history of notes.txt does not exist in the subdataset – it appears as if the file was generated at once, instead of successively over the course:

$ cd midterm_project
$ git log notes.txt
commit 1e273c59✂SHA1
Author: Elena Piscopia <elena@example.net>
Date:   Tue Jun 18 16:13:00 2019 +0000

    moved notes.txt from root of top-ds to midterm subds

(Undo-ing this requires git resets in both datasets)

$ # in midterm_project
$ git reset --hard HEAD~

$ # in DataLad-101
$ cd ../
$ git reset --hard HEAD~
HEAD is now at 4f00ad0 [DATALAD RUNCMD] rerun analysis in container
HEAD is now at 39e55c8 add container and execute analysis within container

The process is a bit more complex for annexed files. Let’s do it wrong, first: What happens if we move an annexed file in the same way as notes.txt?

$ mv books/TLCL.pdf midterm_project
$ datalad status -r
  deleted: books/TLCL.pdf (symlink)
 modified: midterm_project (dataset)
untracked: midterm_project/TLCL.pdf (symlink)
$ datalad save -r -m "move annexed file around"
add(ok): TLCL.pdf (file) [TLCL.pdf is a git-annex symlink.]
save(ok): midterm_project (dataset)
delete(ok): books/TLCL.pdf (symlink)
add(ok): midterm_project (dataset)
add(ok): .gitmodules (file)
save(ok): . (dataset)
action summary:
  add (ok: 3)
  delete (ok: 1)
  save (notneeded: 2, ok: 2)

At this point, this does not look that different to the result of moving notes.txt. Note, though, that the deleted and untracked PDFs are symlinks – and therein lies the problem: What was moved was not the file content (which is still in the annex of the top-level dataset, DataLad-101), but its symlink that was stored in Git. After moving the file, the symlink is broken, and git-annex has no way of finding out where the file content could be:

$ cd midterm_project
$ git annex whereis TLCL.pdf
whereis TLCL.pdf (0 copies) failed
whereis: 1 failed

Let’s rewind, and find out how to do it correctly:

$ git reset --hard HEAD~
$ cd ../
$ git reset --hard HEAD~
HEAD is now at 4f00ad0 [DATALAD RUNCMD] rerun analysis in container
HEAD is now at 39e55c8 add container and execute analysis within container

The crucial step to remember is to get the annexed file out of the annex prior to moving it. For this, we need to fall back to git-annex commands:

$ git annex unlock books/TLCL.pdf
$ mv books/TLCL.pdf midterm_project
$ datalad status -r
unlock books/TLCL.pdf ok
(recording state in git...)
  deleted: books/TLCL.pdf (file)
 modified: midterm_project (dataset)
untracked: midterm_project/TLCL.pdf (file)

Afterwards, a (recursive) datalad save commits the removal of the book from DataLad-101, and adds the file content into the annex of midterm_project:

$ datalad save -r -m "move book into midterm_project"
add(ok): TLCL.pdf (file)
save(ok): midterm_project (dataset)
delete(ok): books/TLCL.pdf (file)
add(ok): midterm_project (dataset)
add(ok): .gitmodules (file)
save(ok): . (dataset)
action summary:
  add (ok: 3)
  delete (ok: 1)
  save (notneeded: 2, ok: 2)

Even though you did split the file’s history, at least its content is in the correct dataset now:

$ cd midterm_project
$ git annex whereis TLCL.pdf
whereis TLCL.pdf (1 copy)
  	✂UUID✂ -- me@appveyor-vm:~/dl-101/DataLad-101/midterm_project [here]
ok

But more than showing you how it can be done, if necessary, this paragraph hopefully convinced you that moving files across dataset boundaries is not convenient. It can be a confusing and potentially “file-content-losing”-dangerous process, but it also dissociates a file from its provenance that is captured in its previous dataset, with no machine-readable way to learn about the move easily. A better alternative may be copying files with the datalad copy-file (manual) command introduced in detail in the online-handbook, and demonstrated in the next but one paragraph. Let’s quickly clean up by moving the file back:

$ # in midterm_project
$ git annex unannex TLCL.pdf
unannex TLCL.pdf ok
(recording state in git...)
$ mv TLCL.pdf ../books
$ cd ../
$ datalad save -r -m "move book back from midterm_project"
delete(ok): TLCL.pdf (symlink)
save(ok): midterm_project (dataset)
add(ok): midterm_project (dataset)
add(ok): .gitmodules (file)
add(ok): books/TLCL.pdf (file)
save(ok): . (dataset)
action summary:
  add (ok: 3)
  delete (ok: 1)
  save (notneeded: 2, ok: 2)

9.2.4. Copying files

Let’s create a copy of an annexed file, using the Unix command cp to copy.

$ cp books/TLCL.pdf copyofTLCL.pdf
$ datalad status
untracked: copyofTLCL.pdf (file)

That’s expected. The copy shows up as a new, untracked file. Let’s save it:

$ datalad save -m "add copy of TLCL.pdf"
add(ok): copyofTLCL.pdf (file)
save(ok): . (dataset)
$ git log -n 1 -p
commit ebbe6b93✂SHA1
Author: Elena Piscopia <elena@example.net>
Date:   Tue Jun 18 16:13:00 2019 +0000

    add copy of TLCL.pdf

diff --git a/copyofTLCL.pdf b/copyofTLCL.pdf
new file mode 120000
index 0000000..34328e2
--- /dev/null
+++ b/copyofTLCL.pdf
@@ -0,0 +1 @@
+.git/annex/objects/jf/3M/✂/MD5E-s2120211--06d1efcb✂MD5.pdf
\ No newline at end of file

That’s it.

Symlinks!

If you have read the additional content in the section Data integrity, you know that the same file content is only stored once, and copies of the same file point to the same location in the object tree.

Let’s check that out:

$ ls -l copyofTLCL.pdf
$ ls -l books/TLCL.pdf
lrwxrwxrwx 1 elena elena 128 2019-06-18 16:13 copyofTLCL.pdf -> .git/annex/objects/jf/3M/✂/MD5E-s2120211--06d1efcb✂MD5.pdf
lrwxrwxrwx 1 elena elena 131 2019-06-18 16:13 books/TLCL.pdf -> ../.git/annex/objects/jf/3M/✂/MD5E-s2120211--06d1efcb✂MD5.pdf

Indeed! Apart from their relative location (.git versus ../.git) their symlink is identical. Thus, even though two copies of the book exist in your dataset, your disk needs to store it only once.

In most cases, this is just an interesting fun-fact, but beware when dropping content with datalad drop (manual) as outlined in Removing annexed content entirely: If you drop the content of one copy of a file, all other copies will lose this content as well.

Finally, let’s clean up:

$ git reset --hard HEAD~1
HEAD is now at de2d23c move book back from midterm_project

9.2.5. Copying files across dataset boundaries

Instead of moving files across dataset boundaries, copying them is an easier and actually supported method. The DataLad command that can be used for this is datalad copy-file. This command allows to copy files (from any dataset or non-dataset location, annexed or not annexed) into a dataset. If the file is copied from a dataset and is annexed, its availability metadata is added to the new dataset as well, and there is no need for unannex’ing the or even retrieving its file contents. Let’s see this in action for a file stored in Git, and a file stored in annex:

$ datalad copy-file notes.txt midterm_project -d midterm_project
copy_file(ok): /home/me/dl-101/DataLad-101/notes.txt
add(ok): notes.txt (file)
save(ok): . (dataset)
$ datalad copy-file books/bash_guide.pdf midterm_project -d midterm_project
copy_file(ok): /home/me/dl-101/DataLad-101/books/bash_guide.pdf [/home/me/dl-101/DataLad-101/midterm_project/bash_guide.pdf]
save(ok): . (dataset)

Both files have been successfully transferred and saved to the subdataset, and no unannexing was necessary. notes.txt was annexed in the subdataset, though, as this subdataset was not configured with the text2git run procedure.

$ tree midterm_project
midterm_project
├── bash_guide.pdf -> .git/annex/objects/31/wQ/SHA256E-s1198170--d08f2c7b✂SHA162d6b24a20cb5abae4d4402c.pdf/SHA256E-s1198170--d08f2c7b✂SHA162d6b24a20cb5abae4d4402c.pdf
├── CHANGELOG.md
├── code
│   ├── README.md
│   └── script.py
├── input
│   └── iris.csv -> .git/annex/objects/qz/Jg/✂/MD5E-s3975--341a3b52✂MD5.csv
├── notes.txt -> .git/annex/objects/p1/jm/✂/MD5E-s5080--73ec6967✂MD5.txt
├── pairwise_relationships.png -> .git/annex/objects/q1/gp/✂/MD5E-s261062--025dc493✂MD5.png
├── prediction_report.csv -> .git/annex/objects/8q/6M/✂/MD5E-s345--a88cab39✂MD5.csv
└── README.md

2 directories, 9 files

The subdataset has two new commits as datalad copy-file can take care of saving changes in the copied-to dataset, and thus the new subdataset state would need to be saved in the superdataset.

$ datalad status -r
 modified: midterm_project (dataset)

Still, just as when we moved files across dataset boundaries, the files’ provenance record is lost:

$ cd midterm_project
$ git log notes.txt
commit 5b8d330f✂SHA1
Author: Elena Piscopia <elena@example.net>
Date:   Tue Jun 18 16:13:00 2019 +0000

    [DATALAD] Recorded changes

Nevertheless, copying files with datalad copy-file is easier and safer than moving them with standard Unix commands, especially so for annexed files. A more detailed introduction to datalad copy-file and a concrete use case can be found in the online-handbook.

Let’s clean up:

$ git reset --hard HEAD~2
HEAD is now at b2b6855 move book back from midterm_project

9.2.6. Moving/renaming a subdirectory or subdataset

Moving or renaming subdirectories, especially if they are subdatasets, can be a minefield. But in principle, a safe way to proceed is using the Unix mv command to move or rename, and the datalad save to clean up afterwards, just as in the examples above. Make sure to not use git mv, especially for subdatasets.

Let’s, for example, rename the books directory:

$ mv books/ readings
$ datalad status
untracked: readings (directory)
  deleted: books/TLCL.pdf (symlink)
  deleted: books/bash_guide.pdf (symlink)
  deleted: books/byte-of-python.pdf (symlink)
  deleted: books/progit.pdf (symlink)
$ datalad save -m "renamed directory"
delete(ok): books/TLCL.pdf (symlink)
delete(ok): books/bash_guide.pdf (symlink)
delete(ok): books/byte-of-python.pdf (symlink)
delete(ok): books/progit.pdf (symlink)
add(ok): readings/TLCL.pdf (file)
add(ok): readings/bash_guide.pdf (file)
add(ok): readings/byte-of-python.pdf (file)
add(ok): readings/progit.pdf (file)
save(ok): . (dataset)

This is easy, and complication free. Moving (as in: changing the location, instead of the name) the directory would work in the same fashion, and a datalad save would fix broken symlinks afterwards. Let’s quickly clean this up:

$ git reset --hard HEAD~1
HEAD is now at de2d23c move book back from midterm_project

But let’s now try to move the longnow subdataset into the root of the superdataset:

$ mv recordings/longnow .
$ datalad status
untracked: longnow (directory)
  deleted: recordings/longnow (dataset)
$ datalad save -m "moved subdataset"
delete(ok): recordings/longnow (dataset)
add(ok): longnow (dataset)
add(ok): .gitmodules (file)
save(ok): . (dataset)
$ datalad status
nothing to save, working tree clean

This seems fine, and it has indeed worked. However, reverting a commit like this is tricky, at the moment. This could lead to trouble if you at a later point try to revert or rebase chunks of your history including this move. Therefore, if you can, try not to move subdatasets around. For now we’ll clean up in a somewhat “hacky” way: Reverting, and moving remaining subdataset contents back to their original place by hand to take care of the unwanted changes the commit reversal introduced.

$ git reset --hard HEAD~1
warning: unable to rmdir 'longnow': Directory not empty
HEAD is now at de2d23c move book back from midterm_project
$ mv -f longnow recordings

The take-home message therefore is that it is best not to move subdatasets, but very possible to move subdirectories if necessary. In both cases, do not attempt moving with the git mv, but stick with mv and a subsequent datalad save.

9.2.7. Moving/renaming a superdataset

Once created, a DataLad superdataset may not be in an optimal place on your file system, or have the best name.

After a while, you might think that the dataset would fit much better into /home/user/research_projects/ than in /home/user/Documents/MyFiles/tmp/datalad-test/. Or maybe at some point, a long name such as My-very-first-DataLad-project-wohoo-I-am-so-excited does not look pretty in your terminal prompt anymore, and going for finance-2019 seems more professional.

These will be situations in which you want to rename or move a superdataset. Will that break anything?

In all standard situations, no, it will be completely fine. You can use standard Unix commands such as mv to do it, and also whichever graphical user interface or explorer you may use.

Beware of one thing though: If your dataset either is a sibling or has a sibling with the source being a path, moving or renaming the dataset will break the linkage between the datasets. This can be fixed easily though. We can try this in the Find-out-more on adjusting sibling URLs.

If a renamed/moved dataset is a sibling…

As section DIY configurations explains, each sibling is registered in .git/config in a “submodule” section. Let’s look at how our sibling “roommate” is registered there:

$ cat .git/config
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	editor = nano
[annex]
	uuid = 46b169aa-bb91-42d6-be06-355d957fb4f7
	version = 10
[filter "annex"]
	smudge = git-annex smudge -- %f
	clean = git-annex smudge --clean -- %f
	process = git-annex filter-process
[submodule "recordings/longnow"]
	active = true
	url = https://github.com/datalad-datasets/longnow-podcasts.git
[remote "roommate"]
	url = ../mock_user/DataLad-101
	fetch = +refs/heads/*:refs/remotes/roommate/*
	annex-uuid = ✂UUID✂
	annex-ignore = false
[submodule "midterm_project"]
	active = true
	url = ./midterm_project
[submodule "longnow"]
	active = true
	url = https://github.com/datalad-datasets/longnow-podcasts.git

As you can see, its “url” is specified as a relative path. Say your room mate’s directory is a dataset you would want to move. Let’s see what happens if we move the dataset such that the path does not point to the dataset anymore:

$ # add an intermediate directory
$ cd ../mock_user
$ mkdir onemoredir
$ # move your room mates dataset into this new directory
$ mv DataLad-101 onemoredir

This means that relative to your DataLad-101, your room mates dataset is not at ../mock_user/DataLad-101 anymore, but in ../mock_user/onemoredir/DataLad-101. The path specified in the configuration file is thus wrong now.

$ # navigate back into your dataset
$ cd ../DataLad-101
$ # attempt a datalad update
$ datalad update
update(error): . (dataset) [Fetch failed: CommandError(CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false fetch --verbose --progress --no-recurse-submodules --prune roommate' failed with exitcode 128 under /home/me/dl-101/DataLad-101 [err: 'fatal: '../mock_user/DataLad-101' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.'])] [CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false fetch --verbose --progress --no-recurse-submodules --prune roommate' failed with exitcode 128 under /home/me/dl-101/DataLad-101 [err: 'fatal: '../mock_user/DataLad-101' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.']]

Here we go:

'fatal: '../mock_user/DataLad-101' does not appear to be a git repository
 fatal: Could not read from remote repository.

Git seems pretty insistent (given the amount of error messages) that it cannot seem to find a Git repository at the location the .git/config file specified. Luckily, we can provide this information. Edit the file with an editor of your choice and fix the path from url = ../mock_user/DataLad-101 to url = ../mock_user/onemoredir/DataLad-101.

Below, we are using the stream editor sed for this operation.

$ sed -i 's/..\/mock_user\/DataLad-101/..\/mock_user\/onemoredir\/DataLad-101/' .git/config

This is how the file looks now:

$ cat .git/config
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	editor = nano
[annex]
	uuid = 46b169aa-bb91-42d6-be06-355d957fb4f7
	version = 10
[filter "annex"]
	smudge = git-annex smudge -- %f
	clean = git-annex smudge --clean -- %f
	process = git-annex filter-process
[submodule "recordings/longnow"]
	active = true
	url = https://github.com/datalad-datasets/longnow-podcasts.git
[remote "roommate"]
	url = ../mock_user/onemoredir/DataLad-101
	fetch = +refs/heads/*:refs/remotes/roommate/*
	annex-uuid = ✂UUID✂
	annex-ignore = false
[submodule "midterm_project"]
	active = true
	url = ./midterm_project
[submodule "longnow"]
	active = true
	url = https://github.com/datalad-datasets/longnow-podcasts.git

Let’s try to update now:

$ datalad update
update(ok): . (dataset)

Nice! We fixed it! Therefore, if a dataset you move or rename is known to other datasets from its path, or identifies siblings with paths, make sure to adjust them in the .git/config file.

To clean up, we’ll redo the move of the dataset and the modification in .git/config.

$ cd ../mock_user && mv onemoredir/DataLad-101 .
$ rm -r onemoredir
$ cd ../DataLad-101 && sed -i 's/..\/mock_user\/onemoredir\/DataLad-101/..\/mock_user\/DataLad-101/' .git/config

9.2.8. Getting contents out of git-annex

Files in your dataset can either be handled by Git or Git-annex. Self-made or predefined configurations to .gitattributes, defaults, or the --to-git option to datalad save allow you to control which tool does what on up to single-file basis. Accidentally though, you may give a file of yours to git-annex when it was intended to be stored in Git, or you want to get a previously annexed file into Git.

Consider you intend to share the cropped .png images you created from the longnow logos. Would you publish your DataLad-101 dataset so GitHub or GitLab, these files would not be available to others, because annexed dataset contents cannot be published to these services. Even though you could find a third party service of your choice and publish your dataset and the annexed data as described in Beyond shared infrastructure, you are feeling lazy today. And since it is only two files, and they are quite small, you decide to store them in Git – this way, the files would be available without configuring an external data store.

To get a file out of the git-annex hands you need to unannex it. This is done with the git-annex command git annex unannex (manual). Let’s see how it works:

$ git annex unannex recordings/*logo_small.jpg
unannex recordings/interval_logo_small.jpg ok
unannex recordings/salt_logo_small.jpg ok
(recording state in git...)

Your dataset notices the unannexing of the files as follows.

$ git status
On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	deleted:    recordings/interval_logo_small.jpg
	deleted:    recordings/salt_logo_small.jpg

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	recordings/interval_logo_small.jpg
	recordings/salt_logo_small.jpg

Once files have been unannexed, they are “untracked” again, and you can save them into Git, either by adding a rule to .gitattributes, or with datalad save --to-git:

$ datalad save --to-git -m "save cropped logos to Git" recordings/*jpg
add(ok): recordings/interval_logo_small.jpg (file)
add(ok): recordings/salt_logo_small.jpg (file)
save(ok): . (dataset)

Note that git-annex keeps the previously annexed file’s content in the annex for safety, to prevent accidental data loss. If it is only few and small files that were unannexed, their size in the annex will not matter much. If it is a lot of files or larger files that were accidentally annexed, you may want to drop the left-behind content using git annex unused and git annex dropunused.

9.2.9. Getting all content out of the annex (removing the annex repo)

In case you want to get all annexed contents out of a Dataset at once, you could turn to git annex uninit. It is a command that can be used to stop using git annex entirely in a given repository/dataset. Running this command will unannex every file in the repository, remove all of git-annex’s other data, and remove the git-annex branch, leaving you with a normal Git repository plus the previously annexed files.

Note a datalad push will reinstate the git-annex branch if your dataset has siblings that still contain the annex branch.

9.2.10. Deleting (annexed) files/directories

Removing annexed file content from a dataset is possible in two different ways: Either by removing the file from the current state of the repository (which Git calls the worktree) but keeping the content in the history of the dataset, or by removing content entirely from a dataset and its history.

9.2.10.1. Removing a file, but keeping content in history

An rm <file> or rm -rf <directory> with a subsequent datalad save will remove a file or directory, and save its removal. The file content however will still be in the history of the dataset, and the file can be brought back to existence by going back into the history of the dataset or reverting the removal commit:

$ # download a file
$ datalad download-url -m "Added flower mosaic from wikimedia" \
  https://upload.wikimedia.org/wikipedia/commons/a/a5/Flower_poster_2.jpg \
  --path flowers.jpg
$ ls -l flowers.jpg
download_url(ok): /home/me/dl-101/DataLad-101/flowers.jpg (file)
add(ok): flowers.jpg (file)
save(ok): . (dataset)
lrwxrwxrwx 1 elena elena 128 2019-06-18 16:13  2013 flowers.jpg -> .git/annex/objects/7q/9Z/✂/MD5E-s4487679--3898ef0e✂MD5.jpg
$ # removal is easy:
$ rm flowers.jpg

This will lead to a dirty dataset status:

$ datalad status
  deleted: flowers.jpg (symlink)

If a removal happened by accident, a git checkout -- flowers.jpg would undo the removal at this stage. To stick with the removal and clean up the dataset state, datalad save will suffice:

$ datalad save -m "removed file again"
delete(ok): flowers.jpg (symlink)
save(ok): . (dataset)

This commits the deletion of the file in the dataset’s history. If this commit is reverted, the file comes back to existence:

$ git reset --hard HEAD~1
$ ls
HEAD is now at 682d11f Added flower mosaic from wikimedia
books
code
flowers.jpg
midterm_project
notes.txt
recordings

In other words, with an rm and subsequent datalad save, the symlink is removed, but the content is retained in the history.

9.2.10.2. Removing annexed content entirely

The command to remove file content entirely and irreversibly from a repository is the datalad drop command. This command will delete the content stored in the annex of the dataset, and can be very helpful to make a dataset more lean if the file content is either irrelevant or can be retrieved from other sources easily. Think about a situation in which a very large result file is computed by default in some analysis, but is not relevant for any project, and can thus be removed. Or if only the results of an analysis need to be kept, but the file contents from its input datasets can be dropped at these input datasets are backed-up else where. Because the command works on annexed contents, it will drop file content from a dataset, but it will retain the symlink for this file (as this symlink is stored in Git).

datalad drop can take any number of files. If an entire dataset is specified, all file content in sub-directories is dropped automatically, but for content in sub-datasets to be dropped, the -r/--recursive flag has to be included. By default, DataLad will not drop any content that does not have at least one verified remote copy that the content could be retrieved from again. It is possible to drop the downloaded image, because thanks to datalad download-url (manual) its original location in the web is known:

$ datalad drop flowers.jpg
drop(ok): flowers.jpg (file)

Currently, the file content is gone, but the symlink still exist. Opening the remaining symlink will fail, but the content can be obtained easily again with datalad get (manual):

$ datalad get flowers.jpg
get(ok): flowers.jpg (file) [from web...]

If a file has no verified remote copies, DataLad will only drop its content if the user enforces it using the --reckless [MODE] option, where [MODE] is either modification (drop despite unsaved modifications), availability (drop even though no other copy is known), undead (only for datasets; would drop a dataset without announcing its death to linked dataset clones) or kill (no safety checks at all are run). We will demonstrate this by generating an empty file:

$ dd if=/dev/zero | head -c 18520 > a.pdf
$ datalad save -m "add some file" a.pdf
add(ok): a.pdf (file)
save(ok): . (dataset)

DataLad will safeguard dropping content that it cannot retrieve again:

$ datalad drop a.pdf
drop(error): a.pdf (file) [unsafe; Could only verify the existence of 0 out of 1 necessary copy; (Use --reckless availability to override this check, or adjust numcopies.)]

But with --reckless availability it will work:

$ datalad drop --reckless availability a.pdf
drop(ok): a.pdf (file)

Note though that this file content is irreversibly gone now, and even going back in time in the history of the dataset will not bring it back into existence.

Finally, let’s clean up:

$ git reset --hard HEAD~2
HEAD is now at c278f6a save cropped logos to Git

9.2.11. Deleting content stored in Git

It is much harder to delete dataset content that is stored in Git compared to content stored in git-annex. Operations such as rm or git rm remove the file from the worktree, but not from its history, and they can be brought back to life just as annexed contents that were solely rm'ed. There is also no straightforward Git equivalent of drop. To accomplish a complete removal of a file from a dataset, we recommend the external tool git-filter-repo. It is a powerful and potentially very dangerous tool to rewrite Git history.

Usually, removing files stored in Git completely is not a common or recommended operation, as it involves quite aggressive rewriting of the dataset history. Sometimes, however, sensitive files, for example private SSH keys or passwords, or too many or too large files are accidentally saved into Git, and need to get out of the dataset history. The command git-filter-repo <path-specification> --force will “filter-out”, i.e., remove all files but the ones specified in <path-specification> from the dataset’s history. An advanced chapter in the online-handbook shows an example invocation.

9.2.12. Uninstalling or deleting subdatasets

Depending on the exact aim, different commands are of relevance for deleting a DataLad subdataset. One way to uninstall a dataset is the datalad drop command. To work on datasets, drop needs to be parametrized with --what all. If needed, add --recursive in case the dataset contains subdatasets, and a fitting --reckless mode, such as datalad drop --what all --reckless kill --recursive.

$ # clone a subdataset - the content is irrelevant, so why not a cloud :)
$ datalad clone -d . \
 https://github.com/datalad-datasets/disneyanimation-cloud.git \
 cloud
[INFO] Remote origin not usable by git-annex; setting annex-ignore
install(ok): cloud (dataset)
add(ok): cloud (dataset)
add(ok): .gitmodules (file)
save(ok): . (dataset)
add(ok): .gitmodules (file)
save(ok): . (dataset)

To uninstall the dataset, you can use

$ datalad drop --what all --reckless kill --recursive cloud
uninstall(ok): cloud (dataset)

Note that the dataset is still known in the dataset, and not completely removed. A datalad get [-n/--no-data] cloud would install the dataset again.

In case one wants to fully delete a subdataset from a dataset, the datalad remove (manual) command is relevant[3]. It needs a pointer to the root of the superdataset with the -d/--dataset flag, a path to the subdataset to be removed, and optionally a commit message (-m/--message) or recursive specification (-r/--recursive). To remove a subdataset, we will install the uninstalled subdataset again, and subsequently remove it with the datalad remove command:

$ datalad get -n cloud
[INFO] Remote origin not usable by git-annex; setting annex-ignore
install(ok): /home/me/dl-101/DataLad-101/cloud (dataset) [Installed subdataset in order to get /home/me/dl-101/DataLad-101/cloud]
$ # delete the subdataset
$ datalad remove -m "remove obsolete subds" -d . cloud
uninstall(ok): cloud (dataset)
remove(ok): cloud (dataset)
add(ok): .gitmodules (file)
save(ok): . (dataset)

Note that for both commands a pointer to the current directory will not work. datalad remove . or datalad drop . will fail, even if the command is executed in a subdataset instead of the top-level superdataset – you need to execute the command from a higher-level directory.

9.2.13. Deleting a superdataset

If for whatever reason you at one point tried to remove a DataLad dataset, whether with a GUI or the command line call rm -rf <directory>, you likely have seen permission denied errors such as

rm: cannot remove '<directory>/.git/annex/objects/Mz/M1/MD5E-s422982--2977b5c6ea32de1f98689bc42613aac7.jpg/MD5E-s422982--2977b5c6ea32de1f98689bc42613aac7.jpg': Permission denied
rm: cannot remove '<directory>/.git/annex/objects/FP/wv/MD5E-s543180--6209797211280fc0a95196b0f781311e.jpg/MD5E-s543180--6209797211280fc0a95196b0f781311e.jpg': Permission denied
 [...]

This error indicates that there is write-protected content within .git that cannot not be deleted. What is this write-protected content? It’s the file content stored in the object tree of git-annex. If you want, you can re-read the section on Data integrity to find out how git-annex revokes write permission for the user to protect the file content given to it. To remove a dataset with annexed content one has to regain write permissions to everything in the dataset. This is done with the Unix chmod command:

$ chmod -R u+w <dataset>

This recursively (-R, i.e., throughout all files and (sub)directories) gives users (u) write permissions (+w) for the dataset.

Afterwards, rm -rf <dataset> will succeed.

However, instead of rm -rf, a faster way to remove a dataset is using either datalad drop or datalad remove: Run datalad drop -d <dataset> or datalad remove -d <dataset> outside of the superdataset to remove a top-level dataset with all its contents. Likely, both --recursive and --reckless [availability|undead|kill] flags are necessary to traverse into subdatasets and to remove content that does not have verified remotes.

Be aware, though, that deleting a dataset in which ever way will irretrievably delete the dataset, it’s contents, and it’s history.

9.2.14. Summary

To sum up, file system management operations are safe and easy. Even if you are currently confused about one or two operations, worry not – the take-home-message is simple: Use datalad save whenever you move or rename files. Be mindful that a datalad status can appear unintuitive or that symlinks can break if annexed files are moved, but all of these problems are solved after a datalad save command. Apart from this command, having a clean dataset status prior to doing anything is your friend as well. It will make sure that you have a neat and organized commit history, and no accidental commits of changes unrelated to your file system management operations. The only operation you should beware of is moving subdatasets around – this can be a minefield. With all of these experiences and tips, you feel confident that you know how to handle your datasets files and directories well and worry-free.

Footnotes