2.2. DataLad, Re-Run!

So far, you created a .tsv file of all speakers and talk titles in the longnow/ podcasts subdataset. Let’s actually take a look into this file now:

$ less recordings/podcasts.tsv
2003-11-15	Brian Eno  The Long Now
2003-12-13	Peter Schwartz  The Art Of The Really Long View
2004-01-10	George Dyson  There s Plenty of Room at the Top  Long term Thinking About Large scale Computing
2004-02-14	James Dewar  Long term Policy Analysis
2004-03-13	Rusty Schweickart  The Asteroid Threat Over the Next 100 000 Years
2004-04-10	Daniel Janzen  Third World Conservation  It s ALL Gardening
2004-05-15	David Rumsey  Mapping Time
2004-06-12	Bruce Sterling  The Singularity  Your Future as a Black Hole
2004-07-10	Jill Tarter  The Search for Extra terrestrial Intelligence  Necessarily a Long term Strategy
2004-08-14	Phillip Longman  The Depopulation Problem
2004-09-11	Danny Hillis  Progress on the 10 000 year Clock
2004-10-16	Paul Hawken  The Long Green
2004-11-13	Michael West  The Prospects of Human Life Extension
2004-12-04	Ken Dychtwald  The Consequences of Human Life Extension

Not too bad, and certainly good enough for the podcast night people. What’s been cool about creating this file is that it was created with a script within a datalad run command. Thanks to datalad run, the output file podcasts.tsv is associated with the script it generated.

Upon reviewing the list you realized that you made a mistake, though: you only listed the talks in the SALT series (the Long_Now__Seminars_About_Long_term_Thinking/ directory), but not in the Long_Now__Conversations_at_The_Interval/ directory. Let’s fix this in the script. Replace the contents in code/list_titles.sh with the following, fixed script:

Here’s a script adjustment for Windows users

Please use an editor of your choice to replace the contents of list_titles.sh inside of the code directory with the following:

for i in recordings/longnow/Long_Now*/*.mp3; do
   # get the filename
   base=$(basename "$i");
   # strip the extension
   base=${base%.mp3};
   # date as yyyy-mm-dd
   printf "${base%%__*}\t" | tr '_' '-';
   # name and title without underscores
   printf "${base#*__}\n" | tr '_' ' ';
done
$ cat << EOT >| code/list_titles.sh
for i in recordings/longnow/Long_Now*/*.mp3; do
   # get the filename
   base=\$(basename "\$i");
   # strip the extension
   base=\${base%.mp3};
   printf "\${base%%__*}\t" | tr '_' '-';
   # name and title without underscores
   printf "\${base#*__}\n" | tr '_' ' ';

done
EOT

Because the script is now modified, save the modifications to the dataset. We can use the shorthand “BF” to denote “Bug fix” in the commit message.

$ datalad status
 modified: code/list_titles.sh (file)
$ datalad save -m "BF: list both directories content" code/list_titles.sh
add(ok): code/list_titles.sh (file)
save(ok): . (dataset)
action summary:
  add (ok: 1)
  save (ok: 1)

What we could do is run the same datalad run command as before to recreate the file, but now with all of the contents:

# do not execute this!
$ datalad run -m "create a list of podcast titles" \
  "bash code/list_titles.sh > recordings/podcasts.tsv"

However, think about any situation where the command would be longer than this, or that is many months past the first execution. It would not be easy to remember the command, nor would it be very convenient to copy it from the run record.

Luckily, a fellow student remembered the DataLad way of re-executing a run command, and he’s eager to show it to you.

“In order to re-execute a datalad run command, find the commit and use its shasum (or a tag, or anything else that Git understands) as an argument for the datalad rerun command (datalad-rerun manual)! That’s it!”, he says happily.

So you go ahead and find the commit shasum in your history:

$ git log -n 2
commit fa3e98aac15a8fc91a015088b1845d2f40278fa1
Author: Elena Piscopia <elena@example.net>
Date:   Fri Jan 29 08:38:37 2021 +0100

    BF: list both directories content

commit ee3c90f1a89b46e4383d4bda1c3663aa6b08fdaa
Author: Elena Piscopia <elena@example.net>
Date:   Fri Jan 29 08:38:35 2021 +0100

    [DATALAD RUNCMD] create a list of podcast titles

Take that shasum and paste it after datalad rerun (the first 6-8 characters of the shasum would be sufficient, here we’re using all of them).

$ datalad rerun ee3c90f1a89b46e4383d4bda1c3663aa6b08fdaa
[INFO] run commit ee3c90f; (create a list of ...)
[INFO] == Command start (output follows) ===== 
[INFO] == Command exit (modification check follows) ===== 
add(ok): recordings/podcasts.tsv (file)
save(ok): . (dataset)
action summary:
  add (ok: 1)
  save (notneeded: 1, ok: 1)
  unlock (notneeded: 1)

Now DataLad has made use of the run record, and re-executed the original command based on the information in it. Because we updated the script, the output podcasts.tsv has changed and now contains the podcast titles of both subdirectories. You’ve probably already guessed it, but the easiest way to check whether a datalad rerun has changed the desired output file is to check whether the rerun command appears in the datasets history: If a datalad rerun does not add or change any content in the dataset, it will also not be recorded in the history.

$ git log -n 1
commit b5919c52323645110d18077191166b7cc58cc6ca
Author: Elena Piscopia <elena@example.net>
Date:   Fri Jan 29 08:38:39 2021 +0100

    [DATALAD RUNCMD] create a list of podcast titles
    
    === Do not change lines below ===
    {
     "chain": [
      "ee3c90f1a89b46e4383d4bda1c3663aa6b08fdaa"
     ],
     "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv",
     "dsid": "c8774a5b-c857-4751-bce4-e58c6be73349",
     "exit": 0,
     "extra_inputs": [],
     "inputs": [],
     "outputs": [],
     "pwd": "."
    }
    ^^^ Do not change lines above ^^^

In the dataset’s history, we can see that a new datalad run was recorded. This action is committed by DataLad under the original commit message of the run command, and looks just like the previous datalad run commit apart from the execution time.

Two cool tools that go beyond the git log are the datalad diff (datalad-diff manual) and git diff commands. Both commands can report differences between two states of a dataset. Thus, you can get an overview of what changed between two commits. Both commands have a similar, but not identical structure: datalad diff compares one state (a commit specified with -f/--from, by default the latest change) and another state from the dataset’s history (a commit specified with -t/--to). Let’s do a datalad diff between the current state of the dataset and the previous commit (called “HEAD~1” in Git terminology1):

please use datalad diff –from master –to HEAD~1

While this example works on Unix file systems, it will not provide the same output on Windows. This is due to different file handling on Windows. When executing this command, you will see all files being modified between the most recent and the second-most recent commit. On a technical level, this is correct given the underlying file handling on Windows, and chapter Under the hood: git-annex will shed light on why that is.

For now, to get the same output as shown in the code snippet below, use the following command where main (or master) is the name of your default branch:

datalad diff --from main --to HEAD~1

The --from argument specifies a different starting point for the comparison - the main or master branch, which would be the starting point on most Unix-based systems.

$ datalad diff --to HEAD~1
 modified: recordings/podcasts.tsv (file)

This indeed shows the output file as “modified”. However, we do not know what exactly changed. This is a task for git diff (get out of the diff view by pressing q):

$ git diff HEAD~1
diff --git a/recordings/podcasts.tsv b/recordings/podcasts.tsv
index f691b53..d77891d 100644
--- a/recordings/podcasts.tsv
+++ b/recordings/podcasts.tsv
@@ -1,3 +1,31 @@
+2017-06-09	How Digital Memory Is Shaping Our Future  Abby Smith Rumsey
+2017-06-09	Pace Layers Thinking  Stewart Brand  Paul Saffo
+2017-06-09	Proof  The Science of Booze  Adam Rogers
+2017-06-09	Seveneves at The Interval  Neal Stephenson
+2017-06-09	Talking with Robots about Architecture  Jeffrey McGrew
+2017-06-09	The Red Planet for Real  Andy Weir
+2017-07-03	Transforming Perception  One Sense at a Time  Kara Platoni
+2017-08-01	How Climate Will Evolve Government and Society  Kim Stanley Robinson
+2017-09-01	Envisioning Deep Time  Jonathon Keats
+2017-10-01	Thinking Long term About the Evolving Global Challenge  The Refugee Reality
+2017-11-01	The Web In An Eye Blink  Jason Scott
+2017-12-01	Ideology in our Genes  The Biological Basis for Political Traits  Rose McDermott
+2017-12-07	Can Democracy Survive the Internet   Nathaniel Persily
+2018-01-02	The New Deal You Don t Know  Louis Hyman

This output actually shows the precise changes between the contents created with the first version of the script and the second script with the bug fix. All of the files that are added after the second directory was queried as well are shown in the diff, preceded by a +.

Quickly create a note about these two helpful commands in notes.txt:

$ cat << EOT >> notes.txt
There are two useful functions to display changes between two
states of a dataset: "datalad diff -f/--from COMMIT -t/--to COMMIT"
and "git diff COMMIT COMMIT", where COMMIT is a shasum of a commit
in the history.

EOT

Finally, save this note.

$ datalad save -m "add note datalad and git diff"
add(ok): notes.txt (file)
save(ok): . (dataset)
action summary:
  add (ok: 1)
  save (ok: 1)

Note that datalad rerun can re-execute the run records of both a datalad run or a datalad rerun command, but not with any other type of datalad command in your history such as a datalad save on results or outputs after you executed a script. Therefore, make it a habit to record the execution of scripts by plugging it into datalad run.

This very basic example of a datalad run is as simple as it can get, but it is already convenient from a memory-load perspective: Now you do not need to remember the commands or scripts involved in creating an output. DataLad kept track of what you did, and you can instruct it to “rerun” it. Also, incidentally, we have generated provenance information. It is now recorded in the history of the dataset how the output podcasts.tsv came into existence. And we can interact with and use this provenance information with other tools than from the machine-readable run record. For example, to find out who (or what) created or modified a file, give the file path to git log (prefixed by --):

use “git log master – recordings/podcasts.tsv”

A previous Windows Wit already advised to append main or master, the common “default branch”, to any command that starts with git log. Here, the last part of the command specifies a file (-- recordings/podcasts.tsv). Please append main or master to git log, prior to the file specification.

$ git log -- recordings/podcasts.tsv
commit b5919c52323645110d18077191166b7cc58cc6ca
Author: Elena Piscopia <elena@example.net>
Date:   Fri Jan 29 08:38:39 2021 +0100

    [DATALAD RUNCMD] create a list of podcast titles
    
    === Do not change lines below ===
    {
     "chain": [
      "ee3c90f1a89b46e4383d4bda1c3663aa6b08fdaa"
     ],
     "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv",
     "dsid": "c8774a5b-c857-4751-bce4-e58c6be73349",
     "exit": 0,
     "extra_inputs": [],
     "inputs": [],
     "outputs": [],
     "pwd": "."
    }
    ^^^ Do not change lines above ^^^

commit ee3c90f1a89b46e4383d4bda1c3663aa6b08fdaa
Author: Elena Piscopia <elena@example.net>
Date:   Fri Jan 29 08:38:35 2021 +0100

    [DATALAD RUNCMD] create a list of podcast titles
    
    === Do not change lines below ===
    {
     "chain": [],
     "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv",
     "dsid": "c8774a5b-c857-4751-bce4-e58c6be73349",
     "exit": 0,
     "extra_inputs": [],
     "inputs": [],
     "outputs": [],
     "pwd": "."
    }
    ^^^ Do not change lines above ^^^

Neat, isn’t it?

Still, this datalad run was very simple. The next section will demonstrate how datalad run becomes handy in more complex standard use cases: situations with locked contents.

But prior to that, make a note about datalad run and datalad rerun in your notes.txt file.

$ cat << EOT >> notes.txt
The datalad run command can record the impact a script or command has
on a Dataset. In its simplest form, datalad run only takes a commit
message and the command that should be executed.

Any datalad run command can be re-executed by using its commit shasum
as an argument in datalad rerun CHECKSUM. DataLad will take
information from the run record of the original commit, and re-execute
it. If no changes happen with a rerun, the command will not be written
to history. Note: you can also rerun a datalad rerun command!

EOT

Finally, save this note.

$ datalad save -m "add note on basic datalad run and datalad rerun"
add(ok): notes.txt (file)
save(ok): . (dataset)
action summary:
  add (ok: 1)
  save (ok: 1)

Footnotes

1

The section Back and forth in time will elaborate more on common git commands and terminology.