Join us at distribits 2024!

Technologies for distributed data management

The first distribits meeting will happen on April 18th to April 20th, 2024, at “Haus der Universität”, Düsseldorf, Germany, with the aim of bringing together enthusiasts of tools and workflows in the domain of distributed data.

Go to distribits.live for more information and to register!

The Handbook

Welcome!

This handbook is a living resource about why and – more importantly – how to use DataLad. It aims to provide novices and advanced users of all backgrounds with both the basics of DataLad and start-to-end use cases of specific applications. If you want to get hands-on experience and learn DataLad, the Basics part of this book will teach you. If you want to know what is possible, the use cases will show you. And if you want to help others to get started with DataLad, the companion repository provides free and open source teaching material tailored to the handbook.

Before you read on, please note that this version of the handbook is based on at least DataLad version 0.17, and the higher your version of DataLad is, the better. The section Installation and configuration will set you up with what you need if you currently do not have DataLad 0.17 or higher installed.

If you’re new here, please start the handbook here. Alternatively, try to identify with one of several user-types in this user specific guide to the handbook.

The handbook is currently in beta stage.

If you would be willing to provide feedback on its contents, please get in touch.

What this is all about

• Introduction
  • 1. A brief overview of DataLad
  • 2. How to use the handbook
  • 3. Installation and configuration
  • 4. General prerequisites
  • 5. What you really need to know

The fundamentals of DataLad

• Basics
  • 1. DataLad datasets
    • 1.1. Create a dataset
    • 1.2. Populate a dataset
    • 1.3. Modify content
    • 1.4. Install datasets
    • 1.5. Dataset nesting
    • 1.6. Summary
  • 2. DataLad, Run!
    • 2.1. Keeping track
    • 2.2. DataLad, Re-Run!
    • 2.3. Input and output
    • 2.4. Clean desk
    • 2.5. Summary
  • 3. Under the hood: git-annex
    • 3.1. Data safety
    • 3.2. Data integrity
  • 4. Collaboration
    • 4.1. Looking without touching
    • 4.2. Where’s Waldo?
    • 4.3. Retrace and reenact
    • 4.4. Stay up to date
    • 4.5. Networking
    • 4.6. Summary
  • 5. Tuning datasets to your needs
    • 5.1. DIY configurations
    • 5.2. More on DIY configurations
    • 5.3. Configurations to go
    • 5.4. Summary
  • 6. Make the most out of datasets
    • 6.1. A Data Analysis Project with DataLad
    • 6.2. YODA: Best practices for data analyses in a dataset
    • 6.3. YODA-compliant data analysis projects
    • 6.4. Summary
  • 7. One step further
    • 7.1. More on Dataset nesting
    • 7.2. Computational reproducibility with software containers
    • 7.3. Summary
  • 8. Third party infrastructure
    • 8.1. Beyond shared infrastructure
    • 8.2. Publishing datasets to Git repository hosting
    • 8.3. Walk-through: Dropbox as a special remote
    • 8.4. Walk-through: Amazon S3 as a special remote
    • 8.5. Walk-through: Git LFS as a special remote on GitHub
    • 8.6. Walk-through: Dataset hosting on GIN
    • 8.7. Built-in data export
    • 8.8. Keeping (some) dataset contents private
    • 8.9. Overview: The datalad push command
    • 8.10. Summary
  • 9. Help yourself
    • 9.1. What to do if things go wrong
    • 9.2. Miscellaneous file system operations
    • 9.3. Back and forth in time
    • 9.4. How to get help
    • 9.5. Gists

Beyond the Basics

• Advanced
  • 1. Advanced options
    • 1.1. How to hide content from DataLad
    • 1.2. DataLad’s result hooks
    • 1.3. Configure custom data access
    • 1.4. Remote Indexed Archives for dataset storage and backup
    • 1.5. Prioritizing subdataset clone locations
    • 1.6. Subsample datasets using datalad copy-file
  • 2. Go big or go home
    • 2.1. Going big with DataLad
    • 2.2. Calculate in greater numbers
    • 2.3. Fixing up too-large datasets
    • 2.4. Summary
  • 3. Computing on clusters
    • 3.1. DataLad on High Throughput or High Performance Compute Clusters
    • 3.2. DataLad-centric analysis with job scheduling and parallel computing
    • 3.3. Walkthrough: Parallel ENKI preprocessing with fMRIprep
  • 4. Better late than never
    • 4.1. Transitioning existing projects into DataLad
  • 5. Special purpose showrooms
    • 5.1. Reproducible machine learning analyses: DataLad as DVC
  • 6. DataLad internals
    • 6.1. DataLad’s internal design
    • 6.2. Contributing to DataLad
  • 7. DataLad Extensions
    • 7.1. DataLad extensions
    • 7.2. What’s next in DataLad?
    • 7.3. Metadata-Management with MetaLad
    • 7.4. DataCat - a shiny front-end for your dataset
    • 7.5. DataLad in a graphical user interface
    • 7.6. Create your own extension

Hands-on real-world applications with step-by-step recipes

• Use cases
  • A typical collaborative data management workflow
  • Basic provenance tracking
  • Writing a reproducible paper
  • Student supervision in a research project
  • A basic automatically and computationally reproducible neuroimaging analysis
  • An automatically and computationally reproducible neuroimaging analysis from scratch
  • Scaling up: Managing 80TB and 15 million files from the HCP release
  • Building a scalable data storage for scientific computing
  • Using Globus as a data store for the Canadian Open Neuroscience Portal
  • DataLad for reproducible machine-learning analyses
  • Encrypted data storage and transport
  • Contributing

Appendix

Further information and references

• Glossary
• Frequently Asked Questions
• DataLad cheat sheet
• Contributing
• Teaching with the DataLad Handbook
• Acknowledgements
• Tell me what you are and I tell you where to start
• Handbook Poster from the 2020 (virtual) OHBM
• OpenNeuro Quickstart Guide: Accessing OpenNeuro datasets via DataLad
• So… Windows… eh?
• How to name a file: Interoperability considerations

Code lists from chapters

Easy access to copy-paste snippets for workshops

• About code lists
• Code from chapter: 01_dataset_basics
• Code from chapter: 02_reproducible_execution
• Code from chapter: 10_yoda
• OHBM Brainhack TrainTrack: DataLad
• OHBM 2020 Open Science Room: Reproducible Research Objects with DataLad
• An introduction to DataLad with a focus on ML
• DataLad tutorial at the MPI Leipzig
• An introduction to DataLad at the MPI Berlin
• An introduction to DataLad for the ABCD ReproNim course week 8b
• An introduction to DataLad for Yale
• An introduction to DataLad at the ReproNim DGPA workshop
• Neurohackdemy 2022: Data Management for Neuroimaging with DataLad