Workshop Syllabus
The Unix Shell
- Files and directories:
pwd, cd, ls, mkdir, ...
- History and tab completion
- Pipes and redirection
- Looping over files
- Creating and running shell scripts
- Finding things:
grep, find, ...
- Reference...
Version Control with Git
- Creating a repository
- Recording changes to files:
add, commit, ...
- Viewing changes:
status, diff, ...
- Ignoring files
- Working on the web:
clone, pull, push, ...
- Resolving conflicts
- Open licenses
- Where to host work, and why
- Reference...
Programming in Python
- Using libraries
- Working with arrays
- Reading and plotting data
- Creating and using functions
- Loops and conditionals:
for, if, else, ...
- Defensive programming
- Using Python from the command line
- Reference...
High Throughput Computing
- Log in to a remote machine - secured shell
ssh
- Data Transfer
scp rsync , globus
- SSH keys: Access without password
- Example: Running Python and unix scripts on a remote server
- Properties of Distributed High-Throughput Computing
- Scheduling Computing Jobs via HTCondor
- Properties of Distributed High-Throughput Computing
- Using software modules and
tutorial
- Examples of running R and Python scripts
- HTCondor job scripts -
condor_submit, condor_q , condor_rm
- Scaling up the resources: Example Unix and Python scripts
-
stash - the data storage solution
- Scientfic workflows -
DAGMan , Pegasus