Bash-tutorial

With this bash tutorial the basics of programming in bash are introduced. At the end some (complicated but useful) examples for scientific use are introduced. This tutorial is not yet finished.

FEEL FREE TO ADD SOMETHING WHEREVER YOU WANT!

Beside the bash there are a bunch of other shells (eg. tsh, chs, tcsh, zsh, ash, psh etc.). Check if you are in the bash. 

                  $ ps
                  PID TTY          TIME CMD
                  12134 pts/5    00:00:00 bash
                  12490 pts/5    00:00:00 ps

If another shell is currently active, start into the bash: 

                  $ bash

Useful programmes

As this list would be far too long, there is an extra webpage where you can look up some of the most useful programmes:

List of useful bash programmes and commands

Some basic bash

Variables

Define a variable 

                  $ var=MPIA

Print a variable on the screen (plus some additional info) 

                  $ echo Insitute: $var
                  MPIA

Careful with empty spaces! You have to use quotation marks! 

                  $ var=MPIA Heidelberg
                  bash: Heidelberg: command not found

There are two different types of quotation marks. The double quotation marks replace variables inside, single quotation marks don't! But you can force special characters like the dollar sign in double quoted environments by using the escape character. 

                  $ echo "$var Heidelberg"
                  MPIA Heidelberg
                  $ echo '$var Heidelberg'
                  $var Heidelberg''
                  $ echo "\$var Heidelberg"
                  $var Heidelberg

If no empty space follows the variable you can tell explicitly the variable name by using curly brackets {} 

                  $ echo "Software: ${var}phot"
                  Software: MPIAphot

You can save terminal output from other programmes into variables using backquotes` ` or $( ) 

                  $ var=`ls`
                  $ echo $var
                  data name.sh test.sh
                  $ var=$(ls)
                  $ echo $var
                  data name.sh test.sh

You can also use variables within backquotes 

                  $ type='.pdf'
                  $ files=`ls *$type`
                  $ echo $files
                  thesis.pdf linux.pdf

You can also show all variables currently defined in the open shell. Most of them are system variables and those you defined before. 

                  $ env
                  SHELL=/bin/bash
                  TERM=xterm
                  EDITOR=/usr/bin/emacs
                  HOME=/home/user
                  PAGER=less
                  ...

Lists and arrays

A list is a string separated by empty spaces 

                  $ list="1 2 3 4 5"
                  $ echo $list
                  1 2 3 4 5

An array is defined by encompassing a list with brackets. Elements are accessed by $array[i]. We cannot simply print the whole array with echo $array (this only returns the first element), but we have to explicitly tell the programme what to do. This is done with the @, which means "everything". Attention: The first element of the array is $array[0]! And even more attention: Here I use the syntax "@ ]" with an empty space between the "@" and the "]" because otherwise the Wiki would close the style-sheet-environment. You have to remove the empty space in order to make this valid code!) 

                  $ array=(1 2 3 4 5)
                  $ echo $array
                  1
                  $ echo ${array[@ ]}
                  1 2 3 4 5
                  $ echo ${array[1]}
                  2

The length of a list can be determined by using the hash #. There is a difference between lists and arrays. For lists the length of the string is returned, for arrays it's the number of elements. (Attention: Here I use the syntax "@ ]" with an empty space between the "@" and the "]" because otherwise the Wiki would close the style-sheet-environment. You have to remove the empty space in order to make this valid code!) 

                  $ echo ${#list}
                  9
                  $ echo ${#array[@ ]}
                  5

Mathematics

Mathematics can be done using the double brackets (or also [ ] ). The bash can only do integer operations. 

                  $ echo $((2+4))
                  6
                  $ echo $[2+4]
                  6
                  $ echo $[2*4]
                  8
                  $ echo $[2/4]
                  0

The bash can also do floating point operations (but this section does not exist yet - sorry)

Bash scripts

You can write all your commands into an external file and then execute it. Open an editor of your choice and write into the very first line 

                  #!/bin/bash

so that the bash knows how to interpret this script. Then the commands of your choice follow. Save this file (e.g. script.sh) and give yourself the execution permissions 

                  $ chmod +x script.sh

Then execute it 

                  $ ./script.sh

Using the bash more efficiently

Everytime you start a new terminal some scripts are automatically executed. The most important one is ~/.bashrc. You can e.g. add some directories to your default path so that you can execute them wherever you want. Open ~/.bashrc in an editor of your choice and add the following line to it. You have to use export in order to make the changes permanent (=after this script ends) 

 export PATH=$PATH:/home/mpiauser

After saving and echoing the system-variable $PATH you will find something like this 

 echo $PATH
                  .:/usr/bin:/usr/local/bin:...:/home/mpiauser

You can write more commands. A useful one is to define some aliases, which are sort of shortcuts for otherwise long commands 

 alias work='cd /home/mpiauser/work/data/telescope/70cm/2009-12-01/calibration/'

This means that by simply typing work at the terminal you will go to that directory

Another useful thing to more efficiently use the bash are control keys. By pressing a certain key combination on the terminal prompt you can do very useful things. Here are some examples 

 CTRL+A      Go to the beginning of the line
                  CTRL+E      Go to the end of the line
                  CTRL+D      Logout
                  CTRL+C      Terminate the programme that is executed
                  CTRL+K      Delete everything after the cursor position
                  CTRL+W      Delete the next word
                  CTRL+R      Search the history
                  CTRL+L      Clears your terminal display (it execute the 'clear' command)

With the 'clear' command you can also clear the display in IDL: 

                  IDL> $clear

A summary of control keys is given here.

Some more basics

Now that you've a bit more experience I'll shock you with some more complicated things.

Redirecting output

You can redirect your terminal output by using ">" or ">>" directly into a file of your choice 

                  $ echo Hello world
                  Hello world
                  $ echo Hello world > hello.txt
                  $ echo How are you? >> hello.txt

Open an editor of your choice and see what is written into the file hello.txt. Careful! The > overwrites existing files! The ">>" appends the text...

You can also redirect your output into the next programme by using the pipe "|". Here bc is a simple calculator which uses the output from the terminal as input for the calculation 

                  $ echo 3+5 | bc

I'm sick of hitting ENTER all the time...

If you want to execute more lines of code at once you can either use the semicolon ";" or the double ampersand "&&" 

                  $ echo Hello; echo World
                  Hello
                  World
                  $ echo Hello && echo World
                  Hello
                  World

The difference between the two is, that "&&" only executes the second command if the first command ended succesfully!

Important programmes

 Still to do...

Conditionals and Loops

TRUE or FALSE

Working out true and false' is not an easy thing, as the syntax can be somewhat different and strange for different cases. Here I will stick to only the syntax that worked out for me. In general the syntax for conditional tests is like 

                  [[ a operator b ]]

It is important to stick to this (especially the empty spaces between all the various parts!). a and b can either be strings or numbers (they can also be variables), and depending on the operator they will be compared as one or the other.

String operators are: 

                  [[ "$a" < "$b" ]]      sorted alphabetically this is true if $a is before $b
                  [[ "$a" = "$b" ]]      true if $a equals $b
                  [[ "$a" > "$b" ]]       opposite of <

For strings it's not too bad to quote the variables (I don't know exactly why, but you can get into trouble in some special cases if it's not).

You can also compare numbers. The number operators are: 

                  [[ $a -gt $b ]]        true if a > b
                  [[ $a -ge $b ]]          "   " a >= b
                  [[ $a -eq $b ]]          "   " a = b
                  [[ $a -le $b ]]          "   " a <= b
                  [[ $a -lt $b ]]          "   " a < b

You can also check the presence/absence of a file or variable by using this syntax 

                  [[ -f $name ]]         file $name exists
                  [[ -d $name ]]         directory $name exists
                  [[ -e $name ]]         a file/directory called $name exists

                  [[ -n $name ]]         a variable $name exists
                  [[ -z $name ]]         a variable $name is not defined

With this I think we can go into conditional business...

IF

The syntax for the if case is 

                  if [[ condition ]]; then
                  ...
                  elif [[ condition2 ]]; then
                  ...
                  else
                  ...
                  fi

Of course the elif and else are optional.

You can get it even shorter by using && (logical AND) and || (logical XOR), were && is follow by the command for the true condition and || by the false cond. command: 

                  $ i=3
                  $ [[ $i -gt 2 ]] && echo "Yes, that's true!" || echo "No, that's untrue!"
                  Yes, that's true

                  $ i=1
                  $ [[ $i -gt 2 ]] && echo "Yes, that's true" || echo "No, that's untrue"
                  No, that's untrue

FOR loop

The syntax for the for loop is 

                  for var in argument; do
                  ...
                  done

The good thing in bash is that it uses as argument for the for-loop a list, where it subsequently goes through the list-elements that are divided by empty spaces. The elements are then assigned to the variable var, which can be used in the loop. Here's an example: 

                  list="1 2 3 4"
                  for i in $list; do
                  echo $i
                  done

which would give 

                  1
                  2
                  3
                  4

BTW, you can also write all this in one line by using the semicolon introduced earlier: 

                  $ list="1 2 3 4"; for i in $list; do echo $i; done
                  1
                  2
                  3
                  4

You can also run a loop for a certain number of circles by using the programme seq 

                  for i in $(seq 1 10); do
                  echo $i
                  done

Or you loop through all the files in a directory 

                  for i in $(ls); do
                  echo $i
                  done

WHILE

You can also run a loop as long as a certain condition is fullfilled. The let is only one of many ways to increment your variable 

                  counter=0
                  while [ $counter -lt 10 ]; do
                  echo $counter
                  let counter+=1
                  done

SED, AWK and other things...

Still to do. Sed and awk are very powerful commands with many functions (too much to explain them all here, so please google for online tutorials or see the references below). Sed is very helpful to manipulate lines in a text file, while awk is more approriate to manipulate columns of a file. However, often you can do the same thing with both commands.

Just a few examples:

To replace the string 'wrong' by 'right' type 

                  sed 's/wrong/right/g' yourfile

To delete lines that contain the string 'bla' 

                  sed '/bla/d' yourfile

or with awk 

                  awk '!/bla/' yourfile

To print lines that contain the string 'bla' at the line begin (emulating grep) 

                  awk '/^bla/' yourfile

To delete empty lines 

                  awk '!/^$/' yourfile

To delete the second column 

                  awk '{$2=""; print}' yourfile

To print the second column 

                  awk '{print $2}' yourfile

or somewhat trickier and shorter 

                  awk '$0=$2' yourfile

To excange the second column and fourth column and appending the line number of each line 

                  awk '{x=$2;$2=$4;$4=x; print $0" line number "NF}' yourfile

Subtract the last column from the first: 

                  awk '{$1=$1-$NF;print}' yourfile

Regular expression:

References:

sed

awk

To paste to files 

                  paste file1 file1

See also the man of 'join' for more functions.

Useful examples

Operations on many files at once

You can rename a lot of files at once 

                  for i in rx003*; do
                  mv $i ${i/rx003/crab_nebula}
                  done

Delete all files mpia.* except one (mpia.fits) 

                  rm `ls mpia.* | grep -v mpia.fits`

Create a ds9 region file from a catalog

If you have a catalog with many columns (RA, Dec, magnitudes, errors, ...) you can easily create a ds9 region file by doing this. 

                  $ echo fk5 > ds9.reg
                  $ cat noh_l1544.dat | awk '{print "circle("$1","$2",2\")"}' >> ds9.reg

Query VizieR database

You can query the VizieR-database from the bash with vizquery (maybe you have to install it). This programme automatically removes the header from the file (35 lines in this case) with sed and then appends the aquired data to the file iras.coord. 

                  rm iras.coord
                  for i in `seq 248`; do
                  echo “# CB$i” >> iras.coord
                  vizquery -c=”CB88 $i” -mime=tsv -source=II/125 | sed '1,35d' >> iras.coord;
                  done

In this case 

 -c CB88 $i        the object of interest
                  -source=II/125    the catalog to be searched
                  -mime=tsv         return format (tab-separated values in this case)

Automatically sort FITS files

This script automatically sorts the fits-files bash_*.fits into different directories. It finds out whether the file is a calibration or science frame plus sorts the different filters. You can download the complete script including the fits-files here Δ. 

                  for i in $(ls *.fits); do

                  echo $i

                  object=`gethead OBJECT $i`
                  filter=`echo $object | awk '{print $1}'`

                  if [[ $filter = "Dome" || $filter = "Sky" ]]; then

                  filter=`echo $object | awk '{print $2}'`
                  [[ ! -e calib/$filter ]] && mkdir -p calib/$filter
                  mv $i calib/$filter

                  else

                  [[ ! -e $filter ]] && mkdir -p $filter
                  mv $i $filter

                  fi

                  done