Shell scripting (1)

Here, we study how to effectively use shell scripting. Once you master how to deal with schell scripting, you can save much time.

Exercise 1

Below is a source code to return DOY (Day-of-Year) from date (year, month and day). Write the source code, save it as "mmdd2doy.c", and confirm whether it can run properly. 

#include <stdio.h>
#include <stdlib.h>

int days[12]={31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};


int main(int argc, char *argv[])
{
    int i, yy, mm, dd, doy=0;

    if (argc != 4){
        fprintf(stderr, "Usage : %s [year] [month] [day]\n", argv[0]);
        exit(1);
    }

    yy = atoi(argv[1]);
    mm = atoi(argv[2]);
    dd = atoi(argv[3]);
    
    // Leap year
    if ((yy % 4 == 0) && (yy % 100 != 0)) days[1] += 1;

    // Sum of days from Jan 1 to the input date
    for (i=0; i<mm-1; i++) doy += days[i];
    doy += dd;
    
    printf("%d\n", doy);
    
    return (0);
}

(Compilation)
$ gcc -o mmdd2doy mmdd2doy.c
$
(Execution example)
$ ./mmdd2doy (arguments are not enough) 
Usage : ./mmdd2doy [year] [month] [day]
$
$ ./mmdd2doy 2006 10 26 (arguments are enough) 
299
$

Exercise 2

Assume that we want to know DOYs of several dates. We can use shell scripting. Here, we use bash, which is quite popular among Linux and UNIX.

First of all, we must know path of the bash, i.e. where the bash is. There are a few ways to know the path, and the simplest way is shown as follows. 

$ which bash
/usr/bin/bash
$

The above result shows that we can access bash under /usr/bin. This information is necessary for shell scripting. Write a below source code and save it as "mmdd2doy-1.sh" 

#!/usr/bin/bash

./mmdd2doy 2006 4 1
./mmdd2doy 2003 2 8
./mmdd2doy 2004 12 24
./mmdd2doy 2002 8 21
./mmdd2doy 2004 10 15

(Execution result)
$ ./mmdd2doy-1.sh
91
39
359
233
289
$

Exercise 3

In Exercise 2, we describe each set of data (year, month and day) and execute five times. If we use array and for-loop in shell scripting, it may be better than "mmdd2doy-1.sh". Below is a source code to use array and for-loop. Write the source code and save it as "mmdd2doy-2.sh". Confirm whether it can run properly.

Caution: In each shell scripting, e.g. shell scripting in bash, c-shell (csh) or b-shell (sh), syntax is a little different. Also, for-loop and how to deal with variables in bash are different from C programming. 

#!/usr/bin/bash

yy=(2006 2003 2004 2002 2004)
mm=(4 8 12 8 10)
dd=(1 8 24 21 15)

for i in 0 1 2 3 4
do
    echo "Year=${yy[$i]}, Month=${mm[$i]}, Day=${dd[$i]}"
	./mmdd2doy ${yy[$i]} ${mm[$i]} ${dd[$i]}
done

(Execution result)
$ ./mmdd2doy-2.sh
Year=2006, Month=4, Day=1
91
Year=2003, Month=8, Day=8
220
Year=2004, Month=12, Day=24
359
Year=2002, Month=8, Day=21
233
Year=2004, Month=10, Day=15
289
$

Exercise 4

Assume that we want to deal with many MODIS data, shown as follows. 

MODIS-MOD03-V1.3-20020307-010408.dat
MODIS-MOD03-V1.3-20020308-014823.dat
MODIS-MOD03-V1.3-20020309-005203.dat
MODIS-MOD03-V1.3-20020310-013543.dat
MODIS-MOD03-V1.3-20020312-012253.dat
MODIS-MOD03-V1.3-20020313-020703.dat
MODIS-MOD03-V1.3-20020314-010958.dat
MODIS-MOD03-V1.3-20020317-014152.dat
MODIS-MOD03-V1.3-20020319-012926.dat
MODIS-MOD03-V1.3-20030804-013326.dat
MODIS-MOD03-V1.3-20030808-010753.dat
MODIS-MOD03-V1.3-20030810-005548.dat
MODIS-MOD03-V1.3-20030823-002556.dat
MODIS-MOD03-V1.3-20030824-010803.dat
MODIS-MOD03-V1.3-20030825-015237.dat
MODIS-MOD03-V1.3-20030829-012722.dat

One of processing is to know each DOY of MODIS data after extracting date information from file name. For example, 

MODIS-MOD03-V1.3-20020307-010408.dat
                 yyyymmdd

it means that the above file was observed on 2002/Mar/07. Please follow below steps.

Step-1

Save the above list as a file. Assume that it is saved as "MODIS_list.txt".

Step-2

Execute as follows to produce a file, in which "touch" command is described. "Touch" command can produce a null file. 

$ awk 'print "touch " $0' MODIS_list.txt > touch_MODIS_list.txt

Confirm the above result by executing "cat" command as follows. 

$ cat touch_MODIS_list.txt
touch MODIS-MOD03-V1.3-20020307-010408.dat
touch MODIS-MOD03-V1.3-20020308-014823.dat
touch MODIS-MOD03-V1.3-20020309-005203.dat
touch MODIS-MOD03-V1.3-20020310-013543.dat
touch MODIS-MOD03-V1.3-20020312-012253.dat
touch MODIS-MOD03-V1.3-20020313-020703.dat
touch MODIS-MOD03-V1.3-20020314-010958.dat
touch MODIS-MOD03-V1.3-20020317-014152.dat
touch MODIS-MOD03-V1.3-20020319-012926.dat
touch MODIS-MOD03-V1.3-20030804-013326.dat
touch MODIS-MOD03-V1.3-20030808-010753.dat
touch MODIS-MOD03-V1.3-20030810-005548.dat
touch MODIS-MOD03-V1.3-20030823-002556.dat
touch MODIS-MOD03-V1.3-20030824-010803.dat
touch MODIS-MOD03-V1.3-20030825-015237.dat
touch MODIS-MOD03-V1.3-20030829-012722.dat

Step-3

Execute as follows to produce many MODIS files with zero-volume size. 

(Execution result)
$ bash touch_MODIS_list.txt
$
$ ls -l MODIS-MOD03* (confirm whether zero-volume size of files exist)
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2002030
7-010408.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2002030
8-014823.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2002030
9-005203.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2002031
0-013543.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2002031
2-012253.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2002031
3-020703.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2002031
4-010958.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2002031
7-014152.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2002031
9-012926.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2003080
4-013326.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2003080
8-010753.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2003081
0-005548.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2003082
3-002556.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2003082
4-010803.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2003082
5-015237.dat
-rw-rw-rw-    1 susaki susaki  0 Oct 26 14:41 MODIS-MOD03-V1.3-2003082
9-012722.dat

Step-4

Write a below source code and save it as "modis_file_list1.sh". Confirm whether each MODIS file is listed as follows. 

#!/usr/bin/bash


for file in `ls MODIS-MOD03*`
do
    echo $file
done

(Execution result)
$ ./modis_file_list1.sh
MODIS-MOD03-V1.3-20020307-010408.dat
MODIS-MOD03-V1.3-20020308-014823.dat
MODIS-MOD03-V1.3-20020309-005203.dat
MODIS-MOD03-V1.3-20020310-013543.dat
MODIS-MOD03-V1.3-20020312-012253.dat
MODIS-MOD03-V1.3-20020313-020703.dat
MODIS-MOD03-V1.3-20020314-010958.dat
MODIS-MOD03-V1.3-20020317-014152.dat
MODIS-MOD03-V1.3-20020319-012926.dat
MODIS-MOD03-V1.3-20030804-013326.dat
MODIS-MOD03-V1.3-20030808-010753.dat
MODIS-MOD03-V1.3-20030810-005548.dat
MODIS-MOD03-V1.3-20030823-002556.dat
MODIS-MOD03-V1.3-20030824-010803.dat
MODIS-MOD03-V1.3-20030825-015237.dat
MODIS-MOD03-V1.3-20030829-012722.dat
$

Step-5

Below is a source code to extract date, year, month and day information from file name using "cut" command. Write and save it as "modis_file_list2.sh". Confirm whether each MODIS file is listed as follows. 

#!/usr/bin/bash


for file in `ls MODIS-MOD03*`
do
    date=`echo $file | cut -f4 -d"-"`
    yy=`echo $date | cut -b1-4`
    mm=`echo $date | cut -b5-6`
    dd=`echo $date | cut -b7-8`
    
    echo "Year=$yy, Month=$mm, Day=$dd"
done

(Execution result)
$ ./modis_file_list2.sh
Year=2002, Month=03, Day=07
Year=2002, Month=03, Day=08
Year=2002, Month=03, Day=09
Year=2002, Month=03, Day=10
Year=2002, Month=03, Day=12
Year=2002, Month=03, Day=13
Year=2002, Month=03, Day=14
Year=2002, Month=03, Day=17
Year=2002, Month=03, Day=19
Year=2003, Month=08, Day=04
Year=2003, Month=08, Day=08
Year=2003, Month=08, Day=10
Year=2003, Month=08, Day=23
Year=2003, Month=08, Day=24
Year=2003, Month=08, Day=25
Year=2003, Month=08, Day=29
$

Step-6

Finally, we want to show each DOY of MODIS file. In a below source code, "mmdd2doy" is used to return DOY from date. Write and save it as "modis_file_list3.sh". Confirm whether each DOY of MODIS file is listed as follows. 

#!/usr/bin/bash


for file in `ls MODIS-MOD03*`
do
    date=`echo $file | cut -f4 -d"-"`
    yy=`echo $date | cut -b1-4`
    mm=`echo $date | cut -b5-6`
    dd=`echo $date | cut -b7-8`
    
    # DOY from yyyy/mm/dd
    doy=`./mmdd2doy $yy $mm $dd`

    echo "$yy/$mm/$dd -> DOY=$doy"
done

(Execution result)
$ ./modis_file_list2.sh
2002/03/07 -> DOY=66
2002/03/08 -> DOY=67
2002/03/09 -> DOY=68
2002/03/10 -> DOY=69
2002/03/12 -> DOY=71
2002/03/13 -> DOY=72
2002/03/14 -> DOY=73
2002/03/17 -> DOY=76
2002/03/19 -> DOY=78
2003/08/04 -> DOY=216
2003/08/08 -> DOY=220
2003/08/10 -> DOY=222
2003/08/23 -> DOY=235
2003/08/24 -> DOY=236
2003/08/25 -> DOY=237
2003/08/29 -> DOY=241
$
Lessons for C programming
Junichi Susaki, Kyoto University