...making Linux just a little more fun!

<-- prev | next -->

Introduction to Shell Scripting - The Basics

By Ben Okopnik

A Blast from the Past!
Originally published in Issue 52 of Linux Gazette, April 2000



Here's a hint. When you think your code to exec a shell function is
just not working, never, repeat NEVER send it "/etc/reboot" just to see
what happens.
 -- Elliott Evans

Introduction

Shell scripting is a fascinating combination of art and science that gives you access to the incredible flexibility and power of Linux with very simple tools. Back in the early days of PCs, I was considered quite an expert with DOS's "batch files", something I now realize was a weak and gutless imitation of Unix's shell scripts. I'm not usually much given to Microsoft-bashing - I believe that they have done some good things in their time, although their attempts to create an operating system have been pretty sad - but their BFL ("Batch File Language") was a joke by comparison. It wasn't even particularly funny.

Since scripting is an inextricable part of understanding shell usage in general, quite a bit of the material in here will deal with shell quirks, methods, and specifics. Be patient; it's all a part of the knowledge that is necessary for writing good scripts.

Philosophy of Scripting

Linux - Unix in general - is not a warm and fuzzy, non-knowledgeable-user oriented system. Rather than specifying exact motions and operations that you must perform (and thus imiting you only to the operations described), it provides you with a myriad of small tools which can be connected in a literally infinite number of combinations to achieve almost any result (I find Perl's motto of "TMTOWTDI" - There's More Than One Way To Do It - highly apropos for all of Unix). That sort of power and flexibility, of course, carries a price - increased complexity and a requirement for higher competence in the user. Just as there is an enormous difference between operating, say, a bicycle versus a super-sonic jet fighter, so is there an enormous difference between blindly following the rigid dictates of a standardized GUI and creating your own program, or shell script, that performs exactly the functions you need in exactly the way you need them.

Shell scripting is programming - but it is programming made easy, with little, if any, formal structure. It is an interpreted language, with its own syntax - but it is only the syntax that you use when invoking programs from your command line; something I refer to as "recyclable knowledge". This, in fact, is what makes shell scripts so useful: in the process of writing them, you continually learn more about the specifics of your shell and the operation of your system - and this is knowledge that truly pays for itself in the long run as well as the short.

Requirements

Since I have a strong preference for Bash, and it happens to be the default shell in Linux, that's what these scripts are written for (although I've tried to keep Bash-isms down to a minimum - most if not all of these scripts should run under plain old "sh".) Even if you use something else, that's still fine: as long as you have Bash installed, these scripts will execute correctly. As you will see, scripts invoke the shell that they need; it's part of what a well-written script does.

I'm going to assume that you're going to do all these exercises in your home directory - you don't want these files scattered all over the place where you can't find them later. I'm also going to assume that you know enough to hit the "Enter" key after each line that you type in, and that, before selecting a name for your shell script, you will check that you do not have an executable with that same name in your path (type "which bkup" to check for an executable called "bkup"). You also shouldn't call your script "test"; that's a Unix FAQ ("why doesn't my shell script/program do anything?") There's an executable in /usr/bin called "test" that does nothing - nothing obvious, that is - when invoked...

It goes without saying that you have to know the basics of file operations - copying, moving, etc. - as well as being familiar with the basic assumptions of the file system, i.e., "." is the current directory, ".." is the parent (the one above the current), "~" is your home directory, etc. You didn't know that? You do now!

Whatever editor you use, whether 'vi', 'emacs', 'mcedit' (the DOS-like editor in Midnight Commander), or any other text editor is fine; just don't save this work in some word-processing format - it must be plain text. If you're not sure, or keep getting "line noise" when you try to run your script, you can check the raw contents of the file you've created with "cat script_name" to be sure.

In order to avoid constant repetition of material, I'm going to number the lines as we go through and discuss different parts of a script file. The line numbers will not, of course, be there in the actual script.

Building a Script

Let's go over the basics of creating a script. Those of you who find this obvious and simplistic are invited to follow along anyway; as we progress, the material will become more complex - and a "refresher" never hurts. The projected audience for this article is a Linux newbie, someone who has never created a shell script before - but wishes to become a Script Guru in 834,657 easy steps. :)

In its simplest form, a shell script is nothing more than a shortcut - a list of commands that you would normally type in, one after another, to be executed at your shell prompt - plus a bit of "magic" to notify the shell that it is indeed a script.

The "magic" consists of two simple things:

  1. A notation at the beginning of the script that specifies the program that is used to execute it, and
  2. A change in the permissions of the file containing the script in order to make it executable.

As a practical example, let's create a script that will "back up" a specified file to a selected directory; we'll go through the steps and the logic that makes it all happen.

First, let's create the script. Start your editor with the filename you want to create:

mcedit bkup
The first line in all of the script files we create will be this one (again, remember to ignore the number and the colon at the start of the line):
1: #!/bin/bash
This line is referred to as the 'shebang'. The interesting thing about it is that the pound character is actually a comment marker - everything following a '#' on a line is supposed to be ignored by the shell - but the '#!' construct is unique in that respect, and is interpreted as a prefix to the name of the executable that will actually process the lines which follow it.

The shebang must:

There's a subtle but important point to all of this, by the way: when a script runs, it actually starts an additional bash process that runs under the current one; that process executes the script and exits, dropping you back in the original shell that spawned it. This is why a script that, for example, changes directories as it executes will not leave you in that new directory when it exits: the original shell has not been told to change directories, and you're right where you were when you started - even though the change is effective while the script runs.

To continue with our script:

2: # "bkup" - copies specified files to the user's ~/Backup
3: # directory after checking for name conflicts.
As I've mentioned, the '#' character is a comment marker. It's a good idea, since you'll probably create a number of shell scripts in the future, to insert some comments in each one to indicate what it does - or at some point, you'll be scratching your head and trying to remember why you wrote it. In later columns, we'll explore ways to make that reminder a bit more automatic... but let's go on.
4: cp -i $1 ~/Backup

The "-i" syntax of the 'cp' command makes it interactive; that is, if we run "bkup file.txt" and a file called "file.txt" already exists in the ~/Backup directory, 'cp' will ask you if you want to overwrite it - and will abort the operation if you hit anything but the 'y' key.

The "$1" is a "positional parameter" - it denotes the first thing that you type after the script name. In fact, there's an entire list of these variables:

$0 - The name of the script being executed - in this case, "bkup".
$1 - The first parameter - in this case, "file.txt"; any parameter may
	be referred to by $<number> in this manner.
#@ - The entire list of parameters - "$1 $2 $3..."
$# - The number of parameters.
There are several other ways to address and manipulate positional parameters (see the Bash man page) - but these will do us for now.

Making it Smarter

So far, our script doesn't do very much; hardly worth bothering, right? All right; let's make it a bit more useful. What if you wanted to both keep the file in the ~/Backup directory and save the new one - perhaps by adding an extension to show the "version"? Let's try that; we'll just add a line, and modify the last line as follows:

4: a=$(date +'%Y%m%d%H%M%S')
5: cp -i $1 ~/Backup/$1.$a
Here, we are beginning to see a little of the real power of shell scripts: the ability to use the results of other Linux tools, called "command substitution". The effect of the $(command) construct is to execute the command inside the parentheses and replace the entire "$(command)" string with the result. In this case, we have asked 'date' to print the current date and time, down to the seconds, and pass the result to a variable called 'a'; then we appended that variable to the filename to be saved in ~/Backup. Note that when we assign a value to a variable, we use its name ( a=xxx ), but when we want to use that value, we must prepend a '$' to that name ($a). The names of variables may be almost anything except the reserved words in the shell, i.e.
case do done elif else esac fi for function if in select then until while time
and may not contain unquoted metacharacters or reserved characters, i.e.
! { } | & * ; ( ) < > space tab
It also should not unintentionally be a standard system variable, such as
PATH PS1 PWD RANDOM SECONDS (see "man bash" for many others)

The effect of the last two lines of this script is to create a unique filename - something like file.txt.20000117221714 - that should not conflict with anything else in ~/Backup. Note that I've left in the "-i" switch as a "sanity" check: if, for some truly strange reason, two file names do conflict, "cp" will give you a last-ditch chance to abort. Otherwise, it won't make any difference - like dead yeast in beer, it causes no harm even if it does nothing useful.

By the way, the older version of the $(command) construct - the `command` (note that "back-ticks" are being used rather than single quotes) - is more or less deprecated. $()s are easily nested - $(cat $($2$(basename file1 txt))), for example; something that cannot be done with back-ticks, since the second back-tick would "close" the first one and the command would fail, or do something unexpected. You can still use them, though - in single, non-nested substitutions (the most common kind), or as the innermost or outermost pair of the nested set - but if you use the new method exclusively, you'll always avoid that error.

So, let's see what we have so far, with whitespace added for readability and the line numbers removed (hey, an actual script!):

#!/bin/bash

# "bkup" - copies specified files to the user's ~/Backup
# directory after checking for name conflicts.

a=$(date +'%Y%m%d%H%M%S')
cp -i $1 ~/Backup/$1.$a
Yes, it's only a two-line script - but one that's starting to become useful. The last thing we need to do to make it into an executable program - although we can execute it already with "bash bkup" - is to change its mode to executable:
chmod +x bkup

Oh yes, there is one last thing; another "Unix FAQ". Should you try to execute your newly-created script by typing bkup at the prompt, you'll get this familiar reproof:

bash: bkup: command not found
-- "HEY! Didn't we just sweat, and struggle, and labor... What happened?"

Unlike DOS, the execution of commands and scripts in the current directory is disabled by default - as a security feature. Imagine what would happen if someone created a script called "ls", containing "rm -rf *" ("erase everything") in your home directory and you typed "ls"! If the current directory (".") came before "/bin" in your PATH variable, you'd be in a sorry state indeed...

Due to this, and a number of similar "exploits" that can be pulled off, you have to specify the path to all executables that you wish to run there - a wise restriction. You can also move your script into a directory that is in your path, once you're done tinkering with it; "/usr/local/bin" is a good candidate for this (Hint: type "echo $PATH" to see which directories are listed).

Meanwhile, in order to execute it, simply type

./bkup file.txt
- the "./" just says that the file to be run is in the current directory. Use "~/", instead, if you're calling it from anywhere else; the point here is that you have to give a complete path to the executable, since it is not in any of the directories listed in your PATH variable.

This assumes, of course, that you have a file in your current directory called "file.txt", and that you have created a subdirectory called "Backup" in your home directory. Otherwise, you'll get an error. We'll continue playing with this script in the next issue.

Review

In this article, we've looked at some of the basics involved in creating a shell script, as well as some specifics:
  • File creation
  • Permissions
  • Spawned subshells
  • Execution in a non-PATHed directory
  • The shebang
  • Comments
  • Positional parameters
  • Command substitution
  • Variables

Wrap-up

Well, that's a good bit of information for a start. Play with it, experiment; shell scripting is a large part of the fun and power of Linux. Next month, we'll talk about error checking - the things your script should do if the person using it makes an error in syntax, for example - as well as getting into loops and conditional execution, and maybe dealing with a few of the "power tools" that are commonly used in shell scripts.

Please feel free to send me suggestions for any corrections or improvements, as well as your own favorite shell-scripting tips or any really neat scripting tricks you've discovered; just like anyone whose ego hasn't swamped their good sense, I consider myself a student, always ready to learn something new. If I use any of your material, you will be credited.

Until then -

Happy Linuxing!


REFERENCES

"man" pages for 'bash', 'cp', 'chmod'

I read the Bash man page each day like a Jehovah's Witness reads the
Bible. No wait, the Bash man page IS the bible. Excuse me...
 -- More on confusing aliases, taken from comp.os.linux.misc

 


picture Ben is the Editor-in-Chief for Linux Gazette and a member of The Answer Gang.

Ben was born in Moscow, Russia in 1962. He became interested in electricity at the tender age of six, promptly demonstrated it by sticking a fork into a socket and starting a fire, and has been falling down technological mineshafts ever since. He has been working with computers since the Elder Days, when they had to be built by soldering parts onto printed circuit boards and programs had to fit into 4k of memory. He would gladly pay good money to any psychologist who can cure him of the recurrent nightmares.

His subsequent experiences include creating software in nearly a dozen languages, network and database maintenance during the approach of a hurricane, and writing articles for publications ranging from sailing magazines to technological journals. After a seven-year Atlantic/Caribbean cruise under sail and passages up and down the East coast of the US, he is currently anchored in St. Augustine, Florida. He works as a technical instructor for Sun Microsystems and a private Open Source consultant/Web developer. His current set of hobbies includes flying, yoga, martial arts, motorcycles, writing, and Roman history; his Palm Pilot is crammed full of alarms, many of which contain exclamation points.

He has been working with Linux since 1997, and credits it with his complete loss of interest in waging nuclear warfare on parts of the Pacific Northwest.

Copyright © 2005, Ben Okopnik. Released under the Open Publication license unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 111 of Linux Gazette, February 2005

Tux