Bash

This article is a cheat sheet for things you learn to do in Bash (The Bourne Again Shell). See also: the page of one-liners.

For Loops

The bash one-liner for doing a for loop looks something like this:

for FILE in $(ls); do [COMMAND]; done

Here is a real example that will lowercase the names of all files in the current directory:

for FILE in $(ls); do mv $FILE $(echo $FILE | tr [A-Z] [a-z]); done

To do simple range looping, use the seq command:

for i in `seq 1 10`; do echo "$i, "; done

Arrays

In Bash v4 you can use multi-dimensional array but if you might want to use a different language. See http://wiki.bash-hackers.org/syntax/arrays for more.

Simple arrays work like the following:

#!/bin/bash

# declare three arrays
# you can use the 'declare' built-in, but don't have to 
# declare -a breakfast=('eggs' 'pancakes' 'cereal')
breakfast=('eggs' 'pancakes' 'cereal')
lunch=('sandwich' 'salad' 'smoothie')
dinner=('pasta' 'stir-fry' 'burritos')

# Now how do we loop through them?
# We could just output the values for breakfast
for i in "${breakfast[@]}"
# for BASH 3+ we have ! to access the numeric index
for i in "${!breakfast[@]}"
# Or, Build a sequence to loop through the other arrays by index
# while using a counter that is more friendly
# Arrays are indexed using integers and are zero-based
# the index number (starting at zero) gives that element
for ((  i=1; i<${#breakfast}; i++ ))
do 
  echo "Here is the menu for day ${i}"
  echo "breakfast: ${breakfast[$i-1]}"
  echo "lunch: ${lunch[$i-1]}"
  echo "dinner: ${dinner[$i-1]}"
done


declare -a fruits=('apples' 'oranges' 'banannas')
declare -a vegetables=('broccoli' 'onions' 'peppers' 'potatoes' 'carrots')

# the # character gives us the count of the array
echo "we have ${#fruits[@]} fruits available"
printf "%s\n" "${fruits[@]}"

echo "we have ${#vegetables[@]} vegetables available"
printf "%s\n" "${vegetables[@]}"

If construct

The then can go on the same line as the if as long as you use a semi-colon to terminate the if clause. Alternately, you can put the then on it's own line

if EXPR; then
  # do stuff
fi

is equivalent to

if EXPR
then
  # do stuff
fi

Adding an else clause

if EXPR; then
  # do stuff
else
  # do other stuff
fi

Adding multiple else clauses with elif; then

if EXPR; then
  # do stuff
elif EXPR; then
  # do other stuff
else
  # final else
fi

Note: sometimes you want to comment out a section of an if/else block, or maybe it does nothing at all. In this case, you'll get an error. To avoid the error, you can use the bash built-in : (colon command)

 : [arguments]

Do nothing beyond expanding arguments and performing redirections. The return status is zero.

if [ -f "/tmp/Non-existing-file.txt" ] ; then
  echo "I found the non-existing file"
else
  : # the colon command prevents an error if there are no other statements in this block
fi

Using Find

The find command in Linux is very powerful, and thus somewhat complex to learn all the syntax and options. Suffice to say that you can read the manpage and info pages to answer your questions.

Mime report

This example finds and counts files by their extension. A "poor-man's mime-report"

for x in \
  $(find . -maxdepth 1 -type d | \
  sort | \
  grep -v ^./$ ); \
do \
  echo -e "\n\n$x\n"; \
  find "$x" -type f | \
  egrep -o '\.(.?.?..)$' | \
  sort | \
  uniq -c ; \
done

Breakdown:

 1. stuff everything up to the semicolon into a loop variable named '$x'
 2. look for directories that are immediate descendants of the current directory
 3. sort them
 4. don't include the current directory
 6. with these: echo the name of the directory as a 'heading' in the report
 7. find the files per directory
 8. match only on the extensions found (between two and four-letter extensions)
 9. sort them
10. count and summarize by unique values

But it looks more impressive as a one-liner:

for x in $(find . -maxdepth 1 -type d|sort|grep -v ^./$); do echo -e "\n\n$x\n"; find "$x" -type f | egrep -o '\.(.?.?..)$' | sort | uniq -c ; done

Prune

In case you are trying to figure out the prune option so that you can efficiently scan a directory for something while also ignoring .svn metadata, here is an example:

find ./ -name .svn -prune -o -name "*html*"

Or, a more complex example: 'wcgrep', from the contrib section of the svn repo:

#!/bin/bash

# Copyright 2004 Ben Reser <ben@reser.org>
# Licensed under the terms subversion ships under or GPLv2.

# Useful for greping in a subversion working copy.
# Essentially it behaves the same way your grep command does (in fact it
# ultimately calls the grep command on your path) with a few exceptions.
# Ignores the subversion admin directories (.svn) and vi(m) backup files.
# Recursive is always on with or without -r.
# Always print filename and line numbers.
# Ignores binary files.
# If no path is given the current working directory is searched not stdin.
# Other than that it will take any parameter or pattern your standard grep
# does.
#
# This script requires GNU findutils and by default GNU grep (though that
# can be changed with environment variables).
#
# There are three environment variables you can set that modify the default
# behavior:
#
# WCGREP_GREP      Controls what command is used for the grep command.
#                  If unset or null wcgrep will use the command named grep.
# WCGREP_GREPARGS  Controls what arguments are always passed to the grep
#                  command before the arguments given on the command line.
#                  If unset or null it defaults to -HnI (always print file
#                  names, line numbers and ignore binary files).  If you wish
#                  to set no default args set the variable to a space (" ").
# WCGREP_IGNORE    Controls what files are ignored by the grep command.
#                  This is a regex that is passed to the find command with
#                  -regex so see find's man page for details.  If unset or
#                  null defaults to '.*~$\|.*/\.svn\(/\|$\)', which will
#                  ignore vim backup files and subversion admin dirs.


arg_count=$#
for (( i=1; i <= $arg_count; i++ )); do
arg="$1"
shift 1
if [ -z "$pattern" ]; then
if [ "$arg" == "--" ]; then
grepargs="$grepargs $arg"
pattern="$1"
shift 1
((i++))
elif [ "${arg:0:1}" != "-" ]; then
pattern="$arg"
else
grepargs="$grepargs $arg"
fi
else
pathargs="$pathargs $arg"
fi
done

find $pathargs -regex ${WCGREP_IGNORE:-'.*~$\|.*/\.svn\(/\|$\)'} -prune -o \
-type f -print0 | xargs -r0 ${WCGREP_GREP:-grep} ${WCGREP_GREPARGS:--HnI} \
$grepargs "$pattern"

Examples

This script sets the svn:executable property on a number of files

#!/bin/bash

DIRECTORIES="./example.org/htdocs/
./www.example.org/htdocs/
./security.example.org/htdocs/
./insurance.example.org/htdocs/
./hr.example.org/htdocs/"

for DIRECTORY in $DIRECTORIES
do
echo
echo "working on $DIRECTORY";
for FILE in $(find $DIRECTORY -name .svn -prune -o -type f -regex '.*s?html$'|\
grep -v .svn | xargs grep -l '\-\-\#include');
do svn propset svn:executable ON $FILE;
# do ls $FILE;
done
echo "$DIRECTORY processing complete"
echo
done

echo
echo "Finished fixing websites"
echo

Sometimes when using find, you end up with "Permission denied" errors that add noise to your output. There are a couple solutions for this. Use the prune option to skip entire trees that you should avoid (e.g. /proc). Use shell redirection to ignore remaining error messages (e.g. 2>/dev/null).

The following example searches all of my hard drive starting at / but skips over the backups directory I have in my external disk drive and also skips over the "process" directory. Any errors like files in /var that I do not have permission to see are discarded by redirecting STDERR to the bitbucket.

  find / -path /media/disk/backups -prune -o -path /proc -prune -o -type d -name soffice.cfg 2>/dev/null
/home/greg/.openoffice.org2/user/config/soffice.cfg
/home/greg/.openoffice/1.1.1/user/config/soffice.cfg
/home/greg/.openoffice.org/3/user/config/soffice.cfg
/home/greg/spidey2/.openoffice.org2/user/config/soffice.cfg
/home/greg/liberty/greg/.openoffice.org2/user/config/soffice.cfg
/home/greg/liberty/greg/.openoffice/1.1.1/user/config/soffice.cfg
/opt/openoffice.org/basis3.0/share/config/soffice.cfg
/usr/lib/openoffice/basis3.0/share/config/soffice.cfg
/proc
/media/disk/backups

Move a directory up one level

Sometimes you can end up with a directory which is nested inside it's intended destination. For example, drush archive-restore (arr) can leave you with /var/www/drush/drush and you want the contents of the sub-directory to be at the location of it's parent. Using the BASH shell options for glob control, you can set dotglob and later unset it to be able to move * up. This worked for me on one host, and didn't work on another. For the one that didn't work, mv kept complaining that destination directories were not empty. I don't care if the destination directories exist.... that's the whole point. Uggh!

cd /var/www/drupal/drupal/
shopt -s dotglob
mv -- * ..
shopt -u dotglob

Resources

• greycat (Greg Wooledge) Wiki http://mywiki.wooledge.org/EnglishFrontPage maintained by lhunath (Maarten Billemont)
• http://bash.cyberciti.biz/guide/Main_Page an excellent BASH and Linux wiki site
• http://wiki.bash-hackers.org/scripting/style
• Maarten Billemont's Bash Guide
• Bash Reference Manual
• Penguin Pete
• Wikipedia page
• Bash tests -f -d -x -w -Z What do all the file test options mean?

Bash vs Python or "BASH is better"

Most job postings that focus on DevOps have requirements for Python, Go programming or some other programming language. I disagree that a DevOps Engineer should also be a programmer. I prioritize quality and workmanship (craftsmanship) which is informed by broad experience, but honed by specialization. As a construction analogy, I prefer individual skilled trades over the general handyman approach. Simply put: DevOps is DevOps, it is not programming. Worse, the requirement for the hot language of the day is a bigger tell-tale sign that the company is either posturing or doesn't know what they're doing. Back when when I first learned Perl (which isn't the hot new language anymore), there was a hilarious t-shirt that said "Be careful or I'll replace you with a line of code". Although you could write the Python example more concisely, it is a real-world example of code that I found that does the same thing as 5 lines of BASH that I wrote.

Bash code to concatenate certbot certificates:

#!/bin/bash

# $RENEWED_DOMAINS will contain a space-delimited list of renewed 
# certificate domains (for example, "example.com www.example.com"
# loop through a dynamic list of directories in 'live'
# for SITE in $(find /etc/letsencrypt/live -mindepth 1 -maxdepth 1 -type d -exec basename {} \;)
# $RENEWED_LINEAGE will contain the live subdirectory
for SITE in $RENEWED_DOMAINS
do
        # move to correct let's encrypt directory
        cd $RENEWED_LINEAGE
        # cat files to make combined .pem for haproxy
        cat fullchain.pem privkey.pem > /etc/haproxy/certs/$SITE.pem
done
# reload haproxy
# systemctl reload haproxy

Python code to concatenate certbot certificates:

#!/usr/bin/env python3

import os
import re
import sys

# Certbot sets an environment variable RENEWED_LINEAGE, which points to the
# path of the renewed certificate. We use that path to determine and find
# the files for the currently renewed certificated
lineage=os.environ.get('RENEWED_LINEAGE')

# If nothing renewed, exit
if not lineage:
    sys.exit()

# From the linage, we strip the 'domain name', which is the last part
# of the path.
result = re.match(r'.*/live/(.+)$', lineage)

# If we can not recognize the path, we exit with 1
if not result:
    sys.exit(1)

# Extract the domain name
domain = result.group(1)

# Define a path for HAproxy where you want to write the .pem file.
deploy_path="/etc/haproxy/ssl/" + domain + ".pem"

# The source files can be found in below paths, constructed with the lineage
# path
source_key = lineage + "/privkey.pem"
source_chain = lineage + "/fullchain.pem"

# HAproxy requires to combine the key and chain in one .pem file
with open(deploy_path, "w") as deploy, \
        open(source_key, "r") as key, \
        open(source_chain, "r") as chain:
    deploy.write(key.read())
    deploy.write(chain.read())

# Here you can add your service reload command. Which will be executed after
# every renewal, which is fine if you only have a few domains.

# Alternative is to add the reload to the --post-hook. In that case it is only
# run once after all renewals. That would be the use-case if you have a large
# number of different certificates served by HAproxy.

Gnu's Not Unix, because Unix is proprietary software while Gnu is software that respects your freedom.