Bash: Difference between revisions

(16 intermediate revisions by 2 users not shown)

Line 1:

This article is a cheat sheet for things you learn to do in Bash (The Bourne Again Shell).

This article is a cheat sheet for things you learn to do in Bash (The Bourne Again Shell). See also: the page of [[one-liners]].

== For Loops ==

The bash one-liner for doing a for loop looks something like this:

<~~source~~ lang="bash">

for FILE in $(ls); do [COMMAND]; done

</~~source~~>

</syntaxhighlight>

Here is a real example that will lowercase the names of all files in the current directory:

<~~source~~ lang="bash">

for FILE in $(ls); do mv $FILE $(echo $FILE | tr [A-Z] [a-z]); done

</~~source~~>

</syntaxhighlight>

To do simple range looping, use the '''seq''' command:

<~~source~~ lang="bash">

for i in `seq 1 10`; do echo "$i, "; done

</~~source~~>

</syntaxhighlight>

== Arrays ==

In Bash v4 you can use multi-dimensional array but if you might want to use a different language. See http://wiki.bash-hackers.org/syntax/arrays for more.

Simple arrays work like the following:

#!/bin/bash

# declare three arrays

# you can use the 'declare' built-in, but don't have to

# declare -a breakfast=('eggs' 'pancakes' 'cereal')

breakfast=('eggs' 'pancakes' 'cereal')

lunch=('sandwich' 'salad' 'smoothie')

dinner=('pasta' 'stir-fry' 'burritos')

# Now how do we loop through them?

# We could just output the values for breakfast

for i in "${breakfast[@]}"

# for BASH 3+ we have ! to access the numeric index

for i in "${!breakfast[@]}"

# Or, Build a sequence to loop through the other arrays by index

# while using a counter that is more friendly

# Arrays are indexed using integers and are zero-based

# the index number (starting at zero) gives that element

for (( i=1; i<${#breakfast}; i++ ))

do

echo "Here is the menu for day ${i}"

echo "breakfast: ${breakfast[$i-1]}"

echo "lunch: ${lunch[$i-1]}"

echo "dinner: ${dinner[$i-1]}"

done

declare -a fruits=('apples' 'oranges' 'banannas')

declare -a vegetables=('broccoli' 'onions' 'peppers' 'potatoes' 'carrots')

# the # character gives us the count of the array

echo "we have ${#fruits[@]} fruits available"

printf "%s\n" "${fruits[@]}"

echo "we have ${#vegetables[@]} vegetables available"

printf "%s\n" "${vegetables[@]}"

</syntaxhighlight>

== If construct ==

The <code>then</code> can go on the same line as the <code>if</code> as long as you use a semi-colon to terminate the if clause. Alternately, you can put the <code>then</code> on it's own line

if EXPR; then

# do stuff

fi

</syntaxhighlight>

is equivalent to

if EXPR

then

# do stuff

fi

</syntaxhighlight>

Adding an <code>else</code> clause

if EXPR; then

# do stuff

else

# do other stuff

fi

</syntaxhighlight>

Adding multiple <code>else</code> clauses with <code>elif; then</code>

if EXPR; then

# do stuff

elif EXPR; then

# do other stuff

else

# final else

fi

</syntaxhighlight>

Note: sometimes you want to comment out a section of an if/else block, or maybe it does nothing at all. In this case, you'll get an error. To avoid the error, you can use the [http://www.gnu.org/software/bash/manual/html_node/Bourne-Shell-Builtins.html bash built-in] <code>:</code> (colon command)

<pre>

: [arguments]

</pre>

Do nothing beyond expanding arguments and performing redirections. The return status is zero.

if [ -f "/tmp/Non-existing-file.txt" ] ; then

echo "I found the non-existing file"

else

: # the colon command prevents an error if there are no other statements in this block

fi

</syntaxhighlight>

== Using Find ==

The find command in Linux is very powerful, and thus somewhat complex to learn all the syntax and options.

Suffice to say that you can read the manpage and info pages to answer your questions. ~~However,~~ in case you are trying to figure out the prune option so that you can efficiently scan a directory for something while also ignoring .svn metadata, here is an example:

Suffice to say that you can read the manpage and info pages to answer your questions.

<~~source~~ lang="bash">

=== Mime report ===

This example finds and counts files by their extension. A "poor-man's mime-report"

for x in \

$(find . -maxdepth 1 -type d | \

sort | \

grep -v ^./$ ); \

do \

echo -e "\n\n$x\n"; \

find "$x" -type f | \

egrep -o '\.(.?.?..)$' | \

sort | \

uniq -c ; \

done

</syntaxhighlight>

Breakdown:

<pre>

1. stuff everything up to the semicolon into a loop variable named '$x'

2. look for directories that are immediate descendants of the current directory

3. sort them

4. don't include the current directory

6. with these: echo the name of the directory as a 'heading' in the report

7. find the files per directory

8. match only on the extensions found (between two and four-letter extensions)

9. sort them

10. count and summarize by unique values

</pre>

But it looks more impressive as a one-liner:

=== Prune ===

In case you are trying to figure out the prune option so that you can efficiently scan a directory for something while also ignoring .svn metadata, here is an example:

find ./ -name .svn -prune -o -name "*html*"

</~~source~~>

</syntaxhighlight>

Or, a more complex example: 'wcgrep', from the contrib section of the svn repo:

<~~source~~ lang="bash">

#!/bin/bash

Line 83:

Line 210:

-type f -print0 | xargs -r0 ${WCGREP_GREP:-grep} ${WCGREP_GREPARGS:--HnI} \

$grepargs "$pattern"

</~~source~~>

</syntaxhighlight>

== Examples ==

This script sets the svn:executable property on a number of files

<~~source~~ lang="bash">

#!/bin/bash

Line 112:

Line 239:

echo "Finished fixing websites"

echo

</~~source~~>

</syntaxhighlight>

Sometimes when using find, you end up with "Permission denied" errors that add noise to your output. There are a couple solutions for this. Use the '''prune''' option to skip entire trees that you should avoid (e.g. /proc). Use shell redirection to ignore remaining error messages (e.g. 2>/dev/null).

The following example searches all of my hard drive starting at / but skips over the backups directory I have in my external disk drive and also skips over the "process" directory. Any errors like files in /var that I do not have permission to see are discarded by redirecting STDERR to the bitbucket.

<~~source~~ lang="bash">

find / -path /media/disk/backups -prune -o -path /proc -prune -o -type d -name soffice.cfg 2>/dev/null

/home/greg/.openoffice.org2/user/config/soffice.cfg

Line 129:

Line 256:

/proc

/media/disk/backups

</~~source~~>

</syntaxhighlight>

=== Move a directory up one level ===

Sometimes you can end up with a directory which is nested inside it's intended destination. For example, <code>drush archive-restore (arr)</code> can leave you with <tt>/var/www/drush/drush</tt> and you want the contents of the sub-directory to be at the location of it's parent. Using the BASH shell options for glob control, you can set dotglob and later unset it to be able to move * up. This worked for me on one host, and didn't work on another. For the one that didn't work, mv kept complaining that destination directories were not empty. I don't care if the destination directories exist.... that's the whole point. Uggh!

cd /var/www/drupal/drupal/

shopt -s dotglob

mv -- * ..

shopt -u dotglob

</syntaxhighlight>

==Resources==

* greycat ([http://wooledge.org/~greg/ Greg Wooledge]) Wiki http://mywiki.wooledge.org/EnglishFrontPage maintained by lhunath (Maarten Billemont)

* http://bash.cyberciti.biz/guide/Main_Page an excellent BASH and Linux wiki site

* http://wiki.bash-hackers.org/scripting/style

* [http://mywiki.wooledge.org/BashGuide Maarten Billemont's Bash Guide]

* [http://www.gnu.org/software/bash/manual/bashref.html Bash Reference Manual]

* [http://penguinpetes.com/b2evo/index.php?title=how_the_one_liner_for_loop_in_bash_goes Penguin Pete]

* [[wp:Bash|Wikipedia page]]

* [https://ss64.com/bash/test.html Bash tests] <code>-f -d -x -w -Z </code> What do all the file test options mean?

== Bash vs Python or "BASH is better" ==

Most job postings that focus on DevOps have requirements for [[Python]], [[Go]] programming or some other programming language. I disagree that a [[DevOps]] Engineer should also be a programmer. I prioritize quality and workmanship (craftsmanship) which is '''informed''' by broad experience, but '''honed''' by specialization. As a construction analogy, I prefer individual skilled trades over the general handyman approach. Simply put: DevOps is DevOps, it is not programming. Worse, the requirement for the hot language of the day is a bigger tell-tale sign that the company is either posturing or doesn't know what they're doing. Back when when I first learned Perl (which isn't the hot new language anymore), there was a hilarious t-shirt that said "Be careful or I'll replace you with a line of code"<ref>Dave Jacoby agrees with me on the broad point that (programming) languages are just different domain dialects, and also cites the ThinkGeek t-shirt phrase "Go Away Or I Will Replace You With a Small Shell Script"

https://jacoby.github.io/2021/11/16/i-will-replace-you-with-a-small-shell-script.html</ref>. Although you ''could'' write the Python example more concisely, it is a real-world example of code that I found that does the same thing as 5 lines of BASH that I wrote.

[[Bash]] code to concatenate [[certbot]] certificates:<syntaxhighlight lang="bash">

#!/bin/bash

# $RENEWED_DOMAINS will contain a space-delimited list of renewed

# certificate domains (for example, "example.com www.example.com"

# loop through a dynamic list of directories in 'live'

# for SITE in $(find /etc/letsencrypt/live -mindepth 1 -maxdepth 1 -type d -exec basename {} \;)

# $RENEWED_LINEAGE will contain the live subdirectory

for SITE in $RENEWED_DOMAINS

do

# move to correct let's encrypt directory

cd $RENEWED_LINEAGE

# cat files to make combined .pem for haproxy

cat fullchain.pem privkey.pem > /etc/haproxy/certs/$SITE.pem

done

# reload haproxy

# systemctl reload haproxy

</syntaxhighlight>Python code to concatenate certbot certificates:<syntaxhighlight lang="python3">

#!/usr/bin/env python3

import os

import re

import sys

# Certbot sets an environment variable RENEWED_LINEAGE, which points to the

# path of the renewed certificate. We use that path to determine and find

# the files for the currently renewed certificated

lineage=os.environ.get('RENEWED_LINEAGE')

# If nothing renewed, exit

if not lineage:

sys.exit()

# From the linage, we strip the 'domain name', which is the last part

# of the path.

result = re.match(r'.*/live/(.+)$', lineage)

# If we can not recognize the path, we exit with 1

if not result:

sys.exit(1)

# Extract the domain name

domain = result.group(1)

# Define a path for HAproxy where you want to write the .pem file.

deploy_path="/etc/haproxy/ssl/" + domain + ".pem"

# The source files can be found in below paths, constructed with the lineage

# path

source_key = lineage + "/privkey.pem"

source_chain = lineage + "/fullchain.pem"

# HAproxy requires to combine the key and chain in one .pem file

with open(deploy_path, "w") as deploy, \

open(source_key, "r") as key, \

open(source_chain, "r") as chain:

deploy.write(key.read())

deploy.write(chain.read())

# Here you can add your service reload command. Which will be executed after

# every renewal, which is fine if you only have a few domains.

# Alternative is to add the reload to the --post-hook. In that case it is only

# run once after all renewals. That would be the use-case if you have a large

# number of different certificates served by HAproxy.

</syntaxhighlight>

[[Category:System Administration]]

@@ Line 1: / Line 1: @@
-This article is a cheat sheet for things you learn to do in Bash (The Bourne Again Shell).
+This article is a cheat sheet for things you learn to do in Bash (The Bourne Again Shell).  See also: the page of [[one-liners]].
 == For Loops ==
 The bash one-liner for doing a for loop looks something like this:
-<source lang="bash">
+<syntaxhighlight lang="bash">
 for FILE in $(ls); do [COMMAND]; done
-</source>
+</syntaxhighlight>
 Here is a real example that will lowercase the names of all files in the current directory:
-<source lang="bash">
+<syntaxhighlight lang="bash">
 for FILE in $(ls); do mv $FILE $(echo $FILE | tr [A-Z] [a-z]); done
-</source>
+</syntaxhighlight>
 To do simple range looping, use the '''seq''' command:
-<source lang="bash">
+<syntaxhighlight lang="bash">
 for i in `seq 1 10`; do echo "$i, "; done
-</source>
+</syntaxhighlight>
+== Arrays ==
+In Bash v4 you can use multi-dimensional array but if you might want to use a different language.  See http://wiki.bash-hackers.org/syntax/arrays for more.
+Simple arrays  work like the following:
+<syntaxhighlight lang="bash">
+#!/bin/bash
+# declare three arrays
+# you can use the 'declare' built-in, but don't have to
+# declare -a breakfast=('eggs' 'pancakes' 'cereal')
+breakfast=('eggs' 'pancakes' 'cereal')
+lunch=('sandwich' 'salad' 'smoothie')
+dinner=('pasta' 'stir-fry' 'burritos')
+# Now how do we loop through them?
+# We could just output the values for breakfast
+for i in "${breakfast[@]}"
+# for BASH 3+ we have ! to access the numeric index
+for i in "${!breakfast[@]}"
+# Or, Build a sequence to loop through the other arrays by index
+# while using a counter that is more friendly
+# Arrays are indexed using integers and are zero-based
+# the index number (starting at zero) gives that element
+for ((  i=1; i<${#breakfast}; i++ ))
+do
+  echo "Here is the menu for day ${i}"
+  echo "breakfast: ${breakfast[$i-1]}"
+  echo "lunch: ${lunch[$i-1]}"
+  echo "dinner: ${dinner[$i-1]}"
+done
+declare -a fruits=('apples' 'oranges' 'banannas')
+declare -a vegetables=('broccoli' 'onions' 'peppers' 'potatoes' 'carrots')
+# the # character gives us the count of the array
+echo "we have ${#fruits[@]} fruits available"
+printf "%s\n" "${fruits[@]}"
+echo "we have ${#vegetables[@]} vegetables available"
+printf "%s\n" "${vegetables[@]}"
+</syntaxhighlight>
+== If construct ==
+The <code>then</code> can go on the same line as the <code>if</code> as long as you use a semi-colon to terminate the if clause.  Alternately, you can put the <code>then</code> on it's own line
+<syntaxhighlight lang="bash">
+if EXPR; then
+  # do stuff
+fi
+</syntaxhighlight>
+is equivalent to
+<syntaxhighlight lang="bash">
+if EXPR
+then
+  # do stuff
+fi
+</syntaxhighlight>
+Adding an <code>else</code> clause
+<syntaxhighlight lang="bash">
+if EXPR; then
+  # do stuff
+else
+  # do other stuff
+fi
+</syntaxhighlight>
+Adding multiple <code>else</code> clauses with <code>elif; then</code>
+<syntaxhighlight lang="bash">
+if EXPR; then
+  # do stuff
+elif EXPR; then
+  # do other stuff
+else
+  # final else
+fi
+</syntaxhighlight>
+Note: sometimes you want to comment out a section of an if/else block, or maybe it does nothing at all.  In this case, you'll get an error.  To avoid the error, you can use the [http://www.gnu.org/software/bash/manual/html_node/Bourne-Shell-Builtins.html bash built-in] <code>:</code> (colon command)
+<pre>
+ : [arguments]
+</pre>
+Do nothing beyond expanding arguments and performing redirections. The return status is zero.
+<syntaxhighlight lang="bash">
+if [ -f "/tmp/Non-existing-file.txt" ] ; then
+  echo "I found the non-existing file"
+else
+  : # the colon command prevents an error if there are no other statements in this block
+fi
+</syntaxhighlight>
 == Using Find ==
 The find command in Linux is very powerful, and thus somewhat complex to learn all the syntax and options.
-Suffice to say that you can read the manpage and info pages to answer your questions.  However, in case you are trying to figure out the prune option so that you can efficiently scan a directory for something while also ignoring .svn metadata, here is an example:
+Suffice to say that you can read the manpage and info pages to answer your questions.
-<source lang="bash">
+=== Mime report ===
+This example finds and counts files by their extension.  A "poor-man's mime-report"
+<syntaxhighlight lang="bash" line>
+for x in \
+  $(find . -maxdepth 1 -type d | \
+  sort | \
+  grep -v ^./$ ); \
+do \
+  echo -e "\n\n$x\n"; \
+  find "$x" -type f | \
+  egrep -o '\.(.?.?..)$' | \
+  sort | \
+  uniq -c ; \
+done
+</syntaxhighlight>
+Breakdown:
+<pre>
+. stuff everything up to the semicolon into a loop variable named '$x'
+. look for directories that are immediate descendants of the current directory
+. sort them
+. don't include the current directory
+. with these: echo the name of the directory as a 'heading' in the report
+. find the files per directory
+. match only on the extensions found (between two and four-letter extensions)
+. sort them
+. count and summarize by unique values
+</pre>
+But it looks more impressive as a one-liner:
+<syntaxhighlight lang="bash">for x in $(find . -maxdepth 1 -type d|sort|grep -v ^./$); do echo -e "\n\n$x\n"; find "$x" -type f | egrep -o '\.(.?.?..)$' | sort | uniq -c ; done</syntaxhighlight>
+=== Prune ===
+In case you are trying to figure out the prune option so that you can efficiently scan a directory for something while also ignoring .svn metadata, here is an example:
+<syntaxhighlight lang="bash">
 find ./ -name .svn -prune -o -name "*html*"
-</source>
+</syntaxhighlight>
 Or, a more complex example: 'wcgrep', from the contrib section of the svn repo:
-<source lang="bash">
+<syntaxhighlight lang="bash">
 #!/bin/bash
@@ Line 83: / Line 210: @@
 -type f -print0 | xargs -r0 ${WCGREP_GREP:-grep} ${WCGREP_GREPARGS:--HnI} \
 $grepargs "$pattern"
-</source>
+</syntaxhighlight>
 == Examples ==
 This script sets the svn:executable property on a number of files
-<source lang="bash">
+<syntaxhighlight lang="bash">
 #!/bin/bash
@@ Line 112: / Line 239: @@
 echo "Finished fixing websites"
 echo
-</source>
+</syntaxhighlight>
 Sometimes when using find, you end up with "Permission denied" errors that add noise to your output.  There are a couple solutions for this.  Use the '''prune''' option to skip entire trees that you should avoid (e.g. /proc).  Use shell redirection to ignore remaining error messages (e.g. 2>/dev/null).
 The following example searches all of my hard drive starting at / but skips over the backups directory I have in my external disk drive and also skips over the "process" directory.  Any errors like files in /var that I do not have permission to see are discarded by redirecting STDERR to the bitbucket.
-<source lang="bash">
+<syntaxhighlight lang="bash">
    find / -path /media/disk/backups -prune -o -path /proc -prune -o -type d -name soffice.cfg 2>/dev/null
 /home/greg/.openoffice.org2/user/config/soffice.cfg
@@ Line 129: / Line 256: @@
 /proc
 /media/disk/backups
-</source>
+</syntaxhighlight>
+=== Move a directory up one level ===
+Sometimes you can end up with a directory which is nested inside it's intended destination.  For example, <code>drush archive-restore (arr)</code> can leave you with <tt>/var/www/drush/drush</tt> and you want the contents of the sub-directory to be at the location of it's parent.  Using the BASH shell options for glob control, you can set dotglob and later unset it to be able to move * up.  This worked for me on one host, and didn't work on another.  For the one that didn't work, mv kept complaining that destination directories were not empty.  I don't care if the destination directories exist.... that's the whole point.  Uggh!
+<syntaxhighlight lang="bash">
+cd /var/www/drupal/drupal/
+shopt -s dotglob
+mv -- * ..
+shopt -u dotglob
+</syntaxhighlight>
 ==Resources==
+* greycat ([http://wooledge.org/~greg/ Greg Wooledge]) Wiki http://mywiki.wooledge.org/EnglishFrontPage maintained by lhunath (Maarten Billemont)
+* http://bash.cyberciti.biz/guide/Main_Page an excellent BASH and Linux wiki site
 * http://wiki.bash-hackers.org/scripting/style
+* [http://mywiki.wooledge.org/BashGuide Maarten Billemont's Bash Guide]
 * [http://www.gnu.org/software/bash/manual/bashref.html Bash Reference Manual]
 * [http://penguinpetes.com/b2evo/index.php?title=how_the_one_liner_for_loop_in_bash_goes Penguin Pete]
 * [[wp:Bash|Wikipedia page]]
+* [https://ss64.com/bash/test.html Bash tests] <code>-f -d -x -w -Z </code> What do all the file test options mean?
+== Bash vs Python or "BASH is better" ==
+Most job postings that focus on DevOps have requirements for [[Python]], [[Go]] programming or some other programming language. I disagree that a [[DevOps]] Engineer should also be a programmer. I prioritize quality and workmanship (craftsmanship) which is '''informed''' by broad experience, but '''honed''' by specialization. As a construction analogy, I prefer individual skilled trades over the general handyman approach. Simply put: DevOps is DevOps, it is not programming.  Worse, the requirement for the hot language of the day is a bigger tell-tale sign that the company is either posturing or doesn't know what they're doing. Back when when I first learned Perl (which isn't the hot new language anymore), there was a hilarious t-shirt that said "Be careful or I'll replace you with a line of code"<ref>Dave Jacoby agrees with me on the broad point that (programming) languages are just different domain dialects, and also cites the ThinkGeek t-shirt phrase "Go Away Or I Will Replace You With a Small Shell Script"
+https://jacoby.github.io/2021/11/16/i-will-replace-you-with-a-small-shell-script.html</ref>. Although you ''could'' write the Python example more concisely, it is a real-world example of code that I found that does the same thing as 5 lines of BASH that I wrote.
+[[Bash]] code to concatenate [[certbot]] certificates:<syntaxhighlight lang="bash">
+#!/bin/bash
+# $RENEWED_DOMAINS will contain a space-delimited list of renewed
+# certificate domains (for example, "example.com www.example.com"
+# loop through a dynamic list of directories in 'live'
+# for SITE in $(find /etc/letsencrypt/live -mindepth 1 -maxdepth 1 -type d -exec basename {} \;)
+# $RENEWED_LINEAGE will contain the live subdirectory
+for SITE in $RENEWED_DOMAINS
+do
+        # move to correct let's encrypt directory
+        cd $RENEWED_LINEAGE
+        # cat files to make combined .pem for haproxy
+        cat fullchain.pem privkey.pem > /etc/haproxy/certs/$SITE.pem
+done
+# reload haproxy
+# systemctl reload haproxy
+</syntaxhighlight>Python code to concatenate certbot certificates:<syntaxhighlight lang="python3">
+#!/usr/bin/env python3
+import os
+import re
+import sys
+# Certbot sets an environment variable RENEWED_LINEAGE, which points to the
+# path of the renewed certificate. We use that path to determine and find
+# the files for the currently renewed certificated
+lineage=os.environ.get('RENEWED_LINEAGE')
+# If nothing renewed, exit
+if not lineage:
+    sys.exit()
+# From the linage, we strip the 'domain name', which is the last part
+# of the path.
+result = re.match(r'.*/live/(.+)$', lineage)
+# If we can not recognize the path, we exit with 1
+if not result:
+    sys.exit(1)
+# Extract the domain name
+domain = result.group(1)
+# Define a path for HAproxy where you want to write the .pem file.
+deploy_path="/etc/haproxy/ssl/" + domain + ".pem"
+# The source files can be found in below paths, constructed with the lineage
+# path
+source_key = lineage + "/privkey.pem"
+source_chain = lineage + "/fullchain.pem"
+# HAproxy requires to combine the key and chain in one .pem file
+with open(deploy_path, "w") as deploy, \
+        open(source_key, "r") as key, \
+        open(source_chain, "r") as chain:
+    deploy.write(key.read())
+    deploy.write(chain.read())
+# Here you can add your service reload command. Which will be executed after
+# every renewal, which is fine if you only have a few domains.
+# Alternative is to add the reload to the --post-hook. In that case it is only
+# run once after all renewals. That would be the use-case if you have a large
+# number of different certificates served by HAproxy.
+</syntaxhighlight>
 [[Category:System Administration]]