Bash: Difference between revisions

mNo edit summary
m Text replacement - "<(\/?)source" to "<$1syntaxhighlight"
 
(3 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This article is a cheat sheet for things you learn to do in Bash (The Bourne Again Shell).  Also, see the page of [[one-liners]]
This article is a cheat sheet for things you learn to do in Bash (The Bourne Again Shell).  See also: the page of [[one-liners]].


== For Loops ==
== For Loops ==
The bash one-liner for doing a for loop looks something like this:
The bash one-liner for doing a for loop looks something like this:
<source lang="bash">
<syntaxhighlight lang="bash">
for FILE in $(ls); do [COMMAND]; done
for FILE in $(ls); do [COMMAND]; done
</source>
</syntaxhighlight>
Here is a real example that will lowercase the names of all files in the current directory:
Here is a real example that will lowercase the names of all files in the current directory:
<source lang="bash">
<syntaxhighlight lang="bash">
for FILE in $(ls); do mv $FILE $(echo $FILE | tr [A-Z] [a-z]); done
for FILE in $(ls); do mv $FILE $(echo $FILE | tr [A-Z] [a-z]); done
</source>
</syntaxhighlight>


To do simple range looping, use the '''seq''' command:
To do simple range looping, use the '''seq''' command:
<source lang="bash">
<syntaxhighlight lang="bash">
for i in `seq 1 10`; do echo "$i, "; done
for i in `seq 1 10`; do echo "$i, "; done
</source>
</syntaxhighlight>


== Arrays ==
== Arrays ==
Line 20: Line 20:


Simple arrays  work like the following:
Simple arrays  work like the following:
<source lang="bash">
<syntaxhighlight lang="bash">
#!/bin/bash
#!/bin/bash


Line 57: Line 57:
echo "we have ${#vegetables[@]} vegetables available"
echo "we have ${#vegetables[@]} vegetables available"
printf "%s\n" "${vegetables[@]}"
printf "%s\n" "${vegetables[@]}"
</source>
</syntaxhighlight>


== If construct ==
== If construct ==
The <code>then</code> can go on the same line as the <code>if</code> as long as you use a semi-colon to terminate the if clause.  Alternately, you can put the <code>then</code> on it's own line
The <code>then</code> can go on the same line as the <code>if</code> as long as you use a semi-colon to terminate the if clause.  Alternately, you can put the <code>then</code> on it's own line
<source lang="bash">
<syntaxhighlight lang="bash">
if EXPR; then
if EXPR; then
   # do stuff
   # do stuff
fi
fi
</source>
</syntaxhighlight>
is equivalent to
is equivalent to
<source lang="bash">
<syntaxhighlight lang="bash">
if EXPR
if EXPR
then
then
   # do stuff
   # do stuff
fi
fi
</source>
</syntaxhighlight>


Adding an <code>else</code> clause
Adding an <code>else</code> clause
<source lang="bash">
<syntaxhighlight lang="bash">
if EXPR; then
if EXPR; then
   # do stuff
   # do stuff
Line 81: Line 81:
   # do other stuff
   # do other stuff
fi
fi
</source>
</syntaxhighlight>


Adding multiple <code>else</code> clauses with <code>elif; then</code>
Adding multiple <code>else</code> clauses with <code>elif; then</code>
<source lang="bash">
<syntaxhighlight lang="bash">
if EXPR; then
if EXPR; then
   # do stuff
   # do stuff
Line 92: Line 92:
   # final else
   # final else
fi
fi
</source>
</syntaxhighlight>


Note: sometimes you want to comment out a section of an if/else block, or maybe it does nothing at all.  In this case, you'll get an error.  To avoid the error, you can use the [http://www.gnu.org/software/bash/manual/html_node/Bourne-Shell-Builtins.html bash built-in] <code>:</code> (colon command)
Note: sometimes you want to comment out a section of an if/else block, or maybe it does nothing at all.  In this case, you'll get an error.  To avoid the error, you can use the [http://www.gnu.org/software/bash/manual/html_node/Bourne-Shell-Builtins.html bash built-in] <code>:</code> (colon command)
Line 101: Line 101:
Do nothing beyond expanding arguments and performing redirections. The return status is zero.
Do nothing beyond expanding arguments and performing redirections. The return status is zero.


<source lang="bash">
<syntaxhighlight lang="bash">
if [ -f "/tmp/Non-existing-file.txt" ] ; then
if [ -f "/tmp/Non-existing-file.txt" ] ; then
   echo "I found the non-existing file"
   echo "I found the non-existing file"
Line 107: Line 107:
   : # the colon command prevents an error if there are no other statements in this block
   : # the colon command prevents an error if there are no other statements in this block
fi
fi
</source>
</syntaxhighlight>


== Using Find ==
== Using Find ==
Line 115: Line 115:
=== Mime report ===
=== Mime report ===
This example finds and counts files by their extension.  A "poor-man's mime-report"
This example finds and counts files by their extension.  A "poor-man's mime-report"
<source lang="bash" line>
<syntaxhighlight lang="bash" line>
for x in \
for x in \
   $(find . -maxdepth 1 -type d | \
   $(find . -maxdepth 1 -type d | \
Line 127: Line 127:
   uniq -c ; \
   uniq -c ; \
done
done
</source>
</syntaxhighlight>
Breakdown:
Breakdown:
<pre>
<pre>
Line 142: Line 142:


But it looks more impressive as a one-liner:
But it looks more impressive as a one-liner:
<source lang="bash">for x in $(find . -maxdepth 1 -type d|sort|grep -v ^./$); do echo -e "\n\n$x\n"; find "$x" -type f | egrep -o '\.(.?.?..)$' | sort | uniq -c ; done</source>
<syntaxhighlight lang="bash">for x in $(find . -maxdepth 1 -type d|sort|grep -v ^./$); do echo -e "\n\n$x\n"; find "$x" -type f | egrep -o '\.(.?.?..)$' | sort | uniq -c ; done</syntaxhighlight>


=== Prune ===
=== Prune ===
In case you are trying to figure out the prune option so that you can efficiently scan a directory for something while also ignoring .svn metadata, here is an example:
In case you are trying to figure out the prune option so that you can efficiently scan a directory for something while also ignoring .svn metadata, here is an example:
<source lang="bash">
<syntaxhighlight lang="bash">
find ./ -name .svn -prune -o -name "*html*"
find ./ -name .svn -prune -o -name "*html*"
</source>
</syntaxhighlight>
Or, a more complex example: 'wcgrep', from the contrib section of the svn repo:
Or, a more complex example: 'wcgrep', from the contrib section of the svn repo:
<source lang="bash">
<syntaxhighlight lang="bash">
#!/bin/bash
#!/bin/bash


Line 210: Line 210:
-type f -print0 | xargs -r0 ${WCGREP_GREP:-grep} ${WCGREP_GREPARGS:--HnI} \
-type f -print0 | xargs -r0 ${WCGREP_GREP:-grep} ${WCGREP_GREPARGS:--HnI} \
$grepargs "$pattern"
$grepargs "$pattern"
</source>
</syntaxhighlight>


== Examples ==
== Examples ==
This script sets the svn:executable property on a number of files
This script sets the svn:executable property on a number of files
<source lang="bash">
<syntaxhighlight lang="bash">
#!/bin/bash
#!/bin/bash


Line 239: Line 239:
echo "Finished fixing websites"
echo "Finished fixing websites"
echo
echo
</source>
</syntaxhighlight>


Sometimes when using find, you end up with "Permission denied" errors that add noise to your output.  There are a couple solutions for this.  Use the '''prune''' option to skip entire trees that you should avoid (e.g. /proc).  Use shell redirection to ignore remaining error messages (e.g. 2>/dev/null).
Sometimes when using find, you end up with "Permission denied" errors that add noise to your output.  There are a couple solutions for this.  Use the '''prune''' option to skip entire trees that you should avoid (e.g. /proc).  Use shell redirection to ignore remaining error messages (e.g. 2>/dev/null).


The following example searches all of my hard drive starting at / but skips over the backups directory I have in my external disk drive and also skips over the "process" directory.  Any errors like files in /var that I do not have permission to see are discarded by redirecting STDERR to the bitbucket.
The following example searches all of my hard drive starting at / but skips over the backups directory I have in my external disk drive and also skips over the "process" directory.  Any errors like files in /var that I do not have permission to see are discarded by redirecting STDERR to the bitbucket.
<source lang="bash">
<syntaxhighlight lang="bash">
   find / -path /media/disk/backups -prune -o -path /proc -prune -o -type d -name soffice.cfg 2>/dev/null
   find / -path /media/disk/backups -prune -o -path /proc -prune -o -type d -name soffice.cfg 2>/dev/null
/home/greg/.openoffice.org2/user/config/soffice.cfg
/home/greg/.openoffice.org2/user/config/soffice.cfg
Line 256: Line 256:
/proc
/proc
/media/disk/backups
/media/disk/backups
</source>
</syntaxhighlight>


=== Move a directory up one level ===   
=== Move a directory up one level ===   
Sometimes you can end up with a directory which is nested inside it's intended destination.  For example, <code>drush archive-restore (arr)</code> can leave you with <tt>/var/www/drush/drush</tt> and you want the contents of the sub-directory to be at the location of it's parent.  Using the BASH shell options for glob control, you can set dotglob and later unset it to be able to move * up.  This worked for me on one host, and didn't work on another.  For the one that didn't work, mv kept complaining that destination directories were not empty.  I don't care if the destination directories exist.... that's the whole point.  Uggh!
Sometimes you can end up with a directory which is nested inside it's intended destination.  For example, <code>drush archive-restore (arr)</code> can leave you with <tt>/var/www/drush/drush</tt> and you want the contents of the sub-directory to be at the location of it's parent.  Using the BASH shell options for glob control, you can set dotglob and later unset it to be able to move * up.  This worked for me on one host, and didn't work on another.  For the one that didn't work, mv kept complaining that destination directories were not empty.  I don't care if the destination directories exist.... that's the whole point.  Uggh!
<source lang="bash">
<syntaxhighlight lang="bash">
cd /var/www/drupal/drupal/
cd /var/www/drupal/drupal/
shopt -s dotglob
shopt -s dotglob
mv -- * ..
mv -- * ..
shopt -u dotglob
shopt -u dotglob
</source>
</syntaxhighlight>


==Resources==
==Resources==
Line 275: Line 275:
* [http://penguinpetes.com/b2evo/index.php?title=how_the_one_liner_for_loop_in_bash_goes Penguin Pete]
* [http://penguinpetes.com/b2evo/index.php?title=how_the_one_liner_for_loop_in_bash_goes Penguin Pete]
* [[wp:Bash|Wikipedia page]]
* [[wp:Bash|Wikipedia page]]
* [https://ss64.com/bash/test.html Bash tests] <code>-f -d -x -w -Z </code> What do all the file test options mean?
== Bash vs Python or "BASH is better" ==
Most job postings that focus on DevOps have requirements for [[Python]], [[Go]] programming or some other programming language. I disagree that a [[DevOps]] Engineer should also be a programmer. I prioritize quality and workmanship (craftsmanship) which is '''informed''' by broad experience, but '''honed''' by specialization. As a construction analogy, I prefer individual skilled trades over the general handyman approach. Simply put: DevOps is DevOps, it is not programming.  Worse, the requirement for the hot language of the day is a bigger tell-tale sign that the company is either posturing or doesn't know what they're doing. Back when when I first learned Perl (which isn't the hot new language anymore), there was a hilarious t-shirt that said "Be careful or I'll replace you with a line of code"<ref>Dave Jacoby agrees with me on the broad point that (programming) languages are just different domain dialects, and also cites the ThinkGeek t-shirt phrase "Go Away Or I Will Replace You With a Small Shell Script"
https://jacoby.github.io/2021/11/16/i-will-replace-you-with-a-small-shell-script.html</ref>. Although you ''could'' write the Python example more concisely, it is a real-world example of code that I found that does the same thing as 5 lines of BASH that I wrote.
[[Bash]] code to concatenate [[certbot]] certificates:<syntaxhighlight lang="bash">
#!/bin/bash
# $RENEWED_DOMAINS will contain a space-delimited list of renewed
# certificate domains (for example, "example.com www.example.com"
# loop through a dynamic list of directories in 'live'
# for SITE in $(find /etc/letsencrypt/live -mindepth 1 -maxdepth 1 -type d -exec basename {} \;)
# $RENEWED_LINEAGE will contain the live subdirectory
for SITE in $RENEWED_DOMAINS
do
        # move to correct let's encrypt directory
        cd $RENEWED_LINEAGE
        # cat files to make combined .pem for haproxy
        cat fullchain.pem privkey.pem > /etc/haproxy/certs/$SITE.pem
done
# reload haproxy
# systemctl reload haproxy
</syntaxhighlight>Python code to concatenate certbot certificates:<syntaxhighlight lang="python3">
#!/usr/bin/env python3
import os
import re
import sys
# Certbot sets an environment variable RENEWED_LINEAGE, which points to the
# path of the renewed certificate. We use that path to determine and find
# the files for the currently renewed certificated
lineage=os.environ.get('RENEWED_LINEAGE')
# If nothing renewed, exit
if not lineage:
    sys.exit()
# From the linage, we strip the 'domain name', which is the last part
# of the path.
result = re.match(r'.*/live/(.+)$', lineage)
# If we can not recognize the path, we exit with 1
if not result:
    sys.exit(1)
# Extract the domain name
domain = result.group(1)
# Define a path for HAproxy where you want to write the .pem file.
deploy_path="/etc/haproxy/ssl/" + domain + ".pem"
# The source files can be found in below paths, constructed with the lineage
# path
source_key = lineage + "/privkey.pem"
source_chain = lineage + "/fullchain.pem"
# HAproxy requires to combine the key and chain in one .pem file
with open(deploy_path, "w") as deploy, \
        open(source_key, "r") as key, \
        open(source_chain, "r") as chain:
    deploy.write(key.read())
    deploy.write(chain.read())
# Here you can add your service reload command. Which will be executed after
# every renewal, which is fine if you only have a few domains.
# Alternative is to add the reload to the --post-hook. In that case it is only
# run once after all renewals. That would be the use-case if you have a large
# number of different certificates served by HAproxy.
</syntaxhighlight>


[[Category:System Administration]]
[[Category:System Administration]]