Difference between revisions of "One-liners"
(Adds example on how to split large files) |
|||
| (5 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
Sometimes one-liners are so cool, you just want to remember them. And good one-liners can also teach you the intricacies and features of the [[Bash]] shell. Although there are better sites on the Internet for [http://www.bashoneliners.com/ finding one-liners], [http://www.catonmat.net/series/bash-one-liners-explained understanding one-liners] or [http://uni.xkcd.com/ playing on the command line], we'd still like to illustrate a few here. | Sometimes one-liners are so cool, you just want to remember them. And good one-liners can also teach you the intricacies and features of the [[Bash]] shell. Although there are better sites on the Internet for [http://www.bashoneliners.com/ finding one-liners], [http://www.catonmat.net/series/bash-one-liners-explained understanding one-liners] or [http://uni.xkcd.com/ playing on the command line], we'd still like to illustrate a few here. | ||
| − | == Perl edit == | + | ==Find big files or directories== |
| + | [[File:Anas platyrhynchos (mixed pair) (32428014687).jpg|alt=ducks|thumb|ducks]] | ||
| + | Help! I'm out of disc space. How do I find out where the big files or directories are that are consuming all storage? | ||
| + | |||
| + | <source lang="bash">du -cks -- * | sort -rn | head</source> | ||
| + | <code>du --total --block-size=1K --summarize</code> and the double dash argument means 'take the arguments from STDIN' then the asterisk is the glob character that matches 'everything in this directory', so each file and directory in the current working directory is summarized. This is the piped to <code>sort</code> with the reverse, numeric options and then piped to <code>head</code> for showing just the top 10. Adjust to taste. | ||
| + | |||
| + | ==Mount remote filesystem== | ||
| + | Using sshfs is a great tool for mounting remote filesystems so that you can use your local tools on them. This example supplies a complex SSH command, including port-forwarding at the same time, to the [[SSHFS]] tool. | ||
| + | <source lang="bash">sshfs -o idmap=user -o ssh_command='ssh -t -i /home/greg/.ssh/eQualityTech-Test.pem -o IdentitiesOnly=true -o ForwardAgent=true -L 127.0.0.1:43306:10.0.50.53:3306 centos@ec2-52-203-160-83.compute-1.amazonaws.com ssh -A' centos@10.0.50.161:/ /mnt/es1</source> | ||
| + | |||
| + | ==Compare two wikis for extensions and skins== | ||
| + | This one-liner invokes the API of two wikis asking for info on siteinfo, general, extensions and skins; in json format. Since that data is returned without any newlines, we use `jq` to pretty-print the json output. Then it's an easy `meld` or `diff` to compare them. The `--silent` option to `curl` just suppresses the connection and retrieval metadata; while the `-L` is customary to follow redirects. | ||
| + | <source lang="bash"> | ||
| + | A='https://freephile.org/' B='https://www.mediawiki.org/' API='w/api.php?action=query&meta=siteinfo&siprop=general%7Cextensions%7Cskins&format=json' meld <(curl --silent -L "${A}${API}" | jq '.') <(curl --silent -L "${B}${API}" | jq '.')</source> | ||
| + | |||
| + | ==Perl edit== | ||
Sometimes you want to make a bunch of changes (substitutions) of the same text across multiple files. Like changing a product name across multiple pages of documentation. With a one-line perl command, you can do just that. Furthermore, the example below uses a <code>ls</code> command to select which files to operate on -- giving you even more powerful control over your one-line edit. | Sometimes you want to make a bunch of changes (substitutions) of the same text across multiple files. Like changing a product name across multiple pages of documentation. With a one-line perl command, you can do just that. Furthermore, the example below uses a <code>ls</code> command to select which files to operate on -- giving you even more powerful control over your one-line edit. | ||
<source lang="perl"> | <source lang="perl"> | ||
| Line 7: | Line 23: | ||
</source> | </source> | ||
| − | == Free Memory == | + | ==Free Memory== |
Use <code>echo</code> to output the result of a sub-shell, and a few extra characters (' - + p'), which is then piped to the (reverse-polish) desk calculator. Con<code>cat</code>enate the /proc/meminfo file, printing it on STDOUT. Using extended-regex <code>grep</code>, we search for lines of output that begin with "MemFree", "Cached" or "Writeback" followed by the colon character. Piping to <code>awk</code>, we can print out the string in position 2 of each line. Those values are ultimately processed in the calculator by popping the last two numbers off the stack (Writeback and Cached), and adding that result to the first number (MemFree).<ref>[http://www.computerweekly.com/feature/Write-through-write-around-write-back-Cache-explained Cache explained]</ref> | Use <code>echo</code> to output the result of a sub-shell, and a few extra characters (' - + p'), which is then piped to the (reverse-polish) desk calculator. Con<code>cat</code>enate the /proc/meminfo file, printing it on STDOUT. Using extended-regex <code>grep</code>, we search for lines of output that begin with "MemFree", "Cached" or "Writeback" followed by the colon character. Piping to <code>awk</code>, we can print out the string in position 2 of each line. Those values are ultimately processed in the calculator by popping the last two numbers off the stack (Writeback and Cached), and adding that result to the first number (MemFree).<ref>[http://www.computerweekly.com/feature/Write-through-write-around-write-back-Cache-explained Cache explained]</ref> | ||
<source lang="bash"> | <source lang="bash"> | ||
| Line 17: | Line 33: | ||
</pre> | </pre> | ||
| − | == Size of Graphical Desktop (X Window System) == | + | ==Size of Graphical Desktop (X Window System)== |
So you think your graphical desktop is slowing things down compared to using a pure console based system. Short of logging in single user mode, how much memory does the graphical desktop consume? Since everything is a file, we can look in the folder for processes (/proc), and specifically the folder created for the process id of "X" ([http://x.org X.org]). <code>grep</code>ping for the line starting with 'VmSize', we can see the Virtual Memory size of our graphical desktop. | So you think your graphical desktop is slowing things down compared to using a pure console based system. Short of logging in single user mode, how much memory does the graphical desktop consume? Since everything is a file, we can look in the folder for processes (/proc), and specifically the folder created for the process id of "X" ([http://x.org X.org]). <code>grep</code>ping for the line starting with 'VmSize', we can see the Virtual Memory size of our graphical desktop. | ||
<source lang="bash">grep ^VmSize /proc/$(pidof X)/status</source> | <source lang="bash">grep ^VmSize /proc/$(pidof X)/status</source> | ||
| Line 25: | Line 41: | ||
</pre> | </pre> | ||
| − | == Delete old stuff == | + | ==Delete old stuff== |
You stumble upon a directory full of backups, which is great. But you also realize that nobody setup <code>logrotate</code> or other command to prune old content. Maybe that's because these backups are produced manually, say during upgrades, and so they are also deleted manually. What's a quick one-liner to remove old files? Use the <code>mtime</code> (modification time) option to <code>find</code> combined with the <code>exec</code> option to execute <code>rm</code> (remove) said files. | You stumble upon a directory full of backups, which is great. But you also realize that nobody setup <code>logrotate</code> or other command to prune old content. Maybe that's because these backups are produced manually, say during upgrades, and so they are also deleted manually. What's a quick one-liner to remove old files? Use the <code>mtime</code> (modification time) option to <code>find</code> combined with the <code>exec</code> option to execute <code>rm</code> (remove) said files. | ||
<source lang="bash"> | <source lang="bash"> | ||
| Line 36: | Line 52: | ||
</source> | </source> | ||
| − | == Reports with Find == | + | ==Reports with Find== |
Want to see all the <code>.htaccess</code> files in your webroot and see what they do? You can use <code>-exec bash -c</code> to perform multiple commands with one exec. (you can also use multiple -exec options in find). The example below echo's out the name of the found file; then cat's it with numbered lines. Note that the underscore is a throwaway value (could be any text, such as 'foobar') which consumes the first positional argument ($0) to <code>bash -c</code> making it "more readable" to reference our found filename as $1 (since $0 is commonly understood to refer to the script itself). | Want to see all the <code>.htaccess</code> files in your webroot and see what they do? You can use <code>-exec bash -c</code> to perform multiple commands with one exec. (you can also use multiple -exec options in find). The example below echo's out the name of the found file; then cat's it with numbered lines. Note that the underscore is a throwaway value (could be any text, such as 'foobar') which consumes the first positional argument ($0) to <code>bash -c</code> making it "more readable" to reference our found filename as $1 (since $0 is commonly understood to refer to the script itself). | ||
<source lang="bash"> | <source lang="bash"> | ||
| Line 47: | Line 63: | ||
<ref>https://stackoverflow.com/questions/5119946/find-exec-with-multiple-commands</ref> | <ref>https://stackoverflow.com/questions/5119946/find-exec-with-multiple-commands</ref> | ||
| − | == Split a big file == | + | And this one in your /opt/conf-meza/public "config" directory |
| + | <source lang="bash"> | ||
| + | find . -name '*yml' -o -name '*php' -exec bash -c 'echo -e "\n$0\n"; grep --perl-regexp --only-matching "^\s*(\\\$[^\[ ]+)" '{}' | sed -e "s/^[[:space:]]*//" | sort -u ' '{}' \; | ||
| + | </source> | ||
| + | |||
| + | ==Split a big file== | ||
Say you have a file with 50,000 lines in it, which becomes unwieldy to deal with in a spreadsheet or otherwise. You can easily split the file into segments with the <code>split</code> command. Be default it uses alpha suffixes (little_file.aa, little_file.ab, etc.) If you add the option <code>--numeric-suffixes</code>, then you'll end up with little_file.00, little_file.01, etc. If you would like to re-add the original suffix, then you must use the option called <code>--additional-suffix</code> | Say you have a file with 50,000 lines in it, which becomes unwieldy to deal with in a spreadsheet or otherwise. You can easily split the file into segments with the <code>split</code> command. Be default it uses alpha suffixes (little_file.aa, little_file.ab, etc.) If you add the option <code>--numeric-suffixes</code>, then you'll end up with little_file.00, little_file.01, etc. If you would like to re-add the original suffix, then you must use the option called <code>--additional-suffix</code> | ||
Latest revision as of 10:25, 26 July 2024
Sometimes one-liners are so cool, you just want to remember them. And good one-liners can also teach you the intricacies and features of the Bash shell. Although there are better sites on the Internet for finding one-liners, understanding one-liners or playing on the command line, we'd still like to illustrate a few here.
Contents
Find big files or directories[edit | edit source]
_(32428014687).jpg)
Help! I'm out of disc space. How do I find out where the big files or directories are that are consuming all storage?
du -cks -- * | sort -rn | head
du --total --block-size=1K --summarize and the double dash argument means 'take the arguments from STDIN' then the asterisk is the glob character that matches 'everything in this directory', so each file and directory in the current working directory is summarized. This is the piped to sort with the reverse, numeric options and then piped to head for showing just the top 10. Adjust to taste.
Mount remote filesystem[edit | edit source]
Using sshfs is a great tool for mounting remote filesystems so that you can use your local tools on them. This example supplies a complex SSH command, including port-forwarding at the same time, to the SSHFS tool.
sshfs -o idmap=user -o ssh_command='ssh -t -i /home/greg/.ssh/eQualityTech-Test.pem -o IdentitiesOnly=true -o ForwardAgent=true -L 127.0.0.1:43306:10.0.50.53:3306 centos@ec2-52-203-160-83.compute-1.amazonaws.com ssh -A' centos@10.0.50.161:/ /mnt/es1
Compare two wikis for extensions and skins[edit | edit source]
This one-liner invokes the API of two wikis asking for info on siteinfo, general, extensions and skins; in json format. Since that data is returned without any newlines, we use `jq` to pretty-print the json output. Then it's an easy `meld` or `diff` to compare them. The `--silent` option to `curl` just suppresses the connection and retrieval metadata; while the `-L` is customary to follow redirects.
A='https://freephile.org/' B='https://www.mediawiki.org/' API='w/api.php?action=query&meta=siteinfo&siprop=general%7Cextensions%7Cskins&format=json' meld <(curl --silent -L "${A}${API}" | jq '.') <(curl --silent -L "${B}${API}" | jq '.')
Perl edit[edit | edit source]
Sometimes you want to make a bunch of changes (substitutions) of the same text across multiple files. Like changing a product name across multiple pages of documentation. With a one-line perl command, you can do just that. Furthermore, the example below uses a ls command to select which files to operate on -- giving you even more powerful control over your one-line edit.
perl -p -i -e "s/lemons/lemonade/" $(/bin/ls my/life*)
Free Memory[edit | edit source]
Use echo to output the result of a sub-shell, and a few extra characters (' - + p'), which is then piped to the (reverse-polish) desk calculator. Concatenate the /proc/meminfo file, printing it on STDOUT. Using extended-regex grep, we search for lines of output that begin with "MemFree", "Cached" or "Writeback" followed by the colon character. Piping to awk, we can print out the string in position 2 of each line. Those values are ultimately processed in the calculator by popping the last two numbers off the stack (Writeback and Cached), and adding that result to the first number (MemFree).[1]
echo $(cat /proc/meminfo | egrep '^(MemFree|Cached|Writeback):' | awk '{print $2}') - + p | dc
Result:
3033240
Size of Graphical Desktop (X Window System)[edit | edit source]
So you think your graphical desktop is slowing things down compared to using a pure console based system. Short of logging in single user mode, how much memory does the graphical desktop consume? Since everything is a file, we can look in the folder for processes (/proc), and specifically the folder created for the process id of "X" (X.org). grepping for the line starting with 'VmSize', we can see the Virtual Memory size of our graphical desktop.
grep ^VmSize /proc/$(pidof X)/status
Result:
VmSize: 158212 kB
Delete old stuff[edit | edit source]
You stumble upon a directory full of backups, which is great. But you also realize that nobody setup logrotate or other command to prune old content. Maybe that's because these backups are produced manually, say during upgrades, and so they are also deleted manually. What's a quick one-liner to remove old files? Use the mtime (modification time) option to find combined with the exec option to execute rm (remove) said files.
# Make sure we've got backups; look for recent files
sudo ls -al /backups
# list everything in the backups folder that's older than 30 days
sudo find /backups -mtime +30 -ls
# OK, delete those files
sudo find /backups -mtime +30 -exec rm {} \;
Reports with Find[edit | edit source]
Want to see all the .htaccess files in your webroot and see what they do? You can use -exec bash -c to perform multiple commands with one exec. (you can also use multiple -exec options in find). The example below echo's out the name of the found file; then cat's it with numbered lines. Note that the underscore is a throwaway value (could be any text, such as 'foobar') which consumes the first positional argument ($0) to bash -c making it "more readable" to reference our found filename as $1 (since $0 is commonly understood to refer to the script itself).
# All give similar output
find _mw -name .htaccess -exec bash -c 'echo -e "\n$1\n"; cat -n "$1"' _ '{}' \;
find _mw -name .htaccess -exec bash -c 'echo -e "\n$0\n"; cat -n "$0"' '{}' \;
find _mw -name .htaccess -exec bash -c 'echo -e "\n$0$1\n"; cat -n "$1"' 'Reporting on ' '{}' \;
find _mw -name .htaccess -exec echo -e "\nReporting on " '{}' "\n" \; -exec cat -n '{}' \;
And this one in your /opt/conf-meza/public "config" directory
find . -name '*yml' -o -name '*php' -exec bash -c 'echo -e "\n$0\n"; grep --perl-regexp --only-matching "^\s*(\\\$[^\[ ]+)" '{}' | sed -e "s/^[[:space:]]*//" | sort -u ' '{}' \;
Split a big file[edit | edit source]
Say you have a file with 50,000 lines in it, which becomes unwieldy to deal with in a spreadsheet or otherwise. You can easily split the file into segments with the split command. Be default it uses alpha suffixes (little_file.aa, little_file.ab, etc.) If you add the option --numeric-suffixes, then you'll end up with little_file.00, little_file.01, etc. If you would like to re-add the original suffix, then you must use the option called --additional-suffix
The following command takes BIG_FILE.txt and for every 10,000 lines of that file, it generates new files called 'little_file.00.txt', 'little_file.01.txt', 'little_file.02.txt', and so on.
split --lines=10000 --numeric-suffixes --additional-suffix='.txt' BIG_FILE.txt little_file.
