catonmat net-Bash One-Liners Explained Part I Working with files

Prévia do material em texto

<p>Bash One-Liners Explained, Part I: Working with files</p><p>catonmat.net/bash-one-liners-explained-part-one</p><p>Last updated 6 weeks ago</p><p>I love being super fast in the shell so I decided to do a new article series called Bash</p><p>One-Liners Explained. It's going to be similar to my other article series - Awk One-Liners</p><p>Explained, Sed One-Liners Explained, and Perl One-Liners Explained. After I'm done with</p><p>this bash series, I'll release an e-book by the same title, just as I did with awk, sed, and</p><p>perl series. The e-book will be available as a pdf and in mobile formats (mobi and epub).</p><p>I'll also be releasing bash1line.txt, similar to perl1line.txt that I made for the perl series.</p><p>In this series I'll use the best bash practices, various bash idioms and tricks. I want to</p><p>illustrate how to get various tasks done with just bash built-in commands and bash</p><p>programming language constructs.</p><p>Also see my other articles about working fast in bash:</p><p>Let's start.</p><p>Part I: Working With Files</p><p>1. Empty a file (truncate to 0 size)</p><p>$ > file</p><p>This one-liner uses the output redirection operator > . Redirection of output causes the</p><p>file to be opened for writing. If the file does not exist it is created; if it does exist it is</p><p>truncated to zero size. As we're not redirecting anything to the file it remains empty.</p><p>If you want to replace the contents of a file with some string or create a file with specific</p><p>content, you can do this:</p><p>$ echo "some string" > file</p><p>This puts the string "some string" in the file.</p><p>2. Append a string to a file</p><p>$ echo "foo bar baz" >> file</p><p>This one-liner uses a different output redirection operator >> , which appends to the</p><p>file. If the file does not exist it is created. The string appended to the file is followed by a</p><p>newline. If you don't want a newline appended after the string, add the -n argument to</p><p>echo :</p><p>$ echo -n "foo bar baz" >> file</p><p>1/6</p><p>https://catonmat.net/bash-one-liners-explained-part-one</p><p>https://catonmat.net/awk-one-liners-explained-part-one</p><p>https://catonmat.net/sed-one-liners-explained-part-one</p><p>https://catonmat.net/perl-one-liners-explained-part-one</p><p>https://catonmat.net/books</p><p>https://catonmat.net/ftp/perl1line.txt</p><p>3. Read the first line from a file and put it in a variable</p><p>$ read -r line < file</p><p>This one-liner uses the built-in bash command read and the input redirection operator</p><p>< . The read command reads one line from the standard input and puts it in the line</p><p>variable. The -r parameter makes sure the input is read raw, meaning the backslashes</p><p>won't get escaped (they'll be left as is). The redirection command < file opens file for</p><p>reading and makes it the standard input to the read command.</p><p>The read command removes all characters present in the special IFS variable. IFS</p><p>stands for Internal Field Separator that is used for word splitting after expansion and to</p><p>split lines into words with the read built-in command. By default IFS contains space,</p><p>tab, and newline, which means that the leading and trailing tabs and spaces will get</p><p>removed. If you want to preserve them, you can set IFS to nothing for the time being:</p><p>$ IFS= read -r line < file</p><p>This will change the value of IFS just for this command and will make sure the first line</p><p>gets read into the line variable really raw with all the leading and trailing whitespaces.</p><p>Another way to read the first line from a file into a variable is to do this:</p><p>$ line=$(head -1 file)</p><p>This one-liner uses the command substitution operator $(...) . It runs the command in</p><p>... , and returns its output. In this case the command is head -1 file that outputs the</p><p>first line of the file. The output is then assigned to the line variable. Using $(...) is</p><p>exactly the same as ... , so you could have also written:</p><p>$ line=`head -1 file`</p><p>However $(...) is the preferred way in bash as it's cleaner and easier to nest.</p><p>4. Read a file line-by-line</p><p>$ while read -r line; do</p><p># do something with $line</p><p>done < file</p><p>This is the one and only right way to read lines from a file one-by-one. This method puts</p><p>the read command in a while loop. When the read command encounters end-of-file,</p><p>it returns a positive return code (code for failure) and the while loop stops.</p><p>Remember that read trims leading and trailing whitespace, so if you want to preserve it,</p><p>clear the IFS variable:</p><p>$ while IFS= read -r line; do</p><p># do something with $line</p><p>done < file</p><p>2/6</p><p>If you don't like the to put < file at the end, you can also pipe the contents of the file to</p><p>the while loop:</p><p>$ cat file | while IFS= read -r line; do</p><p># do something with $line</p><p>done</p><p>5. Read a random line from a file and put it in a variable</p><p>$ read -r random_line < <(shuf file)</p><p>There is no clean way to read a random line from a file with just bash, so we'll need to</p><p>use some external programs for help. If you're on a modern Linux machine, then it</p><p>comes with the shuf utility that's in GNU coreutils.</p><p>This one-liner uses the process substitution <(...) operator. This process substitution</p><p>operator creates an anonymous named pipe, and connects the stdout of the process to</p><p>the write part of the named pipe. Then bash executes the process, and it replaces the</p><p>whole process substitution expression with the filename of the anonymous named pipe.</p><p>When bash sees <(shuf file) it opens a special file /dev/fd/n , where n is a free file</p><p>descriptor, then runs shuf file with its stdout connected to /dev/fd/n and replaces</p><p><(shuf file) with /dev/fd/n so the command effectively becomes:</p><p>$ read -r random_line < /dev/fd/n</p><p>Which reads the first line from the shuffled file.</p><p>Here is another way to do it with the help of GNU sort . GNU sort takes the -R option</p><p>that randomizes the input.</p><p>$ read -r random_line < <(sort -R file)</p><p>Another way to get a random line in a variable is this:</p><p>$ random_line=$(sort -R file | head -1)</p><p>Here the file gets randomly sorted by sort -R and then head -1 takes the first line.</p><p>6. Read the first three columns/fields from a file into variables</p><p>$ while read -r field1 field2 field3 throwaway; do</p><p># do something with $field1, $field2, and $field3</p><p>done < file</p><p>If you specify more than one variable name to the read command, it shall split the line</p><p>into fields (splitting is done based on what's in the IFS variable, which contains a</p><p>whitespace, a tab, and a newline by default), and put the first field in the first variable,</p><p>the second field in the second variable, etc., and it will put the remaining fields in the last</p><p>variable. That's why we have the throwaway variable after the three field variables. if we</p><p>didn't have it, and the file had more than three columns, the third field would also get</p><p>3/6</p><p>the leftovers.</p><p>Sometimes it's shorter to just write _ for the throwaway variable:</p><p>$ while read -r field1 field2 field3 _; do</p><p># do something with $field1, $field2, and $field3</p><p>done < file</p><p>Or if you have a file with exactly three fields, then you don't need it at all:</p><p>$ while read -r field1 field2 field3; do</p><p># do something with $field1, $field2, and $field3</p><p>done < file</p><p>Here is an example. Let's say you want to find out number of lines, number of words,</p><p>and number of bytes in a file. If you run wc on a file you get these 3 numbers plus the</p><p>filename as the fourth field:</p><p>$ cat file-with-5-lines</p><p>x 1</p><p>x 2</p><p>x 3</p><p>x 4</p><p>x 5</p><p>$ wc file-with-5-lines</p><p>5 10 20 file-with-5-lines</p><p>So this file has 5 lines, 10 words, and 20 chars. We can use the read command to get</p><p>this info into variables:</p><p>$ read lines words chars _ < <(wc file-with-5-lines)</p><p>$ echo $lines</p><p>5</p><p>$ echo $words</p><p>10</p><p>$ echo $chars</p><p>20</p><p>Similarly you can use here-strings to split strings into variables. Let's say you have a</p><p>string "20 packets in 10 seconds" in a $info variable and you want to extract 20 and 10 .</p><p>Not too long ago I'd have written this:</p><p>However given the power of read and our bash knowledge, we can now do this:</p><p>$ read packets _ _ time _ <<< "$info"</p><p>Here the <<< is a here-string, which lets you pass strings directly to the standard input of</p><p>commands.</p><p>7. Find the size of a file, and put it</p><p>in a variable</p><p>4/6</p><p>$ size=$(wc -c &lt; file)</p><p>This one-liner uses the command substitution operator $(...) that I explained in one-liner #3.</p><p>It runs the command in ... , and returns its output. In this case the command is wc -c < file</p><p>that prints the number of chars (bytes) in the file. The output is then assigned to size</p><p>variable.</p><p>8. Extract the filename from the path</p><p>Let's say you have a /path/to/file.ext , and you want to extract just the filename file.ext . How</p><p>do you do it? A good solution is to use the parameter expansion mechanism:</p><p>$ filename=${path##*/}</p><p>This one-liner uses the ${var##pattern} parameter expansion. This expansion tries to match</p><p>the pattern at the beginning of the $var variable. If it matches, then the result of the</p><p>expansion is the value of $var with the longest matching pattern deleted.</p><p>In this case the pattern is */ which matches at the beginning of /path/to/file.ext and as it's a</p><p>greedy match, the pattern matches all the way till the last slash (it matches /path/to/ ). The</p><p>result of this expansion is then just the filename file.ext as the matched pattern gets deleted.</p><p>9. Extract the directory name from the path</p><p>This is similar to the previous one-liner. Let's say you have a /path/to/file.ext , and you want to</p><p>extract just the path to the file /path/to . You can use the parameter expansion again:</p><p>$ dirname=${path%/*}</p><p>This time it's the ${var%pattern} parameter expansion that tries to match the pattern at the</p><p>end of the $var variable. If the pattern matches, then the result of the expansion is the value</p><p>of $var shortest matching pattern deleted.</p><p>In this case the pattern is /* , which matches at the end of /path/to/file.ext (it matches</p><p>/file.ext ). The result then is just the dirname /path/to as the matched pattern gets deleted.</p><p>10. Make a copy of a file quickly</p><p>Let's say you want to copy the file at /path/to/file to /path/to/file_copy . Normally you'd write:</p><p>$ cp /path/to/file /path/to/file_copy</p><p>However you can do it much quicker by using the brace expansion {...} :</p><p>$ cp /path/to/file{,_copy}</p><p>Brace expansion is a mechanism by which arbitrary strings can be generated. In this particular</p><p>case /path/to/file{,_copy} generates the string /path/to/file /path/to/file_copy , and the whole</p><p>command becomes cp /path/to/file /path/to/file_copy .</p><p>Similarly you can move a file quickly:</p><p>$ mv /path/to/file{,_old}</p><p>This expands to mv /path/to/file /path/to/file_old .</p><p>5/6</p><p>Enjoy!</p><p>Enjoy the article and let me know in the comments what you think about it! If you think that I</p><p>forgot some interesting bash one-liners related to file operations, let me know in to comments</p><p>also!</p><p>Thanks for reading my post. If you enjoyed it and would like to receive my posts automatically,</p><p>you can subscribe to new posts via rss feed or email.</p><p>Browserling reaches 100 paying customers</p><p>Bash One-Liners Explained, Part II: Working with strings</p><p>6/6</p><p>https://catonmat.net/feed</p><p>https://catonmat.net/browserling-reaches-100-paying-customers</p><p>https://catonmat.net/bash-one-liners-explained-part-two</p><p>Bash One-Liners Explained, Part I: Working with files</p><p>Part I: Working With Files</p><p>Enjoy!</p>