Linux Filtering Programs

Unix/Linux has a special group of software tools called "filters." Some filters cannot directly accept input from the keyboard or be used independently. Instead, they operate through pipes or redirection. Filters primarily work on a line-by-line basis to filter, search, modify, replace, insert, delete, or perform statistical operations on data.

The output of filters is usually directed to stdout, which is the screen. If you want to save the output to a file, you need to redirect it, such as using cat fileA | tr -s '\n' > fileB.

To avoid any misinterpretation of the pattern (PATTERN) when using regular expressions, it is generally recommended to enclose the pattern in single quotes ('') or double quotes (""). Therefore, in the example you provided, it is advised to write it as grep '/home' /etc/passwd.

grep itself is not difficult; the challenge lies in becoming proficient with regular expressions.

grep has two commonly used options: -F and -E. The -F option is used to disable the interpretation of regular expressions, treating the pattern as a literal string. On the other hand, the -E option enables extended regular expression matching.

Here are some possible options and usages of grep:

^ back on top ^

• egrep
  egrep is equivalent to grep -E to interpret patten with Extended Regular Expressions. For example egrep 'goo?' means I want to use extended regular expressions to match the string "go" or "goooo" but not the string "goo?".
  
  Example:

  $ egrep 'goo?' re.txt ← Search string "go" or "goo"... "gooooooooooo"

  
  Example:

  $ ls /etc/ | egrep 'pr(o|e)' ← Search for strings with "pre" or "pro" iproute2 modprobe.conf modprobe.conf~ modprobe.d prelink.cache (The following is omitted)

  
  
• fgrep
  fgrep is equivalent to grep -F. It is used to search for fixed strings (without regular expressions). In addition to being particularly fast in searching, it also avoids the interpretation of special characters such as ".", "*", or "|" as regular expression syntax.
  
  Example:

  $ fgrep 's.' re.txt ←Search for the exact string "s." without using regular expressions google Goggles Solves SUDOKU Puzzles. $ fgrep '.*' /etc/*.conf ←To search for the string ".*" (literally, without interpreting it as a regular expression meaning "zero or more characters")

  
  

^ back on top ^

Let me explain some ambiguous options. When using cut -d to set the field delimiter, if a line does not have a matching field delimiter, by default, the entire line will be outputted. In such cases, you can add the -s option to prevent the line from being outputted.

The option "--output-delimiter" allows you to freely set the output delimiter string. In the following example, we will change the original tab delimiter to the string "---" for output.

• Written in awk as ls -l | awk '{print $8}'。
• Use sedto shrink repeated blank characters or tabs into one and then cut, for example ls -l | sed 's/[[:space:]][[:space:]]*/\t/g' | cut -f8。
• Use tr to shrink repeated blank characters into one, and then cutsuch asls -l | tr -s " " | cut -d " " -f8 (only suitable for formats without tabs.)
  
  

^ back on top ^

When viewing the man (manual) pages for a command, you may notice that the documentation contains formatting elements such as bold text or underlines. This is because the man pages are not plain text files; they contain control characters that control the display or operation of the text.

In this case, you can use the col program to filter out control characters from the file. The "-b" option is used to filter out all known control characters like "RLF" (Reverse Line Feed), "HRLF" (Half Reverse Line Feed), "HFLF" (Half Forward Line Feed), etc.

So, in the given example, using the command man regex | col -b > regex.txt will filter out the control characters that cause garbled text, allowing you to use a regular text editor to read or edit the file.

The name col is an abbreviation of the word "colander," Generally, col is operated through pipelines (e.g., COMMAND | col) or redirection (e.g., col -x < FILE > SAVE_FILE) for operation.

Another useful option is -x, which converts tabs to corresponding spaces. This can be handy when you have a file that needs to convert tabs to spaces for formatting purposes.

^ back on top ^

You can indeed use the format tr 'abcde' 'ijklm' where the characters 'i', 'j', 'k', 'l', and 'm' sequentially replace 'a', 'b', 'c', 'd', and 'e' respectively. However, it's important to note that it's not replacing the string 'ijklm' with 'abcde'.

In the following example, the character '5' is replaced with 'i' and the character '6' is replaced with 's':

Although sed is powerful, it operates on data based on the "pattern space" and removes trailing newline characters, making it unable to handle newline characters directly. In such cases, tr can be used easily.

^ back on top ^

The basic usage of sort is straightforward, and once you see an example, you will understand it. Below is an example with a file named "equip," which is a list of computer equipment purchased by the company this year.

The previous example is much better, but it's still strange that "ASUS" and "acer" with the same starting letter "A" are sorted so differently, as per the ASCII table where uppercase letters come before lowercase letters.

To sort the list without considering case sensitivity, you can use the "-f" option with the sort command. This option tells the sort command to ignore case and treat all letters as uppercase before sorting. In this way, "ASUS" and "acer" will be considered as the same letter starting with "A," and they will be sorted based on other criteria (such as their remaining characters or subsequent sorting rules).

Please note that the previous example still looks a bit strange. For example, when sorting in ascending order, the value "20" in the third column should come after "3." The reason for this behavior is that the sort command sorts based on the first character of each field. If the first character is the same, it proceeds to compare the next character, and so on. Therefore, based on the first character, "1x" and "2x" will always come before "3."

To resolve this issue, you can use the -n option. This option tells the sort command to perform a numerical sort based on the values in the specified field. Here's an example:

the sort command supports sorting based on multiple fields with different priority levels. You can achieve this by adding multiple -k options to specify the fields and their priority.
In your example, you want to sort based on field 2, and in the case of equality in field 2, you want to further sort based on field 5. Here's the command:

Talk about the option "-k", such as "-k 2.3", which means that the sorting starts from the third character of column 2.
The following example assumes that the "id" of the employee user is added before the list you get. For example, if you want to filter out the "id" and then sort it, you can specify field 1 to sort from the 6th character, as shown below.

^ back on top ^

^ back on top ^