1

I am using GNU grep 3.4 to find scripts that contain a certain pattern. For this, I call grep recursively like so

grep -rin . -e "pattern"

The pattern is just a word, no regular expression. The weird thing is that the output does not list occurrences in some files which definitely contain the string.

I have tried opening these files in vim and searching there using /pattern and it finds the pattern. The encoding is displayed in vim as [dos:utf-8:]. When I copy the line and write it to a new file, the above grep command lists it correctly.

Why doesn't grep list the original file?

buddemat
  • 162

2 Answers2

1

Grep (or at least older ones) don't understand UTF8. So composed characters, hyphenation points or other invisible data might discourage grep.

Grep is also affected by the values of $LC_ALL, $LC_CTYPE and $LANG.

Use vim to save a few lines around a word that grep failed to find then make a hexdump of that tiny example file. You'll probably see why grep failed

You can also use vim commands (ga, g8 etc) to examine characters but a hex dump may be clearer

RedGrittyBrick
  • 85,717
1

I found the problem (with the help of another answer). The files 'grep' did not show any output for were not actually utf-8 encoded, but utf-16be. I learned this using hexdump (credits to @RedGrittyBrick):

hd file_for_which_grep_works_as_expected.txt

yielded

00000000  20 20 20 20 50 61 74 74  65 72 6e 0a              |    Pattern.|
0000000c

whereas

hd file_for_which_grep_fails.txt

returned

00000000  fe ff 00 50 00 61 00 74  00 74 00 65 00 72 00 6e  |...P.a.t.t.e.r.n|
00000010  00 0a                                             |..|
00000012

So, double-checking the encoding with

file -i file_for_which_grep_fails.txt

identified it as text/plain; charset=utf-16be.

I failed to recognize that the utf-8 shown by vim were actually the buffer encoding, not the file encoding. Executing :set fileencoding in vim also correctly displayed fileencoding=utf-16 (found here https://superuser.com/a/28783/1210682).

So, the problem is that my grep does not work on utf-16 encoded files. This has already been described here: https://superuser.com/a/231471/1210682. However, the remedy of converting utf-16 files to utf-8 before grep does not work when I use it recursively, as I don't know beforehand which files may be utf-8 and which utf-16 and a am searching through a lot of files.

There are different solutions, two of which I am going to shortly describe here:

  1. A quick-and-dirty solution that worked for me was to expand the search pattern to include one that would match the utf-16 version and search for one of both patterns:

    grep -riPa . -e "pattern|p.a.t.t.e.r.n."

    This is of course very limited in terms of possible patterns.

  2. There are alternatives to grep like ugrep or ripgrep that (among other things) can handle utf-16 files. I ended up using ripgrep which is available in the standard Ubuntu package repositories from 18.04 on:

    rg -i "pattern"

There is a great discussion on alternatives here: https://stackoverflow.com/questions/3752913/grepping-binary-files-and-utf16, among them an interesting approach attempting to convert the search pattern to utf-16 and feeding that to grep. However, I couldn't get it to work.

buddemat
  • 162