how to remove "^@" from text files in unix?

Question

I am trying to remove "^@" from multiple text files using Unix platform. I have already found this solution, but it does not work for my case. I also used sed -i -e 's/^@//g' testfile.txt and dos2unix testfile.txt. sample data are put here.

Any suggestion would be appreciated.

I do not know about literal circumflexes and ats. Can you please clarify it? I redo dos2unix but no success. you can also the file in this link. — Seyed Omid Nabavi, Oct 29 '18 at 12:37
Sorry for bothering you with my little knowledge. ^@ are between all letters. — Seyed Omid Nabavi, Oct 29 '18 at 12:45
@SeyedOmidNabavi You can convert the file to UTF-8 with something like `iconv` or `recode`: `iconv -f UTF-16 -t UTF-8 < test.txt` or `recode utf16le..utf8 < test.txt` — Biffen, Oct 29 '18 at 12:47
Possible duplicate of [Using iconv to convert from UTF-16LE to UTF-8](https://stackoverflow.com/questions/17287713/using-iconv-to-convert-from-utf-16le-to-utf-8) — Biffen, Oct 29 '18 at 12:50
@n.m. Was about to post it, but figured there had to be a duplicate, so I voted to close as such instead. — Biffen, Oct 29 '18 at 12:51
@SeyedOmidNabavi As for ‘*Ambiguous input redirect*’: You probably got a `<` or `>` wrong somewhere, or the paths to the files contain spaces (or other special characters) and need to be quoted. — Biffen, Oct 29 '18 at 13:00

Jim Stewart · Accepted Answer · 2018-10-29T12:51:57.147

2

The ^@ that you're seeing isn't a literal string. It's an escape code for a NUL (character value 0). If you want to remove them all:

tr -d '\0' <test.txt >newfile.txt

To help diagnose this sort of thing, the od (octal dump) utility is handy. I ran this on the test file you linked, to confirm that they were NULs:

$ od -c test.txt | head
0000000  \0   A  \0   i  \0   r  \0   Q  \0   u  \0   a  \0   l  \0   i
0000020  \0   t  \0   y  \0   S  \0   t  \0   a  \0   t  \0   i  \0   o
0000040  \0   n  \0   E  \0   o  \0   I  \0   C  \0   o  \0   d  \0   e
0000060  \0  \n  \0   D  \0   E  \0   H  \0   E  \0   0  \0   4  \0   4
*
0000400  \0  \n  \0   D  \0   E  \0   H      \0   E  \0   0  \0   4  \0
0000420   4  \0  \n  \0   D  \0   E  \0   H  \0   E  \0   0  \0   4  \0
*
0422160   4  \0  \n  \n
0422164

edited Oct 29 '18 at 12:51

answered Oct 29 '18 at 12:44

Jim Stewart

16,964
5
69
89

2

Although this solved the immediate problem, it looks like the data was UTF-16, and you should probably use a real conversion tool (see comments on your question). – Jim Stewart Oct 29 '18 at 12:53
2

The file is in UTF-16. Removing the nulls will screw up any non-ASCII characters. – n. m. could be an AI Oct 29 '18 at 12:53

how to remove "^@" from text files in unix?

1 Answers1