I can delete duplicate lines in files using below commands: 1) sort -u and uniq commands. is that possible using sed or awk ?
Asked
Active
Viewed 6,936 times
3
-
2if you have sort and uniq, why do you want to use sed or awk? – Skriptotajs Feb 27 '14 at 11:33
-
Well, possible it is, since both are turing complete languages, as far as I recall. The question is what for you'd use them, as pointed by @Skriptotajs. – Rubens Feb 27 '14 at 11:34
-
Possible duplicate of [How can I delete duplicate lines in a file in Unix?](https://stackoverflow.com/questions/1444406/how-can-i-delete-duplicate-lines-in-a-file-in-unix) – tripleee Oct 24 '18 at 04:45
3 Answers
11
There's a "famous" awk idiom:
awk '!seen[$0]++' file
It has to keep the unique lines in memory, but it preserves the file order.
glenn jackman
- 238,783
- 38
- 220
- 352
-
This looks awesome but somehow it's not working for me on macOS Sierra. – mherzl Jan 20 '17 at 07:18
-
only for small files, if file bigger then ram + swap - not worked – Alex Muravyov Feb 02 '17 at 13:38
-
0
After sorting we can use this sed command
sed -E '$!N; /^(.*)\n\1$/!P; D' filename
If the file is unsorted then you can use with combination of the command.
sort filename | sed -E '$!N; /^\(.*\)\n\1$/!P; D'
-
Obviously these alternatives are unacceptable if you can't sort the file. – tripleee Mar 09 '21 at 17:22
0
sort and uniq these only need to remove duplicates cat filename | sort | uniq >> filename2
if its file consist of number use sort -n
Arun Binoy
- 327
- 1
- 10
-
Though the [`cat` is useless.](https://stackoverflow.com/questions/11710552/useless-use-of-cat) – tripleee Mar 09 '21 at 17:23
-
The `uniq` is also useless. Just use `sort -u filename` The `-u` invokes `sort`'s unique mode. [ None of which answered the OP's question... ] – dave58 Sep 01 '21 at 17:23