Questions tagged [uniq]
26 questions
24
votes
4 answers
Is there a Windows equivalent to the Unix uniq?
I need remove duplicate lines from a text file, it is simple in Linux using
cat file.txt |sort | uniq
when file.txt contains
aaa
bbb
aaa
ccc
It will output
aaa
bbb
ccc
Is there a Windows equivalent? or how do this in a Windows way?
Yu Jiaao
- 773
7
votes
3 answers
Remove duplicates in each line of a file
How can I remove duplicates in each line, for example here?
1 1 1 2 1 2 3
5 5 4 1 2 3 3
I'd like to get this output:
1 2 3
5 4 1 2 3
There are lots of lines (100,000) and in each line I want unique values.
Perl might be the fastest, but how can I…
Arash
- 736
6
votes
3 answers
Remove non-duplicate lines in Linux
how can I remove non-duplicate lines from text file using any linux program linke sed, awk or any other?
Example:
abc
bbc
abc
bbc
ccc
bbc
Result:
abc
bbc
abc
bbc
bbc
Second list have removed ccc because it didn't have duplicate lines.
Is it also…
qlwik
- 63
4
votes
1 answer
Grep on macOS: find unique occurrences of a capturing group in regular expression
I am on macOS and would like to get into using grep (or a similar tool) to find unique occurrences of a certain pattern in a codebase. For example, for finding all console.somemethod() calls in JavaScript I have devised:
grep -oiER…
Dan Inactive
- 145
3
votes
2 answers
How do I make uniq only consider the first field?
I am using FreeBSD 3.2-RELEASE
If I have some sorted text, like this last…
Daniel
- 55
2
votes
1 answer
Printing unique lines with uniq, OS X Yosemite
I am trying to print the unique entries in a column of a .csv file. For this, I tried the following:
awk -F "," '{print $6}' dataCoarse.csv | uniq -u
which just prints the 6th column as it is. There are still duplicate entries. How can I print only…
sodiumnitrate
- 348
2
votes
1 answer
Counting duplicates lines from a stream
I'm currently parsing apache logs with that command:
tail -f /opt/apache/logs/access/gvh-access_log.1365638400 |
grep specific.stuff. | awk '{print $12}' | cut -d/ -f3 > ~/logs
The output is a list of…
cpa
- 131
2
votes
1 answer
gnu sort/uniq: sorting by number of times
How can I use GNU sort and uniq to have the most common occurrences on top instead of numerical or alphanumerical sorting? Example list.txt:
1
2
2
2
3
3
Since '2' occurs 3 times, should be on top, followed by '3' and '1' like this:
$ cat…
719016
- 4,683
2
votes
1 answer
Count based on unique subset of fields
I have a text file that is structured as follows:
P,ABC,DEF
P,GHI,JKL
B,ABC,DEF
B,MNO,PQR
I want to get a count of how many times a line appears where fields 2 and 3 are the same while preserving field 1. So, the output would look something like…
Brian
- 1,085
1
vote
1 answer
Grep matching lines but return matching key IDs in GnuPG output
Having this input:
rsa2048/C7927B82 2015-08-30
rsa2048/FB2D99F9 2015-08-30
I want grep to only return the key ID, for example: C7927B92. Using the pattern from Grep characters before and after match?, I used
grep -o -P 'rsa.{3,13}'
resulting…
StackAbstraction
- 910
1
vote
3 answers
remove lines with duplicate words
I have a sorted file with lines like this
word1 abca
word1 abcb
word1 abcc
word2 abca
word2 abcb
word3 abbb
...........
and i want to have something like this
word1 abca
word2 abca
word3 abbb
...........
anon123
- 21
0
votes
1 answer
Remove list of items from another file in bash
What would be the most efficient method (no repeated command execution) to remove items listed in one file from another file (unordered) ?
One can easily get the list of non-matching items in the second file by
cat first_file.txt second_file.txt |…
dronus
- 1,978
0
votes
2 answers
Sort on ID, then sort on date, then remove lines which have the older date?
I have a couple of tab delimited files.
Each file is structured like so:
ID Title Rating Date_Rated
What I want to do is merge all these files into one, and keep only the latest rating.
file1 may have…
quickbooks
- 73
0
votes
3 answers
Filtering 2nd field from a data set and then using uniq on the output
I've got a dataset that goes like this:
AAAAA 11111 Data1
AAAAA 11111 Data2
AAAAA 11111 Data3
AAAAA 11112 Data4
AAAAA 11112 Data5
AAAAA 11112 Data6
AAAAA 11112 Data7
AAAAA 11113 Data8
AAAAA 11114 Data9
And so on. I want to filter according to the…
Fyyz
- 13
0
votes
1 answer
