Removing consecutive duplicates words out of text using Regex and displaying the new text

Question

Hy,

I have the following code:

import java.io.*;
import java.util.ArrayList;
import java.util.Scanner;
import java.util.regex.*;

/
public  class RegexSimple4
{

     public static void main(String[] args) {   

          try {
              Scanner myfis = new Scanner(new File("D:\\myfis32.txt"));
              ArrayList <String> foundaz = new ArrayList<String>();
              ArrayList <String> noduplicates = new ArrayList<String>();

              while(myfis.hasNext()) {
                  String line = myfis.nextLine();
                  String delim = " ";
                  String [] words = line.split(delim);

                  for (String s : words) {                    
                      if (!s.isEmpty() && s != null) {
                          Pattern pi = Pattern.compile("[aA-zZ]*");
                          Matcher ma = pi.matcher(s);

                          if (ma.find()) {
                              foundaz.add(s);
                          }
                      }
                  }
              }

              if(foundaz.isEmpty()) {
                  System.out.println("No words have been found");
              }

              if(!foundaz.isEmpty()) {
                  int n = foundaz.size();
                  String plus = foundaz.get(0);
                  noduplicates.add(plus);
                  for(int i=1; i<n; i++) {   
                      if ( !noduplicates.get(i-1) .equalsIgnoreCase(foundaz.get(i))) {
                          noduplicates.add(foundaz.get(i));
                      }
                  }

                  //System.out.print("Cuvantul/cuvintele \n"+i);

              }
              if(!foundaz.isEmpty()) { 
                  System.out.print("Original text \n");
                  for(String s: foundaz) {
                      System.out.println(s);
                  }
              }
              if(!noduplicates.isEmpty()) {
                  System.out.print("Remove duplicates\n");
                  for(String s: noduplicates) {
                      System.out.println(s);
                  }
              }

          } catch(Exception ex) {
              System.out.println(ex); 
          }
      }
  }

With the purpose of removing consecutive duplicates from phrases. The code works only for a column of strings not for full length phrases.

For example my input should be:

Blah blah dog cat mice. Cat mice dog dog.

And the output

Blah dog cat mice. Cat mice dog.

Sincerly,

score 20 · Accepted Answer · answered May 18 '14 at 09:06

20

First of all, the regex [aA-zZ]* doesn't do what you think it does. It means "Match zero or more as or characters in the range between ASCII A and ASCII z (which also includes [, ], \ and others), or Zs". It therefore also matches the empty string.

Assuming that you are only looking for duplicate words that consists solely of ASCII letters, case-insensitively, keeping the first word (which means that you wouldn't want to match "it's it's" or "olé olé!"), then you can do that in a single regex operation:

String result = subject.replaceAll("(?i)\\b([a-z]+)\\b(?:\\s+\\1\\b)+", "$1");

which will change

Hello hello Hello there there past pastures

into

Hello there past pastures

Explanation:

(?i)     # Mode: case-insensitive
\b       # Match the start of a word
([a-z]+) # Match one ASCII "word", capture it in group 1
\b       # Match the end of a word
(?:      # Start of non-capturing group:
 \s+     # Match at least one whitespace character
 \1      # Match the same word as captured before (case-insensitively)
 \b      # and make sure it ends there.
)+       # Repeat that as often as possible

See it live on regex101.com.

answered May 18 '14 at 09:06

Tim Pietzcker

328,213
58
503
561

But how I use your regex in my program. I have a file as an input and I would like to display its contents without redundancy using System.out.print. Thank u:-) – SocketM May 18 '14 at 14:27
Thank u very much, but what $1 means :-) ? – SocketM May 18 '14 at 16:12
@SocketM: That's a special variable that references the contents of the first [capturing group](http://www.regular-expressions.info/brackets.html), in this case the first word (which we want to keep). – Tim Pietzcker May 18 '14 at 17:55
@TimPietzcker I have a same problem. Can you check me question for me ? [QUESTION LINK](http://stackoverflow.com/questions/43351621/check-pressed-space-then-add-diez-tag-using-jquery) . fvbuendia answered my question but his answer not enough because Remove Duplicate function doesn't work. I am searching a solution but i can't. – AlwaysStudent Apr 14 '17 at 20:52

score 1 · Answer 2 · answered Mar 29 '17 at 02:20

Bellow code work fine

import java.util.Scanner;

import java.util.regex.Matcher;

import java.util.regex.Pattern;

public class DuplicateRemoveEx {

public static void main(String[] args){

    String regex="(?i)\\b(\\w+)(\\b\\W+\\1\\b)+";
    Pattern p = Pattern.compile(regex,Pattern.CASE_INSENSITIVE);

    Scanner in = new Scanner(System.in);
    int numSentences = Integer.parseInt(in.nextLine());
    while(numSentences-- >0){
        String input = in.nextLine();
        Matcher m = p.matcher(input);
        while(m.find()){
            input=input.replaceAll(regex, "$1");
        }
        System.out.println(input);
    }
    in.close();
}

}

Thanks. Java is an odd beast. Case insensitive flag sent to compile, yet (?i) is also in regex! — JGFMK, Jan 02 '20 at 05:18

score 0 · Answer 3 · answered May 18 '14 at 16:08

Bellow it is your code. I have used lines to split text and Tim's regular expression.

import java.util.Scanner;
import java.io.*;
import java.util.regex.*;
import java.util.ArrayList;
/**
 *
 * @author Marius
 */
public class RegexSimple41 {

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
       ArrayList <String> manyLines = new ArrayList<String>();
       ArrayList <String> noRepeat = new ArrayList<String>(); 
        try
        {
            Scanner myfis = new Scanner(new File("D:\\myfis41.txt"));

            while(myfis.hasNext())
            {
                String line = myfis.nextLine();
                String delim = System.getProperty("line.separator");
                String [] lines = line.split(delim);

                for(String s: lines)
                {
                    if(!s.isEmpty()&&s!=null)
                    {
                        manyLines.add(s);
                    }
                }
            }
            if(!manyLines.isEmpty())
                    { System.out.print("Original text\n");
                        for(String s: manyLines)
                        {
                            System.out.println(s);
                }
                        }
            if(!manyLines.isEmpty())
                    { 
                        for(String s: manyLines)
                        {
                            String result = s.replaceAll("(?i)\\b([a-z]+)\\b(?:\\s+\\1\\b)+", "$1");
                            noRepeat.add(result);
                }
                        }
             if(!noRepeat.isEmpty())
                    { System.out.print("Remove duplicates\n");
                        for(String s: noRepeat)
                        {
                            System.out.println(s);
                }
                        }

        }

        catch(Exception ex)
        {
            System.out.println(ex);
        }
    }

}

Good luck,

Removing consecutive duplicates words out of text using Regex and displaying the new text

3 Answers3

Linked