periodically flushing new documents to an index using lucene

Question

I need to flush the index periodically.that's mean that the index will be regularly updated as the document being added.what do you reckon is the solution for this? I need a sample source code to be able to flush an index.

ok just like this source code below.

public class SimpleFileIndexer {
    public static void main(String[] args) throws Exception {
           File indexDir = new File("C:/Users/Raden/Documents/lucene/LuceneHibernate/adi");
           File dataDir = new File("C:/Users/Raden/Documents/lucene/LuceneHibernate/adi");
           String suffix = "txt";

           SimpleFileIndexer indexer = new SimpleFileIndexer();

           int numIndex = indexer.index(indexDir, dataDir, suffix);

           System.out.println("Total files indexed " + numIndex);
    }

    private int index(File indexDir, File dataDir, String suffix) throws Exception {
           IndexWriter indexWriter = new IndexWriter(
                           FSDirectory.open(indexDir),
                           new SimpleAnalyzer(),
                           true,
                           IndexWriter.MaxFieldLength.LIMITED);
           indexWriter.setUseCompoundFile(false);

           indexDirectory(indexWriter, dataDir, suffix);

           int numIndexed = indexWriter.maxDoc();
           indexWriter.optimize();
           indexWriter.close();

           return numIndexed;
    }

    private void indexDirectory(IndexWriter indexWriter, File dataDir, String suffix) throws IOException {
           File[] files = dataDir.listFiles();
           for (int i = 0; i < files.length; i++) {
                   File f = files[i];
                   if (f.isDirectory()) {
                           indexDirectory(indexWriter, f, suffix);
                   }
                   else {
                           indexFileWithIndexWriter(indexWriter, f, suffix);
                   }
           }
    }

    private void indexFileWithIndexWriter(IndexWriter indexWriter, File f, String suffix) throws IOException {
           if (f.isHidden() || f.isDirectory() || !f.canRead() || !f.exists()) {
                   return;
           }
           if (suffix!=null && !f.getName().endsWith(suffix)) {
                   return;
           }
           System.out.println("Indexing file " + f.getCanonicalPath());

           Document doc = new Document();
           doc.add(new Field("contents", new FileReader(f)));
           doc.add(new Field("filename", f.getCanonicalPath(), Field.Store.YES, Field.Index.ANALYZED));

           indexWriter.addDocument(doc);
    }
}

the above source code can index documents when given the directory of text files. now what I am asking is how can I made the code to run continuously? what class should I use? so that everytime there is new documents added to that directory then lucene will index those documents automatically, can you help me out on this one. I really need to know what is the best solution.

Just remove all the code you've posted here, and replace it with "Can I watch a directory and periodically index it with Lucene, without re-indexing everything?". — Matthijs Bierman, Sep 28 '10 at 08:13

score 1 · Accepted Answer · edited May 23 '17 at 11:55

1

Lucene can't do this by itself. You will need to monitor the filesystem for that.

Look at How to detect filesystem has changed in java.

edited May 23 '17 at 11:55

Community

1
1

answered Sep 28 '10 at 08:11

Matthijs Bierman

1,720
9
14

can you tell me of how to use Jnotify? I find it would be a great idea to have a class that will monitor any directories that I want, either adding,deleting and removing documents in a directory. I mean how can I include Jnotify in the code I included above. thanks – jacobian Sep 28 '10 at 08:23
If you're not a programmer, you probably shouldn't be on SO. Anyway, here are the snippets: http://jnotify.sourceforge.net/sample.html – Matthijs Bierman Sep 28 '10 at 14:54

periodically flushing new documents to an index using lucene

1 Answers1