How to sort Numeric field in Lucene 6

Question

I want to sort my search result based on a numeric field. In the following example code, I want to sort based on the 'age' field. I start from using the answers from:

[How to sort IntPont or LongPoint field in Lucene 6

But it does sort based SCORE. The age are still not ascending.

And

[Sorting search result in Lucene based on a numeric field

I changed SortField.Type.SCORE to SortField.Type.LONG in the search function. But I get:

unexpected docvalues type NONE for field 'age' (expected=NUMERIC)

Here my code:

public class TestLongPointSort {


    public static void main(String[] args) throws Exception {

        String indexPath = "/tmp/testSort";
        Analyzer standardAnalyzer = new StandardAnalyzer();
        Directory indexDir = FSDirectory.open(Paths.get(indexPath));
        IndexWriterConfig iwc = new IndexWriterConfig(standardAnalyzer);

        iwc.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);

        IndexWriter masterIndex = new IndexWriter(indexDir, iwc);

        Document doc = new Document();

        String name = "bob";
        doc.add(new TextField("name", name, Field.Store.YES));
        doc.add(new SortedDocValuesField("name", new BytesRef(name)));
        doc.add(new SortedNumericDocValuesField("age", 20L));
        doc.add(new StoredField("age", 20L));
        long ts = System.currentTimeMillis();
        doc.add(new SortedNumericDocValuesField("ts", ts));
        doc.add(new StoredField("ts", ts));
        masterIndex.addDocument(doc);
        Thread.sleep(1);

        name = "max";
        doc = new Document();
        doc.add(new TextField("name", name, Field.Store.YES));
        doc.add(new SortedDocValuesField("name", new BytesRef(name)));
        doc.add(new SortedNumericDocValuesField("age", 19L));
        doc.add(new StoredField("age", 19L));
        ts = System.currentTimeMillis();
        doc.add(new SortedNumericDocValuesField("ts", ts));
        doc.add(new StoredField("ts", ts));
        masterIndex.addDocument(doc);
        Thread.sleep(1);

        name = "jim";
        doc = new Document();
        doc.add(new TextField("name", name, Field.Store.YES));
        doc.add(new SortedDocValuesField("name", new BytesRef(name)));
        doc.add(new SortedNumericDocValuesField("age", 21L));
        doc.add(new StoredField("age", 21L));
        ts = System.currentTimeMillis();
        doc.add(new SortedNumericDocValuesField("ts", ts));
        doc.add(new StoredField("ts", ts));
        masterIndex.addDocument(doc);

        masterIndex.commit();
        masterIndex.close();

        IndexReader reader = DirectoryReader.open(FSDirectory.open(Paths.get(indexPath)));
        IndexSearcher searcher = new IndexSearcher(reader);

        Analyzer analyzer = new KeywordAnalyzer();
        QueryParser queryParser = new QueryParser("message", analyzer);

        Sort sort;
        TopDocs docs;
        sort = new Sort(new SortField("name", SortField.Type.STRING));
        docs = searcher.search(new MatchAllDocsQuery(), 100, sort);
        System.out.println("Sorted by name");
        for (ScoreDoc scoreDoc : docs.scoreDocs) {
            Document doc2 = searcher.doc(scoreDoc.doc);
            System.out.println("Name:" + doc2.get("name") + " ; age:" + doc2.get("age") + " ; ts:" + doc2.get("ts"));
        }

        //docs = searcher.search(new MatchAllDocsQuery(), 100, new Sort(new SortField("age", SortField.Type.SCORE, true)));
        docs = searcher.search(new MatchAllDocsQuery(), 100, new Sort(new SortField("age", SortField.Type.LONG, true)));
        System.out.println("Sorted by age");
        for (ScoreDoc scoreDoc : docs.scoreDocs) {

            Document doc2 = searcher.doc(scoreDoc.doc);
            System.out.println("Name:" + doc2.get("name") + " ; age:" + doc2.get("age") + " ; ts:" + doc2.get("ts"));
        }

        reader.close();

    }
}

As we can see, sorting STRING is good but I didn't figure out how I can get my numbers (LONG) sorted.

What is the right way to sort Numeric fields?

Thanks

score 8 · Accepted Answer · answered Feb 16 '17 at 23:39

8

To sort search results using a SortedNumericDocValuesField, you'll need to use a SortedNumericSortField:

Sort sort = new Sort(new SortedNumericSortField("age", SortField.Type.LONG, true));
TopDocs docs = searcher.search(new MatchAllDocsQuery(), 100, sort);

answered Feb 16 '17 at 23:39

femtoRgon

32,893
7
60
87

1

Actually, I have also try this one. Still have an error. Just different: unexpected docvalues type NONE for field 'age' (expected one of [SORTED_NUMERIC, NUMERIC]) – moonbeam Feb 17 '17 at 00:50
@moonbeam - Reasonably sure you are mistaken, being that I ran *your code* with this modification, and it worked just fine. – femtoRgon Feb 17 '17 at 00:52
I did cut paste your code. What version of Lucene are you using? I'm using 6.4.1 – moonbeam Feb 17 '17 at 00:55
@moonbeam - The error you've indicated I would expect to see if you had attempted to sort the "name" field, instead of the "age" field as intended, with a `SortedNumericDocValuesField`. Could that have been the problem? – femtoRgon Feb 17 '17 at 01:07
On line TopDocs docs = searcher.search(new MatchAllDocsQuery(), 100, sort);. Got: Exception in thread "main" java.lang.IllegalStateException: unexpected docvalues type NONE for field 'age' (expected one of [SORTED_NUMERIC, NUMERIC]). Re-index with correct docvalues type. – moonbeam Feb 17 '17 at 01:09
@ femtoRgon - What version of Lucene are you using? I would like to test with the same one...thanks – moonbeam Feb 17 '17 at 02:35
@moonbeam - Just checked to be sure, and it does test out on 6.4.1, so it should work. Best guess is: I notice that you've configured for `CREATE_OR_APPEND`. Is it possible you have documents from previous runs in your index in which a different field type was used? Might be worth trying with `CREATE`, or just changing to a fresh direectory (or use `RAMDirectory`), and see if you get different results. – femtoRgon Feb 17 '17 at 05:14

score -1 · Answer 2 · edited May 23 '17 at 12:25

-1

I would suggest you use ArrayList to store data from Document rather than saving it to another document, then use sort methods of ArrayList.

Please visit these links for your reference.

SO - how to sort arraylist

JAVA ArrayList sort method sample

edited May 23 '17 at 12:25

Community

1
1

answered Feb 16 '17 at 18:29

iamLinker

88
1
10

2

Yes, it could be done like that but it means copying the whole search result into memory and sort it externally from Lucene The idea is to use the Lucene documented sort capacity. This question is to figure out how to use this Lucene capability to sort numeric fields as I’m able to sort String fields – moonbeam Feb 16 '17 at 18:53

How to sort Numeric field in Lucene 6

2 Answers2

Linked