Query Failed [Failed to execute main query] OutOfMemoryError: Java heap space

Question

Hi i have data in elastic 80 M of data have 52 Go and i search in text with this query

$json_doc = '{
                 "query": {
                    "query_string": {
                        "query": "*' . $sujet . '*",
                        "lenient": true
                    }
                    },
     "size": 10000000,
     "from": 0,
     "fields": [
     "association_key","text"
                    ]
                }';

when I search a word that is in the base is looking less 20K I have my data but as his exceeds 20 K I have this error

[2015-11-14 20:03:04,663][DEBUG][action.search.type       ] [Pixx] [myindex3][2], node[x9gOi0WuQviXeE0eM6MsAg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3989d22b] lastShard [true]
org.elasticsearch.search.query.QueryPhaseExecutionException: [myindex3][2]: query[filtered(_all:*startup _all:maroc*)->cache(_type:tweecoms3)],from[0],size[10000000]: Query Failed [Failed to execute main query]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:163)
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:56)
    at org.apache.lucene.search.HitQueue.<init>(HitQueue.java:64)
    at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:275)
    at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:37)
    at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:135)
    at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:133)
    at org.apache.lucene.search.TopScoreDocCollector.create(TopScoreDocCollector.java:262)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:490)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
    ... 8 more
[2015-11-14 20:03:11,764][DEBUG][action.search.type       ] [Pixx] [myindex3][0], node[x9gOi0WuQviXeE0eM6MsAg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3989d22b] lastShard [true]
org.elasticsearch.search.query.QueryPhaseExecutionException: [myindex3][0]: query[filtered(_all:*startup _all:maroc*)->cache(_type:tweecoms3)],from[0],size[10000000]: Query Failed [Failed to execute main query]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:163)
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.util.FixedBitSet.<init>(FixedBitSet.java:187)
    at org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:81)
    at org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
    at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
    at org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:144)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
    ... 8 more

how i can resolve this problem i have dedicate server contains just a Intel Xeon W3520 c 4/8 t + 2.66 GHz RAM 32 GB ECC under Debian

Alain O'Dea · Accepted Answer · 2020-02-15T14:10:00.560

UPDATE: ES_HEAP_SIZE is no longer a documented option

Elastic Search won't use all of that memory without being configured.

You need to configure and tune Elastic Search and Debian's memory settings.

You'll need to configure one of these:

/etc/elasticsearch/jvm.options: a file containing JVM options (file is at config/jvm.options (in a TAR or ZIP distro rather than a Debian package)
ES_JAVA_OPTS: an environment variable with additional JVM settings including memory and GC tuning if needed (pretty much what's in jvm.options but in a single line separated by spaces)

Here's the relevant excerpt from Elastic's latest documentation as of February 2020:

Here are examples of how to set the heap size via the jvm.options file:
-Xms2g ①
-Xmx2g ②
① Set the minimum heap size to 2g.

② Set the maximum heap size to 2g.

It is also possible to set the heap size via an environment variable. This can be done by commenting out the Xms and Xmx settings in the jvm.options file and setting these values via ES_JAVA_OPTS:
ES_JAVA_OPTS="-Xms2g -Xmx2g" ./bin/elasticsearch ①
ES_JAVA_OPTS="-Xms4000m -Xmx4000m" ./bin/elasticsearch ②
① Set the minimum and maximum heap size to 2 GB.

② Set the minimum and maximum heap size to 4000 MB.

See more here:

Running against a Dockerized Elasticsearch 2.3.2 I found that setting `ES_HEAP_SIZE` alone didn't change my instance's heap size from the default. Adding `ES_JAVA_OPTS` did indeed change it, and seemed to ignore a lower setting for `ES_HEAP_SIZE`. — eebbesen, Feb 13 '20 at 20:26
Good catch @eebbesen! I've updated as ES_HEAP_SIZE is no longer a documented option. Thank you! — Alain O'Dea, Feb 15 '20 at 14:10

score 1 · Answer 2 · edited May 23 '17 at 12:03

1

Seems like you don't have enough space for Java's heap. Since it seems like you'd have more hardware wise you may try to increase it: Increase heap size in Java

edited May 23 '17 at 12:03

Community

1
1

answered Nov 14 '15 at 19:32

su-ex

106
7

Query Failed [Failed to execute main query] OutOfMemoryError: Java heap space

2 Answers2