Hi everyone,

My Solr JVM runs out of heap space quite frequently. I'm trying to understand Solr/Lucene's memory usage so I can address the problem correctly. Otherwise, I feel I'm taking random shots in the dark.

I've tried previous troubleshooting suggestions. Here's what I've done:

1) Increased Tomcat's JVM heap space, e.g.: JAVA_OPTS='-Xmx1244m -Xms1244m -server'; # frequent heap space problems JAVA_OPTS='-XX:+AggressiveHeap -server'; # runs out of heap space at 2.0g JAVA_OPTS='-Xmx3072m -Xms3072m -server'; # jvm quickly hits 2.9g on 'top'

Solr is the only webapp deployed on this Tomcat instance.

2) I use Solr collection/distribution to separate indexing and searching. The indexer is stable now and memory problems only occur when searching on the Solr slave.

3) In solrconfig.xml, I reduced mergeFactor and maxBufferedDocs by 50%: <mergeFactor>5</mergeFactor> <maxBufferedDocs>500</maxBufferedDocs>

This helped the indexing server but not the Solr slave.

4) In solrconfig.xml, I set filterCache, queryResultCache, and documentCache to 0.

Now for my index details: - To facilitate highlighting, I currently store doc contents in the index, so the index consumes 24GB on disk. - numDocs : 4,953,736 maxDoc : 4,953,736 (just optimized) - Term files: logs # du -ksh ../solr/data/index/*.t?? 5.9M ../solr/data/index/_1kjb.tii 429M ../solr/data/index/_1kjb.tis - I have 22 fields and yes, they currently have norms.

Other info that may be helpful: - My Solr is from 2006-11-15. We have a few mods, including one extra fieldCache that stores ~40 bytes/doc. - Thread counts from solr/admin/threaddump.jsp: Java HotSpot(TM) 64-Bit Server VM 1.5.0_08-b03 Thread Count: current=37 deamon=34 peak=37

My machine has Gentoo Linux and 4gb RAM. 'top' indicates the JVM reaches 2.9g RAM (3472m virtual memory) after 10-20 searches and ~20 mins of use. It seems just a matter of time before more searches or a snapinstaller 'commit' will make it run out of heap space again.

I have flexibility in the changes we can make. I.e., I can omit norms for most fields, or I can stop storing the doc contents in the index. But before embarking on a new strategy, I need some assurance that the strategy will work (crazy, I know). For example, it doesn't seem that removing norms would save a great deal (I calculate saving 1 byte per norm per field on 21 fields is ~99MB).

So...how do I deduce what's taking up so much memory? Any suggestions would be very helpful to me (and hopefully to others, too).

many thanks, -Graham