atom feed10 messages in org.apache.lucene.solr-userRe: Searching on mulit-core Solr
FromSent OnAttachments
vivek sarApr 3, 2009 1:01 pm 
vivek sarApr 6, 2009 12:32 pm 
Fergus McMenemieApr 6, 2009 2:40 pm 
vivek sarApr 8, 2009 1:57 am 
vivek sarApr 8, 2009 11:07 am 
vivek sarApr 9, 2009 12:00 am 
Erik HatcherApr 9, 2009 2:56 am 
Fergus McMenemieApr 9, 2009 4:06 am 
vivek sarApr 9, 2009 9:49 am 
vivek sarApr 9, 2009 10:01 am.xml
Subject:Re: Searching on mulit-core Solr
From:vivek sar (vive@gmail.com)
Date:Apr 9, 2009 10:01:38 am
List:org.apache.lucene.solr-user
Attachments:
solr.xml - 3k

Attached is the solr.xml - note, the schema and solrconfig are located in the core0 and all other cores point to the same core0 instance for schema.

Searches on individual cores work fine so I'm using the solr.xml is correct - I also get their status correctly. From the "NullPointerException" it seems it fails at,

for (int i=resultSize-1; i>=0; i--) { ShardDoc shardDoc = (ShardDoc)queue.pop(); shardDoc.positionInResponse = i; // Need the toString() for correlation with other lists that must // be strings (like keys in highlighting, explain, etc) resultIds.put(shardDoc.id.toString(), shardDoc); }

I've a unique field (required) in my documents so I'm not sure whether that can be null - could doc itself be null - how? Same search on the same cores individually works fine. Not sure if there is a way to debug this.

I'm not sure on when would I get "Connection reset" exception - would it be if indexing is happening at the same time at hight rate - would that cause problems?

Thanks, -vivek

On Thu, Apr 9, 2009 at 4:07 AM, Fergus McMenemie <fer@twig.me.uk> wrote:

Any help on this issue? Would distributed search on multi-core on same Solr instance even work? Does it has to be different Solr instances altogether (separate shards)?

As best I can tell this works fine for me. Multiple cores on the one machine. Very different schema and solrconfig.xml for each of the cores. Distributed searching using shards works fine. But I am using the trunk version.

Perhaps you should post your solr.xml file.

I'm kind of stuck at this point right now. Keep getting one of the two errors (when running distributed search - single searches work fine) as mentioned in this thread earlier.

Thanks, -vivek

On Wed, Apr 8, 2009 at 1:57 AM, vivek sar <vive@gmail.com> wrote:

Thanks Fergus. I'm still having problem with multicore search.

I tried the following with two cores (they both share the same schema and solrconfig.xml) on the same box on same solr instance,

1) http://10.4.x.x:8080/solr/core0/admin/  - works fine, shows all the cores in admin interface 2) http://10.4.x.x:8080/solr/admin/cores  - works fine, see all the cores in xml 3) http://10.4.x.x:8080/solr/20090407_2/select?q=japan - works fine, gives me top 10 records 4) http://10.4.x.x:8080/solr/20090408_3/select?q=japan - works fine, gives me top 10 records 5)
http://10.4.x.x:8080/solr/20090407_2/select?shards=10.4.x.x:8080/solr/20090407_2,10.4.x.x:8080/solr/20090408_3&indent=true&q=japan  - this FAILS. I've seen two problems with this.

   a) When index are being committed I see,

SEVERE: org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: java.net.SocketException: Connection reset        at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:282)        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)        at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)        at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)        at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)        at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)        at java.lang.Thread.run(Thread.java:637)

   b) Other times I see this,

SEVERE: java.lang.NullPointerException        at
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:432)        at
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:276)        at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:290)        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)        at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)        at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)        at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)        at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)        at java.lang.Thread.run(Thread.java:637)

Any tips on how can I search on multicore on same solr instance?

Thanks, -vivek

On Mon, Apr 6, 2009 at 2:40 PM, Fergus McMenemie <fer@twig.me.uk> wrote:

vivek,

404 from the URL you provided in the message! Similar URLs work OK for me.

hmm try http://localhost:8080/solr/admin/cores?action=status and see if that gives a 404.

Also are you running a nightly build or a svn checkout? Using tomcat? Perhaps it should be

http://localhost:8080/apache-solr-1.4-dev/admin/cores?action=status

Fergus.

Hi,

 Any help on this. I've looked at DistributedSearch on Wiki, but that doesn't seem to be working for me on multi-core and multiple Solr instances on the same box.

Scenario,

1) Two boxes (localhost, 10.4.x.x) 2) Two Solr instances on each box (8080 and 8085 ports) 3) Two cores on each instance (core0, core1)

I'm not sure how to construct my search on the above setup if I need to search across all the cores on all the boxes. Here is what I'm trying,

http://localhost:8080/solr/core0/select?shards=localhost:8080/solr/core0,localhost:8085/solr/core0,localhost:8080/solr/core1,localhost:8085/solr/core1,10.4.x.x:8080/solr/core0,10.4.x.x:8085/solr/core0,10.4.x.x:8080/solr/core1,10.4.x.x:8085/solr/core1&indent=true&q=vivek+japan

I get 404 error. Is this the right URL construction for my setup? How else can I do this?

Thanks, -vivek

On Fri, Apr 3, 2009 at 1:02 PM, vivek sar <vive@gmail.com> wrote:

Hi,

 I've a multi-core system (one core per day), so there would be around 30 cores in a month on a box running one Solr instance. We have two boxes running the Solr instance and input data is feeded to them in round-robin fashion. Each box can have up to 30 cores in a month. Here are questions,

 1) How would I search for a term in multiple cores on same box?

 Single core I'm able to search like,   http://localhost:8080/solr/20090402/select?q=*:*

2) How would I search for a term in multiple cores on both boxes at the same time?

3) Is it possible to have two Solr instances on one box with one doing the indexing and other perform only searches on that index? The idea is have two JVMs with each doing its own task - I'm not sure whether the indexer process needs to know about searcher process - like do they need to have the same solr.xml (for multicore etc). We don't want to replicate the indexes also (we got very light search traffic, but very high indexing traffic) so they need to use the same index.

Thanks, -vivek

--

=============================================================== Fergus McMenemie               Email:fer@twig.me.uk Techmore Ltd                   Phone:(UK) 07721 376021

Unix/Mac/Intranets             Analyst Programmer ===============================================================

--

=============================================================== Fergus McMenemie               Email:fer@twig.me.uk Techmore Ltd                   Phone:(UK) 07721 376021

Unix/Mac/Intranets             Analyst Programmer ===============================================================

<?xml version='1.0' encoding='UTF-8'?><solr persistent='true'> <cores adminPath='/admin/cores'> <core name='20090409_2'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090409_2/data'/> <core name='20090408_5'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090408_5/data'/> <core name='20090407_1'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090407_1/data'/> <core name='20090409_1'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090409_1/data'/> <core name='20090408_4'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090408_4/data'/> <core name='20090407_2'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090407_2/data'/> <core name='20090409_6'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090409_6/data'/> <core name='20090409_3'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090409_3/data'/> <core name='20090409_8'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090409_8/data'/> <core name='20090409_4'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090409_4/data'/> <core name='20090409_7'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090409_7/data'/> <core name='20090409_5'
instanceDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/core0/'
dataDir='/Volumes/ngs/Users/afterchat/demo/afterchat/solr/multicore/20090409_5/data'/> <core name='core0' instanceDir='core0/' dataDir='data'/> </cores> </solr>