atom feed45 messages in org.apache.lucene.solr-devRe: modularization discussion
FromSent OnAttachments
Robert MuirApr 26, 2011 8:07 pm 
Yonik SeeleyApr 26, 2011 8:34 pm 
Grant IngersollApr 26, 2011 8:41 pm 
Chris MaleApr 26, 2011 9:12 pm 
Robert MuirApr 26, 2011 9:14 pm 
Michael McCandlessApr 27, 2011 3:28 am 
Michael McCandlessApr 27, 2011 3:36 am 
Mark MillerApr 27, 2011 5:13 am 
Robert MuirApr 27, 2011 5:34 am 
Steven A RoweApr 27, 2011 6:19 am 
Yonik SeeleyApr 27, 2011 6:25 am 
Steven A RoweApr 27, 2011 6:48 am 
Michael McCandlessApr 27, 2011 8:49 am 
Grant IngersollApr 27, 2011 5:50 pm 
Yonik SeeleyApr 27, 2011 6:23 pm 
Greg SteinApr 27, 2011 8:45 pm 
Grant IngersollMay 2, 2011 3:11 pm 
Ryan McKinleyMay 2, 2011 4:31 pm 
Mark MillerMay 2, 2011 7:28 pm 
Michael McCandlessMay 3, 2011 9:49 am 
Michael McCandlessMay 3, 2011 9:55 am 
Mark MillerMay 3, 2011 10:11 am 
Shai EreraMay 3, 2011 10:29 am 
Ryan McKinleyMay 3, 2011 10:31 am 
Mark MillerMay 3, 2011 10:46 am 
Michael McCandlessMay 4, 2011 5:25 am 
Mark MillerMay 4, 2011 6:11 am 
Robert MuirMay 4, 2011 6:30 am 
Uwe SchindlerMay 4, 2011 6:42 am 
Mark MillerMay 4, 2011 6:49 am 
Simon WillnauerMay 4, 2011 7:01 am 
Simon WillnauerMay 5, 2011 1:15 am 
Grant IngersollMay 5, 2011 7:25 am 
Mark MillerMay 5, 2011 7:41 am 
Simon WillnauerMay 5, 2011 8:03 am 
Grant IngersollMay 5, 2011 8:35 am 
Jason RutherglenMay 5, 2011 9:59 am 
Chris HostetterMay 6, 2011 1:35 pm 
Mark MillerMay 6, 2011 1:45 pm 
Michael McCandlessMay 7, 2011 3:30 am 
Simon WillnauerMay 7, 2011 3:34 am 
Michael McCandlessMay 7, 2011 3:46 am 
Michael McCandlessMay 7, 2011 4:02 am 
Simon WillnauerMay 7, 2011 4:10 am 
Grant IngersollMay 7, 2011 5:20 am 
Subject:Re: modularization discussion
From:Chris Hostetter (
Date:May 6, 2011 1:35:48 pm

: To me, the third camp is just saying the proof is in the pudding. If : you want to refactor, then go for it. Just make sure everything still : works, which of course I know people will (but part of that means : actually running Solr, IMO). Perhaps, more importantly don't get mad : that if I have only one day a week to work on Lucene/Solr that I spend : it putting a specific feature in a specific place. Just because : something can/should be modularized, doesn't mean that a person working : in that area must do it before they add whatever they were working on. : For instance, if and when function queries are a module, I will add to : them there and be happy to do so. In the meantime, I will likely add to : them in Solr if that is something I happen to be interested in at that : time b/c I can certainly add a new function in a day, but I can't : refactor the whole module _and_ add my new function in a day.


I want to get that printed on a t-shirt

the corrolarry issue in my mind...

I am happily in favor of code reuse and modularization in the abstract, and when it works in practice i'm plesantly delighted.

But when people talk about modularization as a goal, and make a laundry list things in solr that people think should be refactored into modules (w/o showing specifics of what that module would look like) then i have a hard time buying into some of these ideas panning out in a way that: a) is a useful module to people in and of itself b) doesn't hamstring the evolution/performance in solr.

To look at "faceting" as a concrete example, there are big the reasons faceting works so well in Solr: Solr has total control over the index, knows exactly when the index has changed to rebuild caches, has a strict schema so it can make sense of field types and pick faceting algos accordingly, has multi-phase distributed search approach to get exact counts efficiently across multiple shards, etc... (and there are still a lot of additional enhancements and improvements that can be made to take even more advantage of knowledge solr has because it "owns" the index that we no one has had time to tackle)

I find it really hard to picture a way that this code could be refactored into a reusable module in such a way that it could have an API that would be easily usable outside of Solr -- and when i do get a glimmer of an inkling of what that might look like, that vision scares me because of how that API might then "hobble" Solr's ability to leverage it's total control of the underlying index to add additional performance/features.

To be crystal clear: I recognize that this is *my* hangup -- I am not suggesting that "I am short sighted and have little imagination therefore this code should never be modularized."

I'm trying to explain why i *personally* am hesitant and sceptical of how well modularizations of features like like this might actually work in practice, and why i'm not eager to jump in and contribute on a goal whose end result is something that i can't fully picture (and when i can picture it, i'm a little scared by what i see)

That doesn't mean i'm opposed to it happening -- i would love to live in the land of candy where houses are made of ginger bread and sugar plums grow on trees, I'm just too skeptical that such a land exists (or is as great as legend describes) to go slogging along on an epic journey to try and reach it -- i'm too old for that shit.

I'm certainly not going to stop anyone else fro going on that quest -- but i am entitled to voice my skepticism and concerns, just as adventursome folks are entitled to ignore me.