|Robert Muir||Apr 26, 2011 8:07 pm|
|Yonik Seeley||Apr 26, 2011 8:34 pm|
|Grant Ingersoll||Apr 26, 2011 8:41 pm|
|Chris Male||Apr 26, 2011 9:12 pm|
|Robert Muir||Apr 26, 2011 9:14 pm|
|Michael McCandless||Apr 27, 2011 3:27 am|
|Michael McCandless||Apr 27, 2011 3:36 am|
|Mark Miller||Apr 27, 2011 5:12 am|
|Robert Muir||Apr 27, 2011 5:33 am|
|Steven A Rowe||Apr 27, 2011 6:18 am|
|Yonik Seeley||Apr 27, 2011 6:24 am|
|Steven A Rowe||Apr 27, 2011 6:48 am|
|Michael McCandless||Apr 27, 2011 8:49 am|
|Grant Ingersoll||Apr 27, 2011 5:50 pm|
|Yonik Seeley||Apr 27, 2011 6:23 pm|
|Greg Stein||Apr 27, 2011 8:44 pm|
|Grant Ingersoll||May 2, 2011 3:11 pm|
|Ryan McKinley||May 2, 2011 4:31 pm|
|Mark Miller||May 2, 2011 7:27 pm|
|Michael McCandless||May 3, 2011 9:49 am|
|Michael McCandless||May 3, 2011 9:55 am|
|Mark Miller||May 3, 2011 10:11 am|
|Shai Erera||May 3, 2011 10:29 am|
|Ryan McKinley||May 3, 2011 10:31 am|
|Mark Miller||May 3, 2011 10:46 am|
|Michael McCandless||May 4, 2011 5:25 am|
|Mark Miller||May 4, 2011 6:10 am|
|Robert Muir||May 4, 2011 6:29 am|
|Uwe Schindler||May 4, 2011 6:41 am|
|Mark Miller||May 4, 2011 6:49 am|
|Simon Willnauer||May 4, 2011 7:01 am|
|Simon Willnauer||May 5, 2011 1:15 am|
|Grant Ingersoll||May 5, 2011 7:25 am|
|Mark Miller||May 5, 2011 7:40 am|
|Simon Willnauer||May 5, 2011 8:03 am|
|Grant Ingersoll||May 5, 2011 8:34 am|
|Jason Rutherglen||May 5, 2011 9:58 am|
|Chris Hostetter||May 6, 2011 1:35 pm|
|Mark Miller||May 6, 2011 1:44 pm|
|Michael McCandless||May 7, 2011 3:30 am|
|Simon Willnauer||May 7, 2011 3:34 am|
|Michael McCandless||May 7, 2011 3:45 am|
|Michael McCandless||May 7, 2011 4:01 am|
|Simon Willnauer||May 7, 2011 4:10 am|
|Grant Ingersoll||May 7, 2011 5:20 am|
|Subject:||Re: modularization discussion|
|From:||Simon Willnauer (simo...@googlemail.com)|
|Date:||May 7, 2011 4:10:15 am|
On Sat, May 7, 2011 at 1:02 PM, Michael McCandless <luc...@mikemccandless.com> wrote:
OK I opened:
On Sat, May 7, 2011 at 6:46 AM, Michael McCandless <luc...@mikemccandless.com> wrote:
I agree! And I think you're saying the same thing as Grant.
Ie, others are fully free to refactor stuff, as long as they don't hurt Solr/Lucene (functionality, performance).
But you are tempering that with a nice dose of reality (successfully factoring out faceting will be insanely hard).
I very much agree with that.
And, I (and other refactor-itchers) very much want to hear the specific technical skepticism/concerns on a given module: that assessment is awesome and very useful. In fact, I love your enumeration of how faceting is so well integrated into Solr so much that I'll go open an issue (to factor out faceting), and put your list in!
I think this will mean, in practice, that the refactoring should itself proceed in baby steps. Ie, birthing a new faceting module, iterating on it, etc., and then at some point cutting Solr over to it, are two events likely spread out substantially in time.
Freedom to refactor/poach is the bread and butter of open source.
On Fri, May 6, 2011 at 4:35 PM, Chris Hostetter <hoss...@fucit.org> wrote:
: To me, the third camp is just saying the proof is in the pudding. If : you want to refactor, then go for it. Just make sure everything still : works, which of course I know people will (but part of that means : actually running Solr, IMO). Perhaps, more importantly don't get mad : that if I have only one day a week to work on Lucene/Solr that I spend : it putting a specific feature in a specific place. Just because : something can/should be modularized, doesn't mean that a person working : in that area must do it before they add whatever they were working on. : For instance, if and when function queries are a module, I will add to : them there and be happy to do so. In the meantime, I will likely add to : them in Solr if that is something I happen to be interested in at that : time b/c I can certainly add a new function in a day, but I can't : refactor the whole module _and_ add my new function in a day.
I want to get that printed on a t-shirt
the corrolarry issue in my mind...
I am happily in favor of code reuse and modularization in the abstract, and when it works in practice i'm plesantly delighted.
But when people talk about modularization as a goal, and make a laundry list things in solr that people think should be refactored into modules (w/o showing specifics of what that module would look like) then i have a hard time buying into some of these ideas panning out in a way that: a) is a useful module to people in and of itself b) doesn't hamstring the evolution/performance in solr.
To look at "faceting" as a concrete example, there are big the reasons faceting works so well in Solr: Solr has total control over the index, knows exactly when the index has changed to rebuild caches, has a strict schema so it can make sense of field types and pick faceting algos accordingly, has multi-phase distributed search approach to get exact counts efficiently across multiple shards, etc... (and there are still a lot of additional enhancements and improvements that can be made to take even more advantage of knowledge solr has because it "owns" the index that we no one has had time to tackle)
I find it really hard to picture a way that this code could be refactored into a reusable module in such a way that it could have an API that would be easily usable outside of Solr -- and when i do get a glimmer of an inkling of what that might look like, that vision scares me because of how that API might then "hobble" Solr's ability to leverage it's total control of the underlying index to add additional performance/features.
To be crystal clear: I recognize that this is *my* hangup -- I am not suggesting that "I am short sighted and have little imagination therefore this code should never be modularized."
I'm trying to explain why i *personally* am hesitant and sceptical of how well modularizations of features like like this might actually work in practice, and why i'm not eager to jump in and contribute on a goal whose end result is something that i can't fully picture (and when i can picture it, i'm a little scared by what i see)
That doesn't mean i'm opposed to it happening -- i would love to live in the land of candy where houses are made of ginger bread and sugar plums grow on trees, I'm just too skeptical that such a land exists (or is as great as legend describes) to go slogging along on an epic journey to try and reach it -- i'm too old for that shit.
I'm certainly not going to stop anyone else fro going on that quest -- but i am entitled to voice my skepticism and concerns, just as adventursome folks are entitled to ignore me.