Yonik Seeley wrote:

True.

Actually, an UnflushedDocumentsIndexReader would not be hard!

DocumentsWriter already has an IndexInput (ByteSliceReader) that can read the postings for a single term from the RAM buffer (this is used when flushing the segment). I think it'd be straightforward to get TermEnum/TermDocs/TermPositions iterators on the buffered docs. Norms are already stored as byte arrays in memory. FieldInfos is already available. The stored fields & term vectors are already flushed to the directory so they could be read normally.

Hmm, buffered delete terms are tricky. I guess freezeDocIDs would have to flush deleted terms (and queries, if we add that) before making a reader accessible,

though, the cost is shared because the readers need to be opened anyway (so the app can find docIDs).

So maybe this approach becomes this:

// Returns a "point in time" frozen view of index... IndexReader reader = writer.getReader(); try { <get docIDs from reader, delete by docID> } finally { writer.releaseReader(); }

?

We may even be able to implement this w/o actually freezing the writer, ie, still allowing add/updateDocument calls to proceed. Merging could certainly still proceed. This way you could at any time ask a writer for a "point in time" reader, independent of what else you are doing with the writer. This would require, on flushing, that writer goes and swaps in a "real" segment reader, limited to a specified docID, for any point in time readers that are open.

This would certainly be less work :) I guess the question is how severely are we limiting the application by requiring that you can only do deletes when IW decides to flush, or, by forcing the application to flush when it wants to do deletes.

True, though emulating the easier API on top of the "you get to delete only when IW flushes" means you are forcing a flush, right?