5 messages in com.mysql.lists.javaRe: Handling huge tables
FromSent OnAttachments
Michael Thaler29 Aug 2001 05:13 
Mark Matthews29 Aug 2001 07:16 
Michael Thaler29 Aug 2001 08:42 
Bart Locanthi29 Aug 2001 08:58 
Mark Matthews29 Aug 2001 14:48 
Subject:Re: Handling huge tables
From:Bart Locanthi (ba@sabl.com)
Date:08/29/2001 08:58:59 AM
List:com.mysql.lists.java

as has been pointed out before on this list, the mysql way is to use the SELECT to create a new temporary table with an auto_increment primary key (the table goes away when your connection does). it's a pretty simple idiom - you only have to specify the index column.

you can now index into this table in constant time, unlike what happens when you use LIMIT, and you aren't locking the original table(s) for the duration of your use of the result set. so you can chunk the rows out, say, 100 at a time, with essentially zero overhead and minimal impact on other processes.

don't be fooled by smoke along the lines of "well, oracle doesn't lock the tables" or "oracle lets you have long-lived result sets". you pay for these things (after you've already paid for larry's jet) by the infamous "rollback segments" that grow without bound (until you hit your resource limit) while the db is waiting for you to finish with your result set. conservation of difficulty.

Michael Thaler wrote:

With MM.MySQL, it does contain the whole result set in memory, because of some limitations with how the MySQL protocol works, and how JDBC is supposed to work, especially with application servers. You might want to look into the LIMIT clauses that MySQL supports to allow your users to "page" through a result set.

is it possible to adjust the size of the ResultSet with statement.setFetchSize or is this just ignored? As far as I understand that this should limit the number of columns that are read and kept in memory when a querry is executed. Having the whole ResultSet in memory just does not work for large tables and to work with LIMIT makes things nasty and complicated.

Sincerly, Michael

To unsubscribe, send a message to the address shown in the List-Unsubscribe header of this message. If you cannot see it, e-mail java@lists.mysql.com instead.