Re: Inserting into BLOB, Memory consumption, and other things. - Mark Matthews - com.mysql.lists.java

7 messages in com.mysql.lists.javaRe: Inserting into BLOB, Memory consu...

From	Sent On	Attachments
Guillaume Vangelder	13 Mar 2000 02:24
Mark Matthews	13 Mar 2000 03:44
Wolfgang Fiedler	13 Mar 2000 08:13
Tim Endres	13 Mar 2000 09:22
Dror Matalon	13 Mar 2000 09:43
Guillaume Vangelder	13 Mar 2000 10:17
Tim Endres	13 Mar 2000 15:01

Subject:	Re: Inserting into BLOB, Memory consumption, and other things.
From:	Mark Matthews (mmat...@worldserver.com)
Date:	03/13/2000 03:44:25 AM
List:	com.mysql.lists.java

There really isn't a way around this. The MySQL protocol sends queries in one packet, and the packet length is in the header. Therefore, the JDBC drivers have to take your BLOB, do all of the escaping, figure out how long the escaped BLOB is, and then send it in one shot to the server. This means that they need to allocate a lot of memory to do this. Ten-times sounds like a lot, but it's hard to know what the driver is doing unless you give us specifics on how you're inserting the BLOB.

You should also ask yourself if it makes sense to store large BLOBs in a database. In many cases it makes a lot more sense to store them in the filesystem, and store pointers to the files in the database.

-Mark (author of MM.MySQL)

On Mon, 13 Mar 2000, Guillaume Vangelder wrote:

Hello everybody,

I have browsed the archives of this list to find an answer to my question, without success.

First here's my setup: MySQL 3.22.25 on Linux 2.2.something & on Solaris 2.7 mm.mysql 2.0pre5

I'm trying to insert large files into the database, but I was getting errors like 'Packet size too large'.

So I started mysql with a maximum packet size of 1024M. That did the trick for small files ( < 5M ).

The problem is that I will have to put files up to several hundreds megs. A test with a 10M file resulted in a Broken Pipe, and before any data was sent on the network, my process size was > 150M (it actually depends on the JVM - tried with 1.2, 1.2 and 1.3 - but it was always > 10 times the size of my file).

What to do?

Thanks in advance,

/Guillaume Vangelder

____________________________________________________ Guillaume Vangelder, Technical Director. BUYONET INTERNATIONAL GROUP - The Global eAlliance Powerhouse & Electronic Software Store -

Phone: +46 (0)31 778 79 00 Fax : +46 (0)31 778 12 30 gui...@buyonet.com

www.buyonet.com

____________________________________________________

To unsubscribe, send a message to the address shown in the List-Unsubscribe header of this message. If you cannot see it, e-mail java...@lists.mysql.com instead.