Re: Memory Usage of NDBCLUSTER tables comparing InnoDB tables - Mikael Ronström - com.mysql.lists.cluster

21 messages in com.mysql.lists.clusterRe: Memory Usage of NDBCLUSTER tables...

From	Sent On	Attachments
Kenji HIROHAMA	27 Oct 2005 00:23
Adam Dixon	27 Oct 2005 00:26
Kenji HIROHAMA	27 Oct 2005 00:50
Adam Dixon	27 Oct 2005 01:02
Stewart Smith	27 Oct 2005 01:21
Kenji HIROHAMA	27 Oct 2005 02:38
Stewart Smith	27 Oct 2005 04:27
Jonas Oreland	27 Oct 2005 08:07
Stewart Smith	27 Oct 2005 19:19
Kenji HIROHAMA	27 Oct 2005 19:47
Kenji HIROHAMA	27 Oct 2005 19:55
Kenji HIROHAMA	10 Nov 2005 04:44
Adam Dixon	10 Nov 2005 14:30
Kenji HIROHAMA	10 Nov 2005 22:40
Kenji HIROHAMA	11 Nov 2005 00:21
Mikael Ronström	13 Nov 2005 23:49
Kenji HIROHAMA	14 Nov 2005 00:31
Mikael Ronström	14 Nov 2005 01:15
Stewart Smith	14 Nov 2005 15:12
Cor...@ SkyVantage	16 Nov 2005 22:50
Stewart Smith	16 Nov 2005 23:29

Subject:	Re: Memory Usage of NDBCLUSTER tables comparing InnoDB tables
From:	Mikael Ronström (mik...@mysql.com)
Date:	11/14/2005 01:15:33 AM
List:	com.mysql.lists.cluster

Hi,

2005-11-14 kl. 09.31 skrev Kenji HIROHAMA:

Hi,

Thanks for you reply. Yes, if it's about only 150MB, the figure from "all dump 1000" is a bit stgange.

BTW, I noticed something when seeing the output of ndb_size.pl; http://www.hirohama.biz/wp/wp-content/dbt1.html

All decimal(x, 0) calculate as 0. However, for instance, MyISAM/InnoDB consumes (x+2) bytes, and NDBCLUSTER must consume anything.

Sounds like a bug very much, I don't know what it should be, but 0 it shouldn't be. I'm not sure if decimals are stored as strings or as numbers in NDBCLUSTER.

Rgrds Mikael

Regards, Kenji

On 11/14/05, Mikael Ronström <mik...@mysql.com> wrote:

Hi,

2005-11-11 kl. 07.40 skrev Kenji HIROHAMA:

Hi,

Here is the output of "all dump 1000" command.

--- 2005-11-11 15:07:07 [MgmSrvr] INFO -- Node 2: Data usage is 95%(229020 32K pages of total 240000) 2005-11-11 15:07:07 [MgmSrvr] INFO -- Node 2: Index usage is 92%(35609 8K pages of total 38432)

---

240000 * 32k = 7,680,000k = about 7.5GB 38432 * 8k = 307,456k = about 300MB

The output seems to be reasonable with the config.ini file. Then, it shows the total memory usage is;

(229020 * 32k + 35609 * 8k ) =7,613,512k = about 7.5GB

Still this figure is far from the estimated size by ndb_size.pl. Of course, the extra space must be needed for "redo log" and "undo log" bofore flushing them to disk.

By default setting, how much memory is needed for not data store purpose?

Around 150 MByte I think, haven't checked lately but it's around this value (might include default data storage values as well).

Anyways this doesn't explain the deviation from the script.

Rgrds Mikael

FYI, I uploaded the output of ndb_size.pl; http://www.hirohama.biz/wp/wp-content/dbt1.html

On 11/11/05, Adam Dixon <adam...@gmail.com> wrote:

The processes are consuming this amount of memory becuase you have configured them to do so. The DataMemory and IndexMemory config params are per datanode process running.

Run 'all dump 1000' and check your cluster log to see actual current memory usage. Or I believe in top one memory usage value will be whats allocated to it, and the next will be how much of it is actually used. 5928m 2.1g 2716 S 0.3 35.8 55:49.18 ndbd Like so.

You probably just need to tune your values.

On 11/10/05, Kenji HIROHAMA <kenj...@gmail.com> wrote:

Hi,

I just used ndb_size.pl right now, and it suggests me the total size of memory is about 4GB.

However, ndbd process seems to use 6.7GB according to top command.

My config.ini file is the following;

----- [NDBD DEFAULT] NoOfReplicas= 1 DataDir=/usr/local/mysql/ndb DataMemory=7500MB IndexMemory=300MB RedoBuffer=10MB NoOfFragmentLogFiles=300

[NDB_MGMD] Hostname=c1 DataDir=/usr/local/mysql/ndb

[NDBD] Hostname=a1

[MYSQLD] [MYSQLD]

----

6.7GB - 4 GB = 3.7GB

I know ndb_size.pl is not perfect now, but for what kind of purpose 3.7GB is used?

Regards, Kenji

On 10/27/05, Adam Dixon <adam...@gmail.com> wrote:

Use ndb_size.pl written by Stewart Smith, his email has good instructions on how to use it. It will tell you exactly what you want. http://lists.mysql.com/cluster/2485

Adam

On 10/27/05, Kenji HIROHAMA <kenj...@gmail.com> wrote:

Hi,

I know I needto calculate carefully, but in general, is the size of NDBCLUSTER type table consumption space almost same as InnoDB tyep table usage of HDD?

Let's say, the InnoDB type tables consume 4GB disk space for their data and index. If we alter those tables to NDBCLUSTER, will they stores in 6GB Memory? Or it need double space, 8GB Memory?

If someone has any sort of experience, please let me know.

Thanks,

-- kenj...@gmail.com Kenji Hirohama

-- MySQL Cluster Mailing List For list archives: http://lists.mysql.com/cluster To unsubscribe:

http://lists.mysql.com/cluster?unsub=adam...@gmail.com

Mikael Ronstrom, Senior Software Architect MySQL AB, www.mysql.com

Jumpstart your cluster: http://www.mysql.com/consulting/packaged/cluster.html My blog: http://mikaelronstrom.blogspot.com

Mikael Ronstrom, Senior Software Architect MySQL AB, www.mysql.com

Jumpstart your cluster: http://www.mysql.com/consulting/packaged/cluster.html My blog: http://mikaelronstrom.blogspot.com