Re: Exploding memory!! - B. Keith Murphy - com.mysql.lists.cluster

2 messages in com.mysql.lists.clusterRe: Exploding memory!!

From	Sent On	Attachments
B. Keith Murphy	20 Sep 2007 13:48
B. Keith Murphy	20 Sep 2007 21:20

Subject:	Re: Exploding memory!!
From:	B. Keith Murphy (kmur...@icontact.com)
Date:	09/20/2007 09:20:18 PM
List:	com.mysql.lists.cluster

Answering all my own items today :)

Turns out that the problem was the mysqld daemon (which, for those who are
trying to keep score at home, is on the same server as the data node). It was
trying to "take over" the server and use all the memory. Of course this caused
badness. I made a few changes to the my.cnf of the mysqld daemon and restarted
it. Percentage of memory consumed dropped from 30% to around a steady 5% for the
mysqld daemon. there is something like 300MB free RAM of the 640MB and I am a
much happier camper.

For those who run into this problem..I added some queue and buffer parameters to
the my.cnf file basically.

thanks,

Keith

----- Original Message ----- From: "B. Keith Murphy" <kmur...@icontact.com> To: "cluster" <clus...@lists.mysql.com> Sent: Thursday, September 20, 2007 4:49:21 PM (GMT-0500) America/New_York Subject: Exploding memory!!

So this is fun. I have a developement cluster set up. Two servers, each server
is running a sql node and a data node. The management node runs on another
computer.

Around noon I restarted the cluster. I am tracking cpu/mem/packets and such
through a tool called ganglia. It gives you pretty little html graphs so you can
see "trends".

I have 640MB of RAM for each server. Not much I know, but we aren't really doing
much on this. The my.cnf files are really straight-forward as is the config.ini
file (which I will show in a sec).

I have watched the total memory usage rise from around 200 MB to around 400MB in
the last 4.5 hours using ganglia and top. Assuming this trend doesn't change it
will reach the end of physical memory in about 4 more hours, start swapping and
crash. So, this really wouldn't be a good thing. I am not finding anything
significant about memory leaks or such in mysql lately so I am hoping against
hope that I have something configured screwey. Which is entirely possible as I
am new to this clustering thing :)

Oh, another tidbit:

2007-09-20 16:07:09 [MgmSrvr] INFO -- Node 3: Data usage is 15%(988 32K pages of
total 6400) 2007-09-20 16:07:09 [MgmSrvr] INFO -- Node 3: Index usage is 3%(260 8K pages of
total 7200) 2007-09-20 16:07:09 [MgmSrvr] INFO -- Node 3: Resource 0 min: 0 max: 639 curr: 0

My data and index memory usage according to ndb_mgm is 15% for data and 3% for
indexes. This has stayed consistent since startup.

My config.ini is following. You will see that I specify two replicas. In this
situation with 2 servers and two data nodes that means I have one copy of the
database on each server right? And the Datamemory and Indexmemory lines specify
that I have allocated a total of 256MB of RAM per data node? Thanks for the help
everyone!!

My config.ini is:

# Options affecting ndbd processes on all data nodes:

[NDBD DEFAULT] NoOfReplicas=2 # Number of replicas DataMemory=200M # How much memory to allocate for data storage IndexMemory=56M # How much memory to allocate for index storage

# TCP/IP options: [TCP DEFAULT] portnumber=2202 # This the default; however, you can use any # port that is free for all the hosts in the cluster # Note: It is recommended that you do not specify the # portnumber at all and allow the default value to be # used instead

# Management process options: [NDB_MGMD] hostname=xxx.xxx.xxx.117 # Hostname or IP address of MGM node datadir=/var/lib/mysql-cluster # Directory for MGM node log files

# Options for data node "A": [NDBD] # (one [NDBD] section per data node) hostname=xxx.xxx.xxx.135 # Hostname or IP address datadir=/usr/local/mysql/data # Directory for this data node's data files

# Options for data node "B": [NDBD] hostname=xxx.xxx.xxx.136 # Hostname or IP address datadir=/usr/local/mysql/data # Directory for this data node's data files

# SQL node "A" options: [MYSQLD] hostname=xxx.xxx.xxx.135 # Hostname or IP address # (additional mysqld connections can be # specified for this node for various # purposes such as running ndb_restore)

# SQL node "B" options: [MYSQLD] hostname=xxx.xxx.xxx.136 # Hostname or IP address # (additional mysqld connections can be # specified for this node for various # purposes such as running ndb_restore)