Re: Help with large database import - Pradeep Chandru - com.mysql.lists.cluster

9 messages in com.mysql.lists.clusterRe: Help with large database import

From	Sent On	Attachments
Anders D. Hansen	01 Mar 2007 07:35
Anders D.Hansen	01 Mar 2007 12:50
Anders D. Hansen	01 Mar 2007 13:38
Pradeep Chandru	15 Oct 2007 00:56
Matthew Montgomery	15 Oct 2007 11:24
Pradeep Chandru	16 Oct 2007 22:56
Matthew Montgomery	19 Oct 2007 10:36
Matthew Montgomery	20 Oct 2007 13:40
Matthew Montgomery	22 Oct 2007 22:34

Subject:	Re: Help with large database import
From:	Pradeep Chandru (chan...@gmail.com)
Date:	10/16/2007 10:56:31 PM
List:	com.mysql.lists.cluster

Hi Matthew,

I have configured few of the Cluster variables. I am listing down some of them:

datadir=/vmware/data/ # Directory for this data node's data files MaxNoOfConcurrentOperations=159360 MaxNoOfConcurrentTransactions=159360 MaxNoOfConcurrentIndexOperations=159360 TimeBetweenLocalCheckpoints=9 NoOfFragmentLogFiles=300 DataMemory=2000M IndexMemory=800M RedoBuffer=32M

now i have a different issue.

I have 13GB of disk partition. The DB size is around 2GB in MyISAM. After importing several times the DB has grown in CLUSTER and occupied up to 13 GB. The disk is full. What are the log files i can clear. I have given NoOfFragmentLogFiles=300 is that the concern here?

Thanks for the response.

regards, Pradeep

On 10/15/07, Matthew Montgomery <mmon...@mysql.com> wrote:

Hello Pradeep,

On Mon, 2007-10-15 at 13:26 +0530, Pradeep Chandru wrote:

Hi, I have the similar kind of issue in MySQL 5.0.45.

ERROR 1114 (HY000) at line 1702: The table 'qp_weightage_1' is full

I am trying the installation of mysql cluster for the first time. I am trying to import a db of size 900MB in to the cluster. The server has got 8GB RAM. I have not configured any of the parameters like data memory, index memory... etc. I have the following doubts:

1. How can i load a huge table. What are the variables to be tuned? what are the parameters to be considered for the import to be successful?

You *must* first define your DataMemory and IndexMemory. The default values for DataMemory is 80M and 18M for IndexMemory.

Optionally you should make sure you have sufficient MaxNoOfTables, MaxNoOfAttributes, MaxNoOfOrderedIndexes and MaxNoOfUniqueHashIndexes

You should also know that tables in cluster can require far more room to store the tables than in other engines. This is especially true in 5.0 because VARCHAR columns are stored in memory with a fixed length. So a VARCHAR(255) column will require a minimum of 255 bytes + 16 bytes overhead of DataMemory per row. VARCHAR columns can require as much as three times this amount for UTF-8 characters.

2. Is there any link where i can find more data on tuning the variables.

You can use the ndb_size.pl script to determine the recommended values for IndexMemory and DataMemory.

http://dev.mysql.com/doc/refman/5.0/en/mysql-cluster-config-params-ndbd.html