7 messages in com.mysql.lists.clusterrandom updates crashes nodes
FromSent OnAttachments
jcwang22 Jan 2008 18:55 
jcwang25 Jan 2008 09:19 
jcwang25 Jan 2008 15:14 
Stewart Smith28 Jan 2008 21:51 
Stewart Smith28 Jan 2008 21:52 
jcwang29 Jan 2008 17:23 
Norbert Tretkowski29 Jan 2008 23:20 
Subject:random updates crashes nodes
From:jcwang (jeff@apple.com)
Date:01/22/2008 06:55:09 PM
List:com.mysql.lists.cluster

Hi,

I have multiple threads running 1000 updates each by primary key. I keep getting these errors:

Got temporary error 266 'Time-out in NDB, probably caused by deadlock' from NDB key= 9319

I updated the TransactionDeadlockDetectionTimeout to 3 seconds and still get the errors. Even worse, my data nodes randomly crash and have gotten into an inconsistent states before. The data node error logs show:

Status: Temporary error, restart node Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug) Error: 2341 Error data: dblqh/DblqhMain.cpp Error object: DBLQH (Line: 6950) 0x0000000e Program: ndbd Pid: 8627 Trace: /Users/dbsp/work/cluster/data/ndb_2_trace.log.9 Version: Version 5.1.22 (rc)

The relevant portion of my config file is as follows :

# Options affecting ndbd processes on all data nodes: [ndbd default] NoOfReplicas=2 # Number of replicas DataMemory=1500M # How much memory to allocate for data storage IndexMemory=150M # How much memory to allocate for index storage StringMemory=99 #expressed as a percentage, 100%=5 MB, values > 99 interpreted as bytes NoOfFragmentLogFiles=64 #increase this number if the number of inserts/updates is large RedoBuffer=32M MaxNoOfAttributes=40000 MaxNoOfTables=1600 MaxNoOfOrderedIndexes=3000 TransactionDeadlockDetectionTimeout=3000

# TCP/IP options: [tcp default]

SendBufferMemory=2M ReceiveBufferMemory=1M Checksum=1 #detect corrupted messages

I am running a 2 node cluster on power pc with mysql cluster 5.1.22.

Please let me know if/how to fix this problem.

Thanks