6 messages in com.mysql.lists.clusterRE: stuck in phase 3, bug ?
FromSent OnAttachments
Yong Lee30 Jan 2008 14:34 
Yong Lee30 Jan 2008 15:01 
Yong Lee30 Jan 2008 15:40 
Yong Lee30 Jan 2008 23:09 
list-mysql (www.tiri.li)31 Jan 2008 13:44 
Yong Lee31 Jan 2008 13:45 
Subject:RE: stuck in phase 3, bug ?
From:Yong Lee (yl@eqo.com)
Date:01/30/2008 03:01:08 PM
List:com.mysql.lists.cluster

Also note that it looks like there hasn't been a local checkpoint in a few hours before the problem started occurring

Yong Lee Developer yl@EQO.com

direct: +1.604.273.8173 x113 mobile: +1.604.418.4470 fax: +1.604.273.8172 web: www.EQO.com EQO ID: yonglee

-----Original Message----- From: Yong Lee [mailto:yl@eqo.com] Sent: January 30, 2008 2:35 PM To: clus@lists.mysql.com Subject: stuck in phase 3, bug ?

Hi all,

Our backups (done every 15 minutes) started failing with :

2008-01-30 13:47:02 [MgmSrvr] ALERT -- Node 3: Backup request from 1 failed to start. Error: 1302

One of our ndb nodes died (we have 2 ndb nodes each co-existing with a mysql node) and now we can't restart it as it is constantly stuck in phase 3.

I see the following in the error log for the ndbd that will not restart:

Time: Wednesday 30 January 2008 - 11:02:53 Status: Temporary error, restart node Message: Assertion (Internal error, programming error or missing error message, please report a bug) Error: 2301 Error data: ArrayPool<T>::getPtr Error object: ../../../../../ndb/src/kernel/vm/ArrayPool.hpp line: 360 (block: BACKUP) Program: /usr/sbin/ndbd Pid: 22434 Trace: /var/lib/mysql-cluster/ndb_4_trace.log.25 Version: Version 5.0.45 ***EOM***

This is the second time this has happened in the last few days. During the first instance, we had to restart the good ndbd taking a full outage in the process (yuch).

Is this a bug that should be reported ?

Is there a way to recover without taking another full outage ?

Note we're running mysql 5.0.45

thanks, Yong.