Hi guys, just a quick one, Cluster which overnight had some issues, 2 nodes failed ~6hours apart. Different reasons. However only one of them came back cleanly. The remaining failed node will not re-join. It appears the cluster still has its nodeid allocated, and I cannot free it.

The oddity about this node is that when the cluster had it set as failed, the ndbd processes were still running, and even a kill -9 would not clear them. So something had hung pretty solidly. In an attempt to get it started before business hours, the server was rebooted but still remains offline.

I have restarted the management node, to no effect.

ndb_mgm> 12 status Node 12: not connected ndb_mgm> PURGE STALE SESSIONS No sessions purged

# ndbd --initial error=2350 2007-02-12 08:31:53 [ndbd] INFO -- Error handler restarting system 2007-02-12 08:31:53 [ndbd] INFO -- Error handler shutdown completed - exiting sphase=0 exit=-1

# perror --ndb 2350 NDB error code 2350: Invalid configuration received from Management Server: Permanent error, external action needed: Configuration error

The configuration error it would be complaining about would be that there is no nodeid available for the connecting node.

/var/log/ndb_mgmd.log 2007-02-12 09:37:18 [MgmSrvr] INFO -- Mgmt server state: node id's 2 3 11 12 13 14 15 16 17 18 connected but not reserved

Just wondering if there is a way to avoid a full cluster shutdown and startup to fix this problem. One thing I thought to try is to restart the Master ndbd, which may or may not hold the coins to which nodeids are allocated? (which also happens to be in a different node group) as it appears that the cluster still has nodeid 12 allocated.

Any thoughts?

Kind regards, Adam