In the documentation it comes across that as cluster size grows, higher speed interconnects like SCI should be used to increase performance.

I am trying to understand where the size of the cluster and dataset requires the use of >1 Gigabit per second speeds.

I perfectly understand that because non-ethernet interconnects commonly have lower latency, no TCP stack, and lower CPU usage on the servers, that they will see a higher TPS, without consideration of the possibly increased bandwidth.

What I am having a hard time understanding is where the walls are within NDB.

Specifically, Does anyone have examples of where NDB nodes are using more than 1 Gigabit of bandwidth just for 'standard' operation. Where/When does the lack of bandwidth become the bottleneck?

In my case I am evaluating building a cluster with 16-32 nodes. We aren't overly concerned with TPS speed -- We make aggressive use of memcached for reads, and writes are a more reasonable TPS. Our biggest limitation is the size of the dataset, and this is why we might be going up to a 32 node cluster -- just to have enough space to store everything. In our case, depending on the bandwidth vs latency needs, it might be significantly cheaper to use !0GigE, rather than SCI.

Thanks,

- Paul