atom feed4 messages in org.apache.helix.userRe: Dynamic partition management
FromSent OnAttachments
kishore gJun 2, 2017 9:51 pm.png
Subramanian RaghunathanJun 7, 2017 2:58 pm.png
Lei XiaJun 7, 2017 10:42 pm.png
Subramanian RaghunathanJun 16, 2017 9:52 am.png
Subject:Re: Dynamic partition management
From:Lei Xia (lx@apache.org)
Date:Jun 7, 2017 10:42:33 pm
List:org.apache.helix.user
Attachments:

Hi, Subramanian

Helix actually allows you to dynamically change the number of partitions in a resource. If you are using your own customized rebalancer, i.e, your rebalance mode set in resource's IdealState is CUSTOMIZED, what you can do is to manipulate the IdealState's MapFields when adding or removing partitions.

An example is given here, say initially, your resource has 3 partitions, your IS should looks like:

{ "id":"myDB" ,"simpleFields":{ ,"NUM_PARTITIONS":"3" ,"REBALANCE_MODE":"CUSTOMIZED" ,"REPLICAS":"2" ,"STATE_MODEL_DEF_REF":"OnlineOffline" } ,"listFields":{ } ,"mapFields":{ "myDB_0":{ "node-1":"ONLINE" ,"node-2":"ONLINE" } ,"myDB_1":{ "node-1":"ONLINE" ,"node-2":"ONLINE" } ,"myDB_2":{ "node-1":"ONLINE" ,"node-2":"ONLINE" } } }

Now you would like to add 2 new partitions. You can update your IS with these new partitions, and Helix will bring all replicas to ONLINE for these two new partitions.

{ "id":"myDB" ,"simpleFields":{ * ,"NUM_PARTITIONS":"5"* ,"REBALANCE_MODE":"CUSTOMIZED" ,"REPLICAS":"2" ,"STATE_MODEL_DEF_REF":"OnlineOffline" } ,"listFields":{ } ,"mapFields":{ "myDB_0":{ "node-1":"ONLINE" ,"node-2":"ONLINE" } ,"myDB_1":{ "node-3":"ONLINE" ,"node-4":"ONLINE" } ,"myDB_2":{ "node-5":"ONLINE" ,"node-6":"ONLINE" } *,"myDB_3":{* * "node-1":"ONLINE"* * ,"node-2":"ONLINE"* * }* * ,"myDB_4":{* * "node-3":"ONLINE"* * ,"node-4":"ONLINE"* * }* } }

Removing an existing partition is a little tricky, you can not simply remove it from IS. You need to first set all replica for that partition to DROPPED state, for example, such as below.

{ "id":"myDB" ,"simpleFields":{ * ,"NUM_PARTITIONS":"5"* ,"REBALANCE_MODE":"CUSTOMIZED" ,"REPLICAS":"2" ,"STATE_MODEL_DEF_REF":"OnlineOffline" } ,"listFields":{ } ,"mapFields":{ *"myDB_0":{* * "node-1":"DROPPED"* * ,"node-2":"DROPPED"* * }* * ,"myDB_1":{* * "node-3":"DROPPED"* * ,"node-4":"DROPPED"* * }* ,"myDB_2":{ "node-5":"ONLINE" ,"node-6":"ONLINE" } ,"myDB_3":{ "node-1":"ONLINE" ,"node-2":"ONLINE" } ,"myDB_4":{ "node-3":"ONLINE" ,"node-4":"ONLINE" } }

Once you confirming the partition has been removed from resource's ExternalView, it is safe to remove it from IdealState now.

{ "id":"myDB" ,"simpleFields":{ * ,"NUM_PARTITIONS":"3"* ,"REBALANCE_MODE":"CUSTOMIZED" ,"REPLICAS":"2" ,"STATE_MODEL_DEF_REF":"OnlineOffline" } ,"listFields":{ } ,"mapFields":{ ,"myDB_2":{ "node-5":"ONLINE" ,"node-6":"ONLINE" } ,"myDB_3":{ "node-1":"ONLINE" ,"node-2":"ONLINE" } ,"myDB_4":{ "node-3":"ONLINE" ,"node-4":"ONLINE" } }

Help this helps on solving your problem.

Thanks Lei

On Wed, Jun 7, 2017 at 2:58 PM, Subramanian Raghunathan < subr@integral.com> wrote:

Hi,

Trying to follow up if there are any thoughts/suggestions over the approaches.

Trying to manage a cluster with a resource that has Dynamic number of partitions please provide me your valuable thoughts on same.

Version(s) :

Helix: 0.7.1

Zookeeper:3.3.4

-Zookeeper quorum (3 nodes)

- State Model: OnlineOffline

- Controller (leader elected from one of the cluster nodes)

- Single resources with partitions.

- Custom rebalancer

Few of the approaches that I could think of ...

Approach 1: *Buffer Partitions*

Let’s say the current number of partitions are N.

1 ) Define N+X (Where x could be the delta where it can expand maximum , expecting to be of few in number (1-10))

2 ) Use property store to identify which are all valid partition(s)

3 ) Don’t act upon invalid ones by looking up property store (or) Disable the unallocated partition(s)

When a new partition to be added -

1. Update property store of new partitions (Delta merge)

2. Enable the partition(s)

--- The balancing algorithm should be able to balance valid & extra partition(s) differently. Lots of customizations.

Approach 2: *New Resource Addition*

Let’s say the current number of partitions are N.

Create Resource 1 with N partitions initially.

When a new partition(s) to be added -

1. Drop the resource

2. Create new resource with N+X (newly added partitions )

--- Caveat that I could see is Dropping the resource would stop the entire workflow (It’s a bigger price to pay for us since we are real time).

Thanks & Regards,

Subramanian.

*From:* kishore g [mailto:g.ki@gmail.com] *Sent:* Friday, June 02, 2017 9:52 PM *To:* Subramanian Raghunathan <subr@integral.com>; us@helix.apache.org *Subject:* Re: Dynamic partition management

+helix

On Jun 2, 2017 6:50 PM, "Subramanian Raghunathan" < subr@integral.com> wrote:

Hi kishore,

Trying to manage a cluster with a resource that has Dynamic number of partitions please provide me your valuable thoughts on same.

Version(s) :

Helix: 0.7.1

Zookeeper:3.3.4

-Zookeeper quorum (3 nodes)

- State Model: OnlineOffline

- Controller (leader elected from one of the cluster nodes)

- Single resources with partitions.

- Custom rebalancer

Few of the approaches that I could think of ...

Approach 1: *Buffer Partitions*

Let’s say the current number of partitions are N.

1 ) Define N+X (Where x could be the delta where it can expand maximum , expecting to be of few in number (1-10))

2 ) Use property store to identify which are all valid partition(s)

3 ) Don’t act upon invalid ones by looking up property store (or) Disable the unallocated partition(s)

When a new partition to be added -

3. Update property store of new partitions (Delta merge)

4. Enable the partition(s)

--- The balancing algorithm should be able to balance valid & extra partition(s) differently. Lots of customizations.

Approach 2: *New Resource Addition*

Let’s say the current number of partitions are N.

Create Resource 1 with N partitions initially.

When a new partition(s) to be added -

3. Drop the resource

4. Create new resource with N+X (newly added partitions )

--- Caveat that I could see is Dropping the resource would stop the entire workflow (It’s a bigger price to pay for us since we are real time).

Thanks & Regards,

Subramanian.

Tel: +1 (650) 424 4655 <(650)%20424-4655>

Mob:+1(650) 656 6006 <(650)%20656-6006>

850 Hansen Way,

Palo Alto, CA 94304

www.integral.com

[image: Logo_signature_block] <http://www.integral.com/fxcloud_features/risk_management.html#ym>