Good catch on the put(), Constantin. I'm not worried about
contention when increasing the shard count so I wouldn't use
a transaction when doing the shard number increase.
If two threads try to put() self.num_shards and one gets
overridden, that's OK. If both succeed, that's OK as well but
a little less optimal.
While the scale up in shards makes sense in theory, have you run
any tests on AppEngine to see when you run into contention and
what appropriate shard numbers might be?
On Aug 14, 12:40 am, conman <cons...@googlemail.com>
wrote:
@Bill,
Thanks for sharing your pastie! I looked at it and I might have found
a problem:
In Line 75 you increment your shards count - but you don't save the
Entity.
So this is incfremented only for the current thread - all other
threads running on the distributed system woun't know that the shard
count is already increased and you will get failed transaction due to
contentions after all.
But if you save it - you will run into the same issue I was talking
about in the first place:
A lot threads might be trying to increase and save the shard count -
which might again lead to a failed put().
This case is handled by the outer try/except block which ignores it.
Also I changed the code for increasing the number of shards to:
num_shards = int(num_shards * 0.1 + 1)
Which increases by 10%. The reason is that if you have already 300
shards running and despite that get failed transactions - adding only
a static number like 5 might be not enough.
Regards,
Constantin
9 messages in com.googlegroups.google-appengine[google-appengine] Re: felxible shard...
