In the question of "how much is enough", a few concerns present
themselves:
1) Increasing the number of shards past 1000 is going to require a re-
think of how to perform an actual count, because of the limit on
results from DataStore queries.
2) Updating the number of shards based on a failed transaction seems
risky - don't you really want to update the number of shards based on
how often transactions are failing? If the contention is transient,
increasing the number of shards will still permanently increase the
time to calculate the count. Heck, in an ideal world - you might
periodically combine/reduce high-number shards, and slowly back down
the shard count to keep the number of shards to count from staying too
large for too long. Can transient problems with Google's DataStore
cause transactions to fail that would otherwise not? These kind of
unbounded auto-adjusting algorithms have a tendency to blow up given
unexpected causes for expected error conditions.
I second the question about how much contention costs - how many
counter writes per second can be expected on a non-contended shard vs.
one with heavy contention? How much contention starts to actually
cause transactions to fail (vs just retry and eventually succeed?).
On Aug 14, 2:39 pm, Bill <bill...@gmail.com> wrote:
I should have mentioned why num_shards doesn't matter
to getting accurate results from the Counter. The get_count()
finds all shards regardless of the value of num_shards
because it's querying on the name property. So it'd be
nice to get a large enough num_shards to spread the
increment transactions around, but it's not necessary
so overwrites on num_shard puts() is OK.
-Bill
9 messages in com.googlegroups.google-appengine[google-appengine] Re: felxible shard...
