 | Start a set with this search |
 | Include this search in one of my sets |
 | Exclude this search from one of my sets |
 | Permalink to these results Paste this link in email or IM: |
 | Atom feed for tracking future search results Paste this URL into your reader: |
9 messages in com.googlegroups.google-appengine[google-appengine] Re: felxible shard...| From | Sent On | Attachments |
|---|---|---|
| conman | Aug 13, 2008 4:11 am | |
| Calvin Spealman | Aug 13, 2008 3:07 pm | |
| Bill | Aug 13, 2008 7:03 pm | |
| conman | Aug 14, 2008 12:16 am | |
| conman | Aug 14, 2008 12:39 am | |
| Bill | Aug 14, 2008 10:38 am | |
| Bill | Aug 14, 2008 11:39 am | |
| Bryan A. Pendleton | Aug 14, 2008 12:15 pm | |
| conman | Aug 14, 2008 11:58 pm |
| Permalink for this message Paste this link in email or IM: |
| Permalink for this thread Paste this link in email or IM: |
| Atom feed for this thread Paste this URL into your reader: |
| Subject: | [google-appengine] Re: felxible sharded counter | Actions... |
|---|---|---|
| From: | Bill (bill...@gmail.com) | |
| Date: | Aug 14, 2008 11:39:27 am | |
| List: | com.googlegroups.google-appengine | |
I should have mentioned why num_shards doesn't matter to getting accurate results from the Counter. The get_count() finds all shards regardless of the value of num_shards because it's querying on the name property. So it'd be nice to get a large enough num_shards to spread the increment transactions around, but it's not necessary so overwrites on num_shard puts() is OK. -Bill
On Aug 14, 10:38 am, Bill <bill...@gmail.com> wrote:
Good catch on the put(), Constantin. I'm not worried about contention when increasing the shard count so I wouldn't use a transaction when doing the shard number increase. If two threads try to put() self.num_shards and one gets overridden, that's OK. If both succeed, that's OK as well but a little less optimal.
While the scale up in shards makes sense in theory, have you run any tests on AppEngine to see when you run into contention and what appropriate shard numbers might be?
Best, Bill
On Aug 14, 12:40 am, conman <cons...@googlemail.com> wrote:
@Bill, Thanks for sharing your pastie! I looked at it and I might have found a problem: In Line 75 you increment your shards count - but you don't save the Entity. So this is incfremented only for the current thread - all other threads running on the distributed system woun't know that the shard count is already increased and you will get failed transaction due to contentions after all.
But if you save it - you will run into the same issue I was talking about in the first place: A lot threads might be trying to increase and save the shard count - which might again lead to a failed put(). This case is handled by the outer try/except block which ignores it.
Also I changed the code for increasing the number of shards to: num_shards = int(num_shards * 0.1 + 1) Which increases by 10%. The reason is that if you have already 300 shards running and despite that get failed transactions - adding only a static number like 5 might be not enough.
Regards, Constantin
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to goog...@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsu...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---