atom feed10 messages in com.googlegroups.memcachedRe: HELP ! How Memcached objects are ...
FromSent OnAttachments
Nicolas MartinezNov 14, 2015 5:49 am 
Nicolas MartinezNov 14, 2015 6:14 am 
dormandoNov 14, 2015 2:54 pm 
Nicolas MartinezNov 15, 2015 3:59 am 
dormandoNov 15, 2015 3:30 pm 
Nicolas MartinezNov 16, 2015 6:37 am 
Nicolas MartinezNov 16, 2015 10:01 am 
Nicolas MartinezNov 16, 2015 11:08 am 
dormandoNov 16, 2015 2:54 pm 
Nicolas MartinezNov 17, 2015 1:23 am 
Subject:Re: HELP ! How Memcached objects are stored in the good SLAB
From:dormando (dorm@rydia.net)
Date:Nov 15, 2015 3:30:59 pm
List:com.googlegroups.memcached

Read carefully:

item_size_ok(const size_t nkey, const int flags, const int nbytes) {

passes:

size_t ntotal = item_make_header(nkey + 1, flags, nbytes, prefix, &nsuffix);

Then conditionally:

if (settings.use_cas) { ntotal += sizeof(uint64_t); }

item_make_header is doing: nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - 2);

Then:

return sizeof(item) + nkey + *nsuffix + nbytes;

It's convoluted but shirt.

the lengths are: key + 1 + data_bytes + 2b appended to data_bytes for an extra "\r\n" + stringified rep of the flags + data length + 4 bytes for spaces and \r\n (these are carriage returns, one byte each) + 8b for CAS if enabled

CAS can be turned off via the -C starttime arg. it takes up 8 bytes.

Example: * Key : 2c * Val : 28b * Flg : 0 (1b)

turns into: * Key : 3b * Val : 30b * Hdr : 4b + 3b == 7b * Itm : 56b => 96b. which is the cap for slab 1 in a default setup.

It's tough to get it exact for small chunks due to the way the header is added. You should ballpark or tune the -f value to align with your observed data.

On Sun, 15 Nov 2015, Nicolas Martinez wrote:

Hi, Is CAS always used? If yes, we have to always add 56 bytes to the KEY and VALUE ? you don't count FLAGS characters??

I've found that  Flags's size (number of characters) impact the storage.

Example: * Key : 2 characters = 2 bytes * Value : 28 characters  = 28 bytes * FLAGS : 1 characters = 1 bytes => 31 bytes

seems to take the same storage as * Key : 1 characters = 1 bytes * Value : 28 characters  = 28 bytes * FLAGS : 2 characters = 2 bytes => 31 bytes ... wich is the limit to be stored in Slab1

ok for the /r/n ... should take 4 bytes no?

So, if we count 56 bytes for CAS : 56(cas)+31(key+value+flags)+4(/r/n)= 91

Not good... :(

where I'm wrong ??

Le samedi 14 novembre 2015 23:55:12 UTC+1, Dormando a écrit : The mysql docs don't speak for the main tree... that's their own thing.

the "sizes" binary that comes with the source tree tells you how many bytes an item will use (though I intend to add this output to the 'stats' output somewhere). With CAS this is 56 bytes.

56 + 2 + 30 == 88. Class 1 by default (in 1.4.24) is 96 bytes, but the item still ends up in class 2.

Why is this? (unfortunately?) because memcached pre-renders part of the text protocol into the item header:

*nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - 2); return sizeof(item) + nkey + *nsuffix + nbytes;

so the flags + length are getting flattened + \r\n added to the end. Together that's just enough to push it over the edge. It'd also be nice to add a highly optimized numerics printf so I could twiddle options to save a few bytes of memory in objects, but don't get your hopes up for that happening soon :)

On Sat, 14 Nov 2015, Nicolas Martinez wrote:

> Add: Memcached version : 1.4.4 (RedHat) > > Le samedi 14 novembre 2015 14:49:37 UTC+1, Nicolas Martinez a écrit : >       Hi, few days i'm reading Memcached documentation and blogs... and
i don't understand how objects are stored. > > My test > >       3 slabs :  > >  *  96.0 Bytes >  *  120.0 Bytes >  *  240.0 Bytes > Everywhere, it's told : >  *  if data is < 96 Bytes, it will be stored in Slabs1 (96B) >  *  if data > 96B and < 120B, it will be stored in Slabs2 (120B) >  *  if data > 120B, it will be stored in Slabs3 (240B) >  *  etc. > BUT, for example, when i'm creating an 30B object, it's stored in
Slab2 (120B), and NOT in Slab1 (96B). > > External sources: >       For example, the default size for the smallest block is 88 bytes
(40 bytes of value, and the default 48 bytes for the key and flag data). If the
size of the first item you store into the cache is less than 40 bytes, then a
slab with a block size of 88 bytes is created and the value stored. >       =>
https://dev.mysql.com/doc/mysql-ha-scalability/en/ha-memcached-using-memory.html > > > WRONG > >       A slab class is a collection of pages divided into same sized
chunks. Each slab class is referenced to by its chunk size. So we’ll have Slab
class 80kb, Slab class 100kb and so on. When an object needs to be stored, its
size determines where it gets stored. So if the object is larger than 80kb but
less than 100kb, it gets stored into Slab >       class 100kb.  >      
=> http://returnfoo.com/2012/02/memcached-memory-allocation-and-optimization-2/ > > > WRONG > > How i create an object: > >       data=$(pwgen 30 -c 1) >       echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server}
11211 > > > So, when 30B object is creating :  >  *  key name : 30 = 2 bytes >  *  value: 30 characters = 30 bytes >  *  tags : 0 = 1 bytes > => All = 33 bytes > If i add the default 48b as explained on Mysql website : 33 + 48 = 81B
... so < Slab1 (91B)... but always stored in Slab2 (120B) > > So, the size used to store object in the good Slab is not: >  *  object value size >  *  sum of KEY, VALUE and TAGS in bytes > KEY size : 1 character = 1 B > VALUE size : 1 character = 1 B > TAGS size : 1 character = 1 B > > ... as read everywhere > > So, It seems that: (SUM of KEY+VALUE+TAGS ) >  *  For slab1 96.0 Bytes, data stored if <= 31 B (SUM of 2+28+1 ) >  *  For slab2 120.0 Bytes, data stored if <= 55 B (SUM of 2+52+1 ) >  *  For slab3 152.0 Bytes, data stored if <= 87 B (SUM of 2+84+1 ) >  *  For slab4 192.0 Bytes, data stored if <= 126 B (SUM of 3+122+1 ) >  *  For slab5 240.0 Bytes, data stored if <= 174 B (SUM of 3+170+1 ) >  *  etc. > > My configuration : >  *  Chunk Size : 48 >  *  Chunk Growth Factore: 1,25 >  *  Max Bytes: 64.0 MBytes  > > So, someone could explain me how the data is stored in the right
Slabs??? > How calculate it???  > > Thank you > > > -- > > --- > You received this message because you are subscribed to the Google
Groups "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send
an email to memcached+...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > >