atom feed10 messages in com.googlegroups.memcachedRe: HELP ! How Memcached objects are ...
FromSent OnAttachments
Nicolas MartinezNov 14, 2015 5:49 am 
Nicolas MartinezNov 14, 2015 6:14 am 
dormandoNov 14, 2015 2:54 pm 
Nicolas MartinezNov 15, 2015 3:59 am 
dormandoNov 15, 2015 3:30 pm 
Nicolas MartinezNov 16, 2015 6:37 am 
Nicolas MartinezNov 16, 2015 10:01 am 
Nicolas MartinezNov 16, 2015 11:08 am 
dormandoNov 16, 2015 2:54 pm 
Nicolas MartinezNov 17, 2015 1:23 am 
Subject:Re: HELP ! How Memcached objects are stored in the good SLAB
From:Nicolas Martinez (mart@gmail.com)
Date:Nov 17, 2015 1:23:36 am
List:com.googlegroups.memcached

Nice, thank you very much.

Are you ok with :

Example : •Key : test •Data : 90 bytes •Flags : 0

Size : (4+1) + (90+2) + (1+2+4) + 48 + 8 = 160 Bytes => Slab 4 (192 Bytes)

Key (characters count) + 1

+ Data (characters count) + 2 bytes ( \r\n )

+ Header

+ Chunk Size

+ CAS Size

Header = Flags (characters count) + nData (characters number of DATA count => 90 = 2 characters)

+ 4 bytes (2 spaces and \r\n)

Chunk Size = 48 bytes (default)

CAS Size = 8 bytes (platform 64 bits)

If CAS disabled:

Size : (4+1) + (90+2) + (1+2+4) + 48 = 152 Bytes => Slab 3 (152 Bytes)

Sounds good :)

Le lundi 16 novembre 2015 23:54:34 UTC+1, Dormando a écrit :

key + 1b val + 2b item struct 48b [optional CAS] 8b then the suffix trailer printf: please read the manpage for snprintf:

snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - 2);

there are four characters in there. two spaces, and then \r and \n. the two %d's change into: "0" for flags, and "90" for bytes (ignore the -2 in there). that's 7b total.

In your no-cas test: 5 + 92 + 48 + 7 == 152. that's the exact size of slab 3. (152b).

On Mon, 16 Nov 2015, Nicolas Martinez wrote:

The test was : •Key : test •Data : 90 bytes •Flags : 0

:(

Le lundi 16 novembre 2015 19:01:50 UTC+1, Nicolas Martinez a écrit : Ok... i think i have understood what you say:

Key (Characters Number) + 1 + Data (Characters Number) + Header + Chunk Size + CAS Size

Header = Flags (Characters Number) + Key (Characters Numbers)

+ 2 bytes ( \r\n )

+ 4 bytes (2 spaces and 1 \r)

Chunk Size = 48 bytes (default)

CAS Size = 8 bytes (64 bits platform)

Seems to be good:

4 + 1 + 90 + (1+4+2+4) + 48 + 8 = 162 Bytes => Slab 4 (192 Bytes)

But, if i start Memcached with -C, it's wrong.

4 + 1 + 90 + (1+4+2+4) + 48 = 154 Bytes => Slab 3 (152 Bytes)

It must be in Slab4 (192 Bytes) No?

Le lundi 16 novembre 2015 15:37:59 UTC+1, Nicolas Martinez a écrit : Thank you very much for yours answers.Ok for CAS... i don't use -C so i have to add 8 bytes

i still don"t understand these lines :

2b appended to data_bytes for an extra "\r\n" +

So, \r\n == 2b ?

+ 4 bytes for spaces and \r\n

which spaces? What is this \r\n ? Isn't already counted before? There is 1 "\r\n" and 1 "\"

echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server} 11211

With your example: * Key : 2c * Val : 28 bytes * Flg : 0 (1bytes)

turns into: * Key : 3b

=> key number characters + 1 * Val : 30b

=> 28 bytes + 2 bytes ("\r\n") * Hdr : 4b + 3b == 7b

=> What are 4b? => 3b are: flags (1b) + ?? * Itm : 56b

=> 48b + 8b (CAS) => 96b. which is the cap for slab 1 in a default setup.

Thank you again.

Le lundi 16 novembre 2015 00:31:07 UTC+1, Dormando a écrit : Read carefully:

item_size_ok(const size_t nkey, const int flags, const int nbytes) {

passes:

size_t ntotal = item_make_header(nkey + 1, flags, nbytes, prefix, &nsuffix);

Then conditionally:

if (settings.use_cas) { ntotal += sizeof(uint64_t); }

item_make_header is doing: nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - 2);

Then:

return sizeof(item) + nkey + *nsuffix + nbytes;

It's convoluted but shirt.

the lengths are: key + 1 + data_bytes + 2b appended to data_bytes for an extra "\r\n" + stringified rep of the flags + data length + 4 bytes for spaces and \r\n (these are carriage returns, one byte each) + 8b for CAS if enabled

CAS can be turned off via the -C starttime arg. it takes up 8 bytes.

Example: * Key : 2c * Val : 28b * Flg : 0 (1b)

turns into: * Key : 3b * Val : 30b * Hdr : 4b + 3b == 7b * Itm : 56b => 96b. which is the cap for slab 1 in a default setup.

It's tough to get it exact for small chunks due to the way the header is added. You should ballpark or tune the -f value to align with your observed data.

On Sun, 15 Nov 2015, Nicolas Martinez wrote:

> Hi, > Is CAS always used? > If yes, we have to always add 56 bytes to the KEY and VALUE ? > you don't count FLAGS characters?? > > I've found that Flags's size (number of characters) impact the storage. > > Example: > * Key : 2 characters = 2 bytes > * Value : 28 characters = 28 bytes > * FLAGS : 1 characters = 1 bytes > => 31 bytes > > seems to take the same storage as > * Key : 1 characters = 1 bytes > * Value : 28 characters = 28 bytes > * FLAGS : 2 characters = 2 bytes > => 31 bytes ... wich is the limit to be stored in Slab1 > > ok for the /r/n ... should take 4 bytes no? > > So, if we count 56 bytes for CAS : 56(cas)+31(key+value+flags)+4(/r/n)= 91 > > Not good... :( > > where I'm wrong ?? > > Le samedi 14 novembre 2015 23:55:12 UTC+1, Dormando a écrit : > The mysql docs don't speak for the main tree... that's their own thing. > > the "sizes" binary that comes with the source tree tells you how many > bytes an item will use (though I intend to add this output to the 'stats' > output somewhere). With CAS this is 56 bytes. > > 56 + 2 + 30 == 88. Class 1 by default (in 1.4.24) is 96 bytes, but the > item still ends up in class 2. > > Why is this? (unfortunately?) because memcached pre-renders part of the > text protocol into the item header: > > *nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - > 2); > return sizeof(item) + nkey + *nsuffix + nbytes; > > so the flags + length are getting flattened + \r\n added to the end. > Together that's just enough to push it over the edge. It'd also be nice to > add a highly optimized numerics printf so I could twiddle options to save > a few bytes of memory in objects, but don't get your hopes up for that > happening soon :) > > On Sat, 14 Nov 2015, Nicolas Martinez wrote: > > > Add: Memcached version : 1.4.4 (RedHat) > > > > Le samedi 14 novembre 2015 14:49:37 UTC+1, Nicolas Martinez a écrit : > > Hi, few days i'm reading Memcached documentation and blogs... and i don't understand how objects are stored. > > > > My test > > > > 3 slabs : > > > > * 96.0 Bytes > > * 120.0 Bytes > > * 240.0 Bytes > > Everywhere, it's told : > > * if data is < 96 Bytes, it will be stored in Slabs1 (96B) > > * if data > 96B and < 120B, it will be stored in Slabs2 (120B) > > * if data > 120B, it will be stored in Slabs3 (240B) > > * etc. > > BUT, for example, when i'm creating an 30B object, it's stored in Slab2 (120B), and NOT in Slab1 (96B). > > > > External sources: > > For example, the default size for the smallest

block is 88 bytes (40 bytes of value, and the default 48 bytes for the key and flag data). If the size of the first item you store into the cache is less than 40 bytes, then a slab with a block size of 88 bytes is created and the value stored.

> > => https://dev.mysql.com/doc/mysql-ha-scalability/en/ha-memcached-using-memory.html > > > > > > WRONG > > > > A slab class is a collection of pages divided into

same sized chunks. Each slab class is referenced to by its chunk size. So we’ll have Slab class 80kb, Slab class 100kb and so on. When an object needs to be stored, its size determines where it gets stored. So if the object is larger than 80kb but less than 100kb, it

gets stored into > Slab > > class 100kb. > > => http://returnfoo.com/2012/02/memcached-memory-allocation-and-optimization-2/ > > > > > > WRONG > > > > How i create an object: > > > > data=$(pwgen 30 -c 1) > > echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server} 11211 > > > > > > So, when 30B object is creating : > > * key name : 30 = 2 bytes > > * value: 30 characters = 30 bytes > > * tags : 0 = 1 bytes > > => All = 33 bytes > > If i add the default 48b as explained on Mysql website : 33 + 48 = 81B ... so < Slab1 (91B)... but always stored in Slab2 (120B) > > > > So, the size used to store object in the good Slab is not: > > * object value size > > * sum of KEY, VALUE and TAGS in bytes > > KEY size : 1 character = 1 B > > VALUE size : 1 character = 1 B > > TAGS size : 1 character = 1 B > > > > ... as read everywhere > > > > So, It seems that: (SUM of KEY+VALUE+TAGS ) > > * For slab1 96.0 Bytes, data stored if <= 31 B (SUM of 2+28+1 ) > > * For slab2 120.0 Bytes, data stored if <= 55 B (SUM of 2+52+1 ) > > * For slab3 152.0 Bytes, data stored if <= 87 B (SUM of 2+84+1 ) > > * For slab4 192.0 Bytes, data stored if <= 126 B (SUM of 3+122+1 ) > > * For slab5 240.0 Bytes, data stored if <= 174 B (SUM of 3+170+1 ) > > * etc. > > > > My configuration : > > * Chunk Size : 48 > > * Chunk Growth Factore: 1,25 > > * Max Bytes: 64.0 MBytes > > > > So, someone could explain me how the data is stored in the right Slabs??? > > How calculate it??? > > > > Thank you > > > > > > -- > > > > --- > > You received this message because you are subscribed to the Google Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > --- > You received this message because you are subscribed to the Google Groups "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > >