atom feed17 messages in net.launchpad.lists.openstackRe: [Openstack] [metering] resources ...
FromSent OnAttachments
Doug HellmannMay 11, 2012 8:55 am 
Loic DacharyMay 11, 2012 12:40 pm 
Loic DacharyMay 11, 2012 12:55 pm 
Doug HellmannMay 11, 2012 1:18 pm 
Doug HellmannMay 11, 2012 1:21 pm 
Loic DacharyMay 14, 2012 5:22 am 
Doug HellmannMay 14, 2012 7:15 am 
Loic DacharyMay 14, 2012 10:03 am 
Doug HellmannMay 14, 2012 1:27 pm 
Julien DanjouMay 15, 2012 3:04 am 
Loic DacharyMay 15, 2012 4:52 am 
Julien DanjouMay 15, 2012 5:21 am 
Doug HellmannMay 15, 2012 7:26 am 
Nick BarcetMay 16, 2012 12:31 am 
Loic DacharyMay 16, 2012 1:09 am 
Julien DanjouMay 16, 2012 2:19 am 
Doug HellmannMay 16, 2012 8:05 am 
Subject:Re: [Openstack] [metering] resources metadata
From:Julien Danjou (juli@enovance.com)
Date:May 15, 2012 3:04:56 am
List:net.launchpad.lists.openstack

On Mon, May 14 2012, Loic Dachary wrote:

Each set of metering data will need to be associated with the appropriate metadata from the resource at the time the metering information was collected. The rate of change of metadata and metering events are different, though, so the timestamps of the metadata records are unlikely to match exactly with the values in the metering records. Depending on the clock resolution, it would be possible to have metadata changes and meter data with the same timestamp, resulting in an incorrect association. Indeed, good point.

We can work around that by maintaining proper foreign key references using the metadata version field as you describe in the schema above (so the resource id and metadata version value point to the correct metadata record). It will make recording the metering data less efficient because we will need to determine the current version for the resource metadata, but we can optimize that eventually through indexes and caching.

Aggregation will also need to take the metadata version into account, so
everywhere in the list of queries we say "by resource_id" we need to change that
to "by resource_id and version".

I added the idea of a format version for when the payload format changes and
tried to write down a description of the metadata storage matching this thread
in the wiki.

http://wiki.openstack.org/EfficientMetering?action=diff&rev2=80&rev1=78

What do you think ?

I'm jumping in a bit late in the discussion, but there may be a point I miss in the current definition because, I think it's getting too complicated.

We now have 2 "payload" fields: one for meter and one for metadata.

For example, if you look at the c1 counter (instance) you need to store the "type" as payload of the meter. This is a metadata of the instance, but it's not currently defined as being stored in metadata, but in the "payload" field of the meter. Moreover, I'm rather sure there will soon be a counter with the need of 2 different "payload" information, and we'll have a problem since we can only store one in the current meter schema, so we'll store the second one as a metadata or something. So clearly the initial "payload" solution is not enough.

OTOH I find the metadata proposal in another table too much complicated. Why not storing what metadata in the meter.payload field in the same table (e.g. as a JSON string)?

I miss the point of the introduction of a dedicated metadata table with version string. It sounds to me like early optimization, which is the root of all evil. :) But I might miss something.

-- Julien Danjou // eNovance http://enovance.com // ✉ juli@enovance.com ☎ +33 1 49 70 99 81

_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : open@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp