Re: [Xen-devel] [PATCH] turn off writable page tables - Andrew Theurer - com.xensource.lists.xen-devel

31 messages in com.xensource.lists.xen-develRe: [Xen-devel] [PATCH] turn off writ...

From	Sent On	Attachments
Andrew Theurer	25 Jul 2006 15:14	.patch
Ian Pratt	25 Jul 2006 15:41
Nivedita Singhvi	25 Jul 2006 15:43
Andrew Theurer	25 Jul 2006 16:19
Andrew Theurer	25 Jul 2006 19:24
Jacob Gorm Hansen	25 Jul 2006 22:31
Gerd Hoffmann	26 Jul 2006 01:17
Keir Fraser	26 Jul 2006 01:40
Andrew Theurer	26 Jul 2006 14:10
Ian Pratt	26 Jul 2006 14:38
Joe Bonasera	26 Jul 2006 16:38
Andrew Theurer	27 Jul 2006 07:43
Keir Fraser	27 Jul 2006 08:30
Ian Pratt	27 Jul 2006 10:31
Keir Fraser	28 Jul 2006 01:54
Andrew Theurer	28 Jul 2006 08:21
Ian Pratt	28 Jul 2006 08:51
Keir Fraser	28 Jul 2006 09:31
Zachary Amsden	28 Jul 2006 14:36	.Other
Andi Kleen	28 Jul 2006 16:05
Zachary Amsden	28 Jul 2006 16:10
Mike Day	28 Jul 2006 16:23
Keir Fraser	31 Jul 2006 02:13
Zachary Amsden	31 Jul 2006 02:31
Keir Fraser	31 Jul 2006 02:53
Zachary Amsden	31 Jul 2006 12:55
Keir Fraser	31 Jul 2006 15:07
Zachary Amsden	31 Jul 2006 15:39
Keir Fraser	02 Aug 2006 02:21
Mike D. Day	03 Aug 2006 13:41
Andrew Theurer	09 Aug 2006 14:14

Subject:	Re: [Xen-devel] [PATCH] turn off writable page tables
From:	Andrew Theurer (haba...@us.ibm.com)
Date:	07/27/2006 07:43:33 AM
List:	com.xensource.lists.xen-devel

fork a quite linear from small number to large number of dirty pages. Below are the min and max:

1280 pages 128000 pages wtpt: 813 usec 37552 usec emulate: 3279 usec 283879 usec

Good, at least that suggests that the code works for the usage it was intended for.

So, in a -perfect-world- this works great. Problem is most workloads don't appear to have a vast percentage of entries that need to be updated. I'll go ahead and expand this test to find out what the threshold is to break even. I'll also see if we can implement a

batched

call in fork to update the parent -I hope this will show just as good performance even when most entries need modification and even better performance over wtpt with a low number of entries modified.

With license to make more invasive changes to core Linux mm it certainly should be possible to optimize this specific case with a batched update fairly easily. You could even go further an implement a 'make all PTEs in pagetable RO' hypercall, possibly including a copy to the child. This could potentially work better than current 'late pin', at least the validation would be incremental rather than in one big hit at the end.

FWIW, I found the threshold for emulate vs wtpt. I ran the fork test with a set number of pages dirtied such that we had x number of PTEs per pte_page.

writable-pt

----------- #pte usec 002 5242 004 5251 006 5373 008 5519 010 5873

emulate

-------- #pte usec 002 4922 004 5265 006 6074 008 6991 010 7806 012 5988

So, the threshold appears to be around 4 PTEs/page. I was a little shocked at first how low this number is, but considering the near identical performance with the various workloads, this make sense. All of the workloads had the vast majority of writable pages flushed with just 2 PTEs/page changed and a handful with more PTEs/page changed. It would not surprise me if the overall average was around 4 PTEs/page.

I am having a hard time finding any "enterprise" workloads which have a lot of PTEs/page right before fork. If anyone can point me to some, that would be great.

I will look into batching next, but I am curious if simply using a hypercall in stead of write fault + emulate will make any difference at all. I'll try that first, then implement the batched update.

Eventually a hypercall which does more would be nice, but I guess we'll have to convince the Linux maintainers it's a good idea.

-Andrew