The dd is running in DOM1. The OOM killer is getting run in DOM0. There is clearly a memory leak in the block I/O path.

DOM0 is curly and DOM1 is xen-vm0.

A large amount of memory has already been leaked:

kmacy@curly cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: 262565888 205619200 56946688 0 23339008 28123136 == [root@xen-vm0 ~]$ dd if=/dev/zero of=/tmp/bwout bs=1024k count=256 == kmacy@curly cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: 262565888 214687744 47878144 0 23339008 28123136 == [root@xen-vm0 ~]$ dd if=/dev/zero of=/tmp/bwout count=256 bs=1024k 256+0 records in 256+0 records out == kmacy@curly cat /proc/meminfo | head -3 total: used: free: shared: buffers: cached: Mem: 262565888 223727616 38838272 0 23339008 28123136 == [root@xen-vm0 ~]$ dd if=/dev/zero of=/tmp/bwout count=256 bs=1024k 256+0 records in 256+0 records out == kmacy@curly cat /proc/meminfo | head -2 total: used: free: shared: buffers: cached: Mem: 262565888 232873984 29691904 0 23339008 28123136

So ~40MB is leaked for every 1GB transferred.

I can give you a stack backtrace of the memory allocation failure in DOM0 if you like, but as far as I can tell the horse has long since left the barn at that point.

On a side note - I don't need suspend/restore, I just need coredump and almost immediately after that PTRACE_STOP. So long as I can stop the domain long enough to write out its state I have what I need.