<div dir="ltr"><div>Alexandre,<br><br>This is all correct but not related to inconsistency issue.<br><br></div>Stanislav<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 28, 2015 at 10:44 AM, Alexandre DERUMIER <span dir="ltr"><<a href="mailto:aderumier@odiso.com" target="_blank">aderumier@odiso.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">>>That is right and you just can't use O_DIRECT without alignment. You would just get an error on "write" system call. If you check drbd_oos_test.c you find posix_memalign there.<br>
</span><a href="http://people.redhat.com/msnitzer/docs/io-limits.txt" target="_blank">http://people.redhat.com/msnitzer/docs/io-limits.txt</a><br>
"Direct I/O best practices<br>
-------------------------<br>
Users must always take care to use properly aligned and sized IO. This<br>
is especially important for Direct I/O access. Direct I/O should be<br>
aligned on a 'logical_block_size' boundary and in multiples of the<br>
'logical_block_size'. With native 4K devices (logical_block_size is 4K)<br>
it is now critical that applications perform Direct I/O that is a<br>
multiple of the device's 'logical_block_size'. This means that<br>
applications that do not perform 4K aligned I/O, but 512-byte aligned<br>
I/O, will break with native 4K devices.<br>
"<br>
<br>
about qemu, (for qcow2, I think raw it's ok)<br>
<br>
<a href="http://lists.gnu.org/archive/html/qemu-discuss/2015-01/msg00051.html" target="_blank">http://lists.gnu.org/archive/html/qemu-discuss/2015-01/msg00051.html</a><br>
<br>
"<br>
qcow2 cannot store the "physical block size" as an explicit<br>
property. But what you can do is the following:<br>
<br>
1. Make sure the host physical disk partition system that stores<br>
the qcow2 file is aligned to a multiple of 4K (or the RAID block<br>
size if on a RAID system).<br>
<br>
2. Make sure the host file system that stores the qcow2 file has<br>
a block size of 4K or a multiple of 4K.<br>
<br>
3. Make sure the internal qcow2 cluster_size is 4K or a multiple<br>
of 4K (I think this is the default). Otherwise this is set using<br>
the "-o" "cluster_size=4096" option to qemu-img create/convert.<br>
<br>
4. Make sure the guest partition on the virtual disk (backed by<br>
the qcow2 file) is aligned on a multiple of the qcow2<br>
cluster_size.<br>
<br>
5. Make sure the guest file system of the guest partition on the<br>
virtual disk has a block size which is a multiple of the qcow2<br>
cluster_size.<br>
<br>
In other words, the usual "4K issue" procedures, but on both the<br>
physical and virtual machine.<br>
<span class="">"<br>
----- Mail original -----<br>
De: "Stanislav German-Evtushenko" <<a href="mailto:ginermail@gmail.com">ginermail@gmail.com</a>><br>
</span>À: "aderumier" <<a href="mailto:aderumier@odiso.com">aderumier@odiso.com</a>><br>
Cc: "dietmar" <<a href="mailto:dietmar@proxmox.com">dietmar@proxmox.com</a>>, "pve-devel" <<a href="mailto:pve-devel@pve.proxmox.com">pve-devel@pve.proxmox.com</a>><br>
Envoyé: Jeudi 28 Mai 2015 09:38:15<br>
<div class="HOEnZb"><div class="h5">Objet: Re: [pve-devel] Default cache mode for VM hard drives<br>
<br>
> not sure it's related, but with O_DIRECT I think that the write need to be aligned with multiple of 4k block. (or 512bytes)<br>
That is right and you just can't use O_DIRECT without alignment. You would just get an error on "write" system call. If you check drbd_oos_test.c you find posix_memalign there.<br>
<br>
On Thu, May 28, 2015 at 10:33 AM, Alexandre DERUMIER < <a href="mailto:aderumier@odiso.com">aderumier@odiso.com</a> > wrote:<br>
<br>
<br>
Hi,<br>
<br>
not sure it's related, but with O_DIRECT I think that the write need to be aligned with multiple of 4k block. (or 512bytes)<br>
<br>
(and I remember some bug with qemu and and 512b-logical/4k-physical disks<br>
<br>
<a href="http://pve.proxmox.com/pipermail/pve-devel/2012-November/004530.html" target="_blank">http://pve.proxmox.com/pipermail/pve-devel/2012-November/004530.html</a><br>
<br>
I'm not an expert so I can't confirm.<br>
<br>
----- Mail original -----<br>
De: "Stanislav German-Evtushenko" < <a href="mailto:ginermail@gmail.com">ginermail@gmail.com</a> ><br>
À: "dietmar" < <a href="mailto:dietmar@proxmox.com">dietmar@proxmox.com</a> ><br>
Cc: "aderumier" < <a href="mailto:aderumier@odiso.com">aderumier@odiso.com</a> >, "pve-devel" < <a href="mailto:pve-devel@pve.proxmox.com">pve-devel@pve.proxmox.com</a> ><br>
Envoyé: Jeudi 28 Mai 2015 09:22:12<br>
Objet: Re: [pve-devel] Default cache mode for VM hard drives<br>
<br>
Hi Dietmar,<br>
<br>
I did it couple of times already and everytime I had the same answer "upper layer problem". Well, as we've done this long way up to this point I would like to continue.<br>
<br>
I have just done the same test with mdadm and not DRBD. And what I found that this problem was reproducible on the software raid too, just as it was claimed by Lars Ellenberg. It means that problem is not only related to DRBD but to O_DIRECT mode generally when we don't use host cache and a block device reads data directly from userspace.<br>
<br>
The testcase is bellow.<br>
<br>
1. Prepare<br>
<br>
dd if=/dev/zero of=/tmp/mdadm1 bs=1M count=100<br>
dd if=/dev/zero of=/tmp/mdadm2 bs=1M count=100<br>
losetup /dev/loop1 /tmp/mdadm1<br>
losetup /dev/loop2 /tmp/mdadm2<br>
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/loop{1,2}<br>
<br>
2. Write data with O_DIRECT<br>
<br>
./a.out /dev/md0<br>
<br>
3. Check consistency with vbindiff<br>
<br>
vbindiff /tmp/mdadm{1,2} #press enter multiple times to skip metadata<br>
<br>
And here we find that data on "physical devices" is different and md raid did not catch this.<br>
<br>
<br>
On Thu, May 28, 2015 at 7:40 AM, Dietmar Maurer < <a href="mailto:dietmar@proxmox.com">dietmar@proxmox.com</a> > wrote:<br>
<br>
<br>
> What this means?<br>
<br>
I still think you should discuss that on the DRBD list.<br>
<br>
<br>
<br>
<br>
<br>
Best regards,<br>
Stanislav German-Evtushenko<br>
<br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><a href="http://www.helplinux.ru" target="_blank">www.helplinux.ru</a> - Найди себе Гуру</div>
</div>