<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:209339466;
mso-list-type:hybrid;
mso-list-template-ids:1519822422 134807567 134807577 134807579 134807567 134807577 134807579 134807567 134807577 134807579;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi all,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>On a particular Proxmox VE cluster (pve-manager/3.3-5/bfebec03 (running kernel: 2.6.32-33-pve)), I’m experiencing corruption inside virtual machines after they’ve been storage-migrated between two NFS mounts. I’m aware there are probably a few avenues of investigation but since this is time consuming, I’m wondering if anyone can think which would be the most telling?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The symptom is that Windows Server VMs will refuse to boot correctly after a live storage migration, or some service on them will fail to load, or Windows will start to complain about corrupt files etc. It seems to be if the machines are migrated while writing data, which is unavoidable for some since they provide email, databases, etc. This can be a migration to another NFS mount, or a migration into another file format on the same mount. Migrations while VMs are powered down do not exhibit this issue, although I’m now calculating MD5 sums before and after just in case.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>To reproduce this intermittent issue, I’ve:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Created four VMs on ‘NFS_A’, 2 virtual CPU, 32GB RAW IDE (writethrough), 2GB memory<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Installed Windows Server 2008R2 into all four<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='mso-list:Ignore'>3.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Joined to the domain and allowed the first round of Windows Updates to download to the machine<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='mso-list:Ignore'>4.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Started installing this round of Windows Updates on all servers simultaneously in order to provide some load which touches system components, as of course Windows Updates do.<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='mso-list:Ignore'>5.<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>A few minutes in, migrate storage to NFS_B, back to NFS_A etc<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The migrations were:<o:p></o:p></p><p class=MsoNormal>VM 100000 - none<o:p></o:p></p><p class=MsoNormal>VM 100001 - none<o:p></o:p></p><p class=MsoNormal>VM 100002 - To NFS_B<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;text-indent:36.0pt'>Back to NFS_A. Failed on “TASK ERROR: storage migration failed: mirroring error: VM 100002 qmp command 'block-job-complete' failed - The active block job for device 'drive-ide0' cannot be completed”<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;text-indent:36.0pt'>Tried migrating to NFS_A again. Success this time.<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;text-indent:36.0pt'>To NFS_B<o:p></o:p></p><p class=MsoNormal>VM 100003 - To NFS_B<o:p></o:p></p><p class=MsoNormal style='margin-left:72.0pt'>Back to NFS_A<o:p></o:p></p><p class=MsoNormal style='margin-left:72.0pt'>To NFS_B<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>At this time, I left the machines to finish their updates.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>On completion, chkdsk c: all of them:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>100000: Windows has checked the file system and found no problems.<o:p></o:p></p><p class=MsoNormal>100001: Windows has checked the file system and found no problems.<o:p></o:p></p><p class=MsoNormal>100002: Windows has checked the file system and found no problems. (this is the one that had a failed migration)<o:p></o:p></p><p class=MsoNormal>100003: <o:p></o:p></p><p class=MsoNormal>The type of the file system is NTFS.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>WARNING! F parameter not specified.<o:p></o:p></p><p class=MsoNormal>Running CHKDSK in read-only mode.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>CHKDSK is verifying files (stage 1 of 3)...<o:p></o:p></p><p class=MsoNormal> 70144 file records processed.<o:p></o:p></p><p class=MsoNormal>File verification completed.<o:p></o:p></p><p class=MsoNormal> 61 large file records processed.<o:p></o:p></p><p class=MsoNormal> 0 bad file records processed.<o:p></o:p></p><p class=MsoNormal> 0 EA records processed.<o:p></o:p></p><p class=MsoNormal> 60 reparse records processed.<o:p></o:p></p><p class=MsoNormal>CHKDSK is verifying indexes (stage 2 of 3)...<o:p></o:p></p><p class=MsoNormal>66 percent complete. (83724 of 101286 index entries processed)<o:p></o:p></p><p class=MsoNormal>Error detected in index $I30 for file 38380.<o:p></o:p></p><p class=MsoNormal>Error detected in index $I30 for file 38380.<o:p></o:p></p><p class=MsoNormal>67 percent complete. (84428 of 101286 index entries processed)<o:p></o:p></p><p class=MsoNormal>Error detected in index $I30 for file 64316.<o:p></o:p></p><p class=MsoNormal>Error detected in index $I30 for file 64316.<o:p></o:p></p><p class=MsoNormal>Error detected in index $I30 for file 64323.<o:p></o:p></p><p class=MsoNormal>Error detected in index $I30 for file 64323.<o:p></o:p></p><p class=MsoNormal>Error detected in index $I30 for file 64324.<o:p></o:p></p><p class=MsoNormal>Error detected in index $I30 for file 64324.<o:p></o:p></p><p class=MsoNormal>Error detected in index $I30 for file 64325.<o:p></o:p></p><p class=MsoNormal>Error detected in index $I30 for file 64325.<o:p></o:p></p><p class=MsoNormal> 101286 index entries processed.<o:p></o:p></p><p class=MsoNormal>Index verification completed.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Errors found. CHKDSK cannot continue in read-only mode.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>What I don't know at this point is how serious these errors are, but I think it does prove that something's going wrong with the migration process and leads me to understand why some machines would suddenly develop serious faults after they’ve been migrated.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>How can I troubleshoot further? I’m keen to maximise the effectiveness of what I do next. Since the problem doesn’t happen to all migrations, I’d ideally like to do something to prove the issue rather than change something which temporarily makes it appear to go away, only to find that I migrate a production server in the future and have to restore from backup/snapshot again. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks in advance,<o:p></o:p></p><p class=MsoNormal>Chris<o:p></o:p></p></div></body></html>