[PVE-User] PVE kernel (sources, build process, etc.)

Uwe Sauter uwe.sauter.de at gmail.com
Wed Aug 22 11:45:15 CEST 2018


Hi Marcus,

no, I haven't disabled Spectre/Meltdown mitigations (yet).

For 4 of my hosts I have the Intel microcode from 2018-05-08 running which is the most up-to-date version from Ubuntu.

Using the spectre-meltdown-checker.sh script from https://github.com/speed47/spectre-meltdown-checker gives below output which
let's me believe that performace should be goot with mitigations enabled.

#### output for sandybridge based hosts ####
CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface:  YES  (Mitigation: __user pointer sanitization)
* Kernel has array_index_mask_nospec:  YES  (1 occurrence(s) found of x86 64 bits array_index_mask_nospec())
* Kernel has the Red Hat/Ubuntu patch:  NO
* Kernel has mask_nospec64 (arm64):  NO
> STATUS:  NOT VULNERABLE  (Mitigation: __user pointer sanitization)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface:  YES  (Mitigation: Full generic retpoline, IBPB, IBRS_FW)
* Mitigation 1
  * Kernel is compiled with IBRS support:  YES
    * IBRS enabled and active:  YES  (for kernel and firmware code)
  * Kernel is compiled with IBPB support:  YES
    * IBPB enabled and active:  YES
* Mitigation 2
  * Kernel has branch predictor hardening (arm):  NO
  * Kernel compiled with retpoline option:  YES
    * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)
> STATUS:  NOT VULNERABLE  (Full retpoline + IBPB are mitigating the vulnerability)

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface:  YES  (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI):  YES
  * PTI enabled and active:  YES
  * Reduced performance impact of PTI:  YES  (CPU supports PCID, performance impact of PTI will be reduced)
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

CVE-2018-3640 [rogue system register read] aka 'Variant 3a'
* CPU microcode mitigates the vulnerability:  YES
> STATUS:  NOT VULNERABLE  (your CPU microcode mitigates the vulnerability)

CVE-2018-3639 [speculative store bypass] aka 'Variant 4'
* Mitigated according to the /sys interface:  YES  (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)
* Kernel supports speculation store bypass:  YES  (found in /proc/self/status)
> STATUS:  NOT VULNERABLE  (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)

CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
* Mitigated according to the /sys interface:  YES  (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable)
> STATUS:  NOT VULNERABLE  (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable)


##### end #####


The other 2 hosts are base on Westmere architecture and won't get microcode updates as fas as I know. Disabling mitigations on
these hosts might be worth a try. Same script from above gives:

#### output of westmere based hosts ####
CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface:  YES  (Mitigation: __user pointer sanitization)
* Kernel has array_index_mask_nospec:  YES  (1 occurrence(s) found of x86 64 bits array_index_mask_nospec())
* Kernel has the Red Hat/Ubuntu patch:  NO
* Kernel has mask_nospec64 (arm64):  NO
> STATUS:  NOT VULNERABLE  (Mitigation: __user pointer sanitization)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface:  YES  (Mitigation: Full generic retpoline)
* Mitigation 1
  * Kernel is compiled with IBRS support:  YES
    * IBRS enabled and active:  NO
  * Kernel is compiled with IBPB support:  YES
    * IBPB enabled and active:  NO
* Mitigation 2
  * Kernel has branch predictor hardening (arm):  NO
  * Kernel compiled with retpoline option:  YES
    * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)
> STATUS:  NOT VULNERABLE  (Full retpoline is mitigating the vulnerability)
IBPB is considered as a good addition to retpoline for Variant 2 mitigation, but your CPU microcode doesn't support it

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface:  YES  (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI):  YES
  * PTI enabled and active:  YES
  * Reduced performance impact of PTI:  YES  (CPU supports PCID, performance impact of PTI will be reduced)
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

CVE-2018-3640 [rogue system register read] aka 'Variant 3a'
* CPU microcode mitigates the vulnerability:  NO
> STATUS:  VULNERABLE  (an up-to-date CPU microcode is needed to mitigate this vulnerability)

CVE-2018-3639 [speculative store bypass] aka 'Variant 4'
* Mitigated according to the /sys interface:  NO  (Vulnerable)
* Kernel supports speculation store bypass:  YES  (found in /proc/self/status)
> STATUS:  VULNERABLE  (Your CPU doesn't support SSBD)

CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
* Mitigated according to the /sys interface:  YES  (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled)
> STATUS:  NOT VULNERABLE  (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled)

##### end #####






Am 22.08.18 um 11:08 schrieb Marcus Haarmann:
> Hi,
> did you try this:
> (taken from the ceph list)
> 
> This is PTI, I think.  Try to add "noibrs noibpb nopti nospectre_v2" to 
> kernel cmdline and reboot.
> 
> 
> Did this make a difference ?
> We are struggeling with proxmox/ceph too and we have the suspect that it is kernel related or network related,
> but could not narrow it down to a specific reason.
> But the effects are different... We encountered stuck I/O on rdb devices.
> And kernel says it is losing a mon connection and hunting for a new mon all the time (when backup takes
> place and heavy I/O is done).
> 
> Marcus Haarmann
> 
> ----------------------------------------------------------------------------------------------------------------------------------
> *Von: *"uwe sauter de" <uwe.sauter.de at gmail.com>
> *An: *"Thomas Lamprecht" <t.lamprecht at proxmox.com>, "pve-user" <pve-user at pve.proxmox.com>
> *Gesendet: *Mittwoch, 22. August 2018 10:50:19
> *Betreff: *Re: [PVE-User] PVE kernel (sources, build process, etc.)
> 
>>>>
>>>>> * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ?
>>>>>
>>>>
>>>> Yes. (Note that this may not get much updates anymore)
>>>>
>>>>> * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ?
>>>>>
>>>>
>>>> Yes. We're normally on the latest stable release tagged on the master branch.
>>>>
>>>
>>> I'll checkout both and compare the myri10ge drivers…
>>>
>>
>> What's your exact issue, if I may ask?
>>
>> cheers,
>> Thomas
>>
>>
>>
> 
> Short story is that since updating from 4.13 to 4.15 I get slow_requests in Ceph with only low load on Ceph. If I boot back, those
> are gone.
> 
> Or at least almost as I was able to produce slow_requests with 4.13 but only if deep scrubing was manually initiated on all PGs.
> 
> 
> Sage Weil (Ceph dev) suggested that this probably is MTU or bonding related and thus I'm currently testing with different
> settings. Another guy on the ceph-devel list suggested different driver versions so I was checking that as well.
> 
> 
> 
> Long story is here:
> 
> https://pve.proxmox.com/pipermail/pve-user/2018-May/169472.html
> 
> https://pve.proxmox.com/pipermail/pve-user/2018-May/169492.html
> 
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/026627.html
> 
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028862.html
> 
> https://marc.info/?l=ceph-devel&m=153449419830984&w=2
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user



More information about the pve-user mailing list