[PVE-User] PVE kernel (sources, build process, etc.)

Uwe Sauter uwe.sauter.de at gmail.com
Wed Aug 22 14:14:48 CEST 2018 

One thing speaks againts this being PTI is that both types of nodes have secondary OSDs causing slow requests.

Though it still is an option to try before giving up completely.



Am 22.08.18 um 11:45 schrieb Uwe Sauter:
> Hi Marcus,
> 
> no, I haven't disabled Spectre/Meltdown mitigations (yet).
> 
> For 4 of my hosts I have the Intel microcode from 2018-05-08 running which is the most up-to-date version from Ubuntu.
> 
> Using the spectre-meltdown-checker.sh script from https://github.com/speed47/spectre-meltdown-checker gives below output which
> let's me believe that performace should be goot with mitigations enabled.
> 
> #### output for sandybridge based hosts ####
> CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
> * Mitigated according to the /sys interface:  YES  (Mitigation: __user pointer sanitization)
> * Kernel has array_index_mask_nospec:  YES  (1 occurrence(s) found of x86 64 bits array_index_mask_nospec())
> * Kernel has the Red Hat/Ubuntu patch:  NO
> * Kernel has mask_nospec64 (arm64):  NO
>> STATUS:  NOT VULNERABLE  (Mitigation: __user pointer sanitization)
> 
> CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
> * Mitigated according to the /sys interface:  YES  (Mitigation: Full generic retpoline, IBPB, IBRS_FW)
> * Mitigation 1
>   * Kernel is compiled with IBRS support:  YES
>     * IBRS enabled and active:  YES  (for kernel and firmware code)
>   * Kernel is compiled with IBPB support:  YES
>     * IBPB enabled and active:  YES
> * Mitigation 2
>   * Kernel has branch predictor hardening (arm):  NO
>   * Kernel compiled with retpoline option:  YES
>     * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)
>> STATUS:  NOT VULNERABLE  (Full retpoline + IBPB are mitigating the vulnerability)
> 
> CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
> * Mitigated according to the /sys interface:  YES  (Mitigation: PTI)
> * Kernel supports Page Table Isolation (PTI):  YES
>   * PTI enabled and active:  YES
>   * Reduced performance impact of PTI:  YES  (CPU supports PCID, performance impact of PTI will be reduced)
> * Running as a Xen PV DomU:  NO
>> STATUS:  NOT VULNERABLE  (Mitigation: PTI)
> 
> CVE-2018-3640 [rogue system register read] aka 'Variant 3a'
> * CPU microcode mitigates the vulnerability:  YES
>> STATUS:  NOT VULNERABLE  (your CPU microcode mitigates the vulnerability)
> 
> CVE-2018-3639 [speculative store bypass] aka 'Variant 4'
> * Mitigated according to the /sys interface:  YES  (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)
> * Kernel supports speculation store bypass:  YES  (found in /proc/self/status)
>> STATUS:  NOT VULNERABLE  (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)
> 
> CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
> * Mitigated according to the /sys interface:  YES  (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable)
>> STATUS:  NOT VULNERABLE  (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable)
> 
> 
> ##### end #####
> 
> 
> The other 2 hosts are base on Westmere architecture and won't get microcode updates as fas as I know. Disabling mitigations on
> these hosts might be worth a try. Same script from above gives:
> 
> #### output of westmere based hosts ####
> CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
> * Mitigated according to the /sys interface:  YES  (Mitigation: __user pointer sanitization)
> * Kernel has array_index_mask_nospec:  YES  (1 occurrence(s) found of x86 64 bits array_index_mask_nospec())
> * Kernel has the Red Hat/Ubuntu patch:  NO
> * Kernel has mask_nospec64 (arm64):  NO
>> STATUS:  NOT VULNERABLE  (Mitigation: __user pointer sanitization)
> 
> CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
> * Mitigated according to the /sys interface:  YES  (Mitigation: Full generic retpoline)
> * Mitigation 1
>   * Kernel is compiled with IBRS support:  YES
>     * IBRS enabled and active:  NO
>   * Kernel is compiled with IBPB support:  YES
>     * IBPB enabled and active:  NO
> * Mitigation 2
>   * Kernel has branch predictor hardening (arm):  NO
>   * Kernel compiled with retpoline option:  YES
>     * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)
>> STATUS:  NOT VULNERABLE  (Full retpoline is mitigating the vulnerability)
> IBPB is considered as a good addition to retpoline for Variant 2 mitigation, but your CPU microcode doesn't support it
> 
> CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
> * Mitigated according to the /sys interface:  YES  (Mitigation: PTI)
> * Kernel supports Page Table Isolation (PTI):  YES
>   * PTI enabled and active:  YES
>   * Reduced performance impact of PTI:  YES  (CPU supports PCID, performance impact of PTI will be reduced)
> * Running as a Xen PV DomU:  NO
>> STATUS:  NOT VULNERABLE  (Mitigation: PTI)
> 
> CVE-2018-3640 [rogue system register read] aka 'Variant 3a'
> * CPU microcode mitigates the vulnerability:  NO
>> STATUS:  VULNERABLE  (an up-to-date CPU microcode is needed to mitigate this vulnerability)
> 
> CVE-2018-3639 [speculative store bypass] aka 'Variant 4'
> * Mitigated according to the /sys interface:  NO  (Vulnerable)
> * Kernel supports speculation store bypass:  YES  (found in /proc/self/status)
>> STATUS:  VULNERABLE  (Your CPU doesn't support SSBD)
> 
> CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
> * Mitigated according to the /sys interface:  YES  (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled)
>> STATUS:  NOT VULNERABLE  (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled)
> 
> ##### end #####
> 
> 
> 
> 
> 
> 
> Am 22.08.18 um 11:08 schrieb Marcus Haarmann:
>> Hi,
>> did you try this:
>> (taken from the ceph list)
>>
>> This is PTI, I think.  Try to add "noibrs noibpb nopti nospectre_v2" to 
>> kernel cmdline and reboot.
>>
>>
>> Did this make a difference ?
>> We are struggeling with proxmox/ceph too and we have the suspect that it is kernel related or network related,
>> but could not narrow it down to a specific reason.
>> But the effects are different... We encountered stuck I/O on rdb devices.
>> And kernel says it is losing a mon connection and hunting for a new mon all the time (when backup takes
>> place and heavy I/O is done).
>>
>> Marcus Haarmann
>>
>> ----------------------------------------------------------------------------------------------------------------------------------
>> *Von: *"uwe sauter de" <uwe.sauter.de at gmail.com>
>> *An: *"Thomas Lamprecht" <t.lamprecht at proxmox.com>, "pve-user" <pve-user at pve.proxmox.com>
>> *Gesendet: *Mittwoch, 22. August 2018 10:50:19
>> *Betreff: *Re: [PVE-User] PVE kernel (sources, build process, etc.)
>>
>>>>>
>>>>>> * pve-kernel 4.13 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-artful.git/ ?
>>>>>>
>>>>>
>>>>> Yes. (Note that this may not get much updates anymore)
>>>>>
>>>>>> * pve-kernel 4.15 is based on http://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/ ?
>>>>>>
>>>>>
>>>>> Yes. We're normally on the latest stable release tagged on the master branch.
>>>>>
>>>>
>>>> I'll checkout both and compare the myri10ge drivers…
>>>>
>>>
>>> What's your exact issue, if I may ask?
>>>
>>> cheers,
>>> Thomas
>>>
>>>
>>>
>>
>> Short story is that since updating from 4.13 to 4.15 I get slow_requests in Ceph with only low load on Ceph. If I boot back, those
>> are gone.
>>
>> Or at least almost as I was able to produce slow_requests with 4.13 but only if deep scrubing was manually initiated on all PGs.
>>
>>
>> Sage Weil (Ceph dev) suggested that this probably is MTU or bonding related and thus I'm currently testing with different
>> settings. Another guy on the ceph-devel list suggested different driver versions so I was checking that as well.
>>
>>
>>
>> Long story is here:
>>
>> https://pve.proxmox.com/pipermail/pve-user/2018-May/169472.html
>>
>> https://pve.proxmox.com/pipermail/pve-user/2018-May/169492.html
>>
>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/026627.html
>>
>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028862.html
>>
>> https://marc.info/?l=ceph-devel&m=153449419830984&w=2
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>

More information about the pve-user mailing list