Debian

禁用 VT-d 和中斷重新映射是否會破壞 MSI-X?

  • December 28, 2015

所以我有一個Dell PowerEdge R610正在執行的伺服器Debian Jessie,根據一些文件,它的晶片組有故障:https ://www.novell.com/support/kb/doc.php?id=7014344 (Intel 5500 rev 13)。

提供的外殼測試確認了此晶片組的存在:lspci -nn | grep -qE '8086:(340[36].*rev 13|3405.*rev (12|13|22))' && echo "Interrupt remapping is broken"

它已VT-d在 BIOS 中禁用並已GRUB_CMDLINE_LINUX_DEFAULT="quiet intremap=off"添加到/etc/grub/default.

我的問題是,這是否會影響MSI-XNIC 卡的功能(應該在核心之間傳播中斷)?

在這個特定的伺服器上,確實只有一個核心在處理中斷,但我不確定這是否是禁用 BIOS 和核心設置的結果。或者還有其他什麼可能在起作用?

(不幸的是,我無法訪問 BIOS 來更改設置並在功能開啟的情況下進行測試)。

謝謝你。

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
  0:         43          0          0          0          0          0          0          0   IO-APIC-edge      timer
  1:          3          0          0          0          0          0          0          0   IO-APIC-edge      i8042
  8:          1          0          0          0          0          0          0          0   IO-APIC-edge      rtc0
  9:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          4          0          0          0          0          0          0          0   IO-APIC-edge      i8042
 16:      15459          0          0          0          0          0          0          0   IO-APIC-fasteoi   megasas
 17:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
 18:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
 19:         28          0          0          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1
 20:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb6
 21:        398          0          0          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb5
 23:      50805          0          0          0          0          0          0          0   IO-APIC-fasteoi   ata_piix
 72:          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
 73:          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
 74:          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
 75:          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
 76:          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
 95:          2          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
 96:          2          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
 97:          2          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
 98:          2          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
 99:          2          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
100:          2          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
101:          2          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
102:          2          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
103:  126464280          0          0          0          0          0          0          0   PCI-MSI-edge      eth2-0
104:   92145436          0          0          0          0          0          0          0   PCI-MSI-edge      eth2-1
105:   89096948          0          0          0          0          0          0          0   PCI-MSI-edge      eth2-2
106:   88057555          0          0          0          0          0          0          0   PCI-MSI-edge      eth2-3
107:   81812555          0          0          0          0          0          0          0   PCI-MSI-edge      eth2-4
108:   89599897          0          0          0          0          0          0          0   PCI-MSI-edge      eth2-5
109:   88031686          0          0          0          0          0          0          0   PCI-MSI-edge      eth2-6
110:   87201395          0          0          0          0          0          0          0   PCI-MSI-edge      eth2-7
112:  377125739          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-0
113:      72913          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-1
114:    1890421          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-2
115:  162242342          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-3
116:      53494          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-4
117:      85112          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-5
118:      44281          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-6
119:      41480          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-7
NMI:      39120        194        382        197        337        184        276        186   Non-maskable interrupts
LOC:  217815785    1972192    4721954    2056033    2252897    1834404    1894099    1739779   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:      39120        194        382        197        337        184        276        186   Performance monitoring interrupts
IWI:          8          0          0          0          0          0          0          0   IRQ work interrupts
RTR:          7          0          0          0          0          0          0          0   APIC ICR read retries
RES:     133678      35960      21042       9616      10918       8463       8607       8143   Rescheduling interrupts
CAL:       1815       3264       1626       3885       1970       3807       1972       4000   Function call interrupts
TLB:        896        110        258        139        200        110        187         81   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:        171        171        171        171        171        171        171        171   Machine check polls
HYP:          0          0          0          0          0          0          0          0   Hypervisor callback interrupts
ERR:          0
MIS:          0

網卡設置:

root@server:~# lspci -v

01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
       Subsystem: Dell PowerEdge R610 BCM5709 Gigabit Ethernet
       Flags: bus master, fast devsel, latency 0, IRQ 36
       Memory at d6000000 (64-bit, non-prefetchable) [size=32M]
       Capabilities: [48] Power Management version 3
       Capabilities: [50] Vital Product Data
       Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
       Capabilities: [a0] MSI-X: Enable+ Count=9 Masked-
       Capabilities: [ac] Express Endpoint, MSI 00
       Capabilities: [100] Device Serial Number 84-2b-2b-ff-fe-fe-b3-bd
       Capabilities: [110] Advanced Error Reporting
       Capabilities: [150] Power Budgeting <?>
       Capabilities: [160] Virtual Channel
       Kernel driver in use: bnx2

01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
       Subsystem: Dell PowerEdge R610 BCM5709 Gigabit Ethernet
       Flags: bus master, fast devsel, latency 0, IRQ 48
       Memory at d8000000 (64-bit, non-prefetchable) [size=32M]
       Capabilities: [48] Power Management version 3
       Capabilities: [50] Vital Product Data
       Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
       Capabilities: [a0] MSI-X: Enable- Count=9 Masked-
       Capabilities: [ac] Express Endpoint, MSI 00
       Capabilities: [100] Device Serial Number 84-2b-2b-ff-fe-fe-b3-bf
       Capabilities: [110] Advanced Error Reporting
       Capabilities: [150] Power Budgeting <?>
       Capabilities: [160] Virtual Channel
       Kernel driver in use: bnx2

02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
       Subsystem: Dell PowerEdge R610 BCM5709 Gigabit Ethernet
       Flags: bus master, fast devsel, latency 0, IRQ 32
       Memory at da000000 (64-bit, non-prefetchable) [size=32M]
       Capabilities: [48] Power Management version 3
       Capabilities: [50] Vital Product Data
       Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
       Capabilities: [a0] MSI-X: Enable+ Count=9 Masked-
       Capabilities: [ac] Express Endpoint, MSI 00
       Capabilities: [100] Device Serial Number 84-2b-2b-ff-fe-fe-b3-c1
       Capabilities: [110] Advanced Error Reporting
       Capabilities: [150] Power Budgeting <?>
       Capabilities: [160] Virtual Channel
       Kernel driver in use: bnx2

02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
       Subsystem: Dell PowerEdge R610 BCM5709 Gigabit Ethernet
       Flags: bus master, fast devsel, latency 0, IRQ 42
       Memory at dc000000 (64-bit, non-prefetchable) [size=32M]
       Capabilities: [48] Power Management version 3
       Capabilities: [50] Vital Product Data
       Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
       Capabilities: [a0] MSI-X: Enable- Count=9 Masked-
       Capabilities: [ac] Express Endpoint, MSI 00
       Capabilities: [100] Device Serial Number 84-2b-2b-ff-fe-fe-b3-c3
       Capabilities: [110] Advanced Error Reporting
       Capabilities: [150] Power Budgeting <?>
       Capabilities: [160] Virtual Channel
       Kernel driver in use: bnx2


root@server:~# ethtool -k eth0

Features for eth0:
rx-checksumming: on
tx-checksumming: on
       tx-checksum-ipv4: on
       tx-checksum-ip-generic: off [fixed]
       tx-checksum-ipv6: on
       tx-checksum-fcoe-crc: off [fixed]
       tx-checksum-sctp: off [fixed]
scatter-gather: on
       tx-scatter-gather: on
       tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
       tx-tcp-segmentation: off
       tx-tcp-ecn-segmentation: off
       tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]


root@server:~# ethtool -i eth0
driver: bnx2
version: 2.2.5
firmware-version: 7.12.17 bc 7.10.0 NCSI 2.0.13
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no 

是的。MSI-X 需要啟用中斷重新映射才能完全工作。

來自英特爾 PDF: 面向定向 I/O 架構規範的英特爾® 虛擬化技術 2014 年 10 月

5.1 中斷重映射

中斷重映射架構使系統軟體能夠控制和審查由所有源產生的外部中斷請求,包括來自中斷控制器 (I/OxAPIC)、支持 MSI/MSI-X 的設備(包括端點、根埠和 Root-Complex 集成終端)的請求。點。

引用自:https://serverfault.com/questions/745593