# External Interrupts in the x86 system. Part 1. Interrupt controller evolution

• Tutorial

• What is PIC and what is it for?
• What is APIC and what is it for? What is the purpose of LAPIC and I/O APIC?
• What are the differences between APIC, xAPIC, and x2APIC?
• What is MSI? What are the differences between MSI and MSI-X?

A specification for the $PIR table was recently on the Microsoft website, but currently is unavailable. It is possible to understand the table's content from PCI BIOS Specification [4.2.2. Get PCI Interrupt Routing Options] or from here (the last link is in Russian, but you can try to google «PCI IRQ Routing Table Specification») ### APIC (wiki, osdev) The last method worked until multiprocessor systems arrived. By nature, the PIC can only send interrupts to one CPU, and in a multiprocessor system it is desired to load CPUs in a balanced way. The solution to this problem was the new APIC interface (Advanced PIC). A special controller called LAPIC (Local APIC) was added for each processor, as well as the I/O APIC controller for routing interrupts from external devices. All of these controllers are combined in a common bus with the name APIC (note that modern systems use a standard system bus instead of a separate APIC bus for this task). When an external interrupt arrives on the I/O APIC input, the controller will send an interrupt message to the LAPIC of one of the system CPUs. In this way the I/O APIC controller helps balance interrupt load between processors. The first APIC chip was the 82489DX, which was a separate chip that had a connected LAPIC and I/O APIC within itself. For a dual processor system three such chips were needed: two for LAPIC and one for I/O APIC. Later LAPIC functionality was directly included in processors, and the I/O APIC part was separated to the 82093AA chip. The I/O APIC 82093AA had 24 inputs, and the APIC architecture could support up to 16 CPUs. Interrupts 0-15 were left for old ISA interrupts for compatibility with older systems, and interrupts 16-23 were meant for all the PCI devices. With this delimitation all conflicts between ISA and PCI interrupts could be easily avoided. With the increased number of free interrupt lines it also became possible to increase the number of PIRQx lines. I/O APIC and LAPIC programming is done with the help of MMIO. LAPIC registers are usually placed on address 0xFEE00000, and I/O APIC registers on address 0xFEС00000, though it is possible to reconfigure them. As in the PIC case, separate chips in the beginning became part of the chipset later. APIC architecture was later modernized, and its new variant was named xAPIC (x — extended). With full backwards compatibility, the total number of possible CPUs in system was increased to 256. The next step in architecture development was named x2APIC. The number of possible CPUs in the system was increased to 2^32. These controllers can work in a backwards compatibility mode with xAPIC, or they can work in the new x2APIC mode. In this new mode controller programming is not done through MMIO, but through MSR registers (which are much faster). According to this link, IOMMU support is necessary for this mode. It is worthwhile to note that it is possible to have several I/O APIC controllers in the system. For example, one for 24 interrupts in a southbridge and the other one for 32 interrupts in a northbridge. In the context of I/O APIC, interrupts are usually called GSI (Global System Interrupt). So, the forementioned system has GSIs 0-55. How can we determine if a CPU has an internal LAPIC and which APIC architecture it supports? It is possible to answer these questions by inspecting bit-flags from CPUID. To help the OS discover LAPIC and I/O APIC, the BIOS should present information about them either through an MPtable (old method) or through an ACPI table (a MADT table in this case). Besides common information, both the MPtable and the ACPI (in this case a DSDT table) should contain information about the interrupt routing. This means information about which device uses which interrupt line (similar to the$PIR table).

You can read about the MPtable in the official specification. Earlier the specification was on the Intel website, but currently it is only possible to find it in an archive version. The ACPI specification can be found on the UEFI website (current version is 6.2). It is worthwhile to notice that with ACPI it is possible to declare interrupt routing for systems without APIC (instead of providing a separate \$PIR table).

### MSI

(wiki)

The last variant of APIC was good, but not without downsides. All of the interrupt lines from devices made the system very complicated and thus increased error probability. The PCI express bus came to replace the PCI bus, which simplified all interrupt systems completely. It doesn't have interrupt lines at all. For backwards compatibility interrupt signals (INTx#) are emulated with a separate kind of messages. With PCI interrupt lines their connection was made with physical wires. With PCI express interrupt lines a connection is logical and is made by PCI express bridges. But this support of legacy INTx interrupts only exists for backwards compatibility with the PCI bus. PCI express introduces a completely new method of interrupt delivery — MSI (Message Signaled Interrupts). In this method a device signals about the interrupt simply by writing to a special place in the MMIO region of the CPUs LAPIC.

Earlier a single PCI device (this means all its functions) could have only 4 interrupts, but now it became possible to address up to 32 interrupts.

In the case of MSI there is no sharing of interrupt lines: every interrupt naturally corresponds to its device.

MSI interrupts also solve one more problem. For example, let's imagine a situation where a device makes a memory-write transaction, and wants to signal about its completion through the interrupt. But a write transaction can be delayed on the bus in the process of its transmission (and the device couldn't know about it). In this case the signal about the interrupt will come to the CPU first, so the processor will read not yet valid data. If MSI is used, information about the MSI is transmitted in the same way as data messages, and so it can't come earlier.

It is worthwhile to notice that MSI interrupts can't work without LAPIC, but MSI's can replace I/O APIC (one more design simplification).

After some time the MSI method was extended to MSI-X. Now every device can have up to 2048 interrupts. It is also now possible to specify which CPU should process which interrupt. It can be very useful for highload devices, like network cards for example.

There is no need for a separate BIOS table for MSI support. But the device should indicate its MSI support through one of the Capabilities in its PCI Config space. Also, a device driver should include all necessary support for working with the MSI.

### Сonclusion

In this article we have studied information about interrupt controller evolution and have got a common theoretical knowledge about interrupt delivery from external devices in the x86 system.

In the next part we will go practice and see how to engage each of forementioned interrupt controllers in Linux.

In the third part we will look into the coreboot code and see what settings are needed in the chipset for correct interrupt routing.

#### Acknowledgments

Special thanks to Jacob Garber from the coreboot community for helping me with this article translation.
Support the author
Share post

## Similar posts

AdBlock has stolen the banner, but banners are not teeth — they will be back