AMD-Vi IOMMU
|
|
AMD-Vi provides a IOMMU to translate "Device Virtual Addresses" (DVAs) to "System Physical Addresses" (SPAs). Without nested paging, it is equivalent to translating linear/virtual addresses to physical addresses. It is also able to remap interrupt lines.
The IOMMU can be located in the PCI configuration space and is identified by checking class=0x8, subclass=0x6 and progif=0x0. None of the BAR registers are used, instead registers in capability space (id=0xf) must be used.
Overview
The IOMMU requires several in-memory structures to operate. Communication with the IOMMU is primarily done through ring buffers to improve throughput.
Device table
The device table describes how to handle transactions for devices behind the IOMMU. There is one entry for each device, up to 65536 (the maximum for a single PCI segment).
By default, the array must be contiguous in memory. As each entry is 32 bytes it means a maximum of 2MiB is required, though it can be shorter if the start/end range contains no devices. The IOMMU may support a segmented device table to save further memory.
I/O page tables
The I/O page tables are compatible with long-mode page tables. It uses a number of the AVL bits to set device permissions and other attributes.
Interrupt remapping
Nested translation
Command buffer
Event log
Device table
| 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 | 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 | 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| +0 | Host Page Table Root Pointer[31:12] | Mode | HAD | res | TV | V | ||||||||||||||||||||||||||
| +4 | res | IW | IR | GCR3 TRP[14:12] | GLX | GV | GIoV | GPRP | PPR | Host Page Table Root Pointer[51:32] | ||||||||||||||||||||||
| +8 | GCR3 Table Root Pointer[30:15] | Domain ID[15:0] | ||||||||||||||||||||||||||||||
| +12 | GCR3 Table Root Pointer[51:31] | SATS | SysMgt | EX | SD | Cache | IoCtl | SA | SE | I | ||||||||||||||||||||||
| +16 | Interrupt Table Root Pointer[31:6] | IG | IntTabLen | IV | ||||||||||||||||||||||||||||
| +20 | Lint1Pass | Lint0Pass | IntCtl | HPTMode | NMIPass | EIntPass | InitPass | GuestPagingMode | res | Interrupt Table Root Pointer[51:32] | ||||||||||||||||||||||
| +24 | GDeviceID[15:0] | vImuEn | reserved | |||||||||||||||||||||||||||||
| +28 | SnoopAttribute | Mode0FC | AttrV | reserved | GuestID[15:0] | |||||||||||||||||||||||||||
I/O page tables
| 63 | 62 | 61 | 60 | 59 | 58:52 | 51:12 | 11:9 | 8:7 | 6 | 5 | 4:1 | 0 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ign | IW | IR | FC | U | resv | physical address | next level | ign | D | A | ign | P |
| 63 | 62 | 61 | 60:52 | 51:12 | 11:9 | 8:6 | 5 | 4:1 | 0 |
|---|---|---|---|---|---|---|---|---|---|
| ign | IW | IR | resv | physical address | next level | ign | A | ign | P |
Commands
Each command entry is 128 bits in size.
| Offset | 63:60 | 59:0 | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| +0 | Opcode | First opcode dependent operand | ||||||||||||||
| +8 | Second opcode dependent operand | |||||||||||||||
To send commands, the Command Buffer Base Register must be programmed. It must point to a ring buffer aligned to a 4KiB boundary. The Command Buffer Head Pointer Register and Command Buffer Tail Pointer Register are automatically reset to 0 when the base register is written to.
| 63:60 | 59:56 | 55:52 | 51:12 | 11:00 | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| / | Length | / | Base | / | |||||||||||
| Encoding | Number of entries | Byte size |
|---|---|---|
| 0-7 | (reserved) | |
| 8 | 256 entries | 4 KiB |
| 9 | 512 entries | 8 KiB |
| ... | ||
| 15 | 32768 entries | 512 KiB |
| 63:19 | 18:4 | 3:0 | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| / | Head pointer | / | |||||||||||||
| 63:19 | 18:4 | 3:0 | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| / | Tail pointer | / | |||||||||||||
To notify the IOMMU, the Tail Pointer Register must be incremented, wrapping if necessary (modulo ring buffer size).
The IOMMU will increment the Head Pointer Register when a command has been fetched. This does not mean the command has been processed! To ensure a command has been processed, use COMPLETION_WAIT.
Commands are only processed when CmdBufRun=1. Modifying the Base Register or the Head Pointer Register is only allowed when CmdBufRun=0.
1. COMPLETION_WAIT
2. INVALIDATE_DEVTAB_ENTRY
3. INVALIDATE_IOMMU_PAGES
4. INVALIDATE_IOTLB_PAGES
5. INVALIDATE_INTERRUPT_TABLE
6. PREFETCH_IOMMU_PAGES
7. COMPLETE_PPR_REQUEST
8. INVALIDATE_IOMMU_ALL
9. INSERT_GUEST_EVENT
10. RESET_VMMIO
Events
1. ILLEGAL_DEV_TABLE_ENTRY
2. IO_PAGE_FAULT
3. DEV_TAB_HARDWARE_ERROR
4. PAGE_TAB_HARDWARE_ERROR
5. ILLEGAL_COMMAND_ERROR
6. COMMAND_HARDWARE_ERROR
7. IOTLB_INV_TIMEOUT
8. INVALID_DEVICE_REQUEST
9. INVALID_PPR_REQUEST
10. EVENT_COUNTER_ZERO
11. GUEST_EVENT_FAULT
12. VIOMMU_HARDWARE_ERROR
13. RMP_PAGE_FAULT
14. RMP_HARDWARE_ERROR
Interrupt Remapping
MSI / MSI-X
To remap MSI the address need to be configured to point to FD_F8xx_xxxx.[1]
Registers
PCI configuration space
All accesses must be 32-bits and aligned to a 32-bit boundary. Software should write zeros to reserved bits.
Base Capability (id=0xf)
| Offset | Name |
|---|---|
| 0x00 | Capability Header |
| 0x04 | Base Address Low Register |
| 0x08 | Base Address High Register |
| 0x0C | Range Register |
| 0x10 | Miscellaneous Information Register 0 |
| 0x14 | Miscellaneous Information Register 1 |
MMIO
All accesses should be 64-bits and properly aligned, but smaller power-of-two accesses (32/16/8-bits) are also permitted. Larger accesses (128/...-bits) are not permitted.
| Offset | Name (+4) | Name (+0) |
|---|---|---|
| 0x0000 to 0x1000 | ||
| 0x0000 | Device Table Base Address Register | |
| 0x0008 | Command Buffer Base Address Register | |
| 0x0010 | Event Log Base Address Register | |
| 0x0018 | IOMMU Control Register | |
| 0x0020 | IOMMU Exclusion Base Register / Completion Store Base Register | |
| 0x0028 | IOMMU Exclusion Range Limit Register / Completion Store Limit Register | |
| 0x0030 | IOMMU Extended Feature Register | |
| 0x0038 | PPR Log Base Address Register | |
| 0x0040 | IOMMU Hardware Event Upper Register | |
| 0x0048 | IOMMU Hardware Event Lower Register | |
| 0x0050 | Hardware Event Status Register | |
| 0x0058 | (unmapped) | |
| 0x0060 | SMI Filter Register 0 | |
| 0x0068 | SMI Filter Register 1 | |
| 0x0070 | SMI Filter Register 2 | |
| 0x0078 | SMI Filter Register 3 | |
| 0x0080 | SMI Filter Register 4 | |
| 0x0088 | SMI Filter Register 5 | |
| 0x0090 | SMI Filter Register 6 | |
| 0x0098 | SMI Filter Register 7 | |
| 0x00A0 | SMI Filter Register 8 | |
| 0x00A8 | SMI Filter Register 9 | |
| 0x00B0 | SMI Filter Register 10 | |
| 0x00B8 | SMI Filter Register 11 | |
| 0x00C0 | SMI Filter Register 12 | |
| 0x00C8 | SMI Filter Register 13 | |
| 0x00D0 | SMI Filter Register 14 | |
| 0x00D8 | SMI Filter Register 15 | |
| 0x00E0 | Guest Virtual APIC Log Base Address Register | |
| 0x00E8 | Guest Virtual APIC Log Tail Address Register | |
| 0x00F0 | PPR Log B Base Address Register | |
| 0x00F8 | Event Log B Base Address Register | |
| 0x0100 | Device Table Segment 1 Base Address Register | |
| 0x0108 | Device Table Segment 2 Base Address Register | |
| 0x0110 | Device Table Segment 3 Base Address Register | |
| 0x0118 | Device Table Segment 4 Base Address Register | |
| 0x0120 | Device Table Segment 5 Base Address Register | |
| 0x0128 | Device Table Segment 6 Base Address Register | |
| 0x0130 | Device Table Segment 7 Base Address Register | |
| 0x0138 | Device-Specific Feature Extension (DSFX) Register | |
| 0x0140 | Device-Specific Control Extension (DSCX) Register | |
| 0x0148 | Device-Specific Status Extension (DSSX) Register | |
| 0x0150 | MSI Vector Register 1 | MSI Vector Register 0 |
| 0x0158 | MSI Address Low Register | MSI Capability Header Register |
| 0x0160 | MSI Data Register | MSI Address High Register |
| 0x0168 | IOMMU Performance Optimization Control Register | MSI Mapping Capability Header Register |
| 0x0170 | XT IOMMU General Interrupt Control Register | |
| 0x0178 | XT IOMMU PPR Interrupt Control Register | |
| 0x0180 | XT IOMMU GA Log Interrupt Control Register | |
| 0x0188 | (reserved) | |
| 0x0190 | vIOMMU Status Register | |
| 0x0198 | (reserved) | |
| 0x01A0 | IOMMU Extended Feature 2 Register | |
| ... | ||
| 0x0200 | MARC Aperture 0 Base Register | |
| 0x0208 | MARC Aperture 0 Relocation Register | |
| 0x0210 | MARC Aperture 0 Length Register | |
| 0x0218 | MARC Aperture 1 Base Register | |
| 0x0220 | MARC Aperture 1 Relocation Register | |
| 0x0228 | MARC Aperture 1 Length Register | |
| 0x0230 | MARC Aperture 2 Base Register | |
| 0x0238 | MARC Aperture 2 Relocation Register | |
| 0x0240 | MARC Aperture 2 Length Register | |
| 0x0248 | MARC Aperture 3 Base Register | |
| 0x0250 | MARC Aperture 3 Relocation Register | |
| 0x0258 | MARC Aperture 3 Length Register | |
| 0x0260 | (reserved) | |
| ... | ||
| 0x1000 to 0x2000 | ||
| 0x1000 | (reserved) | |
| ... | ||
| 0x1FF8 | IOMMU Reserved Register | |
| 0x2000 to 0x3000 | ||
| 0x2000 | Command Buffer Head Pointer Register | |
| 0x2008 | Command Buffer Tail Pointer Register | |
| 0x2010 | Event Log Head Pointer Register | |
| 0x2018 | Event Log Tail Pointer Register | |
| 0x2020 | IOMMU Status Register | |
| 0x2028 | (reserved) | |
| 0x2030 | IOMMU PPR Log Head Pointer Register | |
| 0x2038 | IOMMU PPR Log Tail Pointer Register | |
| 0x2040 | Guest Virtual APIC Log Head Pointer Register | |
| 0x2048 | Guest Virtual APIC Log Tail Pointer Register | |
| 0x2050 | PPR Log B Head Pointer Register | |
| 0x2058 | PPR Log B Tail Pointer Register | |
| 0x2060 | Event Log B Head Pointer Register | |
| 0x2068 | Event Log B Tail Pointer Register | |
| 0x2070 | Event Log B Head Pointer Register | |
| 0x2078 | Event Log B Tail Pointer Register | |
| 0x2080 | PPR Log Auto Response Register | |
| 0x2088 | PPR Log Overflow Early Indicator Register | |
| 0x2090 | PPR Log B Overflow Early Indicator Register | |
| 0x2098 | (reserved) | |
Troubleshooting
QEMU: no DMA translation
DMA translation needs to explicitly enabled:
-device amd-iommu,dma-remap=on
Interrupt remapping, IOTLB and other features also need to be enabled explicitly. Run with help to see all options:
-device amd-iommu,help
See also
- ↑ See Table 3 in specification.