User:Demindiro/RISC-V Notes

Fixing "Fatal error: invalid -march= option" (AKA you're using the wrong assembler)

When trying to build the OS Specific Toolchain & trying to compile a program you may have encountered this error:

Assembler messages:
Fatal error: invalid -march= option: `rv64imafdc'

If you run GCC verbosely (i.e. gcc -v ...) you'll see that the wrong assembler is being used. While I still don't know why this happens, there is a simple fix:

./configure
	...
	--with-as=$(PREFIX)/bin/$(TARGET)-as \
	--with-ld=$(PREFIX)/bin/$(TARGET)-ld \
	--with-build-time-tools=$(PREFIX)/bin \
	...

I don't know how or why other people's toolchains work without these options and it'll probably remain a mystery to me, but It Works(tm) now at least.

RISC-V, PCI and setting valid BARs

In my hairloss-inducing adventure of getting PCI to work I've found I missed an important detail that doesn't seem to be mentioned anywhere, so I figured I'd quickly document it here before I forget about it.

When enumerating the PCI buses in QEMU you may notice that all BARs point to 0, for example:

[kernel/src/main.rs:396] &pci = PCIHeader {
    vendor_id: 6900,
    device_id: 4100,
    command: 0,
    status: 16,
    revision_id: 0,
    prog_if: 0,
    subclass: 0,
    class_code: 1,
    cache_line_size: 0,
    latency_timer: 0,
    header_type: 0,
    bist: 0,
    base_address: [
        1,
        0,
        0,
        0,
        12,
        0,
    ],
    cardbus_cis_pointer: 0,
    subsystem_vendor_id: 6900,
    subsystem_id: 8,
    expansion_rom_base_address: 0,
    capabilities_pointer: 152,
    _reserved_0: 0,
    _reserved_1: 0,
    _reserved_2: 0,
    interrupt_line: 0,
    interrupt_pin: 1,
    min_grant: 0,
    max_latency: 0,
}

This means you need to set it yourself. You may naively think that you can set it to whatever you want, but that is not the case. If you do it anyways, you'll either read garbage or you get a load/store fault:

Invalid access at addr 0x5400000, size 4, region '(null)', reason: rejected

If you open the QEMU console and enter info pci it'll seem that everything is fine, so what gives?

  Bus  0, device   0, function 0:
    Host bridge: PCI device 1b36:0008
      PCI subsystem 1af4:1100
      id ""
  Bus  0, device   1, function 0:
    SCSI controller: PCI device 1af4:1004
      PCI subsystem 1af4:0008
      IRQ 0, pin A
      BAR0: I/O at 0xffffffffffffffff [0x003e].
      BAR1: 32 bit memory at 0x00000000 [0x00000fff].
      BAR4: 64 bit prefetchable memory at 0x05400000 [0x05403fff].
      id "disk0"

If you enter info mtree and inspect the output you may notice something else too:

--- snip ---
    0000000010100010-0000000010100017 (prio 0, i/o): fwcfg.dma
    0000000020000000-0000000021ffffff (prio 0, romd): virt.flash0
    0000000022000000-0000000023ffffff (prio 0, romd): virt.flash1
    0000000030000000-000000003fffffff (prio 0, i/o): alias pcie-ecam @pcie-mmcfg-mmio 0000000000000000-000000000fffffff
    0000000040000000-000000007fffffff (prio 0, i/o): alias pcie-mmio @gpex_mmio 0000000040000000-000000007fffffff
    0000000080000000-000000008fffffff (prio 0, ram): riscv_virt_board.ram
    0000000400000000-00000007ffffffff (prio 0, i/o): alias pcie-mmio-high @gpex_mmio 0000000400000000-00000007ffffffff

address-space: I/O
  0000000000000000-000000000000ffff (prio 0, i/o): io

address-space: cpu-memory-0
  0000000000000000-ffffffffffffffff (prio 0, i/o): system
    0000000000001000-000000000000ffff (prio 0, rom): riscv_virt_board.mrom
    0000000000100000-0000000000100fff (prio 0, i/o): riscv.sifive.test
--- snip ---
memory-region: pcie-mmcfg-mmio
  0000000000000000-000000001fffffff (prio 0, i/o): pcie-mmcfg-mmio

memory-region: gpex_mmio
  0000000000000000-ffffffffffffffff (prio 0, i/o): gpex_mmio
    0000000000000000-0000000000000fff (prio 1, i/o): virtio-scsi-pci-msix
      0000000000000000-000000000000003f (prio 0, i/o): msix-table
      0000000000000800-0000000000000807 (prio 0, i/o): msix-pba
    0000000005400000-0000000005403fff (prio 1, i/o): virtio-pci
      0000000005400000-0000000005400fff (prio 0, i/o): virtio-pci-common-virtio-scsi
      0000000005401000-0000000005401fff (prio 0, i/o): virtio-pci-isr-virtio-scsi
      0000000005402000-0000000005402fff (prio 0, i/o): virtio-pci-device-virtio-scsi
      0000000005403000-0000000005403fff (prio 0, i/o): virtio-pci-notify-virtio-scsi

memory-region: system
  0000000000000000-ffffffffffffffff (prio 0, i/o): system
    0000000000001000-000000000000ffff (prio 0, rom): riscv_virt_board.mrom
    0000000000100000-0000000000100fff (prio 0, i/o): riscv.sifive.test
--- snip ---

Note the pcie-mmio-high with the range 0000000400000000-00000007ffffffff. If you pick any address that is within that range then it'll stop faulting and automagically work.

The next question then is: how do you get that info without the QEMU console? RISC-V platforms usually have a FTD and presumably you got the PCI address from the same tree. If you read the device tree specification (https://github.com/devicetree-org/devicetree-specification/releases/tag/v0.3) you'll notice it says this:

2.3.8 ranges

Property name: ranges

Value type: <empty> or <prop-encoded-array> encoded as an arbitrary number of (child-bus-address,parent-bus-address,length) triplets.

Description: The ranges property provides a means of defining a mapping or translation between the address space of the bus (the child address space) and the address space of the bus node’s parent (the parent address space).

Checking the DTS I got and lo and behold: ranges = < 0x1000000 0x00 0x00 0x00 0x3000000 0x00 0x10000 0x2000000 0x00 0x40000000 0x00 0x40000000 0x00 0x40000000 0x3000000 0x04 0x00 0x04 0x00 0x04 0x00 > .

Note that #address-cells = < 3 > for the pci node, #address-cells = < 2 > for the parent node and #size-cells = < 2 > for the pci node so the triplets are (0x1000000 0x00 0x00, 0x00 0x3000000, 0x00 0x10000), (0x2000000 0x00 0x40000000, 0x00 0x40000000, 0x00 0x40000000), (0x3000000 0x04 0x00, 0x04 0x00, 0x04 0x00)