RISC-V Meaty Skeleton with QEMU virt board

WAIT! Have you read Getting Started, Beginner Mistakes, and some of the related OS theory?

Difficulty level
Beginner

Kernel Designs
Models
Monolithic Kernel Microkernel Hybrid Kernel Exokernel Nano/Picokernel Cache Kernel Virtualizing Kernel Megalithic Kernel
Other Concepts
Modular Kernel Higher Half Kernel 64-bit Kernel

This tutorial assumes you have completed RISC-V Bare Bones on the QEMU virt board, or alternatively, HiFive-1 Bare Bones. If not, you should complete them first for an overview of how to boot your own operating system on RISC-V. This tutorial is deliberately brief on concepts that have already been covered in the bare bones tutorials and their transitive prerequisites.

The bare bones tutorials provide minimal examples that are not structured to enable sustainable mid- to long-term development of the codebase. This tutorial attempts to rectify that by providing a well-structured project that should serve you well through your OSDev journey with the following features:

Hierarchical project structure with make build system for sustainable mid- to long-term development
Includes debug target for debugging with GDB (requires cross-debugger targeting riscv64-elf)
Basic console output through NS16550A UART
Convenience wrappers for powering off and rebooting the device
Working kprintf supporting base format specifiers (no floating point support; no sub-specifiers; no n specifier) to facilitate printf debugging
panic function for kernel panics

RISC-V Bare Bones

Main article: RISC-V Bare Bones

It is assumed you have completed RISC-V Bare Bones or another comparable bare bones tutorial. Though not a strict requirement, it is useful for confirming that your development environment works and explaining a few basic things.

We won't be reusing any code from those tutorials though, so throw it away (or save it to an archive) and we'll start over again.

Building a Cross-Compiler

Main article: GCC Cross-Compiler, Why do I need a Cross Compiler?

You must use a GCC Cross-Compiler in this tutorial as in the RISC-V Bare Bones tutorial, with riscv64-elf as the target.

You must configure your cross-binutils with the --with-sysroot option, otherwise linking will mysteriously fail with the this linker was not configured to use sysroots error message. If you forgot to configure your cross-binutils with that option, you'll have to rebuild it, but you can keep your cross-gcc.

Building a Cross-Debugger (optional)

Main article: GDB

If you wish to debug your kernel with GDB, you'll need to build it separately with riscv64-elf as the target, since it's not included by default with GCC and Binutils. Otherwise, if printf debugging is your style, you may safely skip this section.

The process is similar to building a cross-GCC or cross-binutils, and you may refer to the GDB page in BLFS for most details sans cross-debugging support, but we'll go through the process in detail here anyway.

Fetch the latest version of GDB through https://ftp.gnu.org/gnu/gdb/. The latest version at the time of writing is 12.1:

export GDB_VERSION="12.1"
wget https://ftp.gnu.org/gnu/gdb/gdb-${GDB_VERSION}.tar.xz

Unpack the archive:

tar xvf gdb-${GDB_VERSION}.tar.xz

Now move into the source directory:

pushd gdb-${GDB_VERSION}/

Create a build directory and move into it:

mkdir build
pushd build/

Now export a few variables as with building a cross-GCC or cross-binutils:

export PREFIX="$HOME/opt/cross"
export TARGET=riscv64-elf
export PATH="$PREFIX/bin:$PATH"

Configuration options are mostly the same as with building cross-GCC or cross-binutils. In particular, you may wish to enable the following features:

--enable-tui=yes: Enables TUI mode for debugging. Requires development headers for Ncurses to be installed on the build host
--with-expat: Build with Expat, a library for XML parsing. Requires development headers for Expat to be installed on the build host

../configure --target=$TARGET \
  --prefix="$PREFIX" \
  --with-sysroot \
  --disable-nls \
  --disable-werror \
  --enable-languages=c,c++ \
  --without-headers \
  --enable-tui=yes \
  --with-expat

Now build the source code (this can take a while):

make -j$(nproc)

And install:

make -C gdb install

If you wish to keep the source tree available for conveniently re-building GDB in the future (e.g. with a different set of options), clean up the build files now:

make clean

Move one level up to the project root:

popd

Remove the build/ directory we created:

rm -rf build/

Move up one more level:

popd

Double check our cross-debugger is properly installed:

which -- $TARGET-gdb || echo $TARGET-gdb is not in the PATH

Now you may safely remove the source tree and archive, if you wish:

rm -rf gdb-${GDB_VERSION}/
rm gdb-${GDB_VERSION}.tar.xz

If $PREFIX/bin is not in your path already, you may wish to persist it by writing it to your profile:

echo "export PATH=\"\$HOME/opt/cross/bin:\$PATH\"" >> $HOME/.profile
source $HOME/.profile

Dependencies

You will need these dependencies in order to complete the tutorial:

QEMU full-system emulator for 64-bit RISC-V, of which your distribution-provided package should suffice
riscv64-elf toolchain, as discussed above
(Optional, required for debugging with GDB) GNU debugger targeting riscv64-elf, as discussed above

This tutorial requires a GNU/Linux system, or a similar enough system. The BSD systems may almost work. OS X is not supported but can possibly be made to work with some changes. Windows is not supported, but Windows environments like Cygwin and Windows Subsystem For Linux (WSL) might work.

Acknowledgements

The initial project setup is based on RISC-V from scratch (C runtime, linker script) and The Adventures of OS (overall project structure, Makefile), with the section on building a cross-GDB based on (Beyond) Linux From Scratch.

Source Code

Fetch the source code from the v0.0.1 release of https://github.com/DonaldKellett/marvelos code-named "Meaty Skeleton", using Git:

git clone --branch v0.0.1 https://github.com/DonaldKellett/marvelos.git

Operating systems development is about being an expert. Take the time to read the code carefully through and understand it. Please seek further information and help if you don't understand aspects of it. This code is minimal and almost everything is done deliberately, often to pre-emptively solve future problems.

Project Structure

Makefile: used for building the project with make build system
misc/riscv64-virt.dts: device tree of QEMU RISC-V virt board. See Finding our stack for details
src/: source tree containing files used to build the project
- src/asm/crt0.s: minimal C runtime; see Stop! Runtime! for details
- src/lds/riscv64-virt.ld: custom linker script adapted from the output of riscv64-unknown-elf-ld --verbose; see Link it up for details
- src/common/: common utilities and library functions for use across our project
- src/syscon/: syscon drivers for poweroff and reboot
- src/uart/: UART drivers and I/O-related code
- src/kmain.c: Entry point for our kernel

Makefile

The Makefile makes extensive use of environment variables for conciseness and configurability. The following top-level targets are supported (along with sub-targets for building only a specific subsystem):

all: to build the kernel image
run: to build the kernel image and run it in QEMU
debug: to build the kernel image and run it in debug mode to enable debugging with GDB. The "remote" GDB server runs at port $GDB_PORT on the host (default: 1234). Requires a cross-debugger to be installed; see above for details
clean: to clean up build files so the next build won't be affected by the previous build

Run make $TARGET with TARGET set to your desired target to execute the specified target. For example, to build the kernel image and run it in QEMU:

make run

Additionally, the default target is all if not specified:

make

# Build
CC=riscv64-elf-gcc
CFLAGS=-ffreestanding -nostartfiles -nostdlib -nodefaultlibs
CFLAGS+=-g -Wl,--gc-sections -mcmodel=medany
RUNTIME=src/asm/crt0.s
LINKER_SCRIPT=src/lds/riscv64-virt.ld
KERNEL_IMAGE=kmain

# QEMU
QEMU=qemu-system-riscv64
MACH=virt
MEM=128M
RUN=$(QEMU) -nographic -machine $(MACH) -m $(MEM)
RUN+=-bios none -kernel $(KERNEL_IMAGE)

# QEMU (debug)
GDB_PORT=1234

all: uart syscon common kmain
	$(CC) *.o $(RUNTIME) $(CFLAGS) -T $(LINKER_SCRIPT) -o $(KERNEL_IMAGE)

uart:
	$(CC) -c src/uart/uart.c $(CFLAGS) -o uart.o

syscon:
	$(CC) -c src/syscon/syscon.c $(CFLAGS) -o syscon.o

common:
	$(CC) -c src/common/common.c $(CFLAGS) -o common.o

kmain:
	$(CC) -c src/kmain.c $(CFLAGS) -o kmain.o

run: all
	$(RUN)

debug: all
	$(RUN) -gdb tcp::$(GDB_PORT) -S

clean:
	rm -vf *.o
	rm -vf $(KERNEL_IMAGE)

Kernel source

src/common/common.h

#ifndef COMMON_H
#define COMMON_H

int toupper(int);
void panic(const char *, ...);

#endif

src/common/common.c

#include <stdarg.h>
#include "common.h"
#include "../uart/uart.h"

int toupper(int c) {
  return 'a' <= c && c <= 'z' ? c + 'A' - 'a' : c;
}

void panic(const char *format, ...) {
  kputs("Kernel panic!");
  kputs("Reason:");
  va_list arg;
  va_start(arg, format);
  kvprintf(format, arg);
  va_end(arg);
  asm volatile ("wfi");
}

src/syscon/syscon.h

#ifndef SYSCON_H
#define SYSCON_H

// "test" syscon-compatible device is at memory-mapped address 0x100000
// according to our device tree
#define SYSCON_ADDR 0x100000

void poweroff(void);
void reboot(void);

#endif

src/syscon/syscon.c

#include <stdint.h>
#include "syscon.h"
#include "../uart/uart.h"

void poweroff(void) {
  kputs("Poweroff requested");
  *(uint32_t *)SYSCON_ADDR = 0x5555;
}

void reboot(void) {
  kputs("Reboot requested");
  *(uint32_t *)SYSCON_ADDR = 0x7777;
}

src/uart/uart.h

#ifndef UART_H
#define UART_H

#include <stddef.h>
#include <stdarg.h>

// 0x10000000 is memory-mapped address of UART according to device tree
#define UART_ADDR 0x10000000

void uart_init(size_t);
int kputchar(int);
int kputs(const char *);
void kvprintf(const char *, va_list);
void kprintf(const char *, ...);

#endif

src/uart/uart.c

#include <stddef.h>
#include <stdint.h>
#include <stdarg.h>
#include <limits.h>
#include "uart.h"
#include "../common/common.h"

#define to_hex_digit(n) ('0' + (n) + ((n) < 10 ? 0 : 'a' - '0' - 10))

/*
 * Initialize NS16550A UART
 */
void uart_init(size_t base_addr) {
  volatile uint8_t *ptr = (uint8_t *)base_addr;

  // Set word length to 8 (LCR[1:0])
  const uint8_t LCR = 0b11;
  ptr[3] = LCR;

  // Enable FIFO (FCR[0])
  ptr[2] = 0b1;

  // Enable receiver buffer interrupts (IER[0])
  ptr[1] = 0b1;

  // For a real UART, we need to compute and set the baud rate
  // But since this is an emulated UART, we don't need to do anything
  //
  // Assuming clock rate of 22.729 MHz, set signaling rate to 2400 baud
  // divisor = ceil(CLOCK_HZ / (16 * BAUD_RATE))
  //         = ceil(22729000 / (16 * 2400))
  //         = 592
  //
  // uint16_t divisor = 592;
  // uint8_t divisor_least = divisor & 0xFF;
  // uint8_t divisor_most = divisor >> 8;
  // ptr[3] = LCR | 0x80;
  // ptr[0] = divisor_least;
  // ptr[1] = divisor_most;
  // ptr[3] = LCR;
}

static void uart_put(size_t base_addr, uint8_t c) {
  *(uint8_t *)base_addr = c;
}

int kputchar(int character) {
  uart_put(UART_ADDR, (uint8_t)character);
  return character;
}

static void kprint(const char *str) {
  while (*str) {
    kputchar((int)*str);
    ++str;
  }
}

int kputs(const char *str) {
  kprint(str);
  kputchar((int)'\n');
  return 0;
}

// Limited version of vprintf() which only supports the following specifiers:
//
// - d/i: Signed decimal integer
// - u: Unsigned decimal integer
// - o: Unsigned octal
// - x: Unsigned hexadecimal integer
// - X: Unsigned hexadecimal integer (uppercase)
// - c: Character
// - s: String of characters
// - p: Pointer address
// - %: Literal '%'
//
// None of the sub-specifiers are supported for the sake of simplicity.
// The `n` specifier is not supported since that is a major source of
// security vulnerabilities. None of the floating-point specifiers are
// supported since floating point operations don't make sense in kernel
// space
//
// Anyway, this subset should suffice for printf debugging
void kvprintf(const char *format, va_list arg) {
  while (*format) {
    if (*format == '%') {
      ++format;
      if (!*format)
	return;
      switch (*format) {
      case 'd':
      case 'i':
	{
	  int n = va_arg(arg, int);
	  if (n == INT_MIN) {
	    kprint("-2147483648");
	    break;
	  }
	  if (n < 0) {
	    kputchar('-');
	    n = ~n + 1;
	  }
	  char lsh = '0' + n % 10;
	  n /= 10;
	  char buf[9];
	  char *p_buf = buf;
	  while (n) {
            *p_buf++ = '0' + n % 10;
	    n /= 10;
	  }
	  while (p_buf != buf)
	    kputchar(*--p_buf);
	  kputchar(lsh);
	}
	break;
      case 'u':
        {
	  unsigned n = va_arg(arg, unsigned);
	  char lsh = '0' + n % 10;
	  n /= 10;
	  char buf[9];
	  char *p_buf = buf;
	  while (n) {
            *p_buf++ = '0' + n % 10;
	    n /= 10;
	  }
	  while (p_buf != buf)
	    kputchar(*--p_buf);
	  kputchar(lsh);
	}
	break;
      case 'o':
        {
	  unsigned n = va_arg(arg, unsigned);
	  char lsh = '0' + n % 8;
	  n /= 8;
	  char buf[10];
	  char *p_buf = buf;
	  while (n) {
            *p_buf++ = '0' + n % 8;
	    n /= 8;
	  }
	  while (p_buf != buf)
	    kputchar(*--p_buf);
	  kputchar(lsh);
	}
	break;
      case 'x':
        {
	  unsigned n = va_arg(arg, unsigned);
	  char lsh = to_hex_digit(n % 16);
	  n /= 16;
	  char buf[7];
	  char *p_buf = buf;
	  while (n) {
            *p_buf++ = to_hex_digit(n % 16);
	    n /= 16;
	  }
	  while (p_buf != buf)
	    kputchar(*--p_buf);
	  kputchar(lsh);
	}
	break;
      case 'X':
        {
	  unsigned n = va_arg(arg, unsigned);
	  char lsh = to_hex_digit(n % 16);
	  n /= 16;
	  char buf[7];
	  char *p_buf = buf;
	  while (n) {
            *p_buf++ = to_hex_digit(n % 16);
	    n /= 16;
	  }
	  while (p_buf != buf)
	    kputchar(toupper(*--p_buf));
	  kputchar(toupper(lsh));
	}
	break;
      case 'c':
	kputchar(va_arg(arg, int));
	break;
      case 's':
	kprint(va_arg(arg, char *));
	break;
      case 'p':
	{
          kprint("0x");
	  size_t ptr = va_arg(arg, size_t);
	  char lsh = to_hex_digit(ptr % 16);
	  ptr /= 16;
	  char buf[15];
	  char *p_buf = buf;
	  while (ptr) {
            *p_buf++ = to_hex_digit(ptr % 16);
	    ptr /= 16;
	  }
	  while (p_buf != buf)
	    kputchar(*--p_buf);
	  kputchar(lsh);
	}
	break;
      case '%':
	kputchar('%');
	break;
      default:
	kputchar('%');
	kputchar(*format);
      }
    } else
      kputchar(*format);
    ++format;
  }
}

void kprintf(const char *format, ...) {
  va_list arg;
  va_start(arg, format);
  kvprintf(format, arg);
  va_end(arg);
}

src/kmain.c

#include "uart/uart.h"
#include "syscon/syscon.h"
#include "common/common.h"

#define ARCH "RISC-V"
#define MODE 'M'

void kmain(void) {
  uart_init(UART_ADDR);

  kprintf("Hello %s World!\n", ARCH);
  kprintf("We are in %c-mode!\n", MODE);

  poweroff();
}

Running the project

Invoke make run in the project root. You should see the following output:

Hello RISC-V World!
We are in M-mode!
Poweroff requested

Final remarks and going further

This is by no means an end to your OSDev adventures on RISC-V. Make the OS kernel your own! Add support for memory management, interrupt handling, porting newlib ... you name it. Give your own RISC-V OS a creative name! "maRVelOS" / marvelos is already taken by User:Donaldsebleung though ;-)

Also note that this example RISC-V OS runs in M-mode usually reserved for firmware, rather than the S-mode recommended for RISC-V supervisors (OSes). If you wish to follow the RISC-V conventions closely, you may want to look into RISC-V privilege modes and OpenSBI early on and port your OS kernel accordingly. More information about RISC-V privilege modes available on our wiki.

RISC-V Meaty Skeleton with QEMU virt board

Contents

RISC-V Bare Bones

Building a Cross-Compiler

Building a Cross-Debugger (optional)

Dependencies

Acknowledgements

Source Code

Project Structure

Makefile

Kernel source

src/common/common.h

src/common/common.c

src/syscon/syscon.h

src/syscon/syscon.c

src/uart/uart.h

src/uart/uart.c

src/kmain.c

Running the project

Final remarks and going further

See also

External links

Navigation menu

RISC-V Meaty Skeleton with QEMU virt board

RISC-V Bare Bones

Building a Cross-Compiler

Building a Cross-Debugger (optional)

Dependencies

Acknowledgements

Source Code

Project Structure

Makefile

Kernel source

src/common/common.h

src/common/common.c

src/syscon/syscon.h

src/syscon/syscon.c

src/uart/uart.h

src/uart/uart.c

src/kmain.c

Running the project

Final remarks and going further

See also

External links

Navigation menu

Search