User:Maxvolts
Introduction:
I am a software engineer by trade, and have quite a bit of C/C++ experience, as well as some Assembly and Verilog experience, but haven't done many "serious" projects outside of work. I seek to change that.
These are some of my motivations/inspirations for this experiment (most are also on Youtube):
I develop with Linux for my day job, and every time I use it I ask myself... "Is this really the best it could be? Are we officially declaring this the peak of software engineering?" (Note the lack of even mentioning the other two).
So I want to experiment a little with some thoughts I've had bouncing around. Maybe I'm delusional. I guess time will tell.
I really want to build good abstractions.
If you want to peg me to the developer archetypes, I'm probably Elanore Semaphore (albiet less IPC and synchronization focused) with the downside of Duct Von Tape.
Mono Lizzy and Alta Lang are surprise tools I'm saving for later.
Blueprints OS
Status: Pivoted to Serial output for debug info, debugged the multiboot parser, and drew the OS name on the framebuffer.
Current Goal: Get UEFI functioning. Fix the linking problems preventing 64 bit certainty.
Potential Roadmap (Timescale uncertain):
- Getting UEFI and ACPI functioning
- A graphical environment with mouse, keyboard, and sound
- File System
- Running Applications
- Multitasking
- LLVM Backend (that'll be worth a new wiki article)
- Developer Utilities (LLVM Port)
- Self-Hosting
- Basic Networking
- Basic Games + Non developer Utilities
- DOOM port?
- Release a version to the public. (tirimid?)
And further down the road I really get into some other experiments.
While I'm officially calling this an experiment and a learning experience, I hope to experiment in this direction:
Motivating Theme: An OS that is the best for both developing on and developing for.
If it seems like I want a lot of applications, it's because that'll motivate me to make this very easy to build applications on. After all, very few people actually care about the OS, mostly the applications it supports.
Potential Purpose: Computer Science Education.
Practical Notes on "Modern" Operating System Design:
(Maybe break out into a future wiki article. It'll live here for now.)
Modern hereby being defined as "64-bit with a predominantly graphical environment and UEFI and ACPI support that communicates over PCI".
No DOS-likes here, and little intention on supporting hardware made before 2003 (when x64 was first created. 2011 brings AArch64 and 2014 brings RV64).
I knew going into this that trying to make my first operating system use the newer bells and whistles (as well as straying from the usual gnu stack) would make it harder.
My main hope is that by getting more up-to-date abstractions in place, future work will be made easier.
Here's some lessons learned and sample code to speed past the initial grind. I'm currently still in said grind, so this will need to be updated.
If it seems like a strange chronology, I'm writing this in the order I'm doing it.
Tech Stack in use:
- Windows VSCode running the WSL extension (ubuntu) for IDE and build/run environment
- clang as the compiler (but seemingly ld as the underlying linker) (This is done because it's way easier than trying to coax a cross-compiler out of gcc. WSL and gcc compilation do not mix.)
- GRUB2 as the bootloader, grub-mkrescue to create the iso
- QEMU as the emulator
- OVMF as the firmware
- C++ as the main language
We start from bare bones, so make sure you get that working first.
Swapping gcc for clang is a trivial change for that example. Just make sure to include "-target i686-elf" in your compilation flags, for 32-bit support. 64-bit support is more complicated due to the elf-64 format being different, so hold off on the "-target x86_64-elf" target for now. (he says knowing full well even he hasn't solved this yet).
Swapping C++ for C just needs you to wrap your kernel_main in extern "C" { void kernel_main(){} }, to prevent name mangling.
This gives you an immediate benefit of allowing you to overload functions in the C++. Such as, for example, different terminal_print functions for different things.
Pivoting to Multiboot2:
Perhaps the next easiest changes to make from bare bones is to upgrade your bootloader to multiboot2. This should provide us (later down the line) with utilities such as the EFI system pointer, ACPI RSDP, DHCP information, and a real framebuffer.
Below is effectively a different copy of bare bones, but with multiboot2 support. You can yoink the multiboot2 header from here.
https://www.gnu.org/software/grub/manual/multiboot2/multiboot.html#Example-OS-code
The only details you will need to contribute yourself is a slight modification to your grub.cfg and your linker script .
In grub.cfg: change the multiboot command to multiboot2.
The multiboot2 header above is looking for symbols defining the end of the data and bss sections, but the example code does not provide them, nor does it provide a linker script at all.
Here's where you drop the two new symbols you need
(Pretend this code block is configured for ld instead of asm)
/* Read-write data (initialized) */
.data BLOCK(4K) : ALIGN(4K)
{
*(.data)
}
_edata = .;
/* Read-write data (uninitialized) and stack */
.bss BLOCK(4K) : ALIGN(4K)
{
*(COMMON)
*(.bss)
}
_end = .;
You will ABSOLUTELY need an address tag and an entry address tag to be added. However, try to hold off on the framebuffer tag to preserve your VGA display.
As for the information request tag. You can request the following information
Stuff you can acquire immediately:
1: Boot command line (useless right now)
2: Bootloader name (pretty good sanity check to make sure you're parsing it correctly)
4: Basic memory information
6: Memory Map (though get ready for the annoyance of 64 bit pointers while still in 32 bit mode)
And stuff it might just provide entirely unsolicited:
10 - APM table (soon to be wiped out with ACPI)
Parsing the multiboot2 results:
Parsing multiboot2's provided data is rather simple, even if you don't want to copy their example code. Once you have the multiboot pointer:
Acquire the 32 bit size at that address.
Advance 64 bits (recall that in C, if you're using a pointer to 32 bit data, adding to it adds in intervals of its data size, so += 2).
Loop until you've reached the end of the base pointer + the size:
Get the 32-bit value at your current address (tag type).
Here's what each tag type does:
https://www.gnu.org/software/grub/manual/multiboot2/multiboot.html#Boot-information-format
Switch based on this to determine what to display or do.
- You should expect a load address to always be available (tag 21). Since bare bones uses a relocatable elf file.
- Odds are, your boot command line is just an empty string, so it will have a size of 9.
- Since the Boot command line and bootloader names are literally null-terminated C strings, you can make a char* to their data and print them.
Advance your current address the amount of bytes stored in the 32-bit value after your current address, rounded up to the nearest 8 for alignment purposes.
if(size % 8 != 0){
size += (8 - (size % 8))
}
(With 32 bit pointer addition, make sure to divide this value by four)
64 Bit Notes:
When trying to compile your kernel 64 bits, you may try to use clang as a driver for your linker.
Do not do this. Clang will try to be too smart about the situation and try some optimizations that will not work, resulting in incredibly cryptic errors like
"relocation R_X86_64_32 against `.stack' can not be used when making a shared object; recompile with -fPIC", even when everything has -fPIC.
Call the linker directly and the problem will be solved.
Here's also another place for a sample 64 bit kernel:
https://github.com/davidcallanan/os-series/tree/master
Getting QEMU logs:
Virtually everything we are about to do will wipe out the VGA display we worked so hard to produce in bare bones. In order to get data out of this system, we'll need other routes.
One easy thing to do is to turn on QEMU logging for errors and cpu reset data. Add these flags to your qemu command to get more data.
-d gives the list of things to log, and -D picks the output file.
-d guest_errors,cpu_reset -D ./qemu_log.txt
Serial Port output:
Because VGA text mode will go away with virtually anything we do next, we need another way of getting runtime data out until we can get our Multiboot2-provided framebuffer functional.
/* Print Ok followed by a newline and carriage return */
mov $0x3f8, %dx
/* This is the COM1 port, which in the qemu interface is
tied to the default serial view */
mov $0x4F, %ax /* O */
/* I love magic registers!!1! (Output always comes from ax)*/
outb %dx
mov $0x6B, %ax /* k */
outb %dx
mov $0xA, %ax /* New Line */
outb %dx
mov $0xD, %ax /* We're not in a unix system, so you unironically need to
do the windows thing and also carriage return */
outb %dx
When in your C/C++, use this code from this very wiki: Inline Assembly/Examples#OUTx
static inline void outb(uint16_t port, uint8_t val)
{
__asm__ volatile ( "outb %b0, %w1" : : "a"(val), "Nd"(port) : "memory");
}
/* Set port to 0x3f8 to output to QEMU serial */
Once you've swapped out bare bones printing to screen to printing to serial, might as well throw out the old code and add the framebuffer tag.
Customization Sidetrack:
Who said we can't have a little sidetrack along the way? This is purely for GRUB2, as part of our tech stack.
Looking online for GRUB configuration data is seemingly a fool's errand, as they all are under the assumption you're trying to configure GRUB on an operating system that already exists.
However, with a little digging, you can customize even with as little GRUB as you have, as a treat.
In your grub.cfg, you can add lines of format
set [VARIABLENAME]=[VALUE]
(Note the lack of quotes around the value).
Your options for meaningful variables can be found here:
https://www.gnu.org/software/grub/manual/grub/html_node/Special-environment-variables.html
These four let you change the background and text color on GRUB:
color_normal, color_highlight, menu_color_normal, menu_color_highlight
Specifically, they have a format of textcolor/backgroundcolor, where your color options are pretty slim.
(black, blue, green, cyan, red, magenta, brown, light-gray, dark-gray, light-blue, light-green, light-cyan, light-red, light-magenta, yellow, white)
With a little modification to your menuentry, you can get some neat support.
menuentry "Your OS Name Here" --id yourosnamehere
With that you can add these variables:
set default=yourosnamehere
set savedefault=true
set timeout=5
set timeout_style=hidden
These will cause GRUB to automatically and quietly boot into your OS after a 5 second period. You can jump to the GRUB menu by hitting escape during this process.
If you set the timeout to 0, it won't even wait, but you will need to be pre-emptively hitting esc.
Theoretically you can go further than that
https://www.gnu.org/software/grub/manual/grub/html_node/Shell_002dlike-scripting.html
https://www.gnu.org/software/grub/manual/grub/html_node/Multi_002dboot-manual-config.html
As for themes. You should be able to apply them, but I haven't figured that part out myself because every theme resource assumes you have all of your linux directories and uses GRUB's automatic configuration tools on /etc/default/grub to create your grub.cfg.
In theory you should just put another directory on your iso, then:
set theme=/pathtothemes/themename/theme.txt
With a properly formatted theme.txt file
(Here's formatting)
https://www.gnu.org/software/grub/manual/grub/html_node/Theme-file-format.html
(Here's a more useful page)
https://web.archive.org/web/20241209100014/http://wiki.rosalab.ru/en/index.php/Grub2_theme_tutorial
And that should allow things to work.
But I can't figure that out fully and this sidetrack is already long enough.
To add a splash screen to QEMU before boot, you can add this to the command line
-boot menu=on,splash=Splash.bmp,splash-time=2500
Where Splash.bmp should be 320x240, 640x480 or 800x640. Supposedly it also works with jpeg, but I couldn't see that work.
You can also add your OS name to the window with:
-name YourOSNameHere
And can shift your iso file to be quicker to boot by putting it in a non-CD drive with
-drive file=/PathToIso/YourOSNameHere.iso,format=raw
Getting Firmware in place:
QEMU does not come with a UEFI or ACPI2-supporting Firmware by default. If you were looking for that data in the multiboot tags, you aren't going to find it until you swap the bios out.
We're going to be using the Open Virtual Machine Firmware (OVMF), locatable here: http://www.tianocore.org/tianocore-wiki.github.io/index.html
Something about ubuntu having a signed copy for secure boot purposes. Not yet 100% what that's about, but I'm pretty sure that explains why WSL has a copy of it provided in /usr/share/ovmf.
To be honest, I'm not entirely sure if/how this 100% works. (It also wrecks your GRUB style atm, sorry).
You're going to need grub-efi support on your grub-mkrescue.
Make sure to sudo apt install grub-efi (or grub-efi-amd64 or grub-efi-amd64-bin, I'm not entirely sure which one).
Once you have that all set up, getting qemu to run with the firmware is as simple as adding
-bios /usr/share/ovmf/OVMF.fd
to your qemu command line. (Or whatever path you have to this file. This is where my specific stack makes it easy.)
UEFI:
As for what UEFI is good for. Don't be swayed by the swan song. Once boot services are closed (as any good bootloader should do to give the OS its memory back), like 90% of the specification becomes irrelevant to you.
It's only real applications to an OS are:
- Providing a table of other, more useful, non-uefi tables
- Allowing you to update firmware
- Get/set the system time (with dividing the value up and timezones already handled) and set a wakeup time
- Get the ominous value that increments every single time you boot your system.
- Shutdown/Reboot the system
More info might be coming once I figure out what vendor-defined variables are useful for. The spec doesn't really say.
So while the spec is long and full of tempting abstractions, you can only really use these pages:
https://uefi.org/specs/UEFI/2.10/02_Overview.html
https://uefi.org/specs/UEFI/2.10/04_EFI_System_Table.html
https://uefi.org/specs/UEFI/2.10/08_Services_Runtime_Services.html
Come back to the spec when you're making your own bootloader.
ACPI:
You only really need this one very long page.
64-bit Mode:
This code is cobbled together from a few different tutorials on this very wiki (in progress):
/* Check for Required Features */
/* 1: Can it use cpuid? */
pushfd
pushfd
xorl $0x00200000, (%esp)
popfd
pushfd
popl %eax
xorl %eax, (%esp)
popfd
andl $0x00200000, %eax
jz failure
/* 2: does it have the required cpuid extended parameters? */
mov $0x80000000, %eax
cpuid
cmp $0x80000001, %eax
jl failure
/* 3: Is 64 bit mode available? */
mov $0x80000001, %eax
cpuid
and $0x20000000, %edx
jz failure
/* With 64 bit mode confirmed available, we set things up for it */
/* Set the page table's root */
movl $0x1000, %eax
movl %eax, %cr3
/* put 0x1000 registers worth of 0 in*/
xorl %eax, %eax
movl $0x1000, %ecx
rep stosl
/* Establish Table Links (the 3 indicates it is both present and readable)*/
movl %cr3, %edi
movl $0x2003, (0x1000) /* First entry of top table -> second table */
movl $0x3003, (0x2000) /* First entry of second table -> third table */
movl $0x4003, (0x3000) /* First entry of third table -> fourth table */
/* Set up Physical Address Extension */
mov %cr4, %eax
or $0x20, %eax
mov %eax, %cr4
/* Switch to compatibility mode */
mov (0xC0000080), %ecx
rdmsr
or $0x80, %eax
wrmsr
/* Turn on paging and protected mode */
mov %cr0, %eax
or %eax, 0x80000001
mov %eax, %cr0
/* Load GDT */
lgdt GDT
/* Head to kernel main */
call kernel_main