User:Combuster/notepad

These are my notes from experience. PD code, No warranty, might be inaccurate in places etc - even I mess up things more than once ;). If you feel parts should be in the main namespace, feel free to do so

UEFI

A bunch of notes regarding UEFI and how people have tried hard to make things stupid:

Official documentation

Those morons put a contract on the documentation, so the only one you can download straight from google is an older 2.0 specification. And it's got a fair share of pitfalls.

ABI

The EFI ABI for x86_64 is severely limtedly documented in the UEFI specification, which leads to issues where you are not expecting them. Notable problems:

• You need to "push" all arguments on the stack - including all arguments that you also pass in a register. The function should use the register version, so you can leave anything in those holes, but many functions will simply write back the arguments to their locations. If you're not reserving space, that means your stack gets overwritten.
• It does not state that you should maintain 16-byte alignment.
• The documentation gives you 4K stack minimum, but doesn't describe ownership for anything past the top of the stack. In fact, there's no red zone and interrupts will simply jump to the firmware and overwrite anything you put past the stackpointer. If you internally resort to the AMD64 ABI, make sure you pass -mno-red-zone

Basically, it boils down to the fact that the EFI ABI follows exactly the Microsoft x86_64 ABI, including everything it didn't copy over from the original documentation (and all associated surprises thereof)

EFI_BOOT_SERVICES

In the table, there's a missing pointer, which means everything you use past it has wrong offsets and therefore calls the wrong function

EFI_UNINSTALL_PROTOCOL_INTERFACE UninstallProtocolInterface;
EFI_HANDLE_PROTOCOL HandleProtocol;
EFI_REGISTER_PROTOCOL_NOTIFY RegisterProtocolNotify;

This should be something like

EFI_UNINSTALL_PROTOCOL_INTERFACE UninstallProtocolInterface;
void * PossiblyHandleProtocol_1;
void * PossiblyHandleProtocol_2;
EFI_REGISTER_PROTOCOL_NOTIFY RegisterProtocolNotify;

gnu-efi

This is the biggest hack and worst teaching material in the book. What it does is employs a gigantic series of hacks to make linux binaries run on the EFI ABI. It strips relocations, forges fake relocations to make EFI believe the binary is relocatable, then performs manual ELF loading and the relevant self-modifying-code hacks to get all the addresses back in order when it runs. It also has wrappers to toggle between AMD64 and Microsoft ABIs

Symptoms are broken output binaries for x86_64, and intermediate files that cause buffer overflows in objdump so you can't properly inspect them, and way too much support code needed to even get an empty binary working

Instead, use a PE-targeted cross-compiler. i386-pe works for 32-bits; x86_64-pe gives a functional binutils but no gcc; a mingw/cygwin compiler is technically wrong, but probably works better already. For both platforms, just passing "-shared -dll --subsystem 10" arguments is sufficient to get the expected output. But don't use it together with gnu-efi or the proper relocations get messed up with the fake ones and the firmware suddenly stops being able to read the file (as well as objdump).

emu woes

There has been a broken amd64 build of the OVMF bios posted on the website. Right now it works. Most tutorials around the web are well out of date though: you only get one binary instead of separate firmware and plug-in roms (so there's no VGA rom!)

I've seen both work with both Bochs and qemu. In bochs, you seem to need

• The right bios image
• 440FX support compiled and enabled
• Cirrus Logic compiled and enabled (although not necessarily with the newer bioses)

In qemu:

• "-bios path/to/ovmf"
• "-vga cirrus"
• "-no-kvm" where needed
• "-net none" to cut off minutes of boot time waiting for netboot
• "-hda fat:(...)" to create virtual harddisks without building entire images

HD construction

This can perfectly be automated. No superuser access needed! Tools: mtools, parted, dd (clean up and copy instructions here)

Graphics Card design

A graph I made to show many of the components you might find in a video card. Note that the data routing is in many cases much more specialized, with dedicated routes between certain components (like direct connections between the GPU and dedicated texture memory)

Considerations for graphics development:

• Several chips have fixed pixel clocks, which use a fixed frequency. (VGA, older adapters)
• There may be many kind of devices connected to the output, VGA and vga-capable are common, but digital outputs are not uncommon
• LCDs have a native, if not fixed resolution
• Not everything has a refresh rate (Terminals, Non-VGA LCDs, The BGA, etc.)
• Not everything supports different refresh rates (VGA, TV standards), and those who do have a limit (All CRTs and VGA/DVI LCDs)
• Not all chips and modes output a single pixel per dot clock cycle (VGA does 1 clock per 4 bits, 1 clock/pixel in 16 color, 2 clock/pixel in 256 color, similar effects with SVGA controllers)
• Not everything is a framebuffer, including tile engines (VGA text, VT), sprite engines (console hardware), up to DSP-like engines.
• Video cards have a n:m mapping to display outputs - video cards have zero(!) to multiple outputs, SLI'ed cards have effects on adapters connected not to itself.
• It is 1 am - do you know which display engine(s) on which card(s) are currently pretending to be VGA?

E820 woes

I recently discovered a broken bios that did not fill in the mandatory part of the structure (but left parts of it intact). Turns out GRUB doesn't fix this behaviour and that all multiboot kernels would consequently get incorrect data about the memory map.

A more complete list on bugproofing calling 0xE820:

• EAX = 0x0000E820 (Ralph Brown: call may be ignored if it isn't)
• The list can be terminated by either EBX=0 (where you need to parse the provided result) or CF=1 (where you should not)
• reserve at least 20 bytes (Ralph Brown: some bioses will always write 20 bytes)
• make sure the area receiving the structure is zeroed before use. (Undocumented)

Porting the FreeBasic Runtime

To get past deprecated versions and everything, I updated my FB installation from 0.15 to 0.20(!) As a consequence, I got the updated library and since I couldn't be bothered to merge all the differences, I had a go from scratch. I decided afterwards to record the general process here.

starting off

1. unpacked libfb-0.20b to a temp directory, copied the rtlib directory over to my workspace
2. got rid of the automake garbage :)
3. performed some shell tricks to get a list of c files into the directories Makefile, so that I only needed to add that bit of building rules
4. integrated the stuff into the build system.

Well, to start things off, I decided to compile the first c file in the list to see where things got awry, and was greeted by several screens full of errors.

Shaddap, I'm developing an OS here O.o

Starting at the top to work things out, there was a nice #error message, telling me that the platform was unsupported. No duh. Edited fb.h and removed that silly statement.

Then came the garbage from the C library - wide character support was *not* enabled and FB can get pretty nasty over that. So I stripped the fb_unicode.h header, and all the system-specific stuff not available in freestanding land and ended up with fb_file.h fb_device.h fb_serial.h fb_thread and fb_printer.h commented out. Next I went down the other files and commented out the remaining declarations with wide characters.

Some remaining functions were found and removed, and at some point the first function compiled.

Getting to the bone

Obviously, I tried next to compile all parts of the runtime library, which obviously didn't like the fact that lots of pieces were disabled. And of course the missing OS support. Some 50% of the files refused compilation, and to test for sanity, I shoved the remaining stuff into a single binary to see what was actually working. Of course the linker protested and more files were disabled

After disabling 99% of the source files, making mental notes of what was needed, some basic memory moving stuff was working. most string functions were failing with a undefined reference to __fb_ctx, the array functions with error handling code.

__fb_ctx and fb_hInit()

after a nice grep I located the definition in libfb_init.c this file compiled, but didn't link because of an OS dependency to fb_hInit(), with a comment belonging to host specific initialisation.

Since I needed nothing that the C runtime already provided for me, I got away easy with a small support file.

#include "../fb.h"

void fb_hInit( void )
{
    return;
}

afterwards, most of the string functions could be enabled.

Error numbers

The other keystone to the language are the array functions. The required source file that went about this was libfb_error_getset.c, which would provide the calls, but refused to compile from scratch, due to a missing ERROR constant. Simply setting ERROR to a constant resulted in a missing definition, which was resolved by re-enabling fb_thread in fb.h which in turn (what else) gave trouble. After removing parts of an enum with more missing macro's, all that was left was again a depency to what appeared to be a host support function. This time dealing with thread local storage.

Since I didn't want to bother with the threading support, I added a stub that would fix the messages and do something useful. Basically it returns the error context for this specific thread.

#include "../fb.h"

FB_ERRORCTX __fb_errorctx;

FBCALL void *fb_TlsGetCtx (int index, int len )
{ 
    switch(index)
    {
        case FB_TLSKEY_ERROR:
            return(&__fb_errorctx);
            break;

        default:
            while(1);
    }
    
    while(1);
    return(NULL);
}

Since the ERROR macro made no sense, I changed that too adding this into fb_error.h (replacing the previous ERROR macro I put there):

#define ERROR FB_TLSKEY_ERROR

Afterwards, most array functions were operational, and some other random functions started working too. With runtime support for the two most used primitives, I decided to call it a day and commit the changes.

DIV test

originally devised to tell apart amd/intel/cyrix/whatever 486s and 386s that didn't have cpuid, this still works on current hardware. You can detect spoofs (emulators?) by verifying the div test signature with the expected one.

recent intels maintain all flags after DIV, as do bochs and qemu by default. cyrices clear most flags, AMDs set some and clear some.

PUSHFD
POP EAX
; either clear flags for first test, or set flags for second test. pick one:
; AND AX, (0xffff - 0x8D5)
; OR AX, 0x8D5
PUSH EAX
MOV AX, 5
MOV CL, 2
POPFD
DIV CL 
PUSHFD
POP EAX
AND EAX, 0x8D5
;AX contains signature

in 16-bit mode, use popf and ax instead of popfd and eax

results I've gathered:

CPU	Div test with all flags clear	Div test with all flags set
Intel 686	0x000	0x8D5
AMD K6-II	0x010	0x811
AMD Athlon64	0x010	0x811
Cyrix Cx6x86 / M1	0x000	0x801
Emulator	Div test with all flags clear	Div test with all flags set
Bochs	0x000	0x8D5
Qemu (without kqemu)	0x000	0x8D5

Still looking for information on the 386 and 486 generation...

Bochs messages

A list of messages and likely causes:

interrupt(): gate descriptor is not valid sys seg

You have not loaded an IDT, or the IDT is corrupt

interrupt(): SS selector null

• You have no TSS
• You haven't set SS0 / ESP0 in the TSS

CR0 = 0xe0000001 CR2 = 0xe0000001

• Your page tables are not page-aligned
• Your page tables do not point to the correct parts of memory

MOV_EwSw: can't use this segment register 1

• MOV CS, ... is not a valid instruction, except on an 8086 where you don't want it either because it doesn't set IP.

prefetch: getHostMemAddr vetoed direct read, pAddr=0x0000000a0000

• You jumped to the wrong address.
• Your page tables do not point to the correct parts of memory
• You're trying to run code in VGA memory, which is not allowed

HPPA

That big noisy RISC machine from HP. Oh well. Needs some patching to get a elf toolchain built.

apparently there are two file formats at work here: ELF and SOM. if you mix up those two you get no end of assembly messages (unknown pseudo-op .nsubspa etc). the following got me a successfully built gcc. I have not tested its actual operations for sanity. Just doccing it here so I can reproduce the results on a computer that's closer to my HPPA machine.

add to gcc/config.gcc

hppa-*-elf*)
	target_cpu_default="MASK_PA_11|MASK_NO_SPACE_REGS"
	tm_file="${tm_file} pa/pa32-regs.h dbxelf.h elfos.h pa/elf.h \
		 pa/pa-pro-end.h libgloss.h"
	tmake_file="pa/t-pro"
	use_collect2=yes
	gas=yes
	;;

run

export PREFIX=/usr/cross
export TARGET=hppa-elf
mkdir build-gcc-hppa
mkdir build-binutils-hppa
cd build-binutils-hppa
../binutils-x.x/configure --target=$TARGET --prefix=$PREFIX --disable-nls
make
make install
cd ..
export PATH=$PATH:/usr/cross/bin
cd build-gcc-hppa
../gcc-x.x.x/configure --target=$TARGET --prefix=$PREFIX --without-headers
            --disable-nls --with-newlib --enable-languages=c
make all-gcc
make install-gcc

works with binutils 2.17/gcc 4.2.2 fails on binutils 2.17/gcc 4.1.1

The 15 minutes spent trying to get palo (the opensource bootloader used with linux) to compile resulted in an asm error in crt0.s and several missing system headers (guess what, PA-RISC + LILO = PALO. linux.h missing. d'oh).

And that was one step out of the many that need be taken...