Security in an OS is a very important issue. It not only includes security from outside threats, such as viruses and hacking, but internal security as well. For example, the OS is responsible for making sure that processes don't access memory outside of their own address space. If a program does so, it must be shutdown to protect other 'well behaved' programs. Operating system security can be divided into two basic categories, high-level and low-level security.
To see articles under this category, browse this list.
Low-level Protection Mechanisms
There are several different low-level protection mechanisms at the disposal of the operating system programmer. The first mechanism, called "CPU Rings" or more simply "rings", controls which CPU instructions are allowed to be executed. The second and third protection mechanisms are related to memory access. They are called "Paging" and "Segmentation" respectively. They control which areas of memory are allowed to be accessed and/or how those areas of memory are allowed to be accessed.
Rings offer a protection layer for programs. They allow certain levels of resource access to processes. This is good, because it keeps bad programs from messing things up. There are, however, several downsides: The more CPU rings you use, the more the OS is tied to the architecture. You can, however, have several architectures each with it's own ring switching code. Another issue with this is that you OS must have a TSS set up and several other features, making ring switching much more difficult than just running all programs in kernel mode. There are a total of 4 rings in most common architectures. However, many architectures have only two rings (e.g. x86_64), corresponding to ring 0 and 3 in this description.
This is kernel mode or supervisor mode. This level has the least protection, and the most access to resources. When starting up, the OS runs in this mode unless it switches out. Interrupt handlers run in this mode.
Rings 1 and 2
These rings are mostly used for device drivers. They offer more protection, but not as much as ring 3.
This is the ring that most OS's use for applications. This ring is also called Userland, or Userspace. It has the most protection and the least resource access.
Most OS's use only Ring 0 and 3. This is because rings 1 and 2 are unneeded, as device drivers can run in either ring.
Sometimes applications need access to resources that their ring won't allow. If they try to access them, a General Protection Fault (int 13) will be triggered, and the application shutdown. The application must interface with the kernel somehow, and mostly this is done with System Calls.
I/O Privilege Level
Another important aspect of the ring permission system on the x86 architecture is the I/O Privilege Level (IOPL). It determines which rings have unrestricted access to I/O ports. It is a two bit number set in the EFLAGS register. Rings with numbers less than or equal to it have full I/O permissions, while those greater than it have none. It can be used on a process-by-process basis to grant certain servers I/O port access.
High-level protection Mechanisms
High level security in an operating system can be accomplished in many different ways. One way would be through file permissions in a VFS. Files in *nix have a 'permission' value in their inode entry. This controls which users can read, write, execute or delete a file. These are mostly controlled by the File Systems.
Programming Language Specific Security (Assembly/C/C++)
Many operating systems around the OSDev community trust user input completely. While it is understood that most users are fighting to get a driver working/stable, it should also be understood that production level code should not contain fundamental security problems. For example, many system call implementations take values off the stack from users and store them into registers and then freely dereference them or use them as lengths, offsets, and so on. Looking deeper into the following subtopics, readers should be able to decipher why this poses a problem and how they can be used to gain elevated privileges inside an otherwise 'secure system'.
Also note that ELF parsers and network stack implementations can pose a large threat. Many OSDev projects trust integers and offsets loaded from ELF files and freely execute and store data at these addresses. Similarly, network protocol parsers are riddled with security-related bugs (mostly integer over-and underflows) that are often used to cause real security problems due to heap-based ramifications. Take a look at the following piece of code:
int size = int_from_user(); char *buf = malloc(size + 1); memcpy(buf, data_from_user, size);
As you can see, if the user sets size to ((unsigned) -1), the allocation will allocate 0 bytes, which on most usermode heaps will allocate space for the headers and no data, causing a full blown heap overflow controllable by the user.
Knowing the x86 Stack, you should already be able to deduce why this can pose a security threat. It is (relatively) very easy to make a stack overflow, so special care has to be taken while implementing one in your OS.
Heap-based overflows are fundamentally different from stack-based ones and are specific to each allocator. You can help to detect them inside your OS by setting "poison values" for each member of the control structures on free and allocation and checking them before operations. Linux uses this approach (see LIST_POISON1 and LIST_POISON2 ).
Integer Overflows do not cause a direct impact such as stack-based or heap-based overflows do. Instead they lead to these types of bugs (usually heap-based) later on in the code. For instance, let's say we have an application parsing data from a protocol or file format:
int size = read_int_from_network(); char *buffer = malloc(size + 1); // leave room for NULL. memcpy(buffer, data_from_network(), size); // copies 2^32-1 bytes into buffer.
As you can see, there is no verification on the size read from the user and this can cause large problems later in the code. Examples in OSDev where this arises include ELF parsers (think segment/program/entry point offset into the file), network parsers, read/write/lseek implementations, and so on. If your code takes input from the user you *MUST* verify that it fits within the range of the data you are going to process it with. Also you should check for integer-based problems on each math operation done using user-supplied data.
While obviously these cannot be directly applied to hobby OSes, the ideas can certainly be used and ported to them. The best set of documents for securing systems can be found at PaX Design and Implementation
Pax provides almost all the operating system security you need. Reading the documents pertaining to its implementations should allow Hobby OS developers to use most of these ideas in their systems. Most current Windows/BSD (kernel) protections are based on these schemes. Userland Protections can be attributed to ASLR, Heap Cookies (along with general heap hardening), Stack Canaries, and NX.
Using these mechanisms does not prevent corruption of in memory data, but instead makes using the corruption for elevated privledges/advantage very difficult.