Building Upon Existing Software
Consider not writing your operating system from scratch! There are a lot of possibilities to create your own operating system without doing everything on your own. Great examples would be Mac OS X or Android. Just ask yourself: If even Apple - a great company with billions of turnaround and thousands of employees - decided to make an operating system based on an already existing kernel and userland tools, why would you decide not doing so? Do you think you can do everything better than all those programmers programming Linux, FreeBSD and the like? Sometimes people think they have to write an operating system because of very specific things they don't like. Depending on what you don't like in an existing operating system, there are a number of alternatives to writing your operating system from scratch. If choose one of them, you still can be sure that your task is challenging and you have plenty enough work to do.
If you think you can write a better process scheduler, there would be no need to write a complete new operating system. Instead, you could exchange the one of an existing kernel. An example of this would be the Brainfuck Scheduler that replaces the default scheduler used in the Linux kernel. Most possibly, you will have to use the same programming language than an already existing implementation has.
If you don't like the GUI of your operating system, consider writing anything of these GUI components:
- Window Server: If you don't like, for example, X11, you could write a replacement for that. X11 is currently in use on majority of GNU/Linux operating systems. Examples of new window servers that should replace X11 would be Wayland or Mir.
- Window Manager: If you don't like the way how GUI components are drawn and handled, you could write your own window manager.
- Display Manager: If you think the process of logging in could be made better, you could even write your own display manager or try to enhance an existing one (existing ones are XDM, GDM, KDM or LightDM).
- Desktop Manager: If you think your desktop environment sucks, create a new one or try to replace existing parts of your current desktop manager/environment.
Especially Unix-like systems use the System V method of system initialisation, which includes mounting volumes, starting deamons, bringing up network interfaces and similar tasks. If you don't like how your system is made ready for using, you could replace that init system. Existing alternatives are Upstart and systemd. When the kernel transfers control to the init system, a lot of essential work is already done, like hardware is initialised, you have a running process scheduler, memory management and all those things. But you still have a lot of control over the system and you get it very soonly, a few seconds after the system is powered on. The good thing is that you can use the programming language of your choice, even scripting languages like Python can be used at that point of time. To start your init program if you use Linux, supply "init=/path_to_your_init" as a start parameter to the Linux kernel.