Orion

Compiling Andromeda

2012-05-20T13:04:00.003+02:00

Andromeda is an interesting project, but in order to see what we're up to, you might just want to look beyond the code and into the binary.

As of yet, there's not that much going on, but the mere fact that we've got the possibility to boot, handle key-presses and actually have scrolling text on our screens is already quite a feat, for first time kernel programmers like us. Just imagine that the stuff going on in the background is a thousand times more epic than what you can see on screen.

This article will guide you through the building sequence for the Andromeda kernel project. We will start of by getting our tool chain in place. We won't go into detail on how the tools work by them selves. That's what Google is for.

Once we've got our tool chain in place, we'll go into basic compiling. With that behind us, we'll look at the option for providing flags, and last but certainly not least, we'll go into booting the kernel.

Tool chain

To compile Andromeda, we will need the following:

gcc
nasm
GNU-Make
ld
git

The compiler, assembler, make and linker respectively. For the ones that don't know what the different tasks of those tools are, here's a brief summary:

GCC

The compiler compiles C code, and translates it into binary object files. The C compiler doesn't know about the layout of the final layout of the binary image. All it knows how to do is translate C statements into machine instructions.

NASM

The assembler takes loose human-readable machine instructions and translates them into machine-readable machine instructions. The output is surprisingly similar to that of a compiler.

Make

Make is the tool that is the conductor of this small orchestra. It tells the compiler or assembler to translate their code into binary instructions. Once this is done it tells the linker to put it all together.

LD

The linker is the pasting tool. Until this point our sources have been translated into several binary images, but none of them are executable, because they have references to other files, which need to be linked together first.

Git

Git, the stupid content tracker, is a quick and easy to use version keeping system, that's used by projects such as the Linux kernel and Android. This will be used to get our code, and maybe even do some work on it. Who knows?!

Compiling

In order to build the kernel, we'll be issuing the make command in the src directory of the repository. This command will read the Makefile and based on what we've got there it will determine how to compile which source file.

Make takes some arguments. One of them is "-s". This silences make, and keeps the terminal relatively clean, and easy to read. Only warnings, errors and verbose messages will show up in the terminal, keeping it easy to follow what's happening.

Another argument is the "-j n" option, in which n is the number of jobs with which you want to compile the kernel. Generally to optimize the compilation for your system, it is best practice to take the number of cores (if you have hyperthreading, you can use that) and double it.

For example I have a core 2 dual, with 2 cores, so my most optimized command is:

make -sj 4

Compiler flags

To activate or disable some functionality in the kernel, there is a possibility to add some flags to the compilation. The easiest way to do this is to put the flags into the flags variable you hand to make. To enable a feature, the -D flag can be used. To disable, the -U flag is to be used. So for example to compile the slab allocator into the kernel the following command can be used:

make FLAGS=-D\ SLAB

make FLAGS="-D SLAB"

Work is under way to building a kernel configuration editor, so not all flags will have to either be looked up or be remembered. For now, to find out which flags are available, key in make usage.

Booting

There's a couple of different ways to boot Andromeda. The first and easiest is through the make test command.

Make test builds the source tree (FLAGS variable works with this target as well) and then tries to run it in an Qemu/KVM environment. Qemu has a pretty convenient option to emulate grub, which we take advantage of here. For those that don't know what Qemu is, it is a virtual machine manager, and KVM means kernel-based virtual machine (the commands can pretty much be used interchangably).

Another way is to create a director at /media/loop and download a floppy image from github. First you build the kernel, according to the instructions above, and then the image is put into the src/scripts directory. Next we issue the ./updatefloppy.sh command, which will ask you for your sudo password (please install sudo for this, if you haven't already).

To recap, the options are:

make test

make
cd scripts
wget https://github.com/downloads/Andromeda-Kernel/andromeda/floppy.img
./updatefloppy.sh; kvm -fda floppy.img -m 16M

What is virtual memory?

2012-05-03T16:21:00.000+02:00

Lately I have been working on the virtual memory system. Now most of you out there won’t know what virtual memory is, so here is a brief explanation.

Virtual memory is the mechanism used to make all tasks think they’re the only task running on that system at any given time, unless communication through memory is desired, in which the operating system makes that possible through a mechanism called shared memory.

That’s virtual memory in one full sentence. If you still don’t get it, that’s perfectly understandable. So here is an explanation which is a tiny bit longer. It explains the goals of virtual memory and explains different approaches to solving the very same problem.

When writing the code for any particular task on any system it’s a nuisance, to say the least, to have to keep the possible existence of other tasks on that system into account. The sole role of the operating system is to help other tasks run, and so what it does, is try to give the entire memory space, to each and every task.

While the whole memory space won’t be possible due to technical restrictions (the kernel itself must be somewhere in memory as well), quite a lot can be made available to user space (that’s where user tasks run).

One of the ways to get this all to work is to have each task ask the kernel for more memory each time they need it, while keeping all the data available to all the other processes. Sometimes, on architectures which don’t support memory protection, this is the only way to go forward.

Another way is to have the processor translate the virtual addresses into physical ones while the kernel concerns itself with the allocation of physical pages (a page is a chunk of memory of a predefined size). When a process isn’t allowed to access a particular physical page, it is simply not mapped to any virtual address at the moment that task is running (again, besides the kernel, but this region of memory has other ways to protect itself).

The advantage of the latter approach is that in this model it isn’t possible for one task to modify critical data or code in the other task. When it is desired to have two tasks sharing a region of space, that can be accomplished by mapping virtual pages in both tasks to the very same physical pages.

Printf flavours

2012-04-19T14:27:00.001+02:00

I’ve been working on several things in the kernel lately. Among them are the slab allocator and the different flavours of the printf function (some of them).

I’ve now finished work (mostly) on these printf functions and I’m happy to present:

sprintf,
vsprintf,
fprintf and
vfprintf.

These functions have better formatting support than the current printf function. So in order to use these new formatting features one will have to make a string, format it using sprintf and then print it using printf.

As soon as we have a better self cleaning buffer in the kernel, we’ll implement it in the vga-text driver and make a new printf function that uses fprintf to write to stdout (which is connected to the driver).

Also work on the slab allocator is progressing, albeit slowly. There’s a lot of thinking involved in building a slab allocator, and I’m hoping to do it well the first time. This means there’s even more thinking involved.

Also this slab allocator will have to be used for page allocation, which makes the slab allocator come with a couple of restrictions (we can’t just assume we’ve got space everywhere outside of the text segment and annoying details like that). So for now we’re still using the slob (single list of blocks) allocator to do all the memory allocation for us, but we’re looking forward to a functional slab allocator (probably done in a couple of weeks).

That’s it for now!

OSdev resources

2012-02-11T12:01:00.000+01:00

I know a couple of people interested in OS development but no clue on where to start, and even though I occasionally show them one or two of the sites I get my information from this blog post is supposed to give them a better set of pointers to documentation.

I started programming in C using a good tutorial on cprogramming.com. Now I understand if this isn't enough for you and one book I highly recommend is the C programming language by Kernighan and Ritchie. Those are the two men behind the language and they've found a way to describe the language in a simple and brief way.

Also something worth learning, although not absolutely necessary when joining an existing project is a form of assembly language. I personally got started with the art of assembly, and honestly haven't found a better resource yet.

Now we have some grasp of what it means to be a programmer, we can start thinking about a simple kernel. For that we'll go to a man called James Molloy. He's written a tutorial which is very easy to understand. It might not be the most powerful and flexible kernel out there, but it'll get the job done for a first kernel. Also there's this beautiful wiki on how to do some things not covered in the tutorial.

Another tutorial that may interest you is one that follows the Windows path a little bit more as opposed to the Unix route used by James Molloy. I'm now talking about the brokenthorn web book. This is one of the first tutorials I've found on the subject and although I didn't take it's path it has taught me quite a lot.

The VFS

2012-02-04T21:47:00.002+01:00

The virtual file system(VFS) is one of the most important parts of the operating system. It handles communication with permanent storage. In our case the VFS doesn't handle the file operation them selves but requires the file system to give us function pointers to work with.

A file in our VFS is nothing more than a data structure with a pointer to the file system driver file, which itself is attached to one of the permanent storage devices. This might seem complex but it does provide us with a whole lot of possibilities, which are provided not only by our system but also by others like Linux and BSD.

The file system mounts still have to be written though so I still can't go too deep into the implementation details of those.

One thing we already have working though is a special kind of file we call a buffer. When we open a file with the initialiser function of the buffer it loads all of its functions into the function pointers of the buffer and now allows us to read from, write to, seek in and close the buffer like it's a normal file somewhere on a disk. Internally it keeps all the data in the form of a tree and when it's time to close the buffer, while no other part of the application has opened it, it will remove all of it, which is quite unlike a normal file.

Because of the index variable, which is 64 bits, the buffer supports files of up to 16.000 PiB. If you don't know what that means. Well, lets just say that it's plenty for the coming decade if not more (I personally have trouble filling my 320 GiB, and that's roughly 0.002% of the buffer …).

The driver model

2012-02-04T21:47:00.000+01:00

Since the beginning of this year we've been busy in the Andromeda team. One of the things we've been working at is the driver model. Our design is fairly simple, and I think I can explain it, so here we go.

The device model consists of a tree of devices. Starting with the root device. This is a virtual device to which everything is attached in some form. Attached to this device are for example the CPU's, memory, the PCI bus and some virtual buses.

The reason for choosing a tree instead of a plain list of devices is simple. When the system is to shut down or to suspend, we want to disable the devices first, then the buses and last the power supply (if necessary). If we're to walk a plain list things are to shut down in random order which might get interesting since for example the PCI bus has been shut down before the graphics card is so it never receives the shut down signal.

The reason why the CPU's, Memory and PCI are attached to the root device are simply the fact that they are at the head of the system. The reason for the virtual buses might not be so obvious yet.

There are 2 virtual buses. One for virtual devices which reside in memory and don't really have to be suspended, and can't have physical hardware attached to them, and then there's the legacy devices. These can be the VGA and PS/2 controller to name a few.

Devices in the model are nothing more than data structures with a file pointer, open function, name/unique identifier, driver pointer and a void pointer for special data structures. When interaction with the devices is needed, the device file can be opened and written to. The device might respond and by reading from the device file the answer can be retrieved.

In the case of a permanent storage device this file can hold pointers to partitions which can in turn hold files which can then be mounted to the VFS.

What we've been up to

2012-02-04T21:46:00.000+01:00

I know, it's been a long while, and quite a bit has happened since.

A summary of what's been done:

A driver model has been designed and implemented and is nearing completion (aside from the actual implementation of drivers).
A virtual file system has been designed and implementations are on the way.
Network stack development is currently being done by Michel.
Ideas have formed on a new memory allocation algorithm.

Now this doesn't sound like much but for two people it's quite a bit of work, considering the fact that in the design of core features in the kernel we prefer to guarantee quality.

In following posts we'll go deeper into the separate items.

Cleaning up

2011-12-21T16:34:00.000+01:00

Currently a lot of work is going into getting the repository clean again.

A lot of files have been placed in directories where they shouldn't be. For example, the directory src/sys shoudn't have existed.

We've been moving most files out to where they ought to be. For example, the error directory, which contained the panic function, has been moved to the kern directory, which by the way has been renamed to andromeda.

This generated a file name conflict with the output binary, and as such we renamed that to andromeda.bin.

In the mean time I've been working on support for multitasking. We've covered quite a lot of ground there, but as we speak a lot of code still has to be written. For example, we still don't have the possibility of switching contexts. Something I think is mandatory for multitasking. Now of course it can be done through some weird function pointer using scheduler, but I personally prefer to go with a traditional task switch.

What does that mean?

Well, basically it means that when an interrupt comes along, before we do anything else, we push all our registers to stack. When the interrupt is handled a couple more functions are called, probably pushing more onto the stack.

When the time comes to switch tasks, what we do is we only store the stack pointer, just before the moment we swap the memory context, by loading another page directory pointer.

When we have to load the old task again, all we have to do is first restore the directory pointer, and then use a link in kernel space (which is always present) to restore the stack pointer.

When we then return to the interrupt handler, the registers will be restored to the state when the process was interrupted and the interrupt will be finished.

Basically that's the normal task switch. Does it sound complex? A little. Does it have to be this way? No, but it sure is the most extensible way of doing things. When a new register is added, all we have to do is remember to push in onto the stack in the interrupt handler and the rest can be dealt with in a more generic location.

Out with the ugly hacks

2011-12-07T00:15:00.001+01:00

Considering the present time (0:14, at UTC+1) I'm going to keep this one short.

I've been working on getting scheduling working, and I found the GDT trick to be a little bit of a nuisance. As it turns out, I had a design in mind as a replacement for this trick.

What does it do?

Basically what we want is to have the kernel start at somewhere very high in memory, both tricks allow us to migrate the kernel code without endangering our more volatile code.

The GDT trick requires us to exploit some feature in the CPU which isn't guaranteed (as far as I could find). Now this doesn't sound very nice, and isn't really portable too.

Now what I did was remove this trick and enabled paging very early on. This means that paging will actually replace the function of the GDT trick. Now this was kinda tricky since booting with the GDT trick might require one to do some very nasty stuff later on.

This code still exists, so I had to enable that in the paging trick. Basically what we have now works.

More info on this can be expected later.

Still waiting

2011-11-23T20:17:00.001+01:00

So we didn't quite reach our deadline for November 4th.

I think it has been a little too ambitious, considering the situations all developers are in now.

So we're definitely working towards the 0.1.1 release, and among the things we already mentioned would be in there, we'll also feature file buffer support.

Now that doesn't say a lot though. Basically all it says is that we have something that resembles a file, which you can write to, and read from. Seeks are also supported, but unfortunately, we can't sync with disk, or even clean up read items in a one way stream/pipe ...

Michel still is working on his PCI and APIC support and is thinking of including C++ code into the project (I wish him luck implementing the special operators, I'll continue doing C).

Doubted is whether or not we will be able to put PCI and APIC into the next release. There'll be a small discussion on that and we'll get back to you when we've got more news about the next release.

Until that time, the developers will be working on both school and the kernel.

Also, I've managed to set up gitweb on my own server, so if you like you can subscribe to the rss/atom feeds on there: http://orion-os.eu/gitweb/

I hope to have more news soon.

A new release already?

2011-11-01T20:55:00.001+01:00

A new release is coming up on the horizon. We now have the date of Friday 4th of November 2011 planned for the release of Andromeda-0.1.1.

This will feature a brand new paging system, a new memory mapping system, minimal ACPI support and basic support for the Advanced Programmable Interrupt Controllers (APIC).

The paging system has been made in such a fashion that the memory requirement scales to the pages requested, instead of permanently occupying 4 Megs of RAM.

The same goes for the new memory map, and did I already mention it is based on the size of the actual physical memory!

This means that we always occupy 0.2% of physical memory available for the memory map, and depending of the amount of virtual memory usage a little bit more for the page tables.

Also we've conjured up a way to put the kernel at 3 GiB. It's called the overflow trick. The principle behind it is pretty simple. All you do is set up a segment which maps memory address 0 to the 3GiB location and make it 4 GiB long or so. Then you jump into the higher half code.

Simple as that. All you need to do is page the code to both low and high memory, disable the overflow trick and unmap the low memory code, leaving only high memory in place.

Whenever some low address is required, all you need to do is add 1 GiB to the address and the 32-bits integer flows over when the segment is applied, and the correct pointer is used.

Simple, stable and effective (do watch out with virtual machines that don't properly emulate the 32-bits overflow, they can be a real pain sometimes).

The ACPI support is created by Michel Megens, so not much for me to mention on that part. The same pretty much goes for APIC support.

And speaking of dates! My birthday was last Sunday (October 30th) at which I turned 19!

Continuing to move

2011-10-16T23:28:00.000+02:00

So I found this bug I was telling you about last blog post. It basically boils down to me having forgotten to map the entire lowest 15 MiB.

That means page faults, which cause a double fault (because the page fault isn't mapped), which causes a triple fault (reboot, because the double fault isn't mapped).

That's nice, I think, but now I have that settled, I need to remove it again after I disabled segmentation, otherwise we've got a security leak on our hands, greater than the great wall of china.

In the mean time I am trying to build a way to keep track of the memory that is mapped, and the memory that isn't. At the moment I've got a system in place that produces 0.29296875% overhead on the entire memory system, at MOST versus a huge 6.6666....% using the old system (and yes I used a calculator for that! I'm not that smart).

This is without counting the heap overhead, which go through the roof on very small allocations, but because that's too unpredictable I've chosen to only count the paging system in.

What needs to be done now?

First of all, the page fault system is to be written. This will in the future include swapping, but since file systems aren't supported yet, a swap file would be pretty useless. So if we run out of memory now, we just panic!

Second the lowest 15 MiB must be released again. Once that's done, we can start doing work on getting all the features we broke by this, to get to work again.

I see a mammoth task in the future, which means I'm happy it's nearly autumn brake.

In the mean time Michel is working on the ACPI tables and stuff like that. I hope he'll get to explain it to me in the near future. I sure wish to know this.

Also I'm getting an Arduino board. Just for playing around. Maybe I'll attempt to port Andromeda, if the chip is even capable of the tricks we require.

And there is a new server coming up. It's made with absolutely NO moving parts so that I can keep it running longer.

Up, up and, well, you know the story

2011-10-08T00:28:00.001+02:00

Basically what I've been working on is getting back to the higher half mode.

The trick I've chosen this time is the GDT trick, designed by Tim Robinson.

What this means is that the image is linked to be at 3GiB and then we use paging to trick the CPU into thinking the code is where it should be.

This is nice and all, but it does make things with physical memory addresses a little harder, and since that's the case I still haven't been able to get paging to work.

The weirdest thing is that the debugging values seem to be correct, while the virtual machine still triple-faults (for non OS-developers, that's basically the PC throwing it's hands up in the air, saying: "I can't fix this, do it your self!").

In the mean time Michel has been continuing work on the ACPI tables.

Besides that there is lack of a clearly defined website to go to for the kernel.

We're working on that, but do keep in mind that we have higher priorities at the moment (such as school).

A new release!

2011-09-29T12:02:00.000+02:00

The 0.1.0 release is here! Time to celebrate!

But exactly what are we celebrating?

Well, in the 0.1.0 release we've successfully merged Andromeda and OpenLoader under the name of Andromeda.

If you were to look at the code, there still is a lot with of code with ol_ prefixes. This is legacy, and although re-factoring is a possibility, we're not actively going to search for this.

Also I want to give credit for at least half of the functions implemented in the 0.1.0 release to our new co-project-leader: Michel Megens.

Below is a list of what has changed since the 0.0.4 release in what we call Andromeda.

Better heap allocator

Heaps of bugs have been fixed

New paging system

A complete rewrite
A new page frame allocation scheme
A page mapping scheme
Work is to be done on integrating the frame allocation and mapping

A start on a virtual file system

Files are implemented minimally
Directories have been implemented
File references need to be created
Mount points aren't done yet

A possibility for very very low resolution graphics

VGA at 320*240 like resolutions
Work is being done on 800*600

Better keyboard support

Capitals and backspace working
PS2 controller now supported!

A start on disk drivers

ATA support now has a stub

Error code centralisation

Error codes can now be found in a single file.

The removal of floppy.img

Make test works again.
floppy.img no longer required to test
Repository is cleaner

A start on the ACPI

Added ACPI data structures
Implemented ACPI table search schemes

A start on the APIC

Detection implemented but we're still using the legacy chip.

A better CPU interface

The CPUID instruction now has a better implementation.

PCI support added
Multi processor support now has a stub.

On the move

2011-09-24T13:04:00.000+02:00

The project is going forward steadily and while we're still merging, some things are already moving forward again.

One of them is the paging subsystem. Well, actually it's gone backwards but that's mainly because it's a rewrite (can't clean up without making a mess first).

One thing I'm working on is getting the pages to be put in the page tables to be put in a page directory (yes, Intel couldn't make it simpler).

That's not really my issue here, but what is my issue is that I have to find out a way to get the pages to be marked free and used, without requiring 4MiB as I used to.

I have got some designs ready, one of which is the old design, which was underwhelming to say the least. Another is a simple bitmap, with the processes maintaining their own page locations. The last one is to have a list of regions, which is a doubly linked list so the processes can also mark the pages as used for them selves. This does however put me at a 8MiB max. The sunny side, however is that there also is an 8 byte minimum, and that with a bit of work, the fragmentation could be resolved, to make the entire requirements just a bit smaller.

Now this 8 MiB max is only on 32-bits systems, as on 64-bits this can be doubled. The bitmap approach has got several advantages, one of which is that it's small and easy to maintain. That is until we're going to think about swapping pages out to disk. At that moment, all hell breaks loose, and code is flying around randomly.

So my choice is this: "Am I going for low memory usage, easy swapping or easy maintenance?" You'll hear the answer soon (hopefully).

The dust is settling

2011-09-20T13:40:00.000+02:00

While the dust of the cosmic collision starts to settle we're still a long way away from the total merge of the projects.

At the moment it's more the drivers that are being implemented in the Andromeda kernel. Michel is the one responsible for that.

In the mean time Steven feels like he can't do a thing since the change is going so rapidly.

And me, what am I doing?

Well, I'm primarily focussing on the same stuff as before the merge and that is getting the paging system up to par. At the moment work is being done on the basics of the paging system and while I feel it isn't sufficient in the long run (when this might be ported to 64 bits systems) for now it'll do and will probably even be good (according to the standards we're working with now).

So what is going to change?

To start with work is being done on drivers such as Intel's Advanced Programmable Interrupt Controller or APIC. In the mean time work also is being done on ACPI or Advanced Configuration and Power Interface (which is a disaster to work with). That is what is imported from Openloader.

In the mean time we're also taking over the Openloader boot procedure and this is also a work in progress.

Also we're doing work on getting the kernel into higher half, which is where the new paging system comes in. It has to support identity paging which is that the same page can be referenced by multiple virtual addresses. This, in combination with relocatable code and a whole lot of rather simple arithmetic on relocating the heap, will allow us to relocate the entire kernel into the 3GiB region or the -2 GiB region on AMD 64 systems.

Work also is being done on the graphics systems, but that is basically what has been going on all the time since Steven joined, and we're looking to merge the makefiles.

The heap code had some significant bugs, which are now resolved, but no guarantees can be given that it is now bug free.

So while the dust is settling, the merging is far from complete and the legacy of the two projects will probably remain visible for months, if not years to come.

Cosmic collision

2011-09-11T17:12:00.000+02:00

This blog post marks quite an important moment in what now used to be Andromeda as we know it, since we're doing something bonkers.

Basically what we're doing is calling the boot loader by Michel Megens the Milky way and sped up time.

If you're not aware of what's going to happen in a couple of million years, well, here's a quick explanation: "Boom!"

If that doesn't cut it for you, well, Andromeda and the Milky way are both galaxies (not sweets) and Andromeda is the only galaxy moving towards the Milky way (aka us).

This means the two will collide in a couple of million years and the two galaxies will likely merge.

That's exactly what Openloader and Andromeda (the kernel) are doing. We're fusing.

Since the end result will still be a kernel the name of the kernel will persist, but the project structure will likely change.

In galactic terms this means the two black holes at the centre of the galaxies will revolve around each other but not merge. In project terms, this will mean we have two project leaders.

Michel being involved with the direct hardware interfacing, while Bart will be focussing on the higher level algorithms such as scheduling, paging and other kinds of memory management.

The merging of two galaxies is a lengthy process and often messy. I don't see why the merging of the two projects will be different, but all together we will likely gain more mass and momentum (at least that's the general idea).

You can still use https://github.com/bemk/andromeda as main repository, but the official one now resides at https://github.com/Andromeda-Kernel/andromeda.

We hope the dust of this collision will soon settle down so we can continue what we were doing before this.

Bugs, bugs, bugs and more bugs

2011-08-31T09:34:00.000+02:00

Yeah, the title says enough, I'm afraid.

I think it would be an understatement to say that we found SOME bugs last summer break. We found a lot.

First up was a series of bugs related to the fact that the ELF loader didn't fully comply with the standards. Once that was found it was an easy fix.

Further, what we've unearthed is quite an impressive bug, this bug is related to the paging system, which by the way, is by no means finished yet.

It's this last bug that I've been working on since a while and the last several days I've been getting some small clues of where the bug could be.

At first we thought it could be a memset issue, but by replacing the current memset with a simpler one, we eliminated that as the possible and most obvious cause.

We then continued with some other obvious culprit which was the memory allocation system, which we started testing. During these tests we figured out that the allocator wasn't the issue.

This led me to thinking about the issue in a new light. From the paging perspective, and so I did a little test, to figure out where in memory this bug starts.

As it turns out, once we enter a level of memory above a certain number, we cause page-faults, which in turn cause more page-faults, and so forth.

This issue is probably due to the way I approach paging, but it leads to the more interesting question. What is precisely going wrong, since without knowing that, we can't fix the issue.

That's what we're doing now, so we probably won't meet this sprints target, but that doesn't really matter, as the bugs resolved will have a way higher priority.

That's it for now,

Hope you had a nice summer break, and see you next post.

Chaos in the project?

2011-07-20T14:33:00.000+02:00

Strange title, maybe?

Well, yes. There is a little bit of chaos in the project, mainly because it's vacation season, so that means that everybody will spend some time away from the project and most things which would normally go well, don't now.

Another source of chaos might be the fact that, up until the last release I had a pretty good idea of where I would like to go with the project. I need it to be in high memory and I want that to be done in a simple fashion, which I think I have completed by now.

Next up I thought I'd look at streams and everything, but it turns out now that we need at the very least a simple virtual file system. Now that very simply put means that I don't have a clue what I'm doing.

I have been reading this book written by Andrew S. Tanenbaum and it does give a nice overview, but it's all theory, and like we all know, there is a difference between theory and practice. Oh-yeah, there are some differences all right.

Now I'm not going to sum it all up for you, but it boils down to the fact that I have a lot of questions and pretty much nothing at hand to answer it all, besides the book, which I'm still reading.

I'm at the same time also looking at the OS dev wiki and have found some interesting bits of info there as well.

So, what can one expect to happen until September the first, when this sprint should be done.

Basically just us lagging severely behind. I think I might have overlooked the gravity of some of the functions and that in planning terms will cost us dearly.

Now this doesn't matter anyway, as I still don't have a clue of when the 1.0.0 release should be.

One thing is for sure though, and that is that the 0.0.5 release might have a little less than one'd expect it to have, more because of the gravity of the features, than because of the lack of planning in the vacation (which is why the sprint ends first of September instead of August the first).

Speaking of summer break, what am I going to do?

Well, I'm going to France for a couple of weeks, and the neighbours will be watching over the plants and everything.

My sister is not going with us for the very first time, because she gets to go to language training camp in Cambridge (lucky arse).

In the mean time I'm pretty sure Steven will keep going with the graphics drivers and stuff like that.

So I think that'll be it for now and I plan on posting again when I'm back.

And here comes the fourth release

2011-07-11T20:31:00.001+02:00

Yep, its time for the fourth development release also known as 0.0.4.

This release features the ability to:

Respond to interrupts (again),
Load in the core image and
Compile the core image.

The previous release (the overhaul of the project layout) should actually have been part of this release, but it was so much of a change that I thought it was worth its own release, this making this the fourth release.

I certainly did a short silly dance once I had the elf loading done for sure (probably not a soul that's seen it, lucky me), because this was a point I was working toward from the beginning of the project.

Up next (at least for me) is the revising of the printf functions and the way text is displayed on screen. This actually might get in the way of Steven a little, but I don't care as he is on vacation anyway.

The next release is scheduled in September, and that might seem an awful long time from now, but keep in mind that the summer breaks are in the middle of this, and that I'll be gone for 21 days here.

Steven has been delivering code, but unfortunately this is not testable, and thus is shifted to the next sprint, which actually starts today (Monday, June the 11th of 2011).

We wish Steven a nice vacation and hope he will be able to make some good progress in my absence.

We also wish to thank him for the server space he's provided.

This is a link to the tag formed during the 0.0.4 release.

Major break through

2011-07-09T12:34:00.000+02:00

I know this post is a little early, but I don't care, since we have had this major breakthrough last nigt.

It doesn't seem like much, but if you're a little familiar with the project you'll realise that the fact that we have Elf loading working, is a HUGE thing.

The reason why it such an impressive feat is because everything has been made from scratch.

"So?" You might ask.

Well, We've had to initialise a heap, use that heap (did that with a first fit algorithm), print some text to screen, and set up paging.

The first big breakthrough was when I had a heap working, the second was paging, even though that one isn't really finished yet. The last in this sequence is Elf loading, since it puts every thing together.

It basically pulls the interesting stuff out of the image and pasts it somewhere in memory (defined by the image). The weird thing is that the batch of memory requested might not even exist. That's difficult since we actually need it.

Not to worry, we have a way of doing things to make this all possible. It's called virtual memory. Basically what it does is split the address space up into 4KB pages, and a page is then connected to a physical batch of memory, allowing for pages to exist in memory addresses which don't exist in the physical address space.

This does however come with a problem. Since the virtual address space is 4 GB in size and the actual memory size is something random, that might spawn the problem of a lack of physical memory. Now that's solved by swapping the pages out to disk.

This however hasn't been implemented, which has a very simple reason. We don't have file system support, and thus we aren't able to write to or read from disk.

Guess what's up next in the planning?

Well, yes, graphics, but besides that?

Indeed, file systems support. This will likely be a three tier approach.

Tier 1: The virtual filesystem (vfs), this is what you interface with as user/programmer.

Tier 2: The filesystem driver(fs driver), which is what looks up the physical position on a physical hard drive, regardless of what type of drive you use.

Tier 3: The disk driver, which takes commands from the filesystem driver to access specific blocks on the drive, without the need to know about filesystems.

How this will be implemented needs some thinking though. Are we going to implement the inodes in the vfs, or will that be supported in the fs driver, and if so where will we keep track of mount points.

If you have ideas, they're always welcome of course.

Another thing that desperately needs to be done is the design of a logo. Now for more info go here.

The fourth release already?

2011-07-07T14:06:00.002+02:00

So it's been a while since last blog post. I know. Last time we were still in 0.0.2, and now I'm already talking about 0.0.4!

What's happened you might ask. Well, if we open the new site, we will see several things there. There is a sprint backlog over there, which explains what we're currently working on before the next release. There's a product backlog which represents all the future features, and there's some more general info.

The sprint page shows you what we're doing and what's done. I'll make a quick summary for you, since this page will likely be updated in the near future.

We've:

Set up a site,
Finished the header analysis,
Done a new project layout,
and verified the elf header.

What's more is, I've actually just finished work on making the core image compile.

Now the work on the core image isn't finished yet, as it still needs testing (it's now in to verify) en might thus go back into the in progress, or even the todo if the issues appear to severe.

In the mean time Steven has been doing some work on getting the graphics going, and even though we don't always agree, he does his best, and quite often produces good work. Hence he's also got several topics.

He's also working on a feature which isn't in the sprint. The feature needs to be done for sure, but will probably be introduced in the master branch at around 0.0.5 (Oh my god, I'm already talking about an 0.0.5 release!).

In the mean time I have had official verification that I may continue at school into the second year!

Now for more info check out the site at http://orionos.tk/.

A quick side note though. Because of energy saving and the target audience primarily being in Europe, the server will be down between 23:00 UTC until 7:00 UTC.

What's going on here?

2011-06-24T23:39:00.000+02:00

If you have been following the code lately you will have seen several things changing, quite dramatically.

One of which is that there is a new contributor, named Steven van der Schoot. His name already was in the CREDITS file, but his role actually has changed dramatically.

Previously, all he was known for was the printhex function, which couldn't do much more that print out numbers in hex. I thought that was a bit messy, as I also wanted something that could do binary, decimal and octal, and just hacked them together into a single function. (If you know the maths it's not that dificult, I admit).

Now he has more time because he has passed his final secondary school exams, for which I wish to congratulate him.

But he has decided to do several things with his spare time. One of which is Mincecraft. The other? The VGA driver for the platform.

Yes, we already had text output, and that probably won't change in the near future, but what he is enabling here is much more than basic text printing. Basically the bootloader configures you to use VGA text mode, which enables you to write characters to a location somewhere in memory. Now that a driver is coming up, we still be writing to this buffer, be it in another location, only god (and my paging code) knows where in physical memory. But once this is done, what he can do, is start focussing on drawing shapes to screen. Such as rectangles, triangles and circles.

Now that sounds fantastic and all, we've got graphics working, or ... do we?

Technically, yes, but don't get too impressed. I still don't know anything about resolution, but for all I know VGA is basically crappy, not being able to go over 640x480, at, what was it? 256 or 16 colours.

Now this is nice and all, but it doesn't really have my priority. I'm more worried about getting the source code in a new layout, which enables me to make a difference between the compressed image (or nano kernel. Yes I'm changing the terminology here) and the core image on basis of path, in contrast to a complex, hard to notice, easy to forget while writing ifdef structure.

This is actually the reason why you see these huge spikes in the impact graphs on github. It basically boils down to the entire project being assigned a new location, which generates a huge impact, while in reality, nothing changes in the content. It's a mere process of copying and removing.

I am going to have to do some work on the makefiles though, and I probably am not going to like doing so. The make files need to get more scalable, as I know that they aren't exactly doing so as of now.

But is your elf code done?

Well, no.

I needed to get my loaf away from it for a second. Making the project more scalable had a higher priority than elf code. Despite the elf code being half done. This is due to the fact that I'm no longer the sole developer of this project and I need to make it possible to give the other developer(s) all the space they need.

I still don't know where I should put Stevens drivers for example. The drivers could be necessary for the nano kernel, since the drivers need to be installed from unreal mode, and the core image can't go there any more (unless you have 3,5 gigs or more). I could also do v8086 mode, but I feel like that could pose security issues.

I still haven't figured it out yet, but I'm working on it. Fortunately with a new developer.

Is it me, or has it been a while?

2011-06-16T10:32:00.000+02:00

It's not you. It indeed has been a while and it is all due to the fact that this week is exam week at school. Those things require my attention a little more than the kernel.

So you've done nothing to the kernel since last update?

Well, that's not entirely true. I've been reading the Elf32 specifications and have managed to pull some more info from the Elf headers. Among which is a segment with a type number that doesn't appear to make sense according to the specifications.

However, after doing some googling I found out that Linux uses its own types. One of those is a stack segment, in which the kernel is specified to reserve room for the stack. In normal circumstances that's nice, but since my kernel sets up its own stack I'm just going to ignore this segment.

As for the rest?

Well I've been redoing my printhex function so that it now prints numbers in every base in the range from 2 to 36. All printf has to do now is call this printnum function with the number (named index), base, a boolean called signed and a boolean for capital characters.

So I guess that's all.

Well, not entirely. I have also changed the layout of the Orion project itself. Andromeda now no longer is merged into the Orion source tree. It rather is imported as sub-module. This means that when you import the Orion source tree, all you get is a .gitmodules file and a bunch of empty directories.

You then have to do git submodule init; git submodule update; to get the sources from the sub-projects.

Now that's all.

And what can we expect for the future?

Well, school is about to break up for the summer, which renders me a lot of spare time. I'll be spending some of that spare time in the woods with my dog, and some of that time in France. I will also be spending some of that time on the project, so don't worry about that.

I think the summer break up will give me more time to do work on the project so you might see the impact graphs show a lot more impact.

What? Nothing new?

2011-06-01T23:32:00.000+02:00

Well, according to my dictionary elves are typically irritating little things, and that is exactly the way Elf loading turns out to be. A whole heap of irritating little things.

So I've been busy for sure, but I also have school and other things to keep working on, so unfortunately this one feature will return to this weeks sprint, and I have a very irritating little feeling that it will continue to come back to the next sprint for a while.

This basically means we have to put up with the slowing development for a while. If I have to choose between school and this project, I'll definitely opt for school, and as it turns out, that is exactly the choice I have to make this weekend.

I don't expect to do a lot this week, because all the work I do is really the time I have left when all of my spare time is divided.

I have hacked together a little document on scheduling though. It can be found at humane hours (from 7:00 through 23:00 GMT) behind this link.