Author Archives: Hans-Jörg Frieden

Breaking the Memory Barrier

Posted by on May 22, 2014 Comments Off on Breaking the Memory Barrier

Overview

AmigaOS is a 32 bit OS. There is little we can change about it. The size of an address pointer is intrinsically entangled into the API, and getting rid of this legacy is, for the most part, a matter of replacing all of the API with a new one. Every time a programmer writes something like “sizeof(struct Message)”, the 32 bit nature is fused into his code.

This has some repercussions that cannot be easily ignored. It means that our address space is inherently limited to 32 bits (meaning 4 gigabytes). In reality this space is even smaller than that. PCI space, the kernel, memory buffers, and other memory areas take up a large chunk of the already limited address space, leaving roughly 2 gigabytes for the applications running on the machine – 2 gigs which also are shared between all of the programs running.

Physical Vs. Virtual

A physical address of a memory block is implicitly defined by its position within the memory chips, and the order in which the modules are inserted into the mainboard’s memory slots. They start at zero and go up to a specific maximum.

A virtual address, on the other hand, is what the CPU and hence the application program sees. They might be the same, but as a general rule, they are different. Virtual addresses are given on the fly, but there is a rule that every memory cell must have a unique virtual address, because all references to that cell are stored as the virtual address the application sees.

Modern systems like the X1000 or upcoming models can take more than 4 gigabytes of memory, but so far, the extra memory will never be used. Even in a 4 gigabyte system, there is memory that will never be touched because there is just no free address; and unfortunately, every byte needs to have its own virtual address, and no two bytes can have the same.

Unless…

Extended Memory Object

Extended memory objects (ExtMem) are a means to access memory beyond the 2 gigabyte barrier by applications that are written to make use of them. In a nutshell, an extended memory object is a chunk of physical memory that exists in a “nirvana” state somewhere in the memory of the computer without a virtual address of its own. The memory cannot be accessed by anyone or anything in this state. In order to access it, an application must map part of the object into its own virtual address space. This mapping does make a part of the memory represented by the ExtMem object accessible in a memory window in the application’s own address space.

There is no limit to the number of mappings an application can do. If needed, it can have several mappings active at a time, and add or delete mappings as required. The only restriction is that mappings must not overlap (either in virtual address space or in the memory object itself). Each mappings opens up a view into a part of the memory object, and, depending on how the mapping was performed, the application can read and/or write to the memory as if it were normal memory.

fig2A mapping is defined by the virtual address in application memory (which can be chosen by the application, or picked at random by the OS), the length of the map’s window, and the offset it maps to in the ExtMem object.

There are some caveats though. Most notably, the ExtMem object itself doesn’t have an address. In that sense it should be treated more like a file than a memory block. If an application wants to have permanent references to memory in the ExtMem object, it needs to store them by offset, just like it would with a file. The first offset is zero, so to address the 1000th byte in the memory block, the application needs to reference it by the offset of 1000. Obviously, this offset must be calculated against the base of the mapping’s offset; just like in a file, reading a part of the file into a buffer makes the first byte read the offset zero in the buffer.

As an example, consider the following situation. We want to access byte 3000 of the ExtMem object. We created a mapping that has length 4000 and starts at offset 2000. The resulting address for our byte would be the base address of the mapping plus 1000, since the offset of the beginning is already at 2000.

Downsides of the ExtMem system

If you think now that this all sounds suspiciously like bank switching, then you are right. The method has been used way back in the Home computer age, and even earlier. The Sinclair ZX Spectrum 128K was equipped with twice as much memory as the Z80 CPU could address; the upper 16k of the machine could be swapped between different chunks of the rest of the memory. Similarly, the Commodore 64 used bank switching to address a larger memory than its 6502 CPU could handle. It was the only possibility at the time to add more memory.

This method we employ now is basically the same (with a bit more added comfort).

Obviously, the method is a compromise. A “real” 64 bit system would be better, and much more transparent to use. However, as I already outlined in the beginning, there is a lot of work involved to make AmigaOS 64 bit compatible, and with the method of ExtMem objects, breaking the barrier is possible now as opposed to years down the road.

Who can benefit from ExtMem objects?

Well, every application that, in some way or the other, has to cope with large amounts of data. Even if the dataset is only potentially large (like, for example, a text editor), using an ExtMem object has its advantages. The text editor (or word processor), by its nature, only presents a small subset of the text it is editing to the user. Likewise, a movie editor would only need to have access to a few frames in order to show thumbnails of the movie on a timeline, or display a single frame that the user is working with.

Another example is RAM disk. Plans are currently underway to update the RAM disk to make use of the ExtMem object interface, allowing out-of-the-box usage of those normally unassigned memory blocks without draining the valuable main memory. Since (depending on programmer setting) memory blocks can even be allocated “on-demand” instead of ahead of time, this will make RAM disk have an even lower footprint, on top of making it possible to store larger amounts of data than ever before in it.

It needs to be said that the ExtMem system doesn’t require memory beyond the 4 gigabyte bounds. It can work with normal memory as well, even though that is not its purpose.

So, as you can see, a good number of applications have a natural tendency to only access a very small subset of their memory at a given time. All of these are good candidate for using ExtMem objects to break the memory barrier.

Happy Easter

Posted by on March 31, 2013 Comments Off on Happy Easter

Brussels, March 31, 2013

Easter seems like an appropriate time to provide a short update on the development status of a few key components of the upcoming AmigaOS 4.2.

  • Exec SG  : kernel development has been forked off into an experimental and stable branch, to facilitate field testing of potentially drastically new features such as the scheduler, separate address spaces, 64 bit address space awareness etc.
    The task scheduler was completely rewritten in C and now implements the long planned “pluggable” scheduler functionality  : the scheduling algorithm can be changed on the fly, during runtime depending on factors like task’s preference and CPU load, different CPU cores using different scheduling algorithms. The re-write of the scheduler also allows for load balancing which is required for proper multicore support.
  • Work is in progress on yet another Power ISA implementation.
  • Gallium  : bumped to Mesa 9.1, i.e. OpenGL 3.1. Software renderer already implemented, work on hardware accelerated drivers within the framework of Gallium will benefit substantially by the experience gained developing Warp3D drivers for the AMD Radeon Evergreen range (Radeon HD 5xxx/6xxx).

Please note that whilst work on AmigaOS 4.2 is in progress, work on other components is also ongoing and will be made available via AmiUpdate to registered users of AmigaOS 4.1.

Hyperion Entertainment’s management and the AmigaOS 4.x development team wish to thank all of you who have provided much needed support of our efforts by buying AmigaOS 4.1 (either as a standalone copy or together with suitable Power(PC) based hardware available from our hardware partners A-EON Technology and A-Cube Systems.

We greatly appreciate your support and wish you a Happy Easter  !

The Right Tool for the Job

Posted by on June 29, 2011 Comments Off on The Right Tool for the Job

For any task, using the right tool for the job is always a crucial matter. This applies to driving a nail into a wall as much as developing software. And while nobody would ever try to use a glass bottle for the nail, the tools of the trade of the software developer are a bit more abstract (and sometimes, more brittle too).

Shared != Shared

On AmigaOS the word “shared” is used in two major contexts: Shared Library, and Shared Object. Both are tools for sharing code between applications. However, they have very different methods for doing this, and with that comes a very different approach to using them.

Let’s first look at what they are.

Shared Libraries

Since the early days of AmigaOS, shared libraries have been a means of sharing code and, to a certain degree, data between multiple users. A shared library is, essentially, a structure in memory called the Library Base, and one or more jump tables to functions that are to be shared. Since Version 4.0 of the OS, these jump tables are called Interface, and although their use differs slightly from their setup in AmigaOS 3.9 and earlier, the principles are the same. A program intending to use a library has to do two steps in order to perform any calls into that library:

  • It has to open the library by calling Exec’s OpenLibrary call.
  • It has to obtain at least one interface from the library by calling GetInterface.

The latter step was not needed on the classic AmigaOS 3.x, but has opened up a host of new possibilities on AmigaOS 4.0 (we’ll talk more about that in a later article).

Interfaces, like the classic AmigaOS 3.x jump tables, are a collection of function pointers in a structure. Calling a function in an interface usually involves knowing the offset of that function. We typically call a function like this:

struct Library *library = IExec->OpenLibrary("foo.library", 0);

The variable IExec contains the interface. The OpenLibrary call is a member of the interface. During compile time, the compiler will calculate the offset of the member and generate appropriate code for that. The code will load the IExec interface pointer into a register, load the CTR register with the address at the specified offset, and branch into the routine using the bctrl mnemonic.

The process relies on a few factors. First of all, it requires to have the library open and have the interface ready. It also relies on the fact that an interface, once written, will at most be extended at the end. It will never be possible to remove functions from the interface (at least it will always have to have a ‘dummy’ entry) nor will it be possible to re-order functions.

Data access is done via the library’s base pointer. The implementer might chose to store user-accessible data within the library base and document (at least part) of it for public access. Since this is a compile-time decision to make, again, the organization of data, just like the organization of functions, must not change once it has been published (unless the library base data is private).

Shared Objects

Shared Objects are a relatively recent addition to AmigaOS 4. They work radically different from the traditional shared libraries. A shared object is, as its name implies, somewhat reminiscent of an object file that is used during compilation of a program. In essence, a shared object allows a program to defer complete and final linking of the program until such time as the program is actually executed on the target user’s machine. This means that some symbols in the program remain unresolved until such time as the program is run.

As a matter of fact, even then the symbols might still remain undefined if a feature called Lazy Binding is active. In essence, Lazy Binding delays the linkage of a function until the very moment it is called for the first time. So, suppose you have a function MyGreatFunc that you call. If Lazy Binding is active, the call will jump into a routine called .resolv within elf.library which performs a symbol look-up on the name of the function and, if it finds it, overwrites the call to .resolv with a call to MyGreatFunc which is located somewhere in the shared objects that are bound to this program.

From the programmer’s point of view, though, there is no difference in whether the program was linked completely (statically) during the development cycle, or dynamically during load time or run time. In essence, the programmer can use a shared object like he uses any other object file in his program. This includes referencing data in the shared object file. Even more, the shared object file can reference data and code in any other shared object file bound to the program, and in the program itself.

In fact, this is a common case. If a program uses a shared object implementation of the standard C library, the start-up code in the shared object file will usually reference and call the program’s main function. This works completely transparent for both; there is no need to open the shared object, it just needs to be specified during linking.

Shared Objects can do even more. There is a set of functions to dynamically load shared objects during runtime of the program, and to look up symbols in the program or any of the loaded shared objects, including those that have been loaded at runtime.

Don’t use the Bottle!

Looking at the above, it seems natural to want to use shared objects instead of shared libraries. The shared objects seem to have so many advantages over the traditional AmigaOS shared libraries, so why would you want to use shared libraries in the first place?

The answer is simple: The flexibility and power of shared objects comes at a cost. Depending on what you do, these costs can be quite substantial:

  1. Shared Objects, in their current implementation as of AmigaOS 4.1 Update 3, are not really shared. Each program using them will load the file again into memory, completely, including all code and data associated with the file. This means that e.g. a shared object version of libc will load all math functions, all system functions, in fact, everything, even if the program only ever uses printf.
  2. Shared Objects are not versioned. While you can specify a minimum library version when you open a shared library to make sure that you get a guaranteed set of functionality, shared objects are provided “as-is” and do not have this kind of information.
  3. Shared Objects require a certain amount of infrastructure up and running. Therefore they cannot be used for Kickstart modules, or device drivers.

Especially point 2 is an issue, in theory and in practice. Some libraries that are implemented as shared objects carry their own versioning information, or provide a call to find out what version a program is using, but this is not standardized in any way, and therefore a shared object might or might not implement such a mechanism. A lot of systems using shared object exclusively end up with something commonly referred to as “Dll Hell” – meaning the system has to provide a wide variety of different versions of the same library, mostly encoded in their file name. If you look at the average Linux distribution, you will see what this means. This here is an example from my local network server:

-rw-r--r-- 1 root root  9156 2009-04-10 18:29 libgmodule-2.0.a
lrwxrwxrwx 1 root root    26 2010-05-04 13:05 libgmodule-2.0.so -> libgmodule-2.0.so.0.2000.1
lrwxrwxrwx 1 root root    26 2009-06-02 12:48 libgmodule-2.0.so.0 -> libgmodule-2.0.so.0.2000.1
-rw-r--r-- 1 root root 13644 2009-04-10 18:29 libgmodule-2.0.so.0.2000.1

As you can see, there are four “versions” of the libgmodule library, two of which are soft links to provide somewhat generic names. If this were AmigaOS, you would simply have one gmodule.library file, and nothing more.

Another issue is different “builds” of the same shared object library. For example, suppose a library that can be used to load image files. Depending on the facilities of the original compiler of the library, he might have decided to compile the library to include support for PNG and JPEG but not GIF. They end up with a shared object file called libImageLoad.so which, for all intend and purpose, looks like libImageLoad.so from another programmer. The other programmer might however have decided to use GIF as an image format and has compiled his copy of libImageLoad.so with GIF support enabled. A user trying to run this program on a system that only has the first version installed will not work (at best, the program will not load because of the missing symbols). The decision on which libImageLoad.so to install/keep  is left to the user – not a good idea.

Of course, this is still somewhat possible for AmigaOS shared libraries as well. A bad practice in the past has been to always install libraries in to the LIBS: drawer, even if the library was only ever used by a specific program. That lead to situations where two programs would install a library of identical name into LIBS:, and only one program would still work.

Glass Hammers and Steel Bottles

So, this leaves the question as to what kind of tool to use in what circumstance. In spite of their individual shortcomings, both shared libraries and shared objects provide a lot of power if used right.

Shared Objects are primary intended for porting of programs from other systems. Any sufficiently large or complex program from Linux or Windows will, in one way or the other, rely on shared code, for various reasons. Usually, this type of code sharing does not translate very well (if at all) to the AmigaOS shared library model. Using shared objects for such projects is almost mandatory.

Shared Objects have other possible uses on AmigaOS, even for new, native development. As an example, a concept like plugins or overlays can be easily and efficiently implemented using shared objects. Anything that is specific to an application and does not install on a global scope is probably easier to do with shared objects, more so since the compiler has intrinsic support for them (as opposed to AmigaOS shared libraries that need to be more or less hand-crafted).

For almost any other purpose, it is recommended to not use shared objects. If you are developing a library, chances are good that you should use a library for that purpose. AmigaOS shared libraries have limitations that are easy to work around in newly developed code. They offer a solid framework for versioning and an escape from other systems’ “Dll Hell”. They are independent on the compiler used, and as a rule, their limitation will force the programmer into a better designed interface. Since AmigaOS 4.0, the Interface concept allows for a strict versioning, meaning that changing the layout of an interface is even possible if they have a different version number, and GetInterface can retrieve a specific version of an interface if the library still implements it.

Closing Words

This post has tried to shed some light on a question that has confused some developers for a while. It tried to give a broad overview of how the different mechanisms work, and when to chose which method for implementing a task. AmigaOS 4.x and beyond tries to give you, the programmer, the tools required to build, but choosing the right tool is still your job.

Oh yeah, and we’re not responsible if you hit your thumb 😉

The State of OpenGL on AmigaOS

Posted by on April 5, 2011 Comments Off on The State of OpenGL on AmigaOS

Since a while now I have been working on OpenGL on AmigaOS. Progress has been slow so far – there have been a good number of fundamental issues to be cleared up.  For starters, let me summarize the goals:

  1. Provide a hardware-accelerated OpenGL for applications, games, and the operating system.
  2. Provide a driver interface with maximum flexibility. We do not want to count exclusively on Gallium3D as a basis for drivers, although it will be a fundamental one.
  3. Provide a low-level API that does not depend on high-level constructs like Windows or Screens. While it should be possible to render to these, it should not depend on any of them.
  4. Be able to upgrade to the latest OpenGL version, depending on driver support.
  5. Support multi-threading and multiple CPU’s.

Let’s look at these goals. We want to provide a hardware-accelerated OpenGL that can be readily used by applications and games alike. We would also like the operating system to be able to take advantage of this. Workbench should be able to draw its stuff with OpenGL hardware-acceleration, and even Intuition should be able to use it. There is no point in restricting such an important part to a mere application/game interface. It has to become an integral part of the system.

OpenGL screenshot showing a software-rendered Gears demo

Early software-only prototype

 

The internal driver interface is designed to be usable with or without Gallium3D. Obviously, the focus of our work will be Gallium-based, but what if there is a graphics card that does not support the required level of shader support to be able to work with Gallium? For this purpose, opengl.library (the central hub of the OpenGL system on AmigaOS) is able to load and operate drivers that only provide a minimal set of internal functions plus the required entry points for OpenGL. That way, it is possible to use a Mesa based driver, or even a completely custom driver, for hardware that is not supported by Gallium3D.

Our interface is intentionally very low-level. There are two major reasons for this. At first, developers will likely never encounter or use the low-level interface directly. They’ll more likely use a high-level API like EGL or SDL or even GLUT instead. Which doesn’t mean that the low-level API is clumsy or awkward to use. It is merely building on very low-level blocks – for example, it only renders into bitmaps and nothing else. Of course, bitmaps are at the base of all AmigaOS graphics and GUI constructs. To render to a screen, one can supply two bitmaps for ScreenBuffer-type double buffering. For a window, there’s the classic solution of using the graphics card blitter to copy the image into the window after rendering, or, on compositing enabled screens, to render directly into the window’s own bitmap. In any case, being low-level means that the API is future proof for any changes that might come to Intuition or Reaction.

Obviously, we want to be able to update drivers individually. The version of OpenGL support, as well as any extensions, will greatly depend on the driver. Most drivers support OpenGL 2.1 right now, including GLSL support. The exact possibilities depend on the hardware as well (a Radeon R100 will never support GLSL). The flexible driver interface will make sure, however, that individual drivers can be updated without having to update either the applications using OpenGL, nor opengl.library itself.

Finally, multi-threading and multi-CPU support is an important factor. If you are aiming at making a system an integral part of the OS, then support for multithreaded operation is as much a requirement as support of multiple CPU’s.

Blog Comments

Posted by on January 16, 2011 Comments Off on Blog Comments

Unfortunately, due to massive spamming of advertising comments on the blog, we were forced to disable commenting entirely. It has become way too much work to moderate all spam and advertisement. Since we want to continue keeping you informed instead of having to sift through comments, disallowing commenting was the only viable option.

Thank you for your understanding.

About this Blog

Posted by on December 19, 2010 13 comments

Welcome to the new Hyperion Entertainment Developer Blog.

On these pages, we intend to inform you in irregular intervals about new developments on AmigaOS 4.x, what is going on and what is being worked on right now. To begin with, we’ll have some news and a sneak peek at the Classic version of AmigaOS 4.1 that is going to be released soon.

To make full use of this, you need a CSS-Capable browser. Users of AmigaOS 4.x can use either OWB or the Timberwolf Alpha release. Older browser like IBrowse will very likely not display most of the content correctly.

There is more to come; watch this space.