There are three types of memory in the system: the system RAM, the local memory for each HW task, and the shared RAM. The system RAM is used to hold the OS image and data, a heap memory pool and all SW tasks. The SW task memory region is similar to traditional UNIX systems; the SW data (state) area contains all task global data, both initialized and non initialized, and corresponds to the .data and .bss sections contained in an ELF executable file. Each HWtask includes a variable amount of local memory, in the form of both FPGA embedded block RAM (BRAM) and hardware registers. The HW data (state) area is an exact mirror of the SW data (state) area found in system memory: the same global variables are allocated at the same relative address in both the SW and HW data (state) areas. Finally, the shared RAM holds a shared memory pool. The OS can reserve shared memory blocks in the shared pool for inter-task communication.

By virtue of the mirrored data (state) areas, relocation is possible by using the following key idea: we require that both the software and hardware configuration of each task save the entire task state between successive periodic instances (jobs) in the SW data (state) and HW data (state) areas respectively. Therefore, since tasks can only migrate at the end of a period, the only memory area that needs to be copied is the data (state) area, which is usually relatively small.

To simplify the job of the application developer, we have developed a parametized interface module that is synthesized together with user provided logic to form a HW task. The interface module connects the task to the shared bus, providing both master and slave capability. It automatically moves the data (state) memory when the task is relocated, and periodically starts/stop the user logic. Finally, it provides the user logic with a memory interface to access the shared RAM memory. Since all communication among tasks is carried out through the shared memory pool, communication is transparent: it does not depend on a task being implemented in hardware or software.

Finally, we modified an existing OS (Xilkernel running on a Microblaze soft CPU) to support task relocation. The OS manages scheduling information for all tasks and controls both the ICAP and the interface modules. Synchronization between the OS and HW task is maintained by using a single hardware timer for the whole system. To test the system, we developed test case based on a relocatable cipher/decipher (see downloads).