• configuration tools that allow the system administrator to specify the relationships between SBCs and how they will be booted
• inter-task communication facilities that allow communication between tasks that are executing on separate SBCs
• the ability to have shared memory between tasks executing on separate SBCs
• access to DMA capabilities to allow very efficient transfer of large amounts of data between the memory of separate SBCs
• support within application development tools for viewing multiple SBCs as a single system
• optional hardware support for tighter coupling of SBCs

Support for these closely-coupled capabilities is not confined to a single software package, but is spread across the PowerMAX operating system, system utilities, system libraries and the NightStar tool set.

Diskless booting
An SBC without a system disk in a closely-coupled configuration is known as a client. Note that a client might have attached disks but it does not have a system disk - the disk partition where the operating system executable and system utilities are stored. Booting a client consists of downloading a boot image to the client and initiating execution of the operating system within that boot image. Since the client does not have a system disk, the boot image contains a configured operating system executable and a memory-based file system. The file system contains the system utilities and libraries needed to boot the operating system. The application's executables and data files could be located in the memory-based file system, on a remote disk that is mounted via NFS or on a directly attached non-system disk.

To create a boot image, a diskless client requires a virtual root partition on a file server. The virtual root is used as the environment for configuring and building the boot image file. The boot image is then compressed to create a smaller boot image.

A boot image can be loaded into a client's memory from on-board flash, across the VME bus or via an ethernet network connection. The boot image is not aware of the means used to download it into memory, therefore the same boot image can be loaded via any of the supported download mechanisms. A typical development cycle would involve loading the boot image via VME or ethernet while the application is being modified. In the final deployed version of the system, the boot image can be burned into flash memory. Booting from flash memory has the advantage that it allows very fast boot times (typically 20 seconds to boot a client) and the image is less likely to be corrupted by hardware device failures.

Application programs can be initiated on a client automatically after the system is booted by running the application programs from standard boot scripts.

Layered Processor Coupling
PowerMAX OS is designed to provide optimal real-time determinism on shielded processors* on a tightly-coupled symmetric multiprocessor. Closely-coupled configurations can be built from SBCs that are themselves symmetric multiprocessor systems. Linking SMPs through the closely-coupled mechanism creates a layered multi-processor architecture. The SMP architecture provides deterministic program execution on its shielded processors and the highly efficient inter-task communication available on an SMP system. Tasks on separate SBCs have the same functionality for inter-task communication, albeit with less efficient communication. The development tool set provides a single system view for all tasks on all SBCs.

SBCs in separate VME clusters can also be attached to a cluster via an ethernet network connection. Such remote clusters can be configured to be completely diskless, using the file server in the local cluster to hold the required virtual root partitions for these remote SBCs.

Inter-board Inter-task Data Communication
The simplest programming paradigm for communicating between tasks is shared memory. The closely-coupled architecture allows physical memory on one board to be accessed from other boards. This allows for shared memory regions between processes executing on separate SBCs. Access to memory is done via programmed I/O on the VME bus. System configuration parameters allow reserving physical memory for use in shared memory regions as well as very flexible parameters for configuring VME master or slave windows for accessing the physical memory of other SBCs.

In PowerMAX OS, the Posix inter-process communication facilities are the preferred method of communicating between tasks. The Posix message passing interfaces can be used to pass data between processes. This set of interfaces also supports the ability to wait for data to arrive in a message queue or to asynchronously be notified when data arrives. The same set of message passing interfaces is available for communicating between processes that are executing on separate SBCs. Using a common set of interfaces allows a decreased learning curve for programmers and makes the code non-specific to the closely-coupled architecture.

The most efficient mechanism for transferring large amounts of data between SBCs is to use the on-board DMA controller to transfer a block of memory in a single VME transfer. Physical memory can be reserved for this purpose. A special device allows the standard UNIX I/O calls (open, read, write) to be used for the transfer of data to a particular SBC via the DMA controller.

The VME bus can also be used as a reliable networking medium. This support creates a low level Streams driver, which makes the VME bus appear like a network controller to the rest of the Streams subsystem. This means that all higher level networking protocols including things like UDP, TCP and NFS are available across the VME backplane. In addition, common networking programs like FTP and rlogin can also be run on the VME bus.

Inter-board Inter-task Synchronization
To be able to update shared data from multiple processors, without corrupting the data, it is critical to be able to synchronize access to the shared data. The Posix interfaces for synchronizing access to shared data are counting semaphores. This same set of interfaces is supported for synchronizing access between processes on separate SBCs.

Another need for synchronization arises when a process needs to notify another process that some event has occurred. Perhaps one process has just transferred data via a DMA operation to memory on another SBC, perhaps a new phase of execution has begun and it is necessary to signal processes on other SBCs to switch phases. There are several supported ways to achieve this kind of notification. First, the Posix counting semaphores or message queues could be used as a notification mechanism. However, there are more efficient mechanisms available that allow generation of an interrupt, which can be caught by a process on another SBC. There are three different types of interrupts that can be generated:

• mailbox interrupts: an interrupt generated locally when an on-board register is remotely accessed via the VME bus.
• VME interrupts: this interrupt type is generally not used, since it uses up one of the seven VME interrupt levels
• RCIM interrupts: this requires a special PMC board that will be discussed later in this paper.

A cross board interrupt can be caught on the receiving system via one of several method:

• A process can block waiting for the interrupt to occur.
• A process can request asynchronous notification via a UNIX signal when the interrupt occurs.
• The interrupt can trigger a user-level interrupt. A process registers itself to catch a particular interrupt in a user-level interrupt routine. This routine is then activated whenever the hardware interrupt fires. The user-level interrupt routine runs at interrupt level, thus providing very quick response time to an interrupt. The interrupt routine has full access to the address space of the process, but only a limited number of system calls can be performed at interrupt level.

Real-time Clock and Interrupt Module
The real-time clock and interrupt module (RCIM) is a Concurrent developed PMC module which can be used to provide some of the features of an SMP system that are not normally available in a system composed of disjoint SBCs. These PMC modules are linked together via cables to provide connectivity between SBCs.

One of the functions the RCIM can provide is a shared high-resolution time of day clock. With a resolution of 320 nanoseconds, this clock can be used for fine-grained time stamping of events or for accurately timing time intervals. This clock is also utilized by the NightStar tools that require a common time base on all processors.

On an SMP platform, the occurrence of a received interrupt is easily communicated to multiple programs, regardless of the processor where the program might be executing. This is critical when an interrupt is used to initiate some activity that requires the cooperation of multiple tasks. In a multi-board environment, it is necessary to have the process that catches the interrupt, in turn, fire other interrupts to the connected boards. The RCIM provides a much more efficient mechanism for this by providing distributed interrupts. These interrupts can be configured to interrupt almost simultaneously on all RCIM-connected boards. In this way processes on all SBCs can be almost simultaneously started when a single interrupt fires. This provides a very efficient mechanism for creating a joint response to an interrupt.

Application Development Tools
The NightStar tool set is a set of development tools that is specifically targeted towards developing real-time applications. Features of these tools that allow monitoring of run time behavior are designed to minimize program interference so that the use of the tool does not change the run time behavior of the program. This tool set also has features that allow debugging applications that span more than one SBC. Below a brief description is provided of the tools that are enabled with features that specifically target closely-coupled.

NightView - this tool is a graphical source-level program debugger. It can debug programs written in Ada, C, C++ and Fortran. It can simultaneously debug multiple programs that are running on the local host or running on multiple host computers. One of the features of NightView is the ability to define a family of programs that can then be acted upon as a group. For example you might be running several instances of the same program and want to set a breakpoint on the same source line in all instances of the program. In the case of multiple SBCs in a closely-coupled environment, the family of programs can span more than one computer system.

The GUI user interface to NightView can be run on one SBC to control the execution of tasks that are executing on other target SBCs. The target systems will require far less system resources because a simple debug agent will be the only additional process present for debugging target processes. The GUI generates network traffic, I/O operations for interacting with the user, reads object and source files and evaluates expressions while the only job of the debug agent is to store and retrieve data in the debugged process, control its execution and supply information about the target system. This means that the debug agent causes far less interference when run alone on a target system.

Next Generation Closely-coupled
When coupling multiple processors, the weakest link is generally the bus that links the processors together. This is certainly true for the closely-coupled architecture as outlined in this paper. The VME bus is being utilized as the interconnection mechanism between boards. This is a relatively low speed bus, which has the double duty of being used for I/O devices and interconnecting the separate SBCs. The next generation of the closely-coupled architecture will still utilize multiple SBCs in a single VME bus. However, the bus used to interconnect the SBCs will be a PCI bus connection between boards. This PCI bus will be dedicated to inter-SBC communication and is higher bandwidth than the current VME64 technology being used.

Conclusion
The use of multiple single board computers for complex real-time applications that require multiple processors is a way to significantly reduce the cost of these systems. However, the application designer should be sure that he has the operating system and development tools provide an environment for programming in this distributed SBC environment. For without an environment that specifically targets a multi-SBC environment, the cost savings in computer equipment will quickly be consumed by increased software development costs.