The Module Interface
--------------------


1. INTRO

OpenRADIUS is built around some design ideas that determine most of how
it works; if you build any server using these principles, then a lot of
its architecture will already be defined.

These are no novel ideas at all; those who are comfortable with the Unix
programming environment will probably be already familiar with them:

- modularity is based on pipes and processes, not on libraries or a
  hybrid form of a library and a process such as some DLL designs;

- concurrency is achieved by offloading work to multiple other
  processes, not by multithreading within the same process.

This means that any operation that takes any significant amount of time
to complete, should be done in a subprocess (module). The server itself
has only a single thread, which keeps processing a request until it can
be answered or until it has to wait for an answer from a subprocess.

All this is done on the assumption that it makes no sense for the server
to preempt any of its own processing, which is CPU bound; it's better to
get that over with as soon as possible. But all I/O operations are fully
non-blocking; as soon as a job is passed off to a subprocess, the server
can handle new requests, (partial) answers from subprocesses, anything.

It also means, indeed, that if you run only a single main server,
without running any external modules, you won't take any advantage of a
multi-CPU machine. But who would want to waste that CPU power on such a
simple job? The bottleneck is the network in that case. If it's not,
because you run multiple high-performane ethernet cards in the machine,
you can run multiple instances of the main server, each bound to a
different network interface - thus taking advantage of the extra CPUs
and other hardware.

The module subprocesses themselves can be very simple too - they don't
need to keep track of multiple outstanding requests; that is done by the
server. It only needs to be handling one request at a time and can block
as long as needed. 

That this doesn't block other requests that cause a call to the same
interface is because multiple subprocesses may be started per interface.
That way, you can write a simple module that eg. introduces a latency of
a full second, but by having the server run 10 of them, you can still
achieve a throughput of 10 requests per second.

What makes this different from the 'classic' Unix server design is
that the server doesn't spawn a child for each request as it comes in,
but that it pre-spawns a pool of childs that do not terminate after each
request, communicating with the childs through pipes. This removes the
overhead of creating and destroying processes; see Apache's FastCGI
module for another example of this type of interface.


2. THE INTERFACE

When the server is started, it creates two pipes for each subprocess,
one for sending requests to it and one for receiving answers, and spawns
the module as a child of the main process.

Each subprocess is associated with a certain interface. You could have
an interface to a certain database, which uses three subprocesses; two of
them may connect to the same database, and one may connect to another
database which contains the same information, but that just happens to
run on another machine.

It doesn't matter; they're all part of the same interface because they
all give the same answer when asked the same question. The reason you'd
want to run multiple subprocesses is for load sharing among database
connections and as said above, to be able to have multiple outstanding
requests on the same interface, even though each simple subprocess can
handle only one request at a time.

After a child is started, it waits for a request to come in on file
descriptor 0, and answers it on file descriptor 1. It must keep running
indefinitely, or until it is terminated by a signal the server. Childs
that terminate unexpectedly are automatically restarted after a short
interval, to avoid consuming unnecessary resources if something causes
the subprocess to immediately exit after starting.


2.1. Interface calls

When an interface is called by the behaviour expression, the server
builds a message from the of A/V pairs currently on the job's request
list. (This list initially contains all information from and about the
original request, but there may be attributes added, changed or removed
by the behaviour expression). 

If no send ACL is specified in the configuration file for the interface,
the request message will include all attributes on the request list
present; otherwise it will only have the ones listed in the ACL. 

If the interface has a channel to a subprocess that's currently idle (it
tries to find one in round-robin fashion), it makes that subprocess take
the job immediately; it puts the message on the sending ring buffer and
puts the channel in the 'sending' state, so that each time the operating
system indicates data can be sent through the pipe to that process,
we'll send from the ring that holds the full message as much as it can
take, until the whole message has been transferred.

The request, with its associated state (together called a job here), is
put on the receiving queue associated with that subprocess, so that when
a (partial) answer arrives from it, the server knows which question the
answer is for.

If, on the other hand, no subprocess channel was found idle, the job is
put on a queue that is associated with the interface itself, not with a
particular subprocess. Then, as soon as the server finishes sending a
request message to any subprocess, it checks the shared queue of the
interface it belongs to, and if there is a pending job, it immediately
makes that subprocess channel (which has just become idle) take the job.

This is more efficient than immediately selecting the subprocess that
gets the message as soon as the interface is called, because it allows a
subprocess that happens to finish its work first, to immediately continue
to do useful work. Thus, no process ever sits idle if there are any jobs
waiting for the interface it belongs to.

As said, the server doesn't wait for any specific answer before doing
anything else. After preparing things for a message to be sent to a
subprocess, it just waits for any new request to come in, or another
(part of an) answer to come from any subprocess, or another subprocess
that we're sending data to, to become available to receive more data.

When an answer happens to have fully arrived from a subprocess, the
server gets the job that was waiting for this answer from that
subprocess' pending job list and adds the A/V pairs from the answer to
the reply list of the job. If no read ACL is defined for the interface,
all pairs in the answer are added, otherwise only the ones listed in the
ACL.

Then, the server continues processing the behaviour expression on the
job until it is done or until it makes another interface call.


3. MESSAGE LAYOUT

Before looking at this, it's a good idea if you already have an idea of
how the server handles information about a request. See the file
doc/packet_handling for more details about attribute spaces, fixed
fields, etc.; see doc/language for more information about manipulating 
the request and reply lists.

The important things to remember here are that *all* information that is
available to the behaviour expression and the subprocesses, is
represented as attribute/value pairs; that each job has two lists of
those A/V pairs associated with it: the request list and the reply list;
that a full attribute specification consists of a attribute space
number, a vendor number and an attribute number, and that values can have
various types: integer, string, IP address or date.

As said earlier, when an interface call is made, the server transmits
the job's current request list in a particularly formatted message to
one of the interface's subprocesses. The subprocess prepares its answer
in the same format, and upon receipt, the server adds the A/V pairs
that are present in the answer to the job's reply list.

These messages can either be in binary format, with length specifiers
for all variable-length fields to facilitate modules written in C, Java
or C++; or in ASCII, with field and record separators to facilitate
modules written in text-oriented languages such as Perl, awk, or even
an ordinary unix shell.


3.1. Binary messages

Binary-type interface messages are used if you don't specify the 'Ascii'
flag for an interface. They are formatted as follows:

0               4               8			(offset)
+---+---+---+---+---+---+---+---+----------- -
| Magic         | Length        | A/V pairs...
+---+---+---+---+---+---+---+---+---------- -

Magic:     the hexadecimal value 0xDEADBEEF, in network order (MSB
	   first). 

Length:    the total length of the message, incl. the Magic and Length
	   fields, in network order.

A/V pairs: a list of attribute/value pairs. Each pair starts at an even
	   4-octet boundary, and is formatted as follows:

0		4		8               12		16  
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+----- -
| Space		| Vendor	| Attribute	| Length	| Val...
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---- -

Space:     number that identifies the attribute's space, in network
	   order. See doc/packet_handling and the dictionary for some
	   more information about spaces.

Vendor:    the vendor's Private Enterprise Code as assigned by the IANA
	   if the attribute is vendor-specific, or 0 if it's a standard 
	   attribute. In network order.

Attribute: the attribute number, in network order.

Length:	   the value's real length, i.e. the number of 8-bit characters
	   in a string or the number of significant bytes in an ordinal
	   value, in network order.

	   If you round this length up to a multiple of 4 octets and add
	   16 octets for the value's own offset, you get the total
	   length for this A/V pair entry - i.e. the relative offset of
	   the next pair.

Value:	   formatted depending on the value's type:

	   - string: any number of octets as specified by the Length
	     field, padded on the right up to a multiple of 4 octets.

	   - ordinal: a value sized between 1 and 4 octets inclusive
	     (between 1 and 8 octets inclusive on 64 bit platforms),
	     as specified by the length field, zero-padded on the left 
	     to a multiple of 4 octets, in network order.


If the server detects any anomaly in a message from a subprocess, such
as a wrong magic number, an A/V pair that would extend past the number
of bytes specified by the message length field, or anything else that
would indicate a framing error, it will terminate and restart the
subprocess, as that's the only way to re-synchronise the stream.

In most cases, this indicates a bug in the subprocess, so if the server
logs that it restarted the subprocess because of an invalid received
message, it is recommended to take a good look at the subprocess' code.

The alignment specifications above may seem perhaps a bit complex, but
they make traversing the list and accessing the ordinal values much
faster. In the end, it only takes an 'add' and an 'and' operation to
round up the lengths when traversing the list. This is much cheaper than
accessing non-aligned values, especially on non-intel platforms.


3.2. ASCII messages

ASCII messages are selected when the 'Ascii' interface flag is
specified. Their format can be influenced to a large extent by other
flags, so that the subprocesses need to do as little text mangling as
possible before being able to do useful work.

The messages look like this (optional components are enclosed by square
brackets, angle brackets denote ASCII control charactes):

[<TAB>][Space:Vendor:]Attribute[<SP>]=[<SP>][Type:]Value<LF>
[<TAB>][Space:Vendor:]Attribute[<SP>]=[<SP>][Type:]Value<LF>
...
<LF>

Thus, A/V pairs are separated by <LF> characters, and an empty pair
indicates the end of the message.

The appearance of the leading <TAB> is controlled by the Add-Tab
interface flag; similarly, the appearance of the spaces around the
equals sign is controlled by the Add-Spaces flag.

The space- and vendor names to qualify the attribute are present by
default; to turn them off, use the Short-Attr flag. The data type
specification is only present if you specify the Add-Type flag.

The individual fields have the following meaning and format:

Space:	   the name of the attribute space as associated with its 
	   number by the dictionary. See doc/packet_handling.

Vendor:	   the name of the vendor as specified by the dictionary for
	   this vendor's Private Enterprise Code.

Attribute: the name of this attribute as specified by the dictionary.

Type:	   one of the type names 'Integer', 'String', 'IPAddr' or
	   'Date' that indicate the value's data type.

Value:	   formatted according to the interface flags and the value's
	   data type.

When the 'Hex-Value' flag is set for this interface, all value fields
are written out in hexadecimal, regardless of their actual data type.
This makes parsing by modules extremely easy, but if the module has to
output any attributes in a human-oriented form, it may not be the best
option.

If the Hex-Value flag is not set, the format for the Value field is as 
follows, depending on the data type:

Integers:  If the flag 'Named-Const' is set for this interface, and
	   the value corresponds to one of the constants defined in the
	   dictionary for this attribute, the constant's name is used.
	   In all other cases, the value is written as a decimal number.

Strings:   Always written between double quotes. Non-printable
	   characters (<32 or >126), single and double quotes and
	   backslashes are written using hex escapes, as in \x3f. If the
	   flag 'Double-Backslash' is set, the backslash is output
	   twice, to better accomodate shell script modules.

IP addrs:  Written in standard dotted decimal notation, as in 1.2.3.4.

Dates:	   Written in decimal as the number of seconds since January 1, 
	   1970.

