Scalable Simulation Framework
Protocol Modeling with SSF.OS
Jim Cowie and Andy Ogielski
about the SSF.OS framework
     traffic generation:
     transmission of real and virtual data
modeling of blocking methods
The tutorial is presumably most useful when read together with the source code.

The first version of this tutorial was written for SSFNet 0.8. It has been revised for SSFNet 1.1, but some obsolete fragments may have remained.

The source code decides what is the correct interpretation if the tutorial is unclear.

Please report errors to

SSF.OS framework

Base classes

There are three core cooperating base classes within the framework: ProtocolGraph, ProtocolSession, and ProtocolMessage. A ProtocolGraph installs, configures, and initializes ProtocolSessions. ProtocolSessions process the opening and closing of individual connections, and the incoming and outgoing ProtocolMessages.

This parsimonious design pattern builds on the earlier research on the x-kernel (see also the book Computer Networks, A Systems Approach by Peterson and Davie) and is not restricted to the simulated environment of SSF: It would also provide a lean framework for configurable protocol design in a real operating system.

To visualize the roles of these classes, it may help to think about them as follows: a ProtocolGraph is a container of all protocol services in a Host or router. It creates protocol services at simulation startup (booting), and later can locate them when queried by protocols or applications. In fact, class Host extends ProtocolGraph, thus ProtocolGraph may be thought of as a container of all operating system services, not only the networking services!

A ProtocolSession is the base class from which the main protocol classes are derived. An SSF.OS protocol implementation must have at least one class derived from ProtocolSession: typically, this class provides methods for requesting a protocol service and for demultiplex incoming messages (e.g., for TCP this is the class tcpSessionMaster). Usually, another class derived from ProtocolSession represents individual protocol sessions (e.g., the class tcpSession for a TCP connection endpoint). In some cases (e.g., OSPF) a single protocol service is sufficient, and there are no individual sessions.

The ProtocolSessions are nodes in a ProtocolGraph. It is a directed acyclic graph, with the edges representing the depends on relation. Example: Sockets depend on TCP and UDP, which depend on IP, and so on. In the SSF.OS modeling environment the 'terminal nodes' on a path through a Host's protocol graph, that is a user-level application, and a network device driver, are also modeled by deriving from the ProtocolSession class, in order to have them automatically configured from the network DML database.

A ProtocolMessage is a recursive list of headers and a payload. A companion class is PacketEvent, which converts a ProtocolMessage into an SSF Event (or vice versa) when it's sent on a link (or received). Class PacketEvent is used only when modeling network device drivers, such as the class NIC.

The additional SSF.OS base classes are Timer and Continuation. They are general purpose utility classes for the implementation of protocols: the Timer provides an efficient implementation of a reusable timer with a callback function; Continuation provides a mechanism for modeling blocking system calls and operating system signals. There is also a small Internet-specific class Protocols that provides a translation between protocol names and their numerical codes in the IP header.

Comments on the design

Besides design esthetics, a major reason for such a structure of a protocol implementation framework is computational efficiency. The ProtocolGraph entity can be thought of as a firewall between two distinct programming models: the method- and message-based world of SSF.OS, and the channel- and event-based world of SSF. The SSF.OS framework provides standardized mechanisms for handling packets and coordinating multiple protocol modules without the simulator overhead of sending events on channels.

In a minimal Host modeling scenario, only two categories of ProtocolSession objects need to keep track of the logical time advance: user-level applications (e.g., to model the temporal behavior of a Web client), and device drivers (e.g., to model the timing of outgoing packets). Between these two end nodes in a protocol graph a message is processed by a chain of protocols, and all operations on that message are carried out as a chain of plain method invocations, with no logical-time advance and no direct interaction with the underlying simulation engine. If it is desired to add some delay to message processing, it can be done at a chain-terminating protocol.

This was a critical design decision for getting good simulation performance on very large models. Indeed, it would be entirely natural to model message exchange within a protocol graph as events being sent on inter-protocol channels. However, such a simulation would suffer very significant performance and storage penalties on large network scales.

Not all steps in message processing are logically time-unaware, however. A message stream may be buffered by some session in the protocol graph for an interval of elapsed time. A message sent out may start a retransmission Timer. A system call may need to block, to be later awaken by an interrupt or signal from the operating system. To model situations like these, a ProtocolSession can schedule future actions by creating and setting a Timer, or can imitate blocking (without actually blocking the simulation) by employing a Continuation object. In fact, these are the only provisions for modeling elapsed time in SSF.OS, and so far they appear to be sufficient.

The remainder of this note should be read together with code documentation (javadoc for Java SSF) or, better yet, together with the source code, as explaining some concepts would literally take more space than the source code itself.


ProtocolMessage is the base storage class for all modeled packet fragments. A packet is modeled as a doubly linked list of ProtocolMessages, each representing a header and payload.

A ProtocolMessage is not an SSF Event. Instead, we define a helper class SSF.OS.PacketEvent, which defines an SSF.Event with a ProtocolMessage payload, that can be sent over an outChannel modeling a communication link with an appropriate delay (see the class SSF.Net.NIC for an example).

A shared memory implementation of SSF.OS uses zero-copy ProtocolMessage processing, that is when a sender creates a new ProtocolMessage object, only its reference is encapsulated in a time-aware SSF.Event that travels across a modeled network and accumulates transmission and queueing delays. The ProtocolMessage object itself is not serialized for transmission. Therefore,

if an instance of ProtocolMessage needs to be retransmitted, care must be taken to use a copy constructor rather than naively re-send the reference to the original message instance.
To see this, consider the Time-To-Live (TTL) field in the IP header: it is decremented by each traversed router's IP protocol, thus if the reference to the original message were resent, it would have the decremented rather than fresh TTL field.

Transmission of real data and virtual data in SSF.OS

There are two kinds of data transmissions in SSF.OS models that are used to generate network traffic:

  1. applications and signaling protocols may need to exchange real data objects that don't belong in any header, but in data payload,
  2. applications modeling bulk transport need to exchange virtual data, whose only property is the number of bytes. That determines the number and virtual size of transport protocol packets, and subsequently virtual sizes of all lower protocols' packets.

These data types cannot be treated separately, because in a typical case of a client-server exchange over a TCP connection, the same connection needs to be used both for transmission of real data (e.g., a client-server protocol request for an N-byte virtual file), and for virtual data (server sends N bytes of virtual data to client).

This is resolved by the package SSF.OS.Socket as follows. Modelers interested in transport protocols other than TCP or UDP should examine the source code for details. SSF.OS.Socket defines an application-level class dataMessage

public class dataMessage extends ProtocolMessage {

  public Object data;
  public int size;

  public dataMessage(Object o, int nbytes) {
    data = o;
    size = nbytes;

where 'data' is a reference to an arbitrary Object that actually contains the data to be transmitted.

Use SSF.OS.Socket to send real or virtual data

Application models can send and receive data both real and virtual data using BSD-style socket methods (example from SSF.OS.Socket):

 public void write(Object[] obj, int nbytes, Continuation caller);
 public void read(Object[] obj, int nbytes,  Continuation caller);

 public void write(int num_virtual_bytes, Continuation caller);
 public void read(int num_virtual_bytes, Continuation caller); 

Suppose a TCP socket is involved for the purpose of explanation. The first method writes an application-level object to be transmitted. Following the write(), the reference to this object will be encapsulated in a dataMessage, which will be transmitted as a payload of a TCP packet, while the value of nbytes determines what is the virtual size of this (and possibly following) TCP packets. The receiving TCP eventually hands the dataMessage to receiving socket, and the application can execute the read to obtain the reference to the transmitted object.

The second group of write and read methods are used to send and receive virtual bytes, i.e., no data is really sent, but TCP will form and transmit packets whose virtual payload sizes will be determined by the argument num_virtual_bytes.

read() and write() are blocking methods. For an explanation how the Continuation works to efficiently model blocking calls without actually suspending a thread, see another tutorial section below.

SSF.OS.Socket API also names methods for sending of byte-serialized raw data, using another polymorphic variant of write() and read():

 public void write(byte[] buf, int nbytes, Continuation caller);
 public void read(byte[] buf, int nbytes,  Continuation caller);

ProtocolMessage Storage Management

In any larger simulation a network model may easily generate billions of packets, which obviously creates a potential memory problem if storage is not reclaimed in a timely manner.

The common Java and C++ SSFAPI contains some rather extensive thinking about event allocation and deallocation, to hide significant differences between garbage collection in these two languages. The current version (SSFNet 1.1) of Java SSF simply uses constructors to create instances of ProtocolMessage, and relies on Java garbage collection for memory reclamation, so that modelers need not pay attention to the problem. C++ SSF (DaSSF and CSSF implementations) implement the rule that the SSF kernel has the right to reclaim the storage of any event after time has passed since the moment of its last receipt. Modelers can prevent deletion by calling save() on the received event (and subsequently release() to allow the deletion), or by explicitely making a copy.

However, in future versions for still higher efficiency the ProtocolMessages may implement pool-based storage management of header fragments. The goal is to preallocate messages in the pool, and grow the pool on demand, recycling headers rather than allocating new ones whenever possible.

Correspondence between ProtocolMessages and ProtocolSessions

Each nontrivial protocol model (such as TCP) will usually consist of several classes organized into a package such as SSF.OS.TCP, with at least one class derived from ProtocolSession, and one class derived from ProtocolMessage.

The class derived from ProtocolMessage will implement a minimal header associated with the protocol, containing the fields required by the modeling detail. Different implementations of these protocols may implement further extensions with additional fields. The goal should be to minimize memory usage, and always use the most minimal header class that still satisfies the requirements for detail in a given simulation.

Recommended Versioning and Typing of ProtocolMessages

The ProtocolMessage base class provides a virtual method version() which returns an arbitrary-length string; this provides a very basic dynamic typing mechanism for messages. Protocols can inspect the version() of a given header and compare to known message types in order to perform appropriate casts. (Java supports class inspection directly, but to minimize style differences, it is recommended in SSF.OS to use the version-string approach in both C++ and Java). The goal is to support some very basic dynamic "typechecking" of received packets. For example:

protocol        header          plausible version string
--------        -------         ------------------------
IP              IpHeader            "IPv4 v0.8"

The ProtocolSession class defines a similar version() method; the version strings of a ProtocolSession and the matching ProtocolMessage header class should correspond.

Serialization of ProtocolMessages

The ProtocolMessage base class provides virtual methods tobytes() and frombytes() to support serialization and deserialization on demand. Implementors of derived header classes must implement their own versions of these methods in order to support exchange of packets containing those headers as bytestreams. The default mode of packet exchange within the simulator will be via reference to a doubly-linked list of symbolic headers, reserving more literal (byte-oriented) representations for contexts (e.g., emulation, interoperation with other simulators) where they're worth the copying overhead.


A ProtocolGraph is a 'container' of all protocol services in a host or router. It creates protocol services at simulation startup (booting), and subsequently can locate them (return an object reference) when queried using the method SessionForName(String protocol_name).

Every protocol model must contain a class derived from ProtocolSession that is instantiated by the ProtocolGraph and configured from the DML database.

Configuring a ProtocolGraph

ProtocolGraph is self-configuring, using the DML database support from the package SSF.DML. Each ProtocolSession entry specifies a symbolic name for the protocol it implements, plus the name of a package (implementation) to use.

The configuration files for the protocol models distributed with SSFNet are included in the SSFNet 1.1 DML Reference. Below is the configuration schema for SSF.OS.TCP, see the method config() in the class SSF.OS.TCP.tcpSessionMaster how such an entry in the model's DML configuration database is interpreted:

ProtocolSession [name tcp use SSF.OS.TCP.tcpSessionMaster
    ISS %I                  # initial sequence number
    MSS %I                  # maximum segment size
    RcvWndSize  %I          # receive buffer size in units of MSS
    SendWndSize %I          # maximum send window size in units of MSS
    SendBufferSize %I       # send buffer size in units of MSS
    MaxRexmitTimes %I       # maximum number of retransmission times 
    TCP_SLOW_INTERVAL %F    # granularity of TCP slow timer, sec
    TCP_FAST_INTERVAL %F    # granularity of TCP fast(delay-ack), sec
    MSL %F                  # maximum segment lifetime, sec
    MaxIdleTime %F          # maximum idle time, sec
    delayed_ack %F          # delayed ack option, true or false
    fast_recovery %S        # use fast recovery algorithm, true/false 
    show_report %S          # print summary connection report, true/false

Further, notice that each protocol's class derived from ProtocolSession, such as SSF.OS.TCP.tcpSessionMaster, may define it's private DML attributes that are used to configure a protocol instance. These attributes are handled exclusively by the protocol implementation (see the source code of class tcpSessionMaster and how it uses the attributes contained in tcpinit), and can be defined either globally for all protocol instances in a model, or individually for Host instances.

The design rule that each protocol takes care of its own DML initialization allows new protocols to be implemented independently in a self-contained manner.

Order of protocol creation and initialization

Class SSF.Net.Net creates the entire network model in the following order:

  1. loads a DML configuration and verifies its syntax against the schema
  2. sets the time unit per SSF tick from Net.frequency
  3. constructs IP address blocks (CIDR)
  4. configures the model:
    1. creates the routers and hosts
    2. creates and connects all links to interfaces
    3. adds static routes (IP or NHI) to hosts and routers, if provided
and then starts the simulation.

In phase 1 of the configuration each Host or router calls the config() method of its ProtocolGraph, which parses the DML ProtocolSession attributes to find which protocols to install, and recursively calls each installed ProtocolSession's config() method to find the protocol-specific parameters required in installation.

Once the simulation starts (method startAll() is called from within SSF.Net.Net), each Host's ProtocolGraph calls the method init() of each installed protocol to finish any remaining installation steps and start the operation.

The order of ProtocolSession attributes in a host configuration database is important: Initialization od a higher protocol may depend on the completion of a lower protocol initialization. When the simulation starts, the protocols are initialized in the reverse order of their appearance in the graph attribute, thus for instance we might have a host configured as follows:

  host [
    id 1
    interface [id 0 bitrate 100000000 latency 0.0]
    graph [
    ProtocolSession [
      name server use SSF.OS.TCP.test.tcpServer
      port 1600               # server's well known port
      client_limit  10        # max number of contemporaneously allowed clients
                              # if omitted, default is no limit
      request_size  4         # client request datagram size (in bytes) - the same as in client
      show_report true        # print client-server session summary report
      debug false             # print verbose client/server diagnostics
    ProtocolSession [name socket use SSF.OS.Socket.socketMaster]
    ProtocolSession [name tcp use SSF.OS.TCP.tcpSessionMaster
        ISS 10000               # initial sequence number
        MSS 1000                # maximum segment size
        RcvWndSize  32          # receive buffer size
        SendWndSize 32          # maximum send window size
        SendBufferSize 128      # send buffer size
        MaxRexmitTimes 12       # maximum retransmission times before drop 
        TCP_SLOW_INTERVAL 0.5   # granularity of TCP slow timer
        TCP_FAST_INTERVAL 0.2   # granularity of TCP fast(delay-ack) timer
        MSL 60.0                # maximum segment lifetime
        MaxIdleTime 600.0       # maximum idle time for drop a connection
        delayed_ack false       # delayed ack option
        fast_recovery true      # implement fast recovery algorithm 
        show_report true        # print a summary connection report
    ProtocolSession [name ip use SSF.OS.IP]
assuring that ip is initialized first, followed by tcp, socket, and tcpServer, in this order.

About units of time

The selection of the simulated time "tick unit" (DML attribute Net.frequency represents the number of ticks per second) is very important - it should be equal to or less than the smallest time resolution in the model, otherwise the default time resolution may be too coarse and create unintended event concurrencies.

Remember that SSF advances the logical time in dimensionless "ticks". It is very important that all model components interpret these ticks in the same units. Therefore, when defining time constants in a protocol model, always use a physical time unit (seconds), and translate into ticks at model instatiation using the method SSF.Net.Net.seconds(double time_sec) that returns the translation of time in seconds into an integer number of ticks, (long)(time_sec*frequency). For an example, see how the method config() in class SSF.OS.TCP.tcpSessionMaster converts the DML attribute TCP_SLOW_INTERVAL (given in seconds) into a number of ticks.

This design rule is essential to assure that protocol models work consistently irrespective of a modeler's choice of the simulated time "tick unit" (provided it's not too coarse).


A ProtocolSession is a node in the ProtocolGraph. It defines the rules for processing ProtocolMessages that are pushed into it. A typical rule for incoming ProtocolMessages may be: strip the header, process the message as required, identify the next header in the linked list, and push it to the next ProtocolSession configured along the path in ProtocolGraph that this message should be traversing.

Versioning and Typing of ProtocolSessions

Like ProtocolMessage, the ProtocolSession base class provides a virtual method version() which returns an arbitrary-length string. The syntax of this string is the same as that of ProtocolMessages with regard to subclassing; in fact, the version strings of ProtocolSessions and the ProtocolMessages they handle should correspond.

Message Handling

One of the hardest SSF.OS design questions has been the nature of push() and the directionality of messages. In the layerist conception of a protocol stack, there's a clear preference for just two directions: UP and DOWN. However, as protocols are decomposed into more and smaller cooperating modules, the UP/DOWN distinction tends to make less and less sense. While one can continue to cast everything in light of UP/DOWN, it's better to think in more general terms of all kinds of "sideways" helper relationships among protocol modules.

Therefore, in the current SSF.OS interface, the abstract method

push(ProtocolMessage message, ProtocolSession fromSession)
takes two arguments: the message being pushed, and the session it's being pushed from, if any. (Messages pushed in from outside the protocol graph have a null sender.)

Message handling then begins with comparing the source session to known sessions. If the session is missing or unrecognized, the message's version() string can be inspected to determine what classes it implements; the unknown ProtocolMessage can hopefully then be cast to an appropriate header type and processed.

Internal to a session, the designer has complete freedom to implement whatever packet-handling algorithms are desired. The only constraints are that a session must (1) implement push() in order to determine the type of incoming messages (headers) based on the identity of the sending protocol and/or the version string of the message, (2) use the push() methods of other protocols to send messages on their way, and (3) never block the push() method.

If some delays are desired, create and set Timers or Continuations to implement any packet-handling behaviors that take place after elapsed time, but return from push() without any logical time delay.


Not all steps in ProtocolMessage processing can be implemented using only time-unaware (chains of) method calls. A session may need to start a retransmission or keepalive timer, or suspend until some condition is satified by another session at a later time. In the world of SSF.OS, it means that some auxiliary Entity (Entities are time-aware) has to call back a session's method after some elapsed time.

The SSF.Process methods waitFor() and waitOn() are disallowed from within ProtocolSession code. To understand why, recall that a single protocol graph may hold, at any moment in time, a large collection of messages. Each of these messages was pushed into the graph at some earlier moment in time, as part of the action of one of two processes: one that watches for application commands, and one that watches for packets arriving from a network device. If a ProtocolSession were to somehow execute a waitFor() in the processing of any single event, all processing of other events would come to a halt until the waitFor() expired --- probably a very undesirable behavior. To be consistent, we would have to allocate one new SSF Process for each incoming message, but this would be a massive waste of simulator resources, since Event processing is computationally expensive compared to method execution.

In other words, each event injected into the protocol graph should find its destination --- either exiting the protocol graph, or ending up in a buffer somewhere inside the protocol graph --- in zero elapsed logical time.

ProtocolSessions should schedule future actions by creating and setting a Timer. See the Timer javadoc for a description how to use it. Note that an instance of Timer is aligned to the containing ProtocolGraph, so that all timers in a host or router are guaranteed to share the same timeline in a parallel simulation.

Finally, here is an example of actual usage: A BSD-style TCP slow clock that fires every 500 ms and calls back the method slowTimeout() in each instance of a TCP session. In Java SSF.OS it's implemented as an inner class of SSF.OS.tcpSessionMaster as follows:

 class slowTimer extends Timer {
    public slowTimer(ProtocolGraph g, long t) {
    public void callback(){
      for(Enumeration list = TCPsessnList.elements();
      set(TCP_SLOW_INTERVAL);  // in SSF ticks!
and, when simulation starts, in the method init() of class SSF.OS.tcpSessionMaster the timers are started with a random initial tick offset init_time (so that distinct hosts do not have their clocks synchronized!)
    (new Timer(inGraph(),2){
        public void callback(){
          transTimer = new slowTimer(inGraph(),SSF.Net.Net.seconds(0.001));
          if((tcp_opt & DELACK) != 0){
            ackTimer = new fastTimer(inGraph(),SSF.Net.Net.seconds(0.001));
          System.out.println("slow Timer start at:" + inGraph().now());

Modeling of blocking methods with the Continuation pattern

Another common situation in protocol modeling where simple method calls are insufficient is modeling of blocking system calls, such as blocking read() or write() for sockets in SSF.OS.Socket. Notice that in the spirit of SSF.OS (and for the sake of high performance), the blocking behavior should be modeled with method calls only, and not by employing auxiliary SSF Processes, in/outChannels and Events.

There are several possible solutions, as usual. The one that was successfully used in the implementation of protocol package SSF.OS.Socket is based on a callback interface called Continuation. The idea is to use an instance of this class to represent the client's 'continuation' (that is, its future activity after the blocking call returns). It's perhaps of interest to note that Continuation predates OO, and goes back to the old days of functional languages like Scheme and ML. That in turn is because continuation-passing has a warm place in the heart of language theoreticians who study control flow semantics. Exceptions, threads, and GOTO can all be described and even built using a continuation-passing mechanism.

Continuation defines two methods: success() and failure(). At the right time, one can activate either branch of the continuation, depending on the circumstances.

 interface Continuation {
    void success();        
    void failure(int errno);

For examples of how this interface is used in ssfnet0.8, see the examples of a client and server in directory src/SSF/OS/TCP/test together with the source code of class SSF.OS.TCP.tcpSocket.

Here is the basic idea: in general, to use it, we translate this code sequence:

     a(); b(); c(); d(); e(); return; // only c() is a blocking call 
into this one:
     a(); b(); 
     c(new Continuation() {           // anonymous inner class
        void success() {
           d(); e();
        void failure(int errno) {
           // Woe is me.
     });  return;
A class whose method is c() saves the reference to Continuation, and does something like this:
    Continuation call_waiting = null;
    public c(Continuation caller) {
      call_waiting = caller;

    unblock(int errno) {
      if(errno > 0) {call_waiting.failure(errno);}
      else {call_waiting.success();}
and whoever is controlling the blocking is supposed to call unblock() at a time when c() should "return from block". Note the immediate return, since all remaining work will take place in the future when (if) the continuation fires. Note also that anonymous classes cannot define constructors since they have no names.

SSF.OS API (Java binding) v0.8 - v1.1

public class ProtocolGraph extends Entity 
implements Configurable {
  public ProtocolGraph();
  public void config(Configuration cfg);
  public void init();
  public ProtocolSession SessionForName(String protocol_name);

public abstract class ProtocolSession
implements Configurable {
  public void init();
  public abstract boolean push(ProtocolMessage message, 
                               ProtocolSession fromSession);
  public void config(Configuration cfg);
  public String version();
  public ProtocolGraph inGraph();
  public void setGraph(ProtocolGraph G);
  public void open(ProtocolSession S, Object request);
  public void opened(ProtocolSession S);
  public void close(ProtocolSession S);
  public void closed(ProtocolSession S);

public class ProtocolMessage {
  public float size();
  public String version();
  public static ProtocolMessage fromVersion(String V);
  public ProtocolMessage();
  public void dropPayload();
  public void carryPayload(ProtocolMessage payload);
  public ProtocolMessage payload();
  public ProtocolMessage previous();
  public int bytecount();
  public void tobytes(byte[] buf, int offset);
  public void frombytes(byte[] buf, int offset);

public class PacketEvent extends Event {
  public ProtocolMessage getHeaders();
  public PacketEvent();
  public PacketEvent(ProtocolMessage msg);
  public PacketEvent(Class pktclass);

public class Protocols {
  public static int getProtocolByName(String pname);
  public static String getProtocolByNumber(int pnum);

public abstract class Timer extends Entity {
  public boolean isCancelled();
  public abstract void callback();
  public void set();
  public void set(long dt);
  public Timer(ProtocolGraph e, long dt);
  public final void cancel();

public interface Continuation {
  public void success();
  public void failure(int errno);