Scalable Simulation Framework
Composition of very large networks

back to list of tutorials...

Example of a large size network design

This is the last of SSFNet DML tutorials. It shows multi-stage Net import allowing to build very large models from pre-configured sub-Nets, and the configuration of BGP routers for inter-domain routing. An interesting network to design is a national-size network whose topology is patterned after one of the larger US ISPs, with "corporate AS networks" at each NAP generating the HTTP/TCP traffic using a measurement-based design of a web client application and simple HTTP servers. The actual target of this tutorial is the network shown below.

The complete DML configuration database is contained in three separate files, the usa1.dml that configures the entire model; the database of component Nets networks.dml, and another database of reusable DML fragments dictionary.dml.

Note: To execute this model you need a machine with large memory (over 2 GB) and serious computing power:
it has been tested by BJ Premore on a SUN Enterprise 6500 server.

Network usa1: it consists of 25 virtually identical Autonomous Systems, each containing 1,300 hosts running a web user modeling application, 4 web servers, 27 internal OSPF routers and one AS boundary router running BGP4. The BGP routers are connected with wide area point-to-point links. The wide area topology is a simplified version of the network of one of the large IP network providers. The complete model contains 33,300 multi-protocol hosts and routers. During a simulation, after the initial routing convergence period, each hosts models the empirically measured behavior of a web-browsing user. [ PDF version for printing. ]

Design in stages

You should open (or print) the file networks.dml at this point.

We begin by designing a small database of networks, and recursively composing them into larger ones until we obtain the complete design.

Our smallest buiding block is a router with four LANs, and 25 host machines on each LAN. We call it network100.

network100. The rectangle is an IP router, heavy lines are 100Mbs LANs (implemented as links with multiple interfaces). Each LAN has 25 hosts attached (not drawn). The router has 3 unattached 1 Gbs interfaces which we will use later to connect several such small networks together.

The networks database also contains a very similar network50.

Looking at the DML specification of .networks.network100.Net, you should notice that the protocol graph configurations are stored in the attribute .dictionary.routerGraph.graph for the router, and in the attribute .dictionary.client100Mb.graph for each host. If you now look in the file dictionary.dml, you will find that the router has the OSPF protocol installed. The hosts' protocols are a bit more complex, and when expanded, for example the specification of the host with id = 1 is:

    host [
      id 1
      interface [id 0 bitrate 100000000 latency 0.0]

      graph [
        ProtocolSession [name client use SSF.OS.WWW.httpClient

          # earliest time to send the first request to server (seconds).
          start_time 1000.0

          # HTTP session arrival process in this client. The first session
          # begins at start_time + inter_session_time (in seconds).
          # Each session is with a server chosen uniformly at random from
          # the servers specified in the .Net.traffic for this client.
          inter_session_time [
            distribution [
              name "Exponential"         # distribution class name
              lambda 0.01                # mean = 1/lambda

          # distribution of number of pages per session with selected server.
          # if random variable = 0, round up to 1
          pages_in_session [
            distribution [
              name "Exponential"         # distribution class name
              lambda 0.2                 # mean = 1/lambda

          # distribution of user-think-times (off-times) between consecutive
          # in-session page requests, counted from the time when the previous
          # page delivery is completed (in seconds). Mean = 
          inter_page_time [
            distribution [
              name "Pareto"              # distribution class name
              k 25.0                     # scale (cutoff) parameter
              alpha 2.0                  # shape (exponent) parameter

          # distribution of delays between consecutive requests in a session.
          # counted from the time when the previous request was completed.
          # (In seconds.)
          inter_request_time [
            distribution [
              name "Pareto"              # distribution class name
              k 0.1667                   # scale (cutoff) parameter
              alpha 1.5                  # shape (exponent) parameter
          http_hdr_size 1000  # nominal HTTP header size (virtual bytes)
          persistent true     # if true, one persistent TCP connection per
                              # session (HTTP1.1), if false, one TCP connection
                              # for each request/response (HTTP1.0).
          show_report true    # print client-server session summary report
          debug false         # print verbose client/server diagnostics
        ProtocolSession [name socket use SSF.OS.Socket.socketMaster]
        ProtocolSession [name tcp use SSF.OS.TCP.tcpSessionMaster
                             _find .dictionary.tcpinit]
        ProtocolSession [name ip use SSF.OS.IP]
      nhi_route [dest default interface 0 next_hop 0(3)]

We use the experimental release of the application-level ProtocolSession SSF.OS.WWW.httpClient. See the source code for details of the client's behavior; in a nutshell, the httpClient implements a renewal process alternating user idle periods and user browsing periods, and within each "user browsing period" there is a number of page requests, with a random distribution of "user reading/thinking time" between page requests, etc. The corresponding SSF.OS.WWW.httpServer contains additional server-side configuration parameters such as the distribution of the number of objects in a page request, distribution of sizes of objects, and the distribution of server delays between request and response.

This simplified web user model is a somewhat expanded version of the model published in:

Anja Feldmann, Anna Gilbert, Polly Huang and Walter Willinger. Dynamics of IP traffic: A study of the role of variability and the impact of control. In the Proceedings of the ACM/SIGCOMM'99, 1999, Cambridge MA. Available at Anja's web site.

Design in stages continued

Next, we'll create several more networks.

The configuration of network4, network33, network34, network21 and network1 is very similar to the preceding tutorial 3, and will not be explained further. Instead, we show their topologies in a sequence of diagrams below. (The names of these networks are used for historical reasons - they were pulled from another network database).

network4. A server farm with 10Gbs network interfaces. Rectangles are IP routers, squares are server hosts connected with point-to-point links to routers. [ PDF version for printing. ]

network33 is constructed by importing network100 four times. Note distinct NHI adressing of routers imported with network100, and routers created directly in the DML tuple. [ PDF version for printing. ]

network34 is similar to network33, only a little bigger. [ PDF version for printing. ]

network21 has topology similar to the example in tutorial 3, but instead of having a couple of hosts per lan it imports network50 and network100 multiple times. [ PDF version for printing. ]

Design in stages continued: an Autonomous System

Finally - using a 3-router 10Gbs backbone network1 - we create a modest Autonomous System model. It has the backbone, there are something like three branch offices, and there is our server farm as well. It is stored in file networks.dml as network named as2:

Network as2 imports network1, network4, network33, network34 and network21 to represent a modestly sized Autonomous System with a 10Gbs backbone. Note that the routers have expanded NHI addresses relative to the entire network as2. [ PDF version for printing. ]

A new feature is the AS boundary router running both BGP and OSPF protocols. This is shown in the initial fragment of the DML specification of as2:

   as2 [
      Net [
        AS_status boundary
        ospf_area 0

       # a big BGP router with multiple unattached interfaces
       # that will provide attachment points for inter-domain links
       router [ id 10
        interface [ id 0 _extends .dictionary.100Gb ]
        interface [ id 1 _extends .dictionary.100Gb ]
        interface [ id 2 _extends .dictionary.100Gb ]
        interface [ id 3 _extends .dictionary.100Gb ]
        interface [ id 4 _extends .dictionary.100Gb ]
        interface [ id 5 _extends .dictionary.100Gb ]
        interface [ id 6 _extends .dictionary.100Gb ]
        interface [ id 7 _extends .dictionary.100Gb ]
        interface [ id 8 _extends .dictionary.100Gb ]
        interface [ id 9 _extends .dictionary.100Gb ]
        interface [ id 10 _extends .dictionary.100Gb ]
        interface [ id 11 _extends .dictionary.100Gb ]
        interface [ id 12 _extends .dictionary.100Gb ]
        interface [ id 13 _extends .dictionary.100Gb ] # attached to AS-internal
        _find .dictionary.BGProuterGraph.graph

       (... stuff omitted...)
See what is done here: we gave the router a lot of unattached interfaces (0 through 12) to be able to use them later to connect many clones of the as2 network together to form a large multi-domain model.

Let us expand the line

        _find .dictionary.BGProuterGraph.graph

It is equivalent to the full specification of the router's protocols:

dictionary [

  # AS boundary router
  BGProuterGraph [graph [
    ProtocolSession [ name bgp use SSF.OS.BGP4.BGPSession ]
    ProtocolSession [ name ospf use SSF.OS.OSPF.sOSPF ]
    ProtocolSession [ name socket use SSF.OS.Socket.socketMaster ]
    ProtocolSession [ name tcp use SSF.OS.TCP.tcpSessionMaster ]
    ProtocolSession [ name ip use SSF.OS.IP ]

  (... stuff omitted...)
As you can see above, BGP uses the TCP transport.

The last design stage: connect 25 Autonomous Systems

We show again our design goal:

At this point you should look at the file usa1.dml that configures the entire model.

The configuration should be straightforward to read. There are two interesting code fragments: we use a minimum amount of typing to import network as2 25 times:

      Net [idrange [from 0 to 24]
        _extends .networks.as2.Net

The interesting part is the traffic scenario configuration:

    traffic [
      pattern [ client 0 _extends .dictionary.serverPattern ]
      pattern [ client 1 _extends .dictionary.serverPattern ]
      ... 21 similar lines omitted...
      pattern [ client 23 _extends .dictionary.serverPattern ]
      pattern [ client 24 _extends .dictionary.serverPattern ]
Let us expand the parent attribute .dictionary.serverPattern: with that, every client has the same list of the servers to choose from. Let us illustrate this in the particular case of the "Boston" AS:
      pattern [ client 24 _extends .dictionary.serverPattern ]

The above line, when fully expanded, states that every host whose fully expanded NHI address begins with 24 (such as for instance the host 24:3:4:1 - locate its LAN in the diagrams above!), and which has the httpClient protocol installed, will be choosing servers from the list specified by the servers attributes.

You can easily see that this server list includes all servers in the model. Thus, the traffic scenario specified above states that every client may choose any server, and if you look into the source code of SSF.OS.WWW.httpClient, you can find that each new client's session begins with choosing a random server.

     pattern [
       client 24
       servers [port 80 nhi_range [from 0:1:2(0) to 0:1:5(0)]]
       servers [port 80 nhi_range [from 1:1:2(0) to 1:1:5(0)]]
       servers [port 80 nhi_range [from 2:1:2(0) to 2:1:5(0)]]
       servers [port 80 nhi_range [from 3:1:2(0) to 3:1:5(0)]]
       servers [port 80 nhi_range [from 4:1:2(0) to 4:1:5(0)]]
       servers [port 80 nhi_range [from 5:1:2(0) to 5:1:5(0)]]
       servers [port 80 nhi_range [from 6:1:2(0) to 6:1:5(0)]]
       servers [port 80 nhi_range [from 7:1:2(0) to 7:1:5(0)]]
       servers [port 80 nhi_range [from 8:1:2(0) to 8:1:5(0)]]
       servers [port 80 nhi_range [from 9:1:2(0) to 9:1:5(0)]]
       servers [port 80 nhi_range [from 10:1:2(0) to 10:1:5(0)]]
       servers [port 80 nhi_range [from 11:1:2(0) to 11:1:5(0)]]
       servers [port 80 nhi_range [from 12:1:2(0) to 12:1:5(0)]]
       servers [port 80 nhi_range [from 13:1:2(0) to 13:1:5(0)]]
       servers [port 80 nhi_range [from 14:1:2(0) to 14:1:5(0)]]
       servers [port 80 nhi_range [from 15:1:2(0) to 15:1:5(0)]]
       servers [port 80 nhi_range [from 16:1:2(0) to 16:1:5(0)]]
       servers [port 80 nhi_range [from 17:1:2(0) to 17:1:5(0)]]
       servers [port 80 nhi_range [from 18:1:2(0) to 18:1:5(0)]]
       servers [port 80 nhi_range [from 19:1:2(0) to 19:1:5(0)]]
       servers [port 80 nhi_range [from 20:1:2(0) to 20:1:5(0)]]
       servers [port 80 nhi_range [from 21:1:2(0) to 21:1:5(0)]]
       servers [port 80 nhi_range [from 22:1:2(0) to 22:1:5(0)]]
       servers [port 80 nhi_range [from 23:1:2(0) to 23:1:5(0)]]
       servers [port 80 nhi_range [from 24:1:2(0) to 24:1:5(0)]]
It is rather easy to modify this traffic scenario: for instance, if you'd rather have a highly focused traffic pattern, you might specify that the available servers are only in San Francisco and in New York City. To do that, all you need is to edit the corresponding attribute .dictionary.serverPattern to look as follows:
     serverPattern [
       servers [port 80 nhi_range [from 3:1:2(0) to 3:1:5(0)]]   # San Francisco
       servers [port 80 nhi_range [from 16:1:2(0) to 16:1:5(0)]] # New York City

Final comments

The model described above exercises only external BGP. You presumably noticed that it would be more realistic to create a transit domain with its own routers and network access points, internal BGP, and to connect ASs at the access points rather than connecting them directly via their AS boundary routers (router with id 10 in network as2).

The latest release of SSF BGP4 protocol includes support for both external and internal BGP, for route reflectors, and for policy specification for policy-based routing simulations. See the BGP4 pages for examples of more elaborate SSFNet models involving all aspects of BGP modeling.

The focus of this tutorial was on hierarchical composition of a large network from preconfigured Nets, and not all specifications of routers were provided. In particular, we omitted the specification of packet buffer sizes on the routers' network interfaces (by default, they are of unlimited size).