This document may be reproduced in any form providing you print my name and contact information, the copyright notice, this message, and that you send me details of the publication.

Traffic Traffic Everywhere - An analogy between network traffic and road users.

Consider an IP network consisting of routers and circuits. Packets flow through the network and in doing so, they take a path specified by whatever routing protocol is implemented in the network.

An analogy to this situation is a real road network in a country. Cities can be thought of as routers, the roads connecting cities together can be likened to circuits with cars being the packets.

Traffic through a transit network can be then explained in the following way. Three countries A, B, and C exist with A connected to B, and B connected to C. If a car starts off in country A, and wishes to drive to a point inside country C, it will transit country B. Country B is a transit network for that particular car. That is to say the car does not start from within B, nor does it stop within network B, but the car does travel through network B.

When travelling through B, the car will take a path between the cities of network B, travelling down roads of varying capacities. The first adjustment (of only two) that we have to make in order for the analogy to be accurate is the idea of a routing protocol. The driver of a car on a road can take whatever path they like in order to cross country B. In an IP network, the path is dictated by the routing protocol. From the perspective of the driver of the car, this is somehat like being handed a route by the border control. The border control would specify the route city by city that a driver must take, until the border at the other side of the country is reached. The routing within a country is handled by one organisation, and in this regard the country operates as an Autonomous System, independent from the routing organisations of all other countries.

The second adjustment to the analogy involves neighbours. A transit network is connected to each of its neighbours by a circuit, however the neighbours of a transit network may be geographically separate, may overlap or may even cover the same geographic area. The easiest way to think of this in the road analogy is to imagine the existence of futuristic teleportation units. These are positioned at the border of countries and when a car drives into one, it will pop out at the border of some other country. In such a way it might be possible to drive from Switzerland, through the US, into Japan, and then on to Italy.

So what we end up with is a driver who leaves one point in his original country, drives to the border, is transported to the border of another country, receives instructions from the border control on what route to take, drives to the next border, and so on. Eventually they arrive at their destination.

In an IP network, the network designers are responsible for the choice of where routers are placed, the positioning of links between routers, the speed of said links, as well as the choice of routing protocol, and what other networks to connect to. This can be a very demanding job when taking into account limited budgets, particularly when there are a large number of routers and lots of traffic to deal with. The analogy works well here. Just think, where are we going to put the cities? which cities will we connect with roads? How wide shall we make the roads? What path will we tell cars to take when they want to drive across the country? What countries can we best allow traffic in from, and what countries can we send the traffic out to? In order to answer these questions it is necessary to know where cars will be coming from, where they will be going to, and how many of them there are.

When building a brand new road network it is impossible to know accurately who will want to cross your country. Fortunately countries tend to start out with small traffic volumes and gradually increase in size. This makes expansion easier because the designers can look at how busy the roads currently are and scale accordingly. This is where things start to get interesting. In IP networks the most popular form of statistics related to traffic flows are gathered via SNMP (Simple Network Management Protocol). These statistics tell the designer how many bytes went in or out a router's interface. This is equivalent to the road designer counting the number of cars travelling along a road. This allows the designer to be able to see when the road starts to get full, and when to put in more lanes on the road. Unfortunately it doesn't say anything about the path taken by traffic. Maybe a lot of the cars travelling down the road from city X to city Y are then continuing on from city Y to city Z. If the designer knew this, they would put in a road from city X to city Z. Unfortunately this critical information is unknown to the designer who is using SNMP statistics! So the designers have to guess. Both circuits and roads are expensive to install, and can have lengthy installation lead times. The result is a network where there are many grossly under utilised circuits, and a few over utilised circuits as well. Maintainence is spent on rarely used roads that have been over engineered, while other roads struggle to hold the traffic they are being asked to carry.

Another problem is the connectivity of our network to the neighbour networks. Who do we connect to? Some countries have a particularly large number of cars travelling through them. They become well known through the sheer number of neighbours they have, and frequenty advertise their countries in glossy magazines. The ads make great promises about their country having fast roads and good connectivity to other coutries. Clearly we would like our network to have a connection to this super transit network so driver's from our network can easily get to anywhere they want to go. As always there's no such thing as a free lunch, and the super transit company will probably charge us for every car we send out via their country. That could get expensive! It could also be improved upon, as many of the cars will likely not finish on the super transit network, but may continue off to a smaller country. Perhaps a holiday resort country. If we also connected to that holiday resort country directly, the cars driving across our network will not have to drive as far to get where they're going. We would not be sending nearly as many cars via the super transit network either, and we wouldn't have to pay them as much.

How can we gather this sort of wonderful useful information? Certainly not with SNMP statistics. With SNMP statistis we're limited to the scenario of standing by the roadside counting cars. How futile!

The better way

More recently, Cisco, in a gesture of kindness to network designers everywhere, began to implement what they call Netflow Exports. These contain vast amounts of detail gathered from the IP packets as they pass through a router. Such information allows the design of a more efficient network as they contain useful details such as source and destination IP addresses, Autonomous System numbers, UDP/TCP port numbers, as well as byte and packet counts.

With this information a road planner can see what destination countries the cars are driving to as well as the countries they will pass through on their way there. This is exactly the sort of detail the designer needs in order to efficiently choose the destination countries for the border teleportation units. The paths of drivers' journey can be shortened by minimising the number of countries they must drive through (the number of hops as it is known), while also minimising the number of cars that are sent via expensive super transport networks.

Within the country the statistics also prove useful. The designer can now tell where cars are destined within the network. They can look at the cars on a particular road and see what their desired destination is. This is an important distinction. With SNMP statistics we can look at the cars travelling along a road from city X to city Y, but we know nothing about where they want to go. With Netflow Exports, we can see that most of the cars really aren't interested in city Y at all, but pass through to city Z. When we have this information, we have the power to make accurate design decisions. Depending on exactly how much of the traffic goes to city Z, we may well decide to build a road from city X to city Z, and either reduce or even entirely eliminate the road from city X to city Y. Drivers are happier now because they don't have to drive as far, and we have even saved some bandwidth as the city X to city Y road traffic is reduced.

All of this may sound too good to be true. Indeed you might be wondering about the drawbacks of Netflow Exports. Indeed there are some. Firstly, they increase the load on a router. It is therefore important to make sure the routers have enough spare capacity before enabling Netflow Exports. The other problem is related to the volume of statistics. Potentially, a statistical record could be generated for each packet that flows through a router! This could lead to vast volumes of statistical data being produced. In reality things aren't this bad, with each record (called a flow) typically representing many packets. The volume of data is still significant however, and special techniques are used to reduce and manipulate the statistics. The vast quantity of statistics also means there are often numerous ways to interpret the resulting statistics, and for this reason special tools for their collection, analysis, and visualisation have been built. Network Intelligence is such a product and its visualisation engine is capable of displaying the traffic for an entire transit network. High speed three dimensional graphics based on OpenGL are used to immerse the user in a representation of their own IP network and associated traffic.

For more information visit the online Network Intelligence site

Written by Martin van den Nieuwelaar,
martin at gadgets.co.nz
26 November 2001.