A Distributed Real-time Tool for IP-Flow Measurement

Yoshinori Kitatsuji
Katsuyuki Yamazaki
KDDI R&D Laboratories, Inc.
2-1-15 Ohara Kamifukuoka-shi Saitama, Japan
{kitaji, yamazaki}@kddilabs.jp
Abstract
It is getting more difcult to monitor multiple services as
well as to detect and/or to trace Daniel of Service attacks
with only tools showing graphs of the whole IP layer trafc
like MRTG or by checking counters of router interfaces. In
this paper, we discuss the specication of a software-based
real-time measurement tool for ow which consists of multiple capture devices, a manager device and user interface
devices, enabling exible ow denition on demand without
stopping system and working with IPv4 and/or IPv6, while
also enabling high performance. With this discussion, we
propose its architecture, bit-pattern-based ow denition
method and data structure. Then we report on the performance evaluation of a prototype of proposed real-time ow
measurement tools developed on PC-UNIXs and show that
the number of bit-pattern composing ow denitions impact
on the performance. Lastly we show an example of measuring ows in a real world environment and conrm that the
ow extraction is simplied.

1. Introduction
It is getting more difcult to monitor multiple service
classes as well as to detect and/or to trace Daniel of Service
attacks by using only tools showing graphs of the whole
IP layer trafc like Multi Router Trafc Grapher (MRTG)
[1] or by checking counters of router interfaces. It is much
more useful to use a ow visualization tool that measures
ows using any portion from the IP headers up to application data transported by the transport layer when this kind
of problem exists. However, it is expensive to develop hardware equipment measuring ows on high speed links with
such methods because of the complexity of the ow identication processes and with the number of ows detected in
short time on such high speed links.
In this paper, we propose a specication of softwarebased real-time measurement tools of ow which consists
of multiple capturing devices, a manager device and user
interface devices, and propose its architecture, bit-patternbased ow denition method and data structure. Then we

report on the performance evaluation of a prototype of a

proposed real-time ow measurement tools developed on
PC-UNIXs and show that the number of bit-pattern composing ow denitions impact on the performance. Lastly
we show an example of measuring ows in a real world environment and conrm that the ow extraction is simplied.

2. Requirements and Problems with Flow measurement

2.1. Background
MPLS based network requires monitoring ows to optimize usage of entire administrated domain. The managed
networks such as Grid Computing research networks are expected to progress the optimization of ow controls based
on operational administration policy and QoS requirements.
To optimize usage efciency in these networks, one of key
issues is monitoring what a ow hinder from the other
reaching its target speed. This kind of monitoring requires
ow measurement up to millisecond class accuracy to grasp
behavior how much a ow consumes bandwidth.
On the other hand, Contents of trafc passing networks
such as Internet Service Providers (ISPs) are changing to
P2P applications or unexpected DoS attack trafc from
News, WWW or traditional services. The requirement of
ow-based trafc monitoring is rising for stable operation
under such trafc consisting of new applications or services.
Generally, The following steps are taken several times in
IP network operations to troubleshoot network congestion
lled by new applications or unexpected DoS attacks.
1. Detection of specic trafc pattern
2. Trafc ltering or BGP route exchange restriction
3. Verication how much action affected a problem
Procedure 1 and 3 are performed by checking trafc graphs
or counters of router interfaces. However, it is difcult
to understand what kind of application and how much its
trafc cause a problem, and how much actions affect it by

Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE

checking router interface counters or trafc graphs. In this

case, 1) extracting trafc ows with a specic pattern from
header of the network layer up to application data and 2) visualizing ows extracted from 1) on demand in real-time
help to understand the problem characteristic and reduce
operation time to solve.

Ethernet
Switch

(1) Traffic is mirrored and copied

traffic is forwarded to Distribution device
Distribution device
(2) Traffic is distributed
to multiple capturing
devices

2.2. Flow Measurement

Generally, ow is dened as a set of packets passing an
observation point in a network during a certain time interval
and having the same 5-element-tuple of source IP address,
destination IP address, protocol, source port number and
destination port number[7]. Especially, this denition can
certainly specify ows requrired in ow measurement in the
managed networks described in previous session. However
we take ow in more wide sense to dene as a set of packets having common properties specied by elds between
protocol headers and application data. This enables to comprise both 5-element-tuple and 1) required in IP network
operation.
We dene ow measurement as clarifying the properties
of trafc passing an observation point, derived from statistical length (total number of packets or bytes) and existing time (gap between rst and last packets) of ows. The
ow measurement process consists of trafc capture, ow
identication, statistic calculation and data storing. The
ow identication process requires the high performance
and often expensive hardware when measuring ows passing through high speed links.
Some router products equip packet sampling function
that routers capture one of every multiple packets passing
interfaces. Captured packets are sent to Cowd devices or
stored in storages. This feature enables to show trends of
trafc passing the router, however it cannot keep up to monitor specic ows required in managed networks. Some
routers have a function to capture every packet passing an
interface, however such processes often cause performance
failure of the packet forwarding or is adopted to only slow
speed interfaces.

2.3. Related Works

In IP network operations, MRTG is one of the most famous tools that generate trafc graphs. It collects counter
values of interfaces by using Simple Network Management
Protocol (SNMP) and generates graphs and HTML pages.
The reason that MRTG is used widely is that graphs and
HTML pages are automatically generated and that SNMP
is used to get any values supported by the Management Information Base (MIB) of many network equipment. However, it is difcult to apply MRTG to measure ows because
routers or switches do not provide ow-based statistics.
In the other related works, the Cooperative Association
for Internet Data Analysis (CAIDA) [2] provides tools and

Capturing devices

Manager devices

(3) Measured flow information

is collected to Manager
device
(4) Users view flow
property via user
interface devices
which draw graph

User Interfaces

Figure 1. Basic Architecture of Real-time Flow

Measurement Tool
analysis promoting the engineering and maintenance of a
robust, scalable global Internet infrastructure. Cowd [3]
and NeTraMet [4] are developed as trafc ow analysis
tool in this task. Cowd collects and analyzes the information exported from NetFlow, which was initially developed by Cisco in its Quality of Service program.[5] NeTraMet is a ow accounting tool to identify ows specied
by the datalink layer address, IP address, port number or
any combination of these. However, it cannot identify ows
dened by other header elds or application data. Both systems can measure ows; however, they cannot collect ows
from high speed links by using multiple devices with mechanisms that share ow denitions and cooperating.
In this paper, we propose specications of the softwarebased real-time ow measurement tools consisting of multiple capture devices and implement its prototype that cooperates between multiple capturing devices and a manager
device to measure ows passing through high speed links.

3. Flow Measurement Tool

We propose a design of a software-base ow measurement tool consisting of a relatively inexpensive, multiple
PC-UNIXs based on the discussion in the previous section.

3.1. Specication
We discuss requirements that capturing and ow identication processes are performed on multiple devices in parallel to bring out the high performance in ow identication
process (Figure 1).

Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE

Manager device
Main
thread process
Statistics is saved in
files periodically

Flow definition and statistics are stored.

Memory All thread process refer this memory
shared between them
... ....... ......
....... .... ........

Capturing device
Memory
... ....... ......

Periodical reports are sent to a Manager device

from Capturing devices. Each connection is
managed by a thread process of Capture device.

...... ... .... .. ..

Save thread
...... .. . .........
process
Hard disk
Definition update
and Collection
User interface ....... ...... ......
thread process
service thread
process

Flow identification
thread process

....... .... ........

...... ... .... .. ..

Report and definition ...... .. . .........

update thread
process
....... ...... ......
A Manager device advertises flow definitions when
detecting the difference of definition between a report
from a Capturing devaice and registration.

Buffer

Captured Packets
are buffered
Packet buffering
thread process

NIC
Capturing device
Capture Device

Capturing device
Capture Device
Packets are captured at a Capturing device

User interface devices

Figure 2. Architecture of a Distributed Real-time Flow Measurement Tool

The system consists of a distribution device, multiple
capture devices, a manager device and user interface
devices as described in Figure 1. The A user gives a
manager device ow denitions through user interface
devices. A manager device should have mechanism
to advertise multiple capture devices consistent ow
denitions.
A capturing device imports all ow denitions registered with a manager device and identies the ows
with these denitions. If a distribution device distributes trafc in some specic manner, a capturing
device may be able to work with some of all ow definitions. However this kind of method generally requires the high performance on the distribution device
to supporting capturing device to operates closely with
both manager and capturing devices. As a result the architecture of distribution method tends to be complex.
Therefore it is better for the capture device to identify
ows regardless of the distribution method.
A manager device determines the length of a ow
based on the packet interval dened by users, from
reports sent from multiple capturing devices. A manager device equip buffer to store report and re-calculate
ow length because reports sent from capturing devices dont reach at the same time. A capturing device
needs to put the timestamp of the rst and last packets
in a reporting interval for every ow information for
the re-calculation of ow length at manager device.1
1 There is no connection between the interval of packets to dene a ow
and report interval of capturing devices.

Time synchronization is required between multiple

capturing devices because the timestamp of the rst
and last packets in every report should be consistent
between capturing devices. The time synchronization
accuracy of GPS becomes microseconds and NTP becomes milliseconds as reported in [6]. Therefore, GPS
or more highly accurate time synchronization system
such is required to measure high speed links, because a
packet takes only 512 ns in GigabitEthernet link lled
up by only 64 byte packets.
It is important that ow denitions are registered anytime when the system is operating and that multiple
ow denitions are used in a system. The same led of
patterns specied by multiple ow denitions should
be omitted to improve the ow identication performance.
It is important periodically to detect of how much loss
occurs to notify users. Reporting capturing loss without any luck involved is more important than high performance with some capturing loss that is not detected.

3.2. Architecture
We propose an architecture of distributed real-time ow
measurement tool as shown in Figure 2 based on the requirements of the previous section.
Firstly, trafc is copied by an Ethernet switch or ber
optic coupler and forwarded to a distribution device. Then
distribution device forwards each packet to one of multiple
capturing devices. we assume that a general-purpose distri-

Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE

bution device is used and that it has a round-robin distribution method, in this paper.
There are 3 thread processes running in capturing devices as described below:
Packet buffering process is deployed to absorb burst
trafc occuring in short time.
Flow identication process identies every packet for
all ow denitions and calculates statistics.
Report and denition update process periodically
makes reports to a manager device and receives message to update ow denitions from a manager device.
The use of PCAP library[8] is one of solutions for capturing processes in a capturing device. A PCAP library can
capture both IPv4 and IPv6 concurrently and make it easier to implement/port a capturing device on/to the variety of
OSs. Additionally, it provides an interface to collect capturing loss, to application programs attempting to detect and
count.
The same ow denitions are advertised to multiple capturing devices from a manager device. The report and denition process controls to derive or remove a packet buffering process and a ow identication process when a capturing device starts. All processes share ow denitions and
statistics constructed in memory space.
There are 4 types of thread processes operating in a manager device as described below.
Main process manages the connections establishment
from capturing devices or user interface devices and
derives the 3 processes below.
Denition advertisement and collection process receives reports from capturing devices and advertises
ow denitions when detecting any difference between registered denitions and denitions in a report.
Save process stores data collected from multiple capture devices periodically.
User interface service process receives ow denitions or sends ow data from/to the user interface devices.
A manger accepts connections from capturing devices
at any time to add and remove capturing devices on demand. A manger device has a mechanism to lter connections based on source address at the same time.

3.3. Flow Denition and Data Structure

In a proposed system, we dene a ow denition as a
set of chained bit-patterns. The bit-pattern is dened by

Table 1. Elements Specifying a Bit-pattern

Pattern ID
Position
Length
Mask
Minimum Pattern
Maximum Pattern
Child Pattern ID
Parent Flow ID

Identier used for reference by other

bit-patterns
Position from a top of IP packet up
to the 255th byte
Length of bit-pattern described in
byte unit
Species a valid and invalid bits
Minimum value to specify range of
bit-pattern
Maximum value to specify range of
bit-pattern
Reference list of chained bit-patterns
Condition value of parent pattern match to exceed to use this bit-pattern

elements as described in Table 1. A ow denition consists

of multiple bit-patterns. The chain of bit-pattern implies
the and operation for pattern matching.
When a capturing device receives a packet, it picks up
a bit eld specied by position, length and mask of a bitpattern. We call this bit led a ow identier. The original
packet is judged as matching with the bit-pattern if its ow
identier is composed in a range specied by minimum and
maximum pattern of bit-pattern. A counter having the same
ow indentier is searched from memory space and incremented, and packet timestamp is stored if the packet matchs
with the bit-pattern.
To specify a ow denition conposed by multiple bitpatterns, a child bit-pattern specifys the parent ow id to
judge that the next pattern maching with child bit-pattern
is done. Pattern matching with the following chiled patterns are not performed if a ow identier is not same value
specied by the parent ow id in a chiled bit-pattern.
For example, a ow denition is dened with chained
bit-pattern as shown in Figure 3 when measuring trafc distribution of the well-known TCP source port number. These
chained bit-patterns are used for both IPv4 and IPv6 trafc
measurement. Bit-pattern: IPv4 TCP is used when the
packet is judged as IPv4 packet with bit-pattern: IP version which checks IP version eld. Check of version 4 is
done with parent ow id specied in IPv4 TCP.

3.4. Flow ID Search

A ow denition enables multiple ow detection as we
dened the range specied by minimum and maximum pattern in a bit-pattern. Each ow matched with a ow denition is distinguished by the value of the eld matched with
the last chained bit-pattern. We dene a ow ID as this
value which species a ow.

Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE

ID: "IP version"

PST: 0, SIZE: 1, MASK: 0xf0,
MIN: 0x40, MAX: 0x60,
Child: "IPv4 TCP", "IPv6 TCP"
Parent Flow ID: --

Generate variety of
flows to a Capturing
device

Bit pattern for IPv4 TCP

ID: "IPv4 TCP"
PST: 9, SIZE: 1, MASK: 0xff,
MIN: 0x06, MAX: 0x06,
Child: "IPv4 TCP source well-known"
Parent Flow ID: 0x40

Starting potion of TCP header is 60 byte

from top of IPv6 header
ID: "IPv6 TCP source well-known port"
PST: 40, SIZE: 2, MASK: 0xffff,
MIN: 0x0000, MAX: 0x0400,
Child: (nil)
Parent Flow ID: --

Figure 3. Example of Flow Denition for TCP

Well-known Port Flows
When a packet matches a ow denition, a counter having the same ow identier is searched from the previously
registered counters. If a counter having the same ow identier does not exist at that moment, a new counter with the
ow identier is registered. Many counters are registered
thus the ow identier search tends to take more time if
there is a large range of bit-patterns.
For searches with a certain length key such as ow identier, a binary search can reduce the processing cost to
O(log n). However the ow identication process registers the ow ID and the search tree grow gradually thus
the search tree must be well balanced. The AVL tree2 [9],
which keeps the search tree balanced is a solution that enables faster search. However the insertion of new counters is costly as opposed to normal lists, thus it is better to
discover how much range size of bit-pattern the AVL tree
search works well with.

4. Implementation and Evaluation

We implemented a distributed real-time ow measurement tool proposed in the previous section on PC-UNIXs.
2 The depth AVL tree is

tree.

balanced or its difference is up to 1 at every sub

Switching Hub

Report flow information

Periodically

ID: "IPv4 TCP source well-known port"

PST: 20, SIZE: 2, MASK: 0xffff,
MIN: 0x0000, MAX: 0x0400,
Child: (nil)
Parent Flow ID: --

ID: "IPv6 TCP"

PST: 6, SIZE: 1, MASK: 0xff,
MIN: 0x06, MAX: 0x06,
Child: "IPv6 TCP source well-known
Parent Flow ID: 0x60

Register Flow
definition when
a capturing device
starts

Switching Hub

Starting potion of TCP header is 20 byte

from top of IPv4 header

Bit pattern for IPv6 TCP

Manager Device

Generater

Capturing Device

Figure 4. Conguration of Performance Evaluation

Table 2. Specication of PC-UNIXs Used for

Evaluation
CPU
Memory
HDD
Bus
Network Interface
Operating System

Xeon 2.8GHz
2GB
73GB
PCI-X (64bit, 133MHz)
2 ports, 10/100Base-TX
RedHat 9 Linux kernel 2.4.20

Then we evaluated the capturing performance for the following:

Difference as per the number of bit-patterns in between
where deploying and not deploying packet buffering
process.
Difference as per the size of bit-pattern range in between where deploying AVL tree and sorted list for
ow identier search.

4.1. Implementation
We measured capturing loss at a capturing device when
sending 500 30K packets per second (pps) respectively
from a trafc generator to a capturing device in Figure 4
conguration. Flow denitions were registered with a manager device through a user interface device and manager
device advertising ow denitions to a capturing device.
A capturing device made a report of every ow identier
counter to a manager device every 10 seconds. All devices
had the same specications as shown in Table 2.
We calculated the average capturing loss from 10 examinations for each packet speed in the following performance

Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE

Number of capt
ure loss per seco
nd

Figure 5. Capturing loss per Number of BitPatterns with Packet Buffering Process

tbi nge
of ra
ze rn
Si tte
pa

Number of capt
ure loss per seco
nd

of rns
er te
mb at
Nu t-p
bi

om a
ted fr
ce
genera ure devi
Capt
(Kbps)
speed ter to a
et
ra
ck
Pa
Gene
c
fi
Traf

om a
ted fr device
genera
ing
(Kbps) a Captur
speed
r to
Packet c Generate
Traffi

Figure 7. Capturing Loss per Bit-Pattern

Range Size with AVL Tree Search

Number of capt
ure loss per seco
nd

Number of
capture los
s per second

Figure 6. Capturing Loss per Number of BitPatterns without Packet Buffering Process
tests. Every capturing loss in an examination was calculated
as loss per second during 60-second trafc generation. The
trafc generation was shaped at the user program level to
keep the jitter shorter. However small burst trafc was observed during generation because the nal packet treatment
was controlled by a kernel.

4.2. Performance Difference in Deploying or Not

Deploying Packet Buffering Process
Figure 5 and 6 show the capturing loss for 1 up to 64 bitpatterns respectively. Trafc with the same kind of patterns
was sent to a capturing device in a round-robin manner. The
length of the bit-pattern was 4 bytes and an exact match
dened by providing the same pattern between minimum
and maximum patterns. This implies that the result will be
either matched or unmatched.

t
bi nge
of ra
ze rn
Si tte
pa

of rns
er te
mb at
Nu t-p
bi

om a
ted fr
vice
genera
(Kbps) a Capture de
speed
Packet Generater to
c
Traffi

om a
ted fr vice
de
genera
bps) a Capture
(K
d
spee
r to
Packet c Generate
Traffi

Figure 8. Capturing Loss per Bit-Pattern

Range Size with Sorted List Search
Figure 5 shows that maximum performance is 27K pps
where the number of patterns is 1 and that minimum performance is 2K pps where the number of patterns is 64. The
appropriate number of patterns is expected to be between
4 and 64 in practice, therefore the performance per capture
device would be between 2K to 15K pps. However, Figure 6
shows the capturing loss is present on given 500 pps trafc.
From two experiment, a packet buffering process is effective for absorbing burst trafc occurring in a short period of
time.

4.3. Performance Difference in Deploying AVL Tree

Search or Sorted List Search
Figure 7 and 8 show the capturing loss where the size of
the bit-pattern range is between 64 and 8192. Flow identi-

Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE

1100

Packet size
Traffic speed

96
88

1000
80
72
64

800

56
700
48
600

40

traffic speed (Mbit/s)

average packet size (byte)

900

32

500

24
400
16
300

8
-8 -7 -6 -5 -4 -3 -2 -1

2 3 4 5 6 7
hour of 2003/June/18

9 10 11 12 13 14 15 16

Figure 9. Packet Length of Trafc Passing Between APAN and WIDE

to reduce entire pattern maching overhead.

Figure 9 shows the 1 minute average packet length (+)
and average trafc () of the trafc passing through a GigabitEthernet link between APAN Tokyo XP [10] [11] and
WIDE [12] measured in June 18, 2003. The average packet
length on that day was 861 bytes. The results of this packet
length are only an example, yet the performance of 2k
15K pps per capturing device is estimated at 13M 100M
bit/s for reference.
The idea to use multiple capturing devices in a proposed
system would be key issue to achieve high-performance
measurement of the IP ow on a high-speed link. We imagine that a distribution device and a manager device potencially have bottleneck respectively. Development of distribution device performing light weightedly, and an evaluation of manger devices are needed.

5.2. Time Synchronization and Accuracy

er eld of generated trafc which is picked up by a capturing device is generated in a round-robin manner, so the
number of ow identiers becomes the same. The length of
the bit-pattern is 4 bytes.
From the results of two gures, the criterion to select a
search from AVL tree search or sorted list search on the proposed system was 1024 as the range size. It is a good idea
to select the search method based on the range size for each
bit-pattern from multiple methods in addition to the AVL
tree search and/or sorted list search before pattern maching
starts when a ow denition is registered with a capturing
device, becuase the range size of bit-pattern composing the
ow denition is specied on ow denition registration.
This previous selection of search methid instead of on-they selection is expected to reduce the overhead of ow identication search.

5. Discussions

We proposed that a capturing device use its own clock. In

order to acquire a high precision time device, the synchronization to GPS or a more accurate time capability system
is appropriate because trafc distribution was assumed to
be provided from commercial products in this paper. Distributing GPS pulses from the antenna or accurate pulses
per second generated from some system are some solutions
when multiple capturing devices are used in a real world
environment.
In contrast, if many capturing devices are used, it is more
effective if the distribution device adds a timestamp on the
packet and then distributes to capturing devices instead of
distributing clock pulses from an extra device to all capturing devices. The addition of a timestamp function on a distribution device requires only small modication to a capturing device to get a timestamp from a packet and to shift
the start or end point of packet to read a timestamp. This
distribution using a timestamp method also eliminates the
need for clock synchronization between capturing devices.

5.1. Performance
From the results of Figure 5, it became clear that
the number of bit-patterns signicantly impacts on performance. This implies that the ow denition consisting
of chained bit-patterns proposed in this paper is expected
to prevent drops in capturing performance where multiple
ow denitions are provided and where bit-patterns specifying the same eld comprising different ow denitions
are omitted.
It would be better to modify the order of chained bitpatterns operating as and operations between bit-patterns
while the system is in progress in order to improve the performance. This tuning to reduce average number of pattern
matching per packet depends on the trafc measured at that
moment. The dynamic matching order method is expected

5.3. Example Real World Environment

Figure 10 shows an example of the ow measurement
of packet speed on the same GigabitEthernet link between
APAN Tokyo XP and WIDE described with Figure 9. The
measurement was performed with a 2-capture-device system. The capturing devices made reported every second.
The distribution was performed in a round-robin manner using a prototype we developed with PC-UNIXs for this measurement. The total average and maximum trafc speed was
1.0K pps and 2.8K pps. The bit-pattern denition was for
checking the protocol eld of the IP packet (1-byte length)
and the destination port eld of TCP header (2-byte length)
because the link carried only IPv4 trafc. There was no
capturing loss detected.

Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE

TCP others

UDP
ICMP

Packet/sec

TCP DST
port = NNTP

TCP DST
port >= 1024
2003/June/24

Figure 10. Flows of outgoing trafc to WIDE

The procedures for graph generation of Figure 10 with
the proposed system were as follows:

In the evaluation, we showed that packet buffering process effectively reduce capturing failure, that a capturing
device performs 2K 15K pps where 4 32 bit-patterns
comprising the ow denition are given and that number
of bit-patterns impact signicantly on performance. We
suggested our proposed chained bit-pattern based ow definition is effective for preventing performance failures in
multiple ow denition environments. We also showed the
performance difference between the AVL tree search and
sorted list search as the search method and indicated the criterion at 1024 of range size in selecting one of two searches
where our prototype is used.
Finally, we showed an example of use for our prototype
and were able to conrm that the procedures from ow definition and extracting ows to checking these ows are performed much more easily with a user interface device.

References
[1] Multi Router Trafc Grapher, http://www.mrtg.org/
[2] the Cooperative Association for Internet Data Analysis, http://www.caida.org/
[3] CAIDA, cowd: Trafc Flow Analysis Tool,
http://www.caida.org/tools/measurement/cowd/

1. Register ow denitions with a manager device.

[4] NeTraMet, http://www2.auckland.ac.nz/net/NeTraMet/
2. Measure ows with multiple capturing devices and
store date to a manager device
3. Check graph for each ow denition by using the user
interface device
4. Dump ow information in a specic period to text le
5. Generate a graph with an appropriate tool like RRDTool or Microsoft Excel.
The ow denition registration for checking graphs is
performed on a user interface device in these procedures.
Thus it was easier to extract certain ows at the observation
point.

[5] Cisco Systems, Inc., NetFlow Service and Applications, White paper, http://www.cisco.com/warp/public/cc/pd/iosw/ioft/nect/tech/napps wp.htm, 2002
[6] Mills, D.L., A. Thyagarajan and B.C. Huffman, Internet timekeeping around the globe, Proc. PTTI Applications and Planning Meeting, Long Beach CA, pp.
365371, December 1997.
[7] J. Quittek, T. Zseby, B. Claise, S. Zander:
Requirements for IP Flow Information Export, Internet Draft (draft-ietf-ipx-reqs-10.txt),
http://www.ietf.org/internet-drafts/draft-ietf-ipxreqs-10.txt, June 2003.
[8] Tcpdump/Pcap, http://www.tcpdump.org/

6. Conclusions
We claried the specication for software-based realtime ow measurement tools which identify ows by using multiple devices, designed and implemented the tools
on PC-UNIXs, and evaluated the performance of a capturing device. We proposed the chained bit-pattern based ow
denition method and data structure for ow identication
and process functions in a capturing device and a manager
device enabling exibly to add or remove capturing devices
while the system is operating.

[9] G. M. Adelson-Velskii and Y. M. Landis: An algorithm for the organization of information, Soviet
Math. Dokl., 3:12591262, 1962.
[10] Asia-Pacic
Advanced
http://www.apan.net/

Network,

[11] Kilnam Chon, Asia-Pacic Advanced Network

(APAN), http://apan.net/home/index1.html, 1999
[12] WIDE Project, http://www.wide.ad.jp/

Proceedings of the 2004 International Symposium on Applications and the Internet (SAINT04)
0-7695-2068-5/04 $20.00 2004 IEEE