Re: Computing Traffic Matrix from netflow data

[Jennifer Rexford <jrex@research.att.com>]

Jing,

> I'm starting to do up scripts which could be used to compute traffic
> matrix (node-to-node). But, I found it's a little bit troublesome as
> routing path may change with time.

Many studies of traffic matrices have assumed that routing changes
don't occur, or don't have much impact.  These studies typically
take a daily dump of the BGP tables to determine the BGP next-hop
for each destination prefix at each ingress router.  In some recent
work, we've found that ignoring BGP routing changes can sometimes
cause you to miss significant traffic shifts.  See

  http://www.cse.ucsd.edu/~teixeira/tm.pdf

for details.  For this study, we collected a feed of iBGP update
messages from each of the ingress routers of interest, so we
could track the changes in the BGP next-hops continuously.

> Also, using (source_ip, dest_ip) is hard to compute node level
> metrix because routing table size is really large and amount of raw
> data is huge( my situation, more than 120GB).

If you just want to compute the router-to-router traffic matrix,
you could

  - collect Netflow measurements on the incoming traffic
  - aggregate traffic with the same destination prefix, using
       the BGP prefix information
  - aggregate traffic with the same BGP next-hop, using the
       BGP next-hop information

Hope that helps...

-- Jen