Pages

4/08/2013

Testing how much routes a device could handle with Quagga

Hi,

I wanted to stress-test multilayer switch Juniper EX 4200 and its routing engine. Somehow. I wanted to simulate a number of BGP routes and how will switch react. Importing and filtering BGP routes to Adj-RIB-In table and then to routing table could be stressfull for device and this could lead to 100% cpu usage and for example OSPF adjacency flapping. Most of work in this blog is done by a route generator script. You could also use a spreadsheet editor (Excel) and then edit exported text to create routes in Quagga configuration file.

 What do you need?
  • router (EX4200 with a advanced license. But don't worry. BGP on EX4200 should work even without license. At least it used to work on older releases. It yelds message to console during commit  and regularry to syslog)
  • standard Linux or Ubuntu PC with NIC and Quagga
  • ethernet cable :-)

Juniper EX 4200 specifications [2]

IPv4 Unicast routes: 16,000

Install and configure Quagga on a Ubuntu PC
# apt-get install quagga

Change Ubuntu IP addresses
# ifconfig eth0 192.0.2.1/24

 Edit Quagga configuration file ...
# nano /etc/quagga/daemons

... and you should run at least these two Quagga daemons
zebra=yes
bgp=yes

How to generate many routes
I used script from this site V. Glinsky[1] blog.
Change these two lines in that script:

my $router_id="192.0.2.1"; #bgp router-id
my $remote_ip="192.0.2.2"; #BGP neighbor ip address


Leave other values unchanged. It is going to generate 300,000 routes. Much more than 16,000. Let's say we are all right with AS numbers. Ubuntu AS65099, Juniper AS65001. Actually I generated 599999 routes in my example.
 
Copy generated bgpd.conf file to a directory /etc/quagga/.

Here is how bgpd.conf should look
hostname quagga-host
password zebra
enable password zebra
line vty
router bgp 65099
  bgp router-id 192.0.2.1
  neighbor 192.0.2.2 remote-as 65001
  network 70.0.0.0/24
  network 70.0.1.0/24
  network 70.0.2.0/24
  network 70.0.3.0/24 


Run Quagga
# /etc/init.d/quagga restart

Configure Juniper BGP protocol and interface
root@ex4200> show configuration protocols bgp 
local-as 65001;
group bgp-test {
    type external;

    peer-as 65099;
    neighbor 192.0.2.1;

}


root@ex4200> show configuration interfaces ge-0/0/0   
unit 0 {
    family inet {
        address 192.0.2.2/24;
}


root@ex4200> show configuration routing-options
router-id 192.0.2.2;
autonomous-system 65001; 


There is a warning during commit. You should have a license. Don't worry, it's OK for now. It will work even without it.
{master:0}[edit]
root@ex4200# commit
[edit protocols]
  'bgp'
    warning: requires 'bgp' license
configuration check succeeds
commit complete 

Connect two devices with ethernet cable and observe results
Juniper EX 4200:

root@ex4200> show bgp summary  
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0             
                   16372      16372          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.0.2.1             65099        299          4       0       0          28 16372/16372/16372/0  0/0/0/0

BGP neighbor adjacency status is Established. Highlighted is number of accepted routes in a Adj-Rib-In table. We don't use any import policy.

root@ex4200> show route

inet.0: 16384 destinations, 16384 routes (16384 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2.0.0.0/24         *[BGP/170] 00:00:39, MED 0, localpref 100
                      AS path: 65099 I, validation-state: unverified
                    > to 192.0.2.1 via ge-0/0/0.0
2.0.1.0/24         *[BGP/170] 00:00:39, MED 0, localpref 100
                      AS path: 65099 I, validation-state: unverified
                    > to 192.0.2.1 via ge-0/0/0.0
2.0.2.0/24         *[BGP/170] 00:00:39, MED 0, localpref 100
                      AS path: 65099 I, validation-state: unverified
                    > to 192.0.2.1 via ge-0/0/0.0
2.0.3.0/24         *[BGP/170] 00:00:39, MED 0, localpref 100
                      AS path: 65099 I, validation-state: unverified
                    > to 192.0.2.1 via ge-0/0/0.0
2.0.4.0/24         *[BGP/170] 00:00:39, MED 0, localpref 100
                      AS path: 65099 I, validation-state: unverified
                    > to 192.0.2.1 via ge-0/0/0.0
2.0.5.0/24         *[BGP/170] 00:00:39, MED 0, localpref 100
                      AS path: 65099 I, validation-state: unverified
                    > to 192.0.2.1 via ge-0/0/0.0
2.0.6.0/24         *[BGP/170] 00:00:39, MED 0, localpref 100
---(more)---


Highlighted number is number of active routes in routing table.
 

root@ex4200> show chassis routing-engine
Routing Engine status:
  Slot 0:
    Current state                  Master
    Temperature                 34 degrees C / 93 degrees F
    CPU temperature             34 degrees C / 93 degrees F
    DRAM                      1024 MB
    Memory utilization          44 percent
    CPU utilization:
      User                      23 percent
      Background                 0 percent
      Kernel                    34 percent
      Interrupt                  0 percent
      Idle                      42 percent
    Model                          EX4200-24F
    Serial ID                      BR0210217636
    Start time                     2013-01-19 19:28:27 UTC
    Uptime                         31 minutes, 9 seconds
    Last reboot reason             0x2:watchdog
    Load averages:                 1 minute   5 minute  15 minute
                                       0.45       0.15       0.10

CPU utilization is a bit higher right after establishing adjacency and receiving routes.

You could also use command
> show system processes extended
to see how rpd process is using CPU [3].

Connect to Quagga and show bgp config
root@bt:~# telnet localhost 2605
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Hello, this is Quagga (version 0.99.15).
Copyright 1996-2005 Kunihiro Ishiguro, et al.


User Access Verification
 

Password:  (password is zebra or quagga)
quagga-host>
quagga-host> show ip bgp summary
BGP router identifier 192.0.2.1, local AS number 65099
RIB entries 599999, using 37 MiB of memory
Peers 1, using 2520 bytes of memory

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
192.0.2.2       4 65001       5     304        0    0    0 00:02:00        0

Total number of neighbors 1
 

quagga-host> show ip bgp neighbors
BGP neighbor is 192.0.2.2, remote AS 65001, local AS 65099, external link
  BGP version 4, remote router ID 192.0.2.2
  BGP state = Established, up for 00:02:05
  Last read 00:00:19, hold time is 90, keepalive interval is 30 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    Route refresh: advertised and received(old & new)
    Address family IPv4 Unicast: advertised and received
    Graceful Restart Capabilty: received
      Remote Restart timer is 120 seconds
      Address families by peer:
        none
  Graceful restart informations:
    End-of-RIB send: IPv4 Unicast
    End-of-RIB received:
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  1          0
    Notifications:          0          0
    Updates:              298          0
    Keepalives:             6          5
    Route Refresh:          0          0
    Capability:             0          0
    Total:                305          5
  Minimum time between advertisement runs is 30 seconds

 For address family: IPv4 Unicast
  Community attribute sent to this neighbor(both)
  0 accepted prefixes

  Connections established 1; dropped 0
  Last reset never
Local host: 192.0.2.1, Local port: 179
Foreign host: 192.0.2.2, Foreign port: 60656
Nexthop: 192.0.2.1
Nexthop global: ::
Nexthop local: ::
BGP connection: non shared network
Read thread: on  Write thread: off 


Session is established. 599999 routes loaded from config file.
 
Other ways to BGP routes generation
Nice guide on how to generate routes from real world bgp routes dump:
http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/ 

How to mitigate this 'attack'?
There is a command to change behavior after certain number of routes are received.
http://www.juniper.net/techpubs/en_US/junos12.3/topics/reference/configuration-statement/accepted-prefix-limit-edit-protocols-bgp.html

References:
[1] http://blog.glinskiy.com/2009/10/how-to-generate-lots-of-bgp-routes.html
[2] http://www.juniper.net/us/en/products-services/switching/ex-series/ex4200/#specifications
[3] http://www.juniper.net/techpubs/en_US/junos9.6/information-products/topic-collections/nog-baseline/routing-engine--memory-introduction.html

4/02/2013

Juniper EX simple multicast router (PIM & IGMPv2)

In next few lines I will show you how to set Juniper EX4200 switch with Junos version 12.2 to act as PIM dense-mode router. Dense mode is configured because it is simpler to configure as sparse-mode.

Background

I am currently testing some L2 access switches for 3play services. Some of feature I test is a multicast service handling by switch. Especially IGMP snooping. Because of that, I need IGMP querier - a device that listens and sends IGMP packets. Simple running VLC server will not automatically listens and respond to IGMP packets from client so DUT switch will not hear and investigate IGMP snooping packets exept these from receivers (and that is not enough). You should have two-way communication to get IGMP snooping operational. I use BackTrack 5 or Ubuntu with VLC as video stream server/source and receivers. I use tagged interfaces on both devices EX 4200 ge-0/0/0, ge-0/0/1 and also on multicast stream server. DUT could be any manageable L2 switch with IGMP snooping (with or without IGMP proxy). IGMP snooping feature: http://en.wikipedia.org/wiki/IGMP_snooping. I am not going to provide IGMP snooping test here, but shortly, this feature helps reduce multicast traffic on LAN segment (VLAN), so that multicast stream is received only on interface/port/MC stream receiver that wants it, not on all ports. Switch with this feature listens to IGMP (Query, Report, Leave) packets and behave according it. Try read this http://www.juniper.net/techpubs/en_US/junos9.4/topics/concept/igmp-snooping-ex-series-overview.html or this http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli_rel_4_0_1a/IGMPSnooping.html. There is also other technology for cope somehow with multicast streams - MVR, IGMP Proxy, IGMP querier configured on switch.

Topology
Topology

Setting up a multicast stream server

All commands are run under root. You don't have to run vlc under root on Ubuntu.

(optional) install VLC
apt-get install vlc
 
(Backtrack 5) allow run VLC under root
http://www.backtrack-linux.org/forums/showthread.php?t=44590 - change file /usr/bin/vlc ... 'u should open it with an hex editor and find in file for "geteuid._libc_start_main" without quotes! when u find it change it to "getppid._libc_start_main" without quotes!'


Next five lines is a setup of vlan tagged interface on interface eth0.
Add IP address, change routing
apt-get install vconfig
modprobe 8021q
vconfig add eth0 130
ifconfig eth0.130 130.0.0.10/24
route add 224.0.0.0/4 via 130.0.0.1 dev eth0.130

run VLC stream server video
cvlc /root/Videos/MPEG-2.mpg --sout '#udp{mux=ts,dst=238.1.1.30:1234}' --ttl=4 --repeat

Juniper EX4200 configuration

## Last commit: 2013-03-28 09:55:20 UTC by root
version 12.2R2.5;
interfaces {
    ge-0/0/0 {
        unit 0 {
            family ethernet-switching {
                port-mode trunk;
                vlan {
                    members [ 30 ];


                }
            }
        }
    }
    ge-0/0/1 {
        unit 0 {
            family ethernet-switching {
                port-mode trunk;
                vlan {
                    members [ 130 ];
                }
            }
        }
    }
    vlan {
        mtu 9216;
        unit 30 {
            family inet {
                mtu 1500;
                address 30.0.0.1/24;
            }
        }
        unit 130 {
            family inet {
                mtu 1500;
                address 130.0.0.1/24;
            }
        }
    }
}
protocols {

# you don't have to set igmp when pim is enabled on interface. This will automatically enable IGMPv2. 
    igmp {
        interface vlan.30;
        interface vlan.130;
    }
    pim {
        interface vlan.30 {
            mode dense;
        }
        interface vlan.130 {
            mode dense;
        }
    }
    igmp-snooping {
        vlan all {
            version 2;
        }
    }
}
vlans {
    v130 {
        vlan-id 130;
        l3-interface vlan.130;
    v30 {
        vlan-id 30;
        l3-interface vlan.30;
    }
}



Setting up a DUT switch

Sorry, you should have to set it by yourself. You only have to set one uplink - tagged port with vlan 30 and one downlink client port - untagged port with vlan 30 (port vlan id 30, untagged frames will be tagged to vlan 30. And also you could also set up IGMP snooping on vlan 30 with router port (uplink port) to test it.

Setting up a multicast receiver

Forcing IGMP version 2 on a receiver. Default version today is IGMPv3.
echo "2" > /proc/sys/net/ipv4/conf/eth0/force_igmp_version

Add ip address on untagged port on receiver connected to DUT switch, vlan 30:
ifconfig eth0 30.0.0.123/24
route add 224.0.0.0/4 via 30.0.0.1 dev eth0
apt-get install vlc

Run VLC and open stream on a receiver
menu Applicaton -> Sound & Video -> VLC media player
press CTRL+N (File -> Open Network Stream)
select protocol: UDP
set multicast address: udp://@238.1.1.30:1234

You should be able to ping Juniper from stream receiver. Run ping 30.0.0.1. They are L2-connected with vlan 30. And you also should be able to see your video.


A simple troubleshooting guide

  • Check cabling
  • observe blinking LEDs on switches and NICs
  • try to connect receiver directly to Juniper. Change Juniper config to be untagged interface or add vlan tagged interface to receiver.
  • use tcpdump on receiver and on streamer. Check for vlan tag (tcpdump -ni eth0 -e), destination and source IP and MAC
  • check IGMP snooping on DUT and Juniper
  • check IGMP packets on receiver: tcpdump -ni eth0 igmp