Articles in the Networking category

  1. Creating Configuration Backups of HP Procurve Switches

    January 12, 2015

    I've created a tool called procurve-watch. It creates a backup of the running switch configuration through secure shell (using scp).

    It also diffs backed up configurations against older versions, in order to keep track of changes. If you run the script from cron every hour or so, you will be notified by email of any (running) configuration changes.

    The tool can backup hundreds of switches in seconds as it is running the configuration copy in parallel.

    A tool like Rancid may actually be the best choice for this task, but it didn't work. The latest version of Rancid doesn't support HP Procurve switches (yet) and older versions created backups containing garbled characters.

    I've released it on github, check it out and let me know if it works for you and you have suggestions to improve it further.

    Tagged as : Networking
  2. Configuring, Attacking and Securing VRRP on Linux

    January 02, 2015

    The VRRP or Virtual Router Redundancy Protocol helps you create a reliable network by using multiple routers in an active/passive configuration. If the primary router fails, the backup router takes over almost seamlessly.

    This is how VRRP works:

    vrrp

    Clients connect to a virtual IP-address. It is called virtual because the IP-address is not hard-coded to a particular interface on any of the routers.

    If a client asks for the MAC-address that is tied to the virtual IP, the master will respond with its MAC-address. If the master dies, the backup router will notice and start responding to ARP-requests.

    Let's take a look at the ARP table on the client to illustrate what is happening.

    Master is active:

    (10.0.1.140) at 0:c:29:a7:7d:f2 on en0 ifscope [ethernet]
    (10.0.1.141) at 0:c:29:a7:7d:f2 on en0 ifscope [ethernet]
    (10.0.1.142) at 0:c:29:b2:5b:7c on en0 ifscope [ethernet]
    

    Master has failed and backup has taken over:

    (10.0.1.140) at 0:c:29:b2:5b:7c on en0 ifscope [ethernet]
    (10.0.1.141) at 0:c:29:a7:7d:f2 on en0 ifscope [ethernet]
    (10.0.1.142) at 0:c:29:b2:5b:7c on en0 ifscope [ethernet]
    

    Notice how the MAC-address of the virtual IP (.140) is now that of the backup router.

    Configuring VRRP on Linux

    1. configure static IP-addresses on the primary and backup router. Do not configure the virtual IP on any of the interfaces. In my test environment, I used 10.0.1.141 for the master and 10.0.1.142 for the backup router.

    2. Because the virtual IP-address is not configured on any of the interfaces, Linux will not reply to any packets destined for this IP. This behaviour needs to be changed or VRRP will not work. Edit /etc/sysctl.conf and add this line:

      net.ipv4.ip_nonlocal_bind=1
      
    3. Run this command to active this setting:

      sysctl -p
      
    4. Install Keepalived

      apt-get install keepalived
      
    5. Sample configuration of /etc/keepalived/keepalived.conf

      vrrp_instance VI_1 {
          interface eth0
          state MASTER
          virtual_router_id 51
          priority 101
      
          authentication {
              auth_type AH
              auth_pass monkey
          }
      
          virtual_ipaddress {
              10.0.1.140
          }
      }
      
    6. Start keepalived:

      service keepalived start
      

    The only configuration difference regarding keepalived between the master and the standby router is the 'priority' setting. The master server should have a higher priority than the backup router (101 vs. 100).

    As there can be multiple VRRP configurations active within the same subnet, it is important that you make sure that you set a unique virtual_router_id.

    Please do not forget to set your own password in case you enable authentication.

    VRRP failover example

    This is what happens if the master is shutdown:

    64 bytes from 10.0.1.140: icmp_seq=148 ttl=64 time=0.583 ms
    64 bytes from 10.0.1.140: icmp_seq=149 ttl=64 time=0.469 ms
    64 bytes from 10.0.1.140: icmp_seq=150 ttl=64 time=0.267 ms
    Request timeout for icmp_seq 151
    Request timeout for icmp_seq 152
    Request timeout for icmp_seq 153
    Request timeout for icmp_seq 154
    64 bytes from 10.0.1.140: icmp_seq=155 ttl=64 time=0.668 ms
    64 bytes from 10.0.1.140: icmp_seq=156 ttl=64 time=0.444 ms
    64 bytes from 10.0.1.140: icmp_seq=157 ttl=64 time=0.510 ms
    

    After about five seconds (default) the standby router takes over and starts responding to the virtual IP.

    Security

    A host within the same subnet could just spoof VRRP packets and disrupt service.

    An attack on VRRP is not just theoretical. A tool called Loki allows you to take over the virtual IP-address and become the master router. This will allow you to create a DoS or sniff all traffic.

    VRRP security is also discussed in this document from the Loki developers.

    According to rfc3768 authentication and security has been deliberately omitted (see section 10 Security Considerations) from newer versions of the VRRP protocol RFC.

    The main argument is that any malicious device in a layer 2 network can stage similar attacks focussing on ARP-spoofing and ARP-poisoning so as the fundament is already insecure, why care about VRRP?

    I understand the reasoning but I disagree. If you do have a secure Layer 2 environment, VRRP becomes the weakest link. Either you really need to filter out VRRP traffic originating from untrusted ports/devices, or implement security on VRRP itself.

    Attacking VRRP with Loki

    I have actually used Loki on VRRP and I can confirm it works (at least) as a Denial-of-Service tool.

    I used Kali (Formerly known as Back-Track) and installed Loki according to these instructions. Please note the bottom of the page.

    What I did on Kali Linux:

    apt-get install python-dpkt python-dumbnet
    wget http://c0decafe.de/svn/codename_loki/packages/kali-1/pylibpcap_0.6.2-1_amd64.deb
    wget http://c0decafe.de/svn/codename_loki/packages/kali-1/loki_0.2.7-1_amd64.deb
    dpkg -i pylibpcap_0.6.2-1_amd64.deb
    dpkg -i loki_0.2.7-1_amd64.deb
    

    Then just run:

    loki.py
    

    vrrp attack

    This is only an issue if you already protected yourself against ARP- and IP-spoofing attacks.

    Protecting VRRP against attacks

    Keepalived offers two authentication types regarding VRRP:

    1. PASS (plain-text password)
    2. AH (IPSEC-AH (authentication header))

    The PASS option is totally useless from a security perspective.

    pass authentication

    As you can see, the password 'monkey' is visible and easily obtained from the VRRP multicast advertisements. So to me, it does not make sense to use this option. Loki just replayed the packets and could still create a DoS.

    So we are left with IPSEC-AH, wich is more promising as it actually does some cryptography using the IPSEC protocol, so there is no clear-text password to be captured. I'm not a crypto expert, so I'm not sure how secure this implementation is. Here is some more info on IPSEC-AH as implemented in Keepalived.

    AH authentication

    If I configure AH authentication, the Loki tool does not recognise the VRRP trafic anymore and it's no longer possible to use this simple script-kiddie-friendly tool to attack your VRRP setup.

    IPSEC-AH actually introduces an IPSEC-AH header between the IP section and the VRRP section of a packet, so it changes the packet format, which probably makes it unrecognisable for Loki.

    Running VRRP multicast traffic on different network segments

    It has been pointed out to me by XANi_ that it is possible with Keepalived to keep the virtual IP-address and the VRRP multicast traffic in different networks. Clients will therefore not be able to attack the VRRP traffic.

    In this case, security on the VRRP traffic is not relevant anymore and you don't really need to worry about authentication, assuming that untrusted devices don't have access to that 'VRRP' VLAN.

    Th first step is that both routers should have their physical interface in the same (untagged) VLAN. The trick is then to specify the virtual IP-addresses in the appropriate VLANs like this example:

    virtual_ipaddress {
    
        10.0.1.1/24 dev eth0.100
        10.0.2.1/24 dev eth0.200
    }
    

    In this example, virtual IP 10.0.1.1 is tied to VLAN 100 and 10.0.2.1 is tied to VLAN 200.

    If the physical router interfaces are present in the untagged VLAN 50 (example), the VRRP multicast traffic will only be observed in this VLAN.

    Some background information on working with VLANs and Keepalived.

    Closing words

    VRRP can provide a very simple solution to setup a high-availability router configuration. Security can be a real issue if untrusted devices reside in the same layer 2 network so implementing security with IPSEC-AH or network segmentation is recommended.

    Tagged as : VRRP
  3. Getting the Sitecom AC600 Wi-Fi Adapter Running on Linux

    November 01, 2014

    TL;DR Yes it works with some modifications of the driver source.

    A USB Wi-Fi adapter I used with a Raspberry Pi broke as I dropped it on the floor, so I had to replace it. I just went to a local shop and bought the Sitecom AC600 adapter as that's what they had available (with support for 5Ghz networking).

    I had some hope that I would just plug it in and it would 'just work™'. But no. Linux. In the end, the device cost me 30 euro's including taxes, but the time spend to get it to work may have made this a very expensive USB Wi-Fi dongle. And it's funny to think about the fact that the Wi-Fi dongle is almost the same price as the Raspberry Pi board itself.

    But I did get it working and I'd like to show you how.

    It started with a google for 'sitecom ac600 linux' which landed me on this page. This page told me the device uses a MediaTek chipset (MT7610U).

    So you need to download the driver from MediaTek. Here is a direct link

    So you may do something like this:

    cd /usr/src
    wget http://s3.amazonaws.com/mtk.cfs/Downloads/linux/mt7610u_wifi_sta_v3002_dpo_20130916.tar.bz2
    tar xjf mt7610u_wifi_sta_v3002_dpo_20130916.tar.bz2
    cd mt7610u_wifi_sta_v3002_dpo_20130916
    

    Now you would hope that it's just like this:

    make
    make install
    

    And we're happy right? Linux FTW! Well, NO! We're using Linux so we have to work for stuff that works right out of the box on Windows and Mac OS.

    So we first start with editing "include/os/rt_linux.h" and go to line ~279. There we make sure that we edit the struct like this:

        typedef struct _OS_FS_INFO_
     {
        kuid_t              fsuid;
        kgid_t              fsgid;
        mm_segment_t    fs;
     } OS_FS_INFO;
    

    Basically, the words int are replaced by kuid_t and kgid_t, or else, compilation will abort with an error.

    Ofcourse, the Sitecom AC600 has an USB identifier that is unknown to the driver, so after compilation, it still doesn't work.

    lsusb output:

    Bus 001 Device 004: ID 0df6:0075 Sitecom Europe B.V.
    

    So google landed me on this nice thread by 'praseodym' that explained the remaining steps. I stole the info below from this thread.

    So while we are in the source directory of the module, we are going to edit "common/rtusb_dev_id.c" and add

    {USB_DEVICE(0x0DF6,0x0075)}, /* MT7610U */
    

    So this will make the AC600 gets recognised by the driver. Now we also need to edit "os/linux/confik.mk" and change these lines like this:

    HAS_WPA_SUPPLICANT=y
    HAS_NATIVE_WPA_SUPPLICANT_SUPPORT=y
    

    So no, we are still not ready yet. I'm not 100 percent sure that this is required anymore, but I found this nice thread in Italian and a very small comment by 'shoe rat' tucked away at the end that may make the difference between a working device or not.

    We need to edit the file "os/linux/config.mk" and go to line ~663. Then, around that line, change

    CHIPSET_DAT = 2860
    

    to:

    CHIPSET_DAT = 2870
    

    Yes. Finally! Now you can do:

    make
    make install
    

    Imagine that such a 'make' takes about 20 minutes on a Raspbery Pi. No joke.

    Now you can either do this:

    modprobe mt7650u_sta
    

    You should see something like this:

    root@raspberrypi:/usr/src# lsmod
    Module                  Size  Used by
    snd_bcm2835            16181  0 
    snd_pcm                63684  1 snd_bcm2835
    snd_page_alloc          3604  1 snd_pcm
    snd_seq                43926  0 
    snd_seq_device          4981  1 snd_seq
    snd_timer              15936  2 snd_pcm,snd_seq
    snd                    44915  5 snd_bcm2835,snd_timer,snd_pcm,snd_seq,snd_seq_device
    soundcore               4827  1 snd
    mt7650u_sta           895786  1 
    pl2303                  7951  0 
    usbserial              19536  1 pl2303
    

    You should be able to see a 'ra0' device when entering ifconfig -a or iwconfig and just configure it like any wireless device (out-of-scope).

    So once up-and-running, the Sitecom AC600 works fine under Linux and even sees and connects to 5 GHz networks. But not without a caveat of-course. I needed to configure a 5 GHz channel below 100 (I chose 48) on my Apple Airport Extreme, or the Wi-Fi dongle would not see the 5GHz network and would not be able to connect to it.

    So I hope somebody else is helped by this information.

    Tagged as : Wi-Fi
  4. How Traffic Shaping Can Dramatically Improve Internet Responsiveness

    March 08, 2014

    At work, access to the internet is provided by a 10 Mbit down / 1 Mbit up ADSL-connection. As we are a mid-size company, bandwidth is clearly a severe constraint. But it was not our biggest problem. Even simple web-browsing was very slow.

    As I was setting up a monitoring environment based on Nagios and pnp4nagios, I started to graph the latency of our internet connection, just to prove that we have a problem.

    Boy did we get that proof:

    bad latency

    Just look at the y-axis, which scale is in milliseconds. For most of the day, the average latency is 175 ms, with some high spikes. Just browsing the web was a pain during times of high-latency, which was clearly almost all of the time.

    I became so fed up with our slow internet access that I decided to take matters in my own hands and resolve the low latency issue. The solution? Traffic shaping.

    I learned that as ADSL-connections are saturated, especially their upload capacity, you will experience high latency and packet loss. So the trick is to never saturate the connection.

    I grabbed a Linux box with two network interfaces and placed it between our internet router and our firewall in bridge mode.

    For actual traffic shaping I used wondershaper which is part of Debian or Ubuntu.

    apt-get install wondershaper
    

    The wondershaper script is extremely simple, it's specifically build to resolve the problem we face with our ADSL connection. It not only prioritises traffic, it allows you to limit bandwidth usage and thus prevent you from saturating the connection.

    This simple example limits bandwidth a bit below full capacity, which dramatically improved latency.

    Syntax:

    wondershaper <interface> <rx> <tx>
    

    Example:

    wondershaper eth1 9500 700
    

    As you can see, latency improved dramatically:

    good latency

    Again, look at the y-axis. We went from an average latency of 175 ms to an average of 35 ms. That's quite an improvement.

    Can you spot on which day I implemented traffic shaping?

    week latency

    At the time of writing this blog post, the company is working on fiber internet access, resolving our internet woes, but it will take quite some time before that will be installed, so this is a nice intermediate solution.

    Tagged as : traffic shaping
  5. Achieving 450 MB/s Network File Transfers Using Linux Bonding

    January 07, 2014

    Linux Bonding

    In this article I'd like to show the results of using regular 1 Gigabit network connections to achieve 450 MB/s file transfers over NFS.

    I'm again using Linux interface bonding for this purpose.

    Linux interface bonding can be used to create a virtual network interface with the aggregate bandwidth of all the interfaces added to the bond. Two gigabit network interfaces will give you - guess what - two gigabit or ~220 MB/s of bandwidth.

    This bandwidth can be used by a single TCP-connection.

    So how is this achieved? The Linux bonding kernel module has special bonding mode: mode 0 or round-robin bonding. In this mode, the kernel will stripe packets across the interfaces in the 'bond' like RAID 0 with hard drives. As with RAID 0, you get additional performance with each device you add.

    So I've added HP NC364T quad-port network cards to my servers. Thus each server has a theoretical bandwidth of 4 Gigabit. These HP network cards cost just 145 Euro and I even found the card for 105 Dollar on Amazon.

    hp quad port nic

    With two servers, you just need four UTP cable's to connect the two network interfaces and you're done. This would cost you ~300 Euro or ~200 dollar in total.

    If you want to connect additional servers, you need a managed gigabit switch with VLAN-support and sufficient ports. Each additional server will use 4 ports on the switch, excluding interfaces for remote access and other purposes.

    Managed gigabit switches are quite inexpensive these days. I bought a 24 port switch: HP 1810-24G v2 (J9803A) for about 180 euros (209 Dollars on Newegg) and it can even be rack-mounted.

    switch

    Using VLANS

    So you can't just use a gigabit switch and connect all these quad-port network cards to a single VLAN. I tested this scenario first and only got a maximum transfer speed of 270 MB/s while copying a file between servers over NFS.

    The trick is to create a separate VLAN for every network port. So if you use a quad-port network card, you need four VLANs. Also, you must make sure that every port on every network card is in the same VLAN. For example, port 1 on every card needs to be in VLAN 21, port 2 in VLAN 22, and so on. You also must add the appropriate switch port to the correct VLAN. Last, you must add the network interfaces to the bond in the right order.

    bondingschema

    So why do you need to use VLANs? The reason is quite simple. Bonding works by spoofing the same hardware or MAC-address on all interfaces. So the switch sees the same hardware address on four ports, and thus gets confused. To which port should the packet be sent?

    If you put each port in it's own VLAN, the 'spoofed' MAC-address is seen only once in each VLAN. So the switch won't be confused. What you are in fact doing by creating VLANs is creating four separate switches. So if you have - for example - four cheap 8-port unmanaged gigabit switches, this would work too.

    So assuming that you have four ethernet interfaces, this is an example of how you can create the bond:

    ifenslave bond0 eth1 eth2 eth3 eth4
    

    Next, you just assign an IP-address to the bond0 interface, like you would with a regular eth(x) interface.

    ifconfig bond0 192.168.2.10 netmask 255.255.255.0
    

    Up to this point, I only achieved about 350 MB/s. I needed to enable jumbo frames on all interfaces and the switch to achieve 450 MB/s.

    ifconfig bond0 mtu 9000 up
    

    Next, you can just mount any NFS share over the interface and start copying files. That's all.

    Once I added each interface to the appropriate VLAN, I got about 450 MB/s for a single file copy with 'cp' over NFS.

    450 MB/s

    I did not perform a 'cp' but a 'dd' because I don't have a disk array fast enough (yet) that can write at 450 MB/s.

    So for three servers, this solution will cost me 600 Euro or 520 Dollar.

    10 Gigabit ethernet?

    Frankly, it's quite expensive if you need to connect more than two servers. An entry-level 10Gbe NIC like the Intel X540-T1 does about 300 Euros or 450 Dollar. This card allows you to use Cat 6e UTP Cabling. (Pricing from Dutch Webshops in Euros and Newegg in Dollars).

    With two of those, you can have 10Gbit ethernet between two servers for 600 Euro or 700 Dollar. If you need to connect more servers, you would need a switch. The problem is that 10Gbit switches are not cheap. An 8-port unmanaged switch from Netgear (ProSAFE Plus XS708E) does about 720 Euro's or 900 Dollar.

    If you want to connect three servers, you need three network cards and a switch. So three network cards and a switch will cost you 900 Euro (1050 Dollar) for the network cards and 720 Euro (900 Dollar) for the switch, totalling 1800 euro or 1950 Dollar.

    You will get higher transfer speeds, but at a significantly higher price.

    For many business purposes this higher price can be easily justified and I would select 10 Gb over 1 Gb bonding in a heart-beat. Less cables, higher performance, lower latency.

    However, bonding gigabit interfaces allows you to use off-the-shelf equipment and maybe a nice compromis between cost, usability and performance.

    Operating System Support

    As far as I'm aware, round-robin bonding is only supported on Linux. Other operating systems do not support it.

Page 1 / 4