skip to primary navigationskip to content
 

IP routing on institutional connections

An institution connects to the Cambridge University Data Network (CUDN) physically through one or more interfaces on its PoP switch and logically through one or more VLANs which carry the IP subnets used by the institution, either directly or as a routed link between the CUDN router and the institutional router.

This page describes how these VLANs and subnets are configured, including how redundant routing is configured. The content of this page is of a technical nature and a good understanding of TCP/IP is required to understand some of details; the intended audience is institutional network managers.

Contents

Connection models

The IP configuration on the institutional VLAN presented on the PoP switch typically follows one of two models:

  • edge - the VLAN provides a direct connection for edge devices (e.g. computers, printers, etc.), perhaps with a proxy ARP firewall
  • routed link - the VLAN provides a routed connection between an institutional router or firewall and the upstream CUDN router

In practice, the configuration can be a hybrid of these models but care must taken by the local network manager to ensure traffic is routed correctly. The two models are described below.

Most institutions have all of their IP addresses delivered across a single VLAN, but they can request that their routed IP service be split across multiple VLANs (in addition to any service VLANs for VoIP clients, University Wireless Service access points or Managed Cluster Services, etc.). This is the additional routed VLAN service and is described separately.

Edge connections

Diagram showing edge connection

The most common and simplest type of connection has the IP subnet(s) used by the institution presented directly on the VLAN served by the CUDN router(s). The VLAN is usually taken from the PoP and carried through the institutional network directly to the edge ports serving networked client devices such as computers and printers.

This type of connection is also used with proxy ARP firewalls as the outside or untrusted VLAN. The firewall responds to the CUDN routers on behalf of client devices and selectively permits or denies traffic between this outside VLAN and an internal institutional inside or trusted VLAN where the client devices are connected. To the CUDN routers, it appears that the client devices are directly connected to the outside VLAN and do not need any special configuration to route traffic through the firewall.

CUDN edge address organisation

For subnets routed directly by the CUDN, some of the addresses are reserved for CUDN use. Which addresses are reserved depends on the size of the subnet and when the allocation was made; the table below summarises the different schemes in use (the scheme names are internal terms to refer to the layout). Some of the schemes shown here are used on routed links and not on regular edge subnets, but are shown here for completeness).

reserved
schemedefault gatewayprimary routersecondary routeravailable to institutioncomments
"62" base + 62 top - 1 base + 61 base + 1 → base + 60
base + 63 → top - 2
default prior to May 2008
"62-alt" base + 62 top - 1 top - 2 base + 1 → base + 61 
base + 63 → top - 3
useful when base + 61 has already been used by the institution
"top" top - 1 top - 2 top - 3 base + 1 → top - 4 default since May 2008; also corresponds to routed link arrangement
"bottop" base + 1 top - 1 top - 2 base + 2 → top - 3 special arrangement for legacy routed links
"bottom" base + 1 base + 2 base + 3 base + 4 → top - 1 special arrangement for some routed links
"254" base + 254 base + 253 base + 252 base + 1 → base + 251
base + 255 → top - 1
special arrangement for enlarged subnets using the top scheme

The remaining addresses are usually available for institutional use although this should be confirmed in the appropriate IP registry (e.g. IP Register). Note that the CUDN PoP equipment (switch and UPS) no longer reside on institutional subnets but on separate management VLANs and so do not take addresses from the institutional allocations.

The primary router and secondary router addresses should not be configured in institutional devices; they are used internally as part of the redundant routing configuration. The default gateway provides a redundant router address which should be transparent to clients and always be provided, with the same IP address and MAC address, even in the event of a router failure.

Multinetting and proxy ARP optimisation

Address ranges are added to a VLAN using a technique called multinetting: this overlays the new addresses on top of the same VLAN by adding secondary addresses to the router interface serving that VLAN. While this avoids having to renumber existing hosts by retaining the ranges used by existing hosts, there are a number of subtle issues presented.

The most obvious issue is that hosts in different subnets using their natural netmask (the one appropriate to the allocated subnet range for that address) won't be aware that they can actually reach each other directly on the same local network and relay traffic through their default gateway, resulting in the packets ricocheting off the upstream router - this is commonly called tromboning for obvious reasons:

Traffic flow diagram illustrating tromboning

To resolve this, the recommended netmask for subnets on multinetted VLANs routed directly on the CUDN is adjusted from the natural/actual netmask to be 255.255.0.0 (/16) and proxy ARP enabled on the CUDN router interface. Proxy ARP causes the router to respond with its own MAC address to ARP requests for IP addresses which it can reach and are NOT on the VLAN upon which the request was received. The effect of this is to make hosts believe that the whole of the /16 (either the public 131.111.0.0/16 or private 172.x.0.0/16 blocks) is reachable on the local network but have the router respond on behalf of hosts which are not and route them as normal:

Traffic flow diagram showing use of proxy ARP to avoid tromboning

This arrangement results in several potential issues:

  • If two hosts in different address ranges have mismatching netmasks configured, the traffic flow between them may be asymmetric, resulting in flooding or performance issues. For example, if host B in the above diagram had a 255.255.255.0 netmask, traffic from A to B would go directly, but the returning traffic from B to A would trombone through the router. While this usually isn't immediately a problem, optimal traffic flow will not be achieved (avoiding tromboning) until all hosts are configured with the wider netmask.
  • Traffic between public and CUDN-wide private IP addresses will still pass through the router.  This is described in more detail in the next section.
  • If an institution makes use of institution private addresses (e.g. 10.0.0.0/17, 172.31.0.0/16 or 192.168.0.0/16 - NOT CUDN-wide private addresses), the proxy ARP responses from the CUDN router must be blocked to avoid the router responding on behalf of addresses in these ranges. It may be possible to prevent the router generating responses for these addresses in future, but it is not possible to do this with the current router equipment. However, it is generally more appropriate to create a local institution private VLAN for such devices, which will avoid this issue.
  • As the device believes that the whole of the /16 is reachable directly on the local network, it will ARP for any address in this range, potentially building a very large ARP table. For the average client machine, this is generally not an issue as it doesn't talk to a large number of internal hosts; however, for a server presenting content mainly to clients elsewhere inside the CUDN, it may be preferable to configure the natural netmask.

Just because the configured netmask is much larger than the natural netmask of the subnet, local subnet broadcast traffic does not get forwarded by the router to other subnets in the same, wider range. For example, when host B above sends a broadcast to 131.111.255.255, this does not get forwarded to all subnets in 131.111.0.0/16.

If an institution wishes to avoid the use of proxy ARP, it may be possible to allocate a new, larger contiguous block to avoid multiple disparate blocks. However, the institution must manage the renumbering process from the old range(s) into the new range in a timely fashion; this is much easier if DHCP is utilised. Obtaining a large block of public addresses is often not possible due to the shortage of these; however, large CUDN-wide private blocks are normally not an issue.

Traffic between public and private addresses on the same VLAN

It should be noted that proxy ARP does not optimise the route between public and CUDN-wide private IP addresses on the same VLAN (unless the netmask was set to an unworkably large range). As such, traffic between public and private addresses will continue to trombone through the upstream router. If large traffic flows are expected between devices on different types of address, this can be solved in one of three ways:

  • renumbering the hosts to all use the same address type
  • adding a secondary address of the other type to certain hosts (e.g. adding a secondary, private IP address to a server to optimise communication with local clients on private IP addresses - the public address may need to be retained if the server needs to be accessible from the internet, e.g. it's an institutional webserver)
  • using an internal router

Routed link connection

This connection type is used when an institution has a router or firewall which operates at layer 3 (i.e. routes traffic explicitly, rather than relying on techniques such as proxy ARP). A small link subnet is configured on the VLAN between the CUDN router and the institutional router and the IP subnet(s) used by the institution are routed to the IP address(es) used by the institutional router at the end of the link. The institution then takes responsibility for routing the client subnet(s) itself and can configure them however it wishes.

There are two types of routed link; the choice of which to use depends on the complexity of the institution's connection with the CUDN:

  • Static routed links are used in the vast majority of cases, with a simple, static configuration for their routing: the institution has a single border router (or multiple border routers which redundantly provide the same IP address to the CUDN routers) and doesn't need to dynamically update the CUDN backbone routing tables, in the event of failover or network reconfiguration
  • Dynamic routed links are generally needed in more complex situations, for institutions with multiple links into the CUDN and need to dynamically adjust the routing for their networks, in the event of failover or other reconfiguration

Proxy ARP is usually disabled on such links as routing will be handled within the institution. Whether an institution uses proxy ARP internally, on their own router(s), is a matter of local preference.

Static routed link

Diagram showing routed link topology

Typically, the link subnet is a /29 providing 6 usable addresses which are divided equally between the institution and the CUDN (note that this corresponds to the top scheme used for edge connections, shown above; alternative schemes bottop or bottom may be used for some existing links, to avoid renumbering).  For example:

subnet baseownertypical use
+ 0 reserved (base address)
+ 1 institution institutional router
+ 2 institution spare, primary institutional router or device
+ 3 institution spare, secondary institutional router or device
+ 4 CUDN secondary CUDN router - physical address
+ 5 CUDN primary CUDN router - physical address
+ 6 CUDN default gateway to CUDN / internet
+ 7 reserved (broadcast address)

The institution need not configure anything with regards the primary and secondary CUDN router physical addresses; they are used internally by the redundant routing configuration and fail over in just the same way as a plain edge connection.  The default CUDN gateway should be configured on the institution equipment as the default router.

An institution can adopt the same redundancy mechanism in reverse, protecting a redundant institutional router address (perhaps on +1) with two physical router addresses on other addresses (+2 and +3).  The CUDN routes all traffic to the redundant address; if one router fails, the other takes over without the CUDN needing reconfigure anything.

In the subnets routed down to the institutional router, all addresses are typically available to the institution in the IP Register database, with only the base and top addresses reserved for the network and broadcast addresses. If an institution chooses to reorganise the routed space into subnets, there is no requirement to notify the UCS, but IP Register can create the appropriate subnets in the database to better reflect the actual organisation and make the netmasks and router addresses for the internal subnets report correctly.

Dynamic routed link

This connection type involves the institution running a dynamic routing protocol between their border routers and the CUDN routers. The CUDN routers will accept routes to institutional subnets via this protocol, allowing the institution to dynamically adjust the routing to their subnets. Where an institution has multiple connections (for example, through multiple PoP switches), the address range on each will be different and the active path(s) controlled by the institution through their use of routing advertisements.

Typically no first hop redundancy is implemented on routed links as redundancy and failover will be handled as an intrinsic part of the dynamic routing protocol.

Dynamic routed links are provided by the CUDN BGP Service.

Redundant connections

For resilience against physical cabling and equipment faults, all PoP switches are connected to two separate upstream CUDN routers via physically diverse cable paths, as far as possible.  In the event of a failure of either of these links (or faults elsewhere on the CUDN), the routers should detect this and cause traffic to divert via the remaining usable link:

  • Inbound redundancy – protects the first hop (default router / gateway address) used by client hosts for upstream traffic
  • Outbound redundancy – provided multiple paths into an institution (from the CUDN out into an institution) for downstream traffic

These are described below.

Inbound redundancy via first hop (gateway) redundancy

When sending traffic in to the CUDN from an institutional network, hosts (or a statically-configured router) will send it to a single default router / gateway address.  To improve resilience against the failure of a single router, the CUDN provides a First Hop Redundancy Protocol (FHRP) on institutional network connections.  In contrast to the more complex protocols used on the CUDN backbone, first hop redundancy does not require any special configuration of client hosts and the failure of a router should only result in an outage of a few seconds (around 3-5).

The actual protocol used to implement first hop redundancy on the CUDN may change over time, depending on the manufacturer, model and configuration of routing equipment used.  However, the basic principle of all such protocols is the same: the routers providing the redundancy cooperate to offer a single default router / gateway address with a virtual MAC/ethernet address; in the event of failure of the currently active router, a backup takes over the IP and MAC address within a few seconds meaning the clients do not need to make any adjustments to their configuration.  The only device which will tend to notice the changeover, aside from the routers themselves, is the PoP switch, which will see the MAC address of the default gateway move from one uplink port to the other.

Examples of FHRPs include the standards-based VRRP (Virtual Router Redundancy Protocol) and proprietary protocols such as Cisco's HSRP (Hot Standby Router Protocol) and GLBP (Gateway Load Balancing Protocol).

This facility is largely transparent but it should be noted that:

  • The routers will frequently (every second or so) emit status announcements to indicate they are operating: if they stop, the other router(s) will detect that one has failed and take appropriate action.  These announcements are typically made using multicast.
  • Traffic from a host to the CUDN router will go to the virtual MAC address described above.  However, traffic from the CUDN router to a host will typically come from the real (burnt in / physical) address of the router emitting it.
  • The address of the first hop router that shows up in a traceroute can be the redundant address or one of the physical addresses of the upstream routers. Which appears depends on the network equipment in use on the CUDN.
  • The active first hop router can change to a different one if the upstream CUDN router loses or has degraded connectivity to the rest of the CUDN, even it is still otherwise functioning.
  • In the future, it is possible that a load-sharing protocol will be introduce that provides different gateways to different hosts, perhaps using different MAC addresses.

Because of these reasons, care should be taken, particularly when applying filters to inbound traffic from the CUDN. Some firewalls may also have issues if the return path for traffic doesn't exactly match the outbound path.

Group numbers for redundancy protocols

Most FHRPs have some form of group ID which is used to differentiate between multiple instances of the same protocol operating on the same network.  In addition, the group ID tends to be used to construct the virtual MAC address.  As such, the group IDs must be unique for each group of addresses being managed by a FHRP (a group is typically a set of addresses which all move between routers as they move from a standby to active state).  Some institutions may use FHRPs to protect internal services, or their router's outside address on a link subnet with the CUDN; the group IDs chosen by the institution for these must not clash with the ID used by the CUDN, if that network is routed by the CUDN routers.

The range of group IDs varies for each protocol, but are typically an 8- or 12-bit number.  To avoid clashes between groups IDs used by the CUDN routers and institutional routers, ranges of these numbers are allocated for different purposes (rather like VLAN numbers):

Group ID rangeStatusUse
0-15 Reserved Unused on the CUDN (in case of accidental configuration)
16-127 Global Available for use by the CUDN
128-255 Local Available for use by institutions
256-2047 Global Available for use by the CUDN
2048-4095 Local Available for use by institutions
4096+ Reserved All other numbers reserved

Typically, group numbers need only be unique on a particular VLAN, so can (and are, on the CUDN) reused across VLANs.  However, it is always good practice to select a number conforming with the above range, even if a VLAN is not directly connected to the CUDN, as routing may be changed in future.

Outbound redundancy

As stated above, each institutional network is connected to two upstream CUDN routers.  When sending traffic out onto an institutional network from the CUDN, traffic can come from either of these two routers, not necessarily just the one which is providing the currently active default router / gateway.

The CUDN employs equal-cost multipath routing (ECMP), whereby traffic is shared amongst the available best paths across the backbone, including both downlinks to the PoP, not just a single active one.  This increases the effective bandwidth into a particular site and better utilises the available capacity of the backbone and downlinks.

It should be noted that:

  • Traffic between any particular pair of hosts (a single source-destination pair) should always take the same path (unless a routing topology change causes this to be recalculated).
  • If traffic is distributed perfectly, it could theoretically result in an effective downstream bandwidth increase to use all available links (e.g. if a PoP switch has 2x 1Gbit/s connections to the routers, it is technically possible to get 2Gbit/s of downstream bandwidth).  However, it is more likely that speeds somewhere between that of one link and both links will be achieved (i.e. between 1Gbit/s and 2Gbit/s, for the above example PoP).
  • Multipath has no effect on upstream bandwidth: that will still all go via a single uplink.  Work may be done in future to better distribute traffic across the two links.
  • Because of blocks to prevent the spoofing of source addresses, it may be the case that neither, one or both of the physical subnet addresses of the routers will be unreachable from elsewhere on the CUDN (returning an 'administratively prohibited' ICMP error from the target router); the redundant gateway address itself may or may not be reachable.  Which routers (and the gateway) are reachable at any point can vary over time (although usually only as a result of topology changes).  However, from a host on the client subnet, there will be no trouble reaching either router - this situation is normal and should be ignored: connectivity to the hosts themselves on the subnet will work without issues.

The last issue is important as the failure of a ping to the gateway address at another site may not indicate there is a problem with that site.

Voice client networks

At present, all the client networks used by the University Telephone Network Voice-over-IP (VoIP) system are routed directly on the CUDN routers and carried into institutions as layer 2 VLANs, separate from their main data VLAN(s).

In future, however, once the telephone network has stabilised, it is expected that institutions will be able to route their voice subnet(s) themselves. This will typically be done using the routed link method. If this is done, the institution will need to apply and maintain the access control lists and DHCP server addresses (for the IP/UDP helper address relay agents) as advised by the UCS Network Division.

Regardless of the method used, it is not recommended that voice VLANs pass through any stateful firewalls, used within or at the border of an institution, to avoid any delay or jitter being added to the voice traffic. In any case, the protocols used on the telephone system are awkward to firewall correctly.

Services available on edge connections

In addition to regular IP routing, there are a number of services which can be enabled on an edge connection.

These services need to operate on the first hop router for a particular connection: if an institution has a routed link connection, these should be implemented on the institution internal router(s).

DHCP relaying

If an institution wishes to use DHCP to configure client hosts on a particular connection, with servers located elsewhere on the CUDN (on a different VLAN), the first hop routers serving the clients can be configured to relay the DHCP messages between the clients and servers.

To configure this service, the IP address(es) of the DHCP servers will need to be supplied, along with the details of the client network to be served (typically by VLAN ID, or by giving one of the IP ranges in use on that connection).

There are several things to note about this service, as used on the CUDN:

  • DHCP relaying is configured on a per-VLAN, not a per-subnet, basis: the routers do not know to which subnet a host will eventually belong, until the DHCP process has completed.  This is important if multinetting is in use on a VLAN.
  • The relaying routers will record their physical interface addresses in the gateway IP address field of the DHCP packet, when they forward it on.  This allows the DHCP server to know which VLAN the request originated on.
  • If multinetting is in use, the gateway IP address could be in any of the configured ranges (although will typically be the primary IP range, so should be consistent, unless this is changed through necessity).  The DHCP server should be configured with all ranges in use as a shared network.
  • Because of the first hop redundancy protocol configuration on the CUDN, DHCP servers will typically receive two copies of each request (one via each router).

No configuration is required if the DHCP server is located on the same VLAN as the clients: they will see the request via local broadcast.

Directed broadcast forwarding (for Wake-on-LAN)

Similarly to DHCP relaying, if an institution wishes to make use of directed broadcast across VLANs, this can be enabled.  The most common use for this is to allow a Wake-on-LAN magic packet to be sent from a server to a client.

Directed broadcasts are normally disabled due to their potentially disruptive nature (if a rogue hosts transmits data to the directed broadcast address, this can generate significant traffic on the target network).  To restrict what data is forwarded as a directed broadcast, this is typically enabled with an access list specifying:

  • The source address(es) of the packets (who can send the directed broadcast).
  • The destination port(s).  For magic packets, UDP port 9 (discard) is often used, as it should have no effect on a host which is awake.

Institutions should state the target network to enabled for directed broadcasts (either by VLAN ID or IP range), along with the above restriction information.

Multicast forwarding

If multicast services are required on an edge subnet, these can be easily enabled.  This process is described on the multicast page.

Hostnames for router addresses

Because the IP addresses used for the routers are held by different routers at different times, depending on failover status, it doesn't make sense for the names of the gateway/router addresses to reflect the actual routers they are assigned to.

As such the router addresses are registered with names such as the following (where <id> is usually a number corresponding to the VLAN ID for this subnet):

  • gw-<id>.net[.private].cam.ac.uk - for the default router / gateway (e.g. gw-789.net.primary.cam.ac.uk for VLAN 789)
  • gw-<id>.route-<primary>.net[.private].cam.ac.uk - for the primary router (e.g. gw-789.route-nms.net.private.cam.ac.uk for the VLAN 789 interface on route-nms, the primary [normal] router)
  • gw-<id>.route-<secondary>.net[.private].cam.ac.uk - for the secondary router (e.g. gw-789.route-oadd.net.private.cam.ac.uk for the VLAN 789 interface on route-oadd, the secondary [backup] router)

Note that these names may resolve into multiple addresses, if the VLAN is multinetted.

Last updated: 12 August 2016