IP routing on institutional connections

An institution connects to the University Data Network (UDN) physically through one or more interfaces on its PoP switch and logically through one or more VLANs which carry the IP subnets used by the institution, either directly or as a routed link between the UDN router and the institutional router.

This page describes how these VLANs and subnets are configured, including how redundant routing is configured. The content of this page is of a technical nature and a good understanding of TCP/IP is required to understand some of details; the intended audience is institutional network managers.

Contents

• Connection models
  • Edge connection
    • UDN edge address organisation
    • Multinetting and proxy ARP optimisation
    • Traffic between public and private addresses on the same VLAN
  • Routed link connection
    • Static routed link
    • Dynamic routed link
• Redundant connections
  • Inbound redundancy via first hop (gateway) redundancy
    • Group numbers for redundancy protocols
  • Outbound redundancy
• Services available on edge connections
  • DHCP relaying
  • Directed broadcast forwarding (for Wake-on-LAN)
  • Multicast forwarding
• Voice client networks
• Hostnames for router addresses

Connection models

The IP configuration on the institutional VLAN presented on the PoP switch typically follows one of two models:

• edge - the VLAN provides a direct connection for edge devices (e.g. computers, printers, etc.), perhaps with a proxy ARP firewall
• routed link - the VLAN provides a routed connection between an institutional router or firewall and the upstream UDN router

In practice, the configuration can be a hybrid of these models but care must taken by the local network manager to ensure traffic is routed correctly. The two models are described below.

Most institutions have all of their IP addresses delivered across a single VLAN, but they can request that their routed IP service be split across multiple VLANs (in addition to any service VLANs for VoIP clients, University Wireless Service access points or Managed Cluster Services, etc.). This is the additional routed VLAN service and is described separately.

It should be noted that a VLAN/subnet is only routed to a single PoP switch and cannot be extended to other sites.  Traffic between  sites will be routed across the UDN backbone.

Edge connections

The most common and simplest type of connection has the IP subnet(s) used by the institution presented directly on the VLAN served by the UDN router(s). The VLAN is usually taken from the PoP and carried through the institutional network directly to the edge ports serving networked client devices such as computers and printers.  The subnet(s) can be public and/or UDN-wide private, including a mix of the two types.

This type of connection is also used with proxy ARP firewalls as the outside or untrusted VLAN. The firewall responds to the UDN routers on behalf of client devices and selectively permits or denies traffic between this outside VLAN and an internal institutional inside or trusted VLAN where the client devices are connected. To the UDN routers, it appears that the client devices are directly connected to the outside VLAN and do not need any special configuration to route traffic through the firewall.

UDN edge address organisation

For subnets routed directly by the UDN, some of the addresses are reserved for UDN use. Which addresses are reserved depends on the size of the subnet and when the allocation was made; the table below summarises the different schemes in use (the scheme names are internal terms to refer to the layout). Some of the schemes shown here are used on routed links and not on regular edge subnets, but are shown here for completeness).

The remaining addresses are usually available for institutional use although this should be confirmed in the appropriate IP registry (e.g. IP Register). Note that the UDN PoP equipment (switch and UPS) no longer reside on institutional subnets but on separate management VLANs and so do not take addresses from the institutional allocations.

The primary router and secondary router addresses should not be configured in institutional devices; they are used internally as part of the redundant routing configuration. The default gateway provides a redundant router address which should be transparent to clients and always be provided, with the same IP address and MAC address, even in the event of a router failure.

Multinetting and proxy ARP optimisation

Note: the scheme described below is deprecated although still supported and can continue to be used.  Instead, when institutions run out of space, it is recommended that a single new, larger subnet of the appropriate type (public or private) is used and hosts renumbered in it.  Use of DHCP, such as that provided by the IP Register DHCP Service can assist with the renumbering process.  In particular, the use of proxy ARP is not supported by the Managed Firewall Service.

Typically, when an institution requires more address space, new address ranges are added to a VLAN using a technique called multinetting: this overlays the new addresses on top of the same VLAN by adding secondary addresses to the router interface serving that VLAN. While this avoids having to renumber existing hosts by retaining the ranges used by existing hosts, there are a number of subtle issues presented.

The most obvious issue is that hosts in different subnets using their natural netmask (the one appropriate to the allocated subnet range for that address) won't be aware that they can actually reach each other directly on the same local network and relay traffic through their default gateway, resulting in the packets ricocheting off the upstream router - this is commonly called tromboning for obvious reasons:

To resolve this, the recommended netmask for subnets on multinetted VLANs routed directly on the UDN is adjusted from the natural/actual netmask to be 255.255.0.0 (/16) and proxy ARP enabled on the UDN router interface. Proxy ARP causes the router to respond with its own MAC address to ARP requests for IP addresses which it can reach and are NOT on the VLAN upon which the request was received. The effect of this is to make hosts believe that the whole of the /16 (either the public 131.111.0.0/16 or private 172.x.0.0/16 blocks) is reachable on the local network but have the router respond on behalf of hosts which are not and route them as normal:

This arrangement results in several potential issues:

• If two hosts in different address ranges have mismatching netmasks configured, the traffic flow between them may be asymmetric, resulting in flooding or performance issues. For example, if host B in the above diagram had a 255.255.255.0 netmask, traffic from A to B would go directly, but the returning traffic from B to A would trombone through the router. While this usually isn't immediately a problem, optimal traffic flow will not be achieved (avoiding tromboning) until all hosts are configured with the wider netmask.
• Traffic between public and UDN-wide private IP addresses will still pass through the router.  This is described in more detail in the next section.
• If an institution makes use of institution private addresses (e.g. 10.0.0.0/17, 172.31.0.0/16 or 192.168.0.0/16 - NOT UDN-wide private addresses), the proxy ARP responses from the UDN router must be blocked to avoid the router responding on behalf of addresses in these ranges. It may be possible to prevent the router generating responses for these addresses in future, but it is not possible to do this with the current router equipment. However, it is generally more appropriate to create a local institution private VLAN for such devices, which will avoid this issue.
• As the device believes that the whole of the /16 is reachable directly on the local network, it will ARP for any address in this range, potentially building a very large ARP table. For the average client machine, this is generally not an issue as it doesn't talk to a large number of internal hosts; however, for a server presenting content mainly to clients elsewhere inside the UDN, it may be preferable to configure the natural netmask.

Just because the configured netmask is much larger than the natural netmask of the subnet, local subnet broadcast traffic does not get forwarded by the router to other subnets in the same, wider range. For example, when host B above sends a broadcast to 131.111.255.255, this does not get forwarded to all subnets in 131.111.0.0/16.

If an institution wishes to avoid the use of proxy ARP, it may be possible to allocate a new, larger contiguous block to avoid multiple disparate blocks. However, the institution must manage the renumbering process from the old range(s) into the new range in a timely fashion; this is much easier if DHCP is utilised. Obtaining a large block of public addresses is often not possible due to the shortage of these; however, large UDN-wide private blocks are normally not an issue.

Traffic between public and private addresses on the same VLAN

It should be noted that proxy ARP does not optimise the route between public and UDN-wide private IP addresses on the same VLAN (unless the netmask was set to an unworkably large range). As such, traffic between public and private addresses will continue to trombone through the upstream router. If large traffic flows are expected between devices on different types of address, this can be solved in one of three ways:

• renumbering the hosts to all use the same address type
• adding a secondary address of the other type to certain hosts (e.g. adding a secondary, private IP address to a server to optimise communication with local clients on private IP addresses - the public address may need to be retained if the server needs to be accessible from the internet, e.g. it's an institutional webserver)
• using an internal router

Routed link connection

This connection type is used when an institution has a router or firewall which operates at layer 3 (i.e. routes traffic explicitly, rather than relying on techniques such as proxy ARP). A small link subnet is configured on the VLAN between the UDN router and the institutional router and the IP subnet(s) used by the institution are routed to the IP address(es) used by the institutional router at the end of the link. The institution then takes responsibility for routing the client subnet(s) itself and can configure them however it wishes.

Regardless of the type used, the addresses used on these subnets must not be used as the outside NAT address or for access control — they are allocated purely to set up the routed link and can change as the UDN is reorganised.  If a known public IP address is required, this should come from a range behind the router.

There are two types of routed link; the choice of which to use depends on the complexity of the institution's connection with the UDN:

• Static routed links are used in the vast majority of cases, with a simple, static configuration for their routing: the institution has a single border router (or multiple border routers which redundantly provide the same IP address to the UDN routers) and doesn't need to dynamically update the UDN backbone routing tables, in the event of failover or network reconfiguration
• Dynamic routed links are generally needed in more complex situations, for institutions with multiple links into the UDN and need to dynamically adjust the routing for their networks, in the event of failover or other reconfiguration

Proxy ARP is usually disabled on such links as routing will be handled within the institution. Whether an institution uses proxy ARP internally, on their own router(s), is a matter of local preference.

Where the subnets routed over a link include public addresses (exclusively, or in addition to UDN-wide private addresses), internet best practice is that the addresses used on the link subnet itself should themselves be public, so they appear correctly in traceroutes and other cases where they are exposed (such as ICMP errors).  If the subnets used on a link will exclusively use UDN-wide private addresses, the addresses used on the link can be UDN-wide private.  The UIS will not route public addresses over a link using UDN-wide private addressing, but this does not stop institutions from doing this internally (although it is strongly advised against, similarly if institution private addresses are used).

Static routed link

Typically, the link subnet is a /29 providing 6 usable addresses which are divided equally between the institution and the UDN (note that this corresponds to the top scheme used for edge connections, shown above; alternative schemes bottop or bottom may be used for some existing links, to avoid renumbering).  For example:

The institution need not configure anything with regards the primary and secondary UDN router physical addresses; they are used internally by the redundant routing configuration and fail over in just the same way as a plain edge connection.  The default UDN gateway should be configured on the institution equipment as the default router.

An institution can adopt the same redundancy mechanism in reverse, protecting a redundant institutional router address (perhaps on +1) with two physical router addresses on other addresses (+2 and +3).  The UDN routes all traffic to the redundant address; if one router fails, the other takes over without the UDN needing reconfigure anything.

In the subnets routed down to the institutional router, all addresses are typically available to the institution in the IP Register database, with only the base and top addresses reserved for the network and broadcast addresses. If an institution chooses to reorganise the routed space into subnets, there is no requirement to notify the UCS, but IP Register can create the appropriate subnets in the database to better reflect the actual organisation and make the netmasks and router addresses for the internal subnets report correctly.

Dynamic routed link

This connection type involves the institution running a dynamic routing protocol between their border routers and the UDN routers. The UDN routers will accept routes to institutional subnets via this protocol, allowing the institution to dynamically adjust the routing to their subnets. Where an institution has multiple connections (for example, through multiple PoP switches), the address range on each will be different and the active path(s) controlled by the institution through their use of routing advertisements.

Typically no first hop redundancy is implemented on routed links as redundancy and failover will be handled as an intrinsic part of the dynamic routing protocol.

Dynamic routed links are provided by the UDN BGP Service.

Redundant connections

For resilience against physical cabling and equipment faults, all PoP switches are connected to two separate upstream UDN routers via physically diverse cable paths, as far as possible.  In the event of a failure of either of these links (or faults elsewhere on the UDN), the routers should detect this and cause traffic to divert via the remaining usable link:

• Inbound redundancy – protects the first hop (default router / gateway address) used by client hosts for upstream traffic
• Outbound redundancy – provided multiple paths into an institution (from the UDN out into an institution) for downstream traffic

These are described below.

Inbound redundancy via first hop (gateway) redundancy

When sending traffic in to the UDN from an institutional network, hosts (or a statically-configured router) will send it to a single default router / gateway address.  To improve resilience against the failure of a single router, the UDN provides a First Hop Redundancy Protocol (FHRP) on institutional network connections.  In contrast to the more complex protocols used on the UDN backbone, first hop redundancy does not require any special configuration of client hosts and the failure of a router should only result in an outage of a few seconds (around 3-5).

The actual protocol used to implement first hop redundancy on the UDN may change over time, depending on the manufacturer, model and configuration of routing equipment used.  However, the basic principle of all such protocols is the same: the routers providing the redundancy cooperate to offer a single default router / gateway address with a virtual MAC/ethernet address; in the event of failure of the currently active router, a backup takes over the IP and MAC address within a few seconds meaning the clients do not need to make any adjustments to their configuration.  The only device which will tend to notice the changeover, aside from the routers themselves, is the PoP switch, which will see the MAC address of the default gateway move from one uplink port to the other.

Examples of FHRPs include the standards-based VRRP (Virtual Router Redundancy Protocol) and proprietary protocols such as Cisco's HSRP (Hot Standby Router Protocol) and GLBP (Gateway Load Balancing Protocol).

This facility is largely transparent but it should be noted that:

• The routers will frequently (every second or so) emit status announcements to indicate they are operating: if they stop, the other router(s) will detect that one has failed and take appropriate action.  These announcements are typically made using multicast.
• Traffic from a host to the UDN router will go to the virtual MAC address described above.  However, traffic from the UDN router to a host will typically come from the real (burnt in / physical) address of the router emitting it.
• The address of the first hop router that shows up in a traceroute can be the redundant address or one of the physical addresses of the upstream routers. Which appears depends on the network equipment in use on the UDN.
• The active first hop router can change to a different one if the upstream UDN router loses or has degraded connectivity to the rest of the UDN, even it is still otherwise functioning.
• In the future, it is possible that a load-sharing protocol will be introduce that provides different gateways to different hosts, perhaps using different MAC addresses.

Because of these reasons, care should be taken, particularly when applying filters to inbound traffic from the UDN. Some firewalls may also have issues if the return path for traffic doesn't exactly match the outbound path.

Group numbers for redundancy protocols

Most FHRPs have some form of group ID which is used to differentiate between multiple instances of the same protocol operating on the same network.  In addition, the group ID tends to be used to construct the virtual MAC address.  As such, the group IDs must be unique for each group of addresses being managed by a FHRP (a group is typically a set of addresses which all move between routers as they move from a standby to active state).  Some institutions may use FHRPs to protect internal services, or their router's outside address on a link subnet with the UDN; the group IDs chosen by the institution for these must not clash with the ID used by the UDN, if that network is routed by the UDN routers.

The range of group IDs varies for each protocol, but are typically an 8- or 12-bit number.  To avoid clashes between groups IDs used by the UDN routers and institutional routers, ranges of these numbers are allocated for different purposes (rather like VLAN numbers):

Typically, group numbers need only be unique on a particular VLAN, so can (and are, on the UDN) reused across VLANs.  However, it is always good practice to select a number conforming with the above range, even if a VLAN is not directly connected to the UDN, as routing may be changed in future.

Outbound redundancy

As stated above, each institutional network is connected to two upstream UDN routers.  When sending traffic out onto an institutional network from the UDN, traffic can come from either of these two routers, not necessarily just the one which is providing the currently active default router / gateway.

The UDN employs equal-cost multipath routing (ECMP), whereby traffic is shared amongst the available best paths across the backbone, including both downlinks to the PoP, not just a single active one.  This increases the effective bandwidth into a particular site and better utilises the available capacity of the backbone and downlinks.

It should be noted that:

• Traffic between any particular pair of hosts (a single source-destination pair) should always take the same path (unless a routing topology change causes this to be recalculated).
• If traffic is distributed perfectly, it could theoretically result in an effective downstream bandwidth increase to use all available links (e.g. if a PoP switch has 2x 1Gbit/s connections to the routers, it is technically possible to get 2Gbit/s of downstream bandwidth).  However, it is more likely that speeds somewhere between that of one link and both links will be achieved (i.e. between 1Gbit/s and 2Gbit/s, for the above example PoP).
• Multipath has no effect on upstream bandwidth: that will still all go via a single uplink.  Work may be done in future to better distribute traffic across the two links.
• Because of blocks to prevent the spoofing of source addresses, it may be the case that neither, one or both of the physical subnet addresses of the routers will be unreachable from elsewhere on the UDN (returning an 'administratively prohibited' ICMP error from the target router); the redundant gateway address itself may or may not be reachable.  Which routers (and the gateway) are reachable at any point can vary over time (although usually only as a result of topology changes).  However, from a host on the client subnet, there will be no trouble reaching either router - this situation is normal and should be ignored: connectivity to the hosts themselves on the subnet will work without issues.

The last issue is important as the failure of a ping to the gateway address at another site may not indicate there is a problem with that site.  If a way is required to monitor the status of the link from another location, the institution will need to monitor a "reliable" host — something which is likely to be online at all required times: perhaps a server or other dedicated host.

Voice client networks

At present, all the client networks used by the University Telephone Network Voice-over-IP (VoIP) system are routed directly on the UDN routers and carried into institutions as layer 2 VLANs, separate from their main data VLAN(s).

In future, however, once the telephone network has stabilised, it is expected that institutions will be able to route their voice subnet(s) themselves. This will typically be done using the routed link method. If this is done, the institution will need to apply and maintain the access control lists and DHCP server addresses (for the IP/UDP helper address relay agents) as advised by the UCS Network Division.

Regardless of the method used, it is not recommended that voice VLANs pass through any stateful firewalls, used within or at the border of an institution, to avoid any delay or jitter being added to the voice traffic. In any case, the protocols used on the telephone system are awkward to firewall correctly.

Services available on edge connections

In addition to regular IP routing, there are a number of services which can be enabled on an edge connection.

These services need to operate on the first hop router for a particular connection: if an institution has a routed link connection, these should be implemented on the institution internal router(s).

DHCP relaying

If an institution wishes to use DHCP to configure client hosts on a particular connection, with servers located elsewhere on the UDN (on a different VLAN), the first hop routers serving the clients can be configured to relay the DHCP messages between the clients and servers.

To configure this service, the IP address(es) of the DHCP servers will need to be supplied, along with the details of the client network to be served (typically by VLAN ID, or by giving one of the IP ranges in use on that connection).

There are several things to note about this service, as used on the UDN:

• DHCP relaying is configured on a per-VLAN, not a per-subnet, basis: the routers do not know to which subnet a host will eventually belong, until the DHCP process has completed.  This is important if multinetting is in use on a VLAN.
• The relaying routers will record their physical interface addresses in the gateway IP address field of the DHCP packet, when they forward it on.  This allows the DHCP server to know which VLAN the request originated on.
• If multinetting is in use, the gateway IP address could be in any of the configured ranges (although will typically be the primary IP range, so should be consistent, unless this is changed through necessity).  The DHCP server should be configured with all ranges in use as a shared network.
• Because of the first hop redundancy protocol configuration on the UDN, DHCP servers will typically receive two copies of each request (one via each router).

No configuration is required if the DHCP server is located on the same VLAN as the clients: they will see the request via local broadcast.

Directed broadcast forwarding (for Wake-on-LAN)

Similarly to DHCP relaying, if an institution wishes to make use of directed broadcast across VLANs, this can be enabled.  The most common use for this is to allow a Wake-on-LAN magic packet to be sent from a server to a client.

Directed broadcasts are normally disabled due to their potentially disruptive nature (if a rogue hosts transmits data to the directed broadcast address, this can generate significant traffic on the target network).  To restrict what data is forwarded as a directed broadcast, this is typically enabled with an access list specifying:

• The source address(es) of the packets (who can send the directed broadcast).
• The destination port(s).  For magic packets, UDP port 9 (discard) is often used, as it should have no effect on a host which is awake.

Institutions should state the target network to enabled for directed broadcasts (either by VLAN ID or IP range), along with the above restriction information.

Multicast forwarding

If multicast services are required on an edge subnet, these can be easily enabled.  This process is described on the multicast page.

Hostnames for router addresses

Because the IP addresses used for the routers are held by different routers at different times, depending on failover status, it doesn't make sense for the names of the gateway/router addresses to reflect the actual routers they are assigned to.

As such the router addresses are registered with names such as the following (where <id> is usually a number corresponding to the VLAN ID for this subnet):

• gw-<id>.net[.private].cam.ac.uk - for the default router / gateway (e.g. gw-789.net.primary.cam.ac.uk for VLAN 789)
• gw-<id>.d-<primary-short-code>.net[.private].cam.ac.uk - for the primary router (e.g. gw-789.d-nm.net.private.cam.ac.uk for the VLAN 789 interface on dist-nms, the primary [normal] router)
• gw-<id>.d-<secondary-short-code>.net[.private].cam.ac.uk - for the secondary router (e.g. gw-789.d-oa.net.private.cam.ac.uk for the VLAN 789 interface on dist-oadd, the secondary [backup] router)

Note that these names may resolve into multiple addresses, if the VLAN is multinetted.

The short codes for routers are the first characters of the full name of the router (e.g. "d-nm" is short for "dist-nms").  This, along with shortening "dist" to "d" just shortens the names to use up less screen width in outputs (which is especially important when router names are used at both ends, in a backbone link on the network).

Many names were created prior to this scheme and have the full "route-<router-name>.net[.private].cam.ac.uk" forms of the name.  These will be adjusted, in due course.

Last updated: 17th January 2023