Connection to the CUDN (Cambridge University Data Network) from an institutional network is made through a PoP (Point-of-Presence) switch physically located on the client site. The PoP switch provides an interface between the CUDN core and the client institution, presenting the CUDN services in a form appropriate to that institution (e.g. which physical ports have which VLANs tagged and untagged).
This page describes the PoP switch hardware and software configuration, as well as the other equipment installed at the client site.
Note: the information here documents the current (as of November 2008) equipment; existing installations are being gradually upgraded to this configuration. Some of the features and services listed here will be unavailable until the upgrade programme is complete.
- Client site equipment
- PoP switch connections
- Institutional connections (ports 1-16 and 25+26 [3560E: 25+27])
- Reserved interfaces (ports 17+18)
- Diagnostic connections (ports 19+20)
- CUDN monitor interface (port 21)
- Institutional interface monitor (port 22)
- CUDN diagnostic interface (port 23)
- UPS management interface (port 24)
- CUDN uplinks (ports 27+28 [3560E: 26+28])
- Institutional connection configuration
- Pinging the PoP switch
- Miscellaneous PoP configuration settings
- Services available on the PoP
- Maintenance of the PoP equipment
- Scheduled power downs
- Upgrade during academic year 2016–2017
In most cases, three pieces of equipment are installed as part of a CUDN connection (in addition to any cabinets or fibre termination):
This equipment is described below.
Historically, a variety of PoP switches have been used, including 3Com 1100 and 3300 as well as Cisco 3500, 2950 and 2960. However, at the time of writing (November 2008), a programme of upgrades is in progress to standardise all PoP switches as a Cisco Catalyst 3560G-24PS. A small number of sites receive a Cisco Catalyst 3560E-24PD; any applicable differences between the models will be described.
Cisco Catalyst 3560G-24PS switch with Cisco RPS-2300 underneath
The Cisco 3560G-24PS is a switch with the following core features:
- 24x ports of 10/100/1000Mbit/s auto-negotiating copper ethernet (16 available to the institution)
- 4x ports of 1000Mbit/s (1Gbit/s) fibre presented as 4x SFP / mini-GBIC module slots with fibre LC connection (2 available to the institution)
- all copper ports support PoE (when enabled) in Class I, II and III, up to a maximum power budget of 370W
- maximum power draw of 540W [3560E: 750W] (including PoE)
- 1U form-factor, 37.8cm (14.9") deep [3560E: 46.0cm (18.1")]
Note: the required depth is greater, in particular due to the size of the connectors on the cable linking the RPS to the switch. For a 3560G-24PS, a minimum of 50cm (19.7") is required [3560E: 60cm (23.6")].
Rear of Cisco 3560G-24PS and RPS-2300 showing interconnecting cable
In addition to the PoP switch itself, a Cisco RPS-2300 Redundant Power Supply is installed to provide a backup in the event of the failure of either the primary supply or internal power supply unit in the switch. Please note the RPS is NOT a UPS and merely provides an alternative power supply route for the switch. In addition, it is only capable of supporting the PoP switch itself and no ancillary equipment.
- powers the attached switch including PoE devices
- maximum power draw of 750W
- 1U form-factor, 43.6cm (17.15") deep
As noted above, the cable connecting the switch and the RPS is fairly large and requires appropriate clearance - approximately 50cm (19.7").
The internal and RPS power supplies run in an active-standby (not load-sharing). The 3560E switches will revert to the internal power supply, when it is restored.
However, the 3560G switches will continue to run from the RPS unless that is interrupted. The transition from the RPS to the (UPS-protected) internal supply may cause the switch to reboot; it is recommended that this transition is forced by disconnecting the RPS power for a few seconds at a convenient "at risk" time.
To power the switch in the event of a mains power failure, an APC Smart-UPS RT 1000VA Uninterruptable Power Supply is provided, along with SURT48XLBP battery pack. This unit is connected between the mains power supply and the internal power supply unit in the switch, protecting the primary source of power to the switch; the RPS is connected directly to the mains supply.
- 2U form-factor, 48.3cm (19") deep for each of UPS and battery pack
- 55.3kg (122lb) total weight: 23.0kg (51lb) for UPS, 32.3kg (72lb) for battery pack
- can be rack-mounted or free-standing tower
Approximate run-times in the event of mains power loss are as follows:
|Devices powered||Approximate load||Approximate run-time|
|3560G-24PS (170W) only||170W||3hr:20min|
|3560G-24PS and 8x 7911G phones (2.6W)||200W||3hr:00min|
|3560G-24PS, 8x 7941G phones (4.7W) and 4x Wireless APs (15.4W)||270W||2hr:20min|
Institutions must not connect their own devices to the CUDN PoP UPS except by prior arrangement with Information Services. Typically, the only other devices which may be attached are VG224 analogue voice gateways for the VoIP project.
At the bottom of this page is some advice on handling scheduled power outages in order to preserve the operational life of UPS batteries.
Most of the 28 (24 copper + 4 fibre) network connections on the PoP are available for use by the client institution as connections into the institutional network. However, some ports are reserved for use by Information Services:
|1-16||Institution||Connection to institutional edge devices or network equipment|
|17-18||UIS / third-party||Reserved for future or 'special' CUDN use (e.g. third-party connection)|
|19-20||Institution||Diagnostic ports with institution main VLAN and voice VLAN (via CDP/LLDP-MED) for temporary connection of test devices|
|21||UIS||Monitoring port for attachment of CUDN traffic capture probes (usually temporarily)|
|22||Institution||Monitoring port for attachment of Institutional traffic capture probes (either temporarily or permanently)|
|23||UIS||Temporary attachment of UIS equipment (e.g. engineer's laptop)|
|24||UIS||Uninterruptable Power Supply (UPS) management|
|Institution||Fibre connection to Institutional network (if required) - one SX connection can be provided as standard; second one available at cost|
|UIS||Fibre uplinks to CUDN core|
The different arrangement of ports 25-28 on the 3560E-24PD is due to the physical orientation of the fibre ports on that model.
Ports 1-16 on the PoP are available for the connection of any devices the institution requires. These can be edge devices (e.g. phones, computers, wireless APs) or network devices (e.g. internal switches, firewalls, routers). Essentially, the institution can request any configuration of the available ports (with appropriate VLANs, PoE, QoS, etc.) it desires. However, for consistency, it is recommended that ports 1-12 are used for the connection of edge devices and ports 13-16 are used for connections to network equipment. It is understood that this might not be appropriate in all cases.
Ports 25 and 26 [3560E: 25 and 27] are largely analogous to ports 13-16 except that they're available as fibre connections. They can be used for connecting either network equipment or edge equipment, but the fibre presentation makes them more likely to be used for connecting network equipment. One multimode fibre SFP (with LC presentation) is available as part of the PoP install, if requested, but additional modules must be purchased from Information Services; third-party modules supplied by the institution must not be connected without prior approval from UIS.
Historically, Gigabit Ethernet connections from the PoP switches were made using multimode fibre as that was all that was available. However, many devices now feature copper Gigabit connections and insitutions may prefer this for convenience. If this is done, it is usually via ports 13-16, rather than a copper SFP module.
For more information on how these ports can be set up, see institutional connection configuration, below.
These ports are generally unused and are reserved for cases where some additional ports may be required which should not come from the institution's allocation. An example might be a slow-speed secondary connection to a third-party institution (e.g. DSL or leased line to a College hostel) where the link is piggy-backed on an existing PoP.
It is recommended institutions keep ports 19 and 20 free for the temporary connection of diagnostic equipment (e.g. a voice handset or test laptop). Essentially, these ports just have the main institutional VLAN untagged and the institutional voice VLAN tagged and advertised with CDP/LLDP-MED so two devices (or daisy-chained phone and PC) can be attached.
These ports are useful, in the event of a network fault, to perform tests from outside an institution's network to confirm the fault is from the PoP upwards to the CUDN. If these ports are used for the permanent connection of devices, an available test point is removed. Note that the port profile is that of an edge port, so only a single device should be connected.
When diagnosing faults, it can be useful to direct a copy of the network traffic to a probe to analyse it. Often this can be done remotely but, occasionally (as with many Quality of Service [QoS] problems) it can be useful to install a traffic capture probe on-site. Port 21 provides a location where such a probe can be attached and a copy of the relevant network traffic directed.
This port is for the attachment of a probe by Information Services. A similar port (22 - see next) is provided for institutional use.
Please see monitoring service, below.
In the event of the Computing Service needing to attach a device to the network at the client site, the diagnostic port (23) is available. It generally sits on a VLAN outside the client network so will not interfere with it or require IP addresses from the institution.
If a monitoring probe (see CUDN monitoring port 21) is attached, the management side of the probe (where captures can be retrieved from and the probe controlled) will be connected here.
The management card of the Uniterruptable Power Supply will be attached here, enabling the UPS to be controlled and monitored.
Institutional PoP switches will have multiple connections to the CUDN core for redundancy. These will go to different CUDN core nodes to help remove the impact of a single core device failing and, where possible, will take diverse physical routes across the GBN (Granta Backbone Network).
Redundancy on routed VLANs will be handled by a combination of Spanning Tree and a First Hop Redundancy Protocol such as VRRP or HSRP. Switched VLANs (e.g. the CUDN 'VLAN Service') will be redundantly handled across the core using Spanning Tree to the corresponding destination PoPs. Institutions should not need to perform any reconfiguration or notice any outage, other than a brief (maximum 3 seconds) interruption while backup connections take over.
Note: redundant connections are being rolled out in stages across the CUDN; it is likely that many institutions will only have a single uplink for the coming few months.
There are four main ways in which the client institution ports (1-16, 25 and 26 [3560E: 25 and 27]) can be configured. These affect parameters other than just the VLANs, including Quality of Service, Spanning Tree and CDP (Cisco Discovery Protocol).
The different configurations between these port types are summarised below:
|Purpose||A single device - computer, phone, 2-port ATA (e.g. Cisco ATA-186) or computer/phone daisy-chained); mini-hubs/switches over 4 ports should not be attached (a Trunk port should be used here).||24-port analogue telephony adapter (e.g. Cisco VG224).||Direct connection of a single Lapwing access point (multiple APs should use a Trunk port).||An institutional network, either entire or part, with multiple downstream devices (computers, phones, wireless access points, etc.); this would be connected to a switch, firewall or other networking device.|
|VLANs||Optional untagged ('access') and optional tagged voice VLAN||Untagged VLAN (voice)||One untagged (for AP management) and, if required, a local institutional VLAN tagged||Optional untagged ('trunk native') and any number tagged ('trunk')|
|Spanning Tree||Enabled with 'portfast' and 'root guard'||Disabled with 'portfast' and BPDUs ignored|
|VoIP QoS profile||Single phone (128kbit/s payload, 32kbit/s signalling)||Up to 24 phones (3Mbit/s payload, 768kbit/s signalling)||Many (>1000) phones (200Mbit/s payload, 50Mbit/s signalling)|
|MAC address limit||5 addresses, 15 minute aging upon inactivity||None|
It is important to use the appropriate type of connection for the device being connected to a particular port. Using the wrong type of port configuration may appear to work at first but present problems later (e.g. issues with Spanning Tree or QoS thresholds). It is also possible additional features may be enabled in future that will cause a port being used in an incorrect way to stop working; if in doubt, please contact the UIS.
Ports can be reconfigured by contacting Network Support at Information Services. Note that unconfigured ports will be shutdown/disabled. If asking for a port to be reconfigured that is physically up, it is best to confirm you know this, when contacting the UIS, to avoid additional delay while we make sure you haven't given the wrong port number (and reconfigure an active device).
When an institution wishes for a port on the PoP to be changed to a different type (e.g. edge to trunk) or a change in the VLANs presented, they should contact Information Services with the following information:
- the port number (1-16, 25+26 [3560E: 25+27]), or a range of numbers
- the type of port required (edge, ATA-24, Lapwing, trunk)
- for edge ports:
- the untagged/access VLAN to be used (if required)
- if the voice VLAN is to be served tagged, via CDP/LLDP-MED
- for trunk ports:
- which VLAN is to be untagged/native (if any)
- which VLAN(s) is/are to be tagged/trunk (if any)
Note that untagged VLANs lose their 802.1p CoS (Class of Service) information, which may affect the ability of other network devices to determine the appropriate Quality of Service. Note that the CUDN uses DSCP (DiffServ) to determine QoS, so will not be affected by this.
Whilst any arrangement of ports of different type and VLAN is available, it is strongly recommended that ports configured similarly are physically grouped on the switch. For example, if there are two voice downlinks to separate switches, or as a backup pair, they use physically adjacent ports.
Also, while it is possible to use untagged VLAN presentation as a way to feed a VLAN into an institutional network and retag it, this can result in the wrong VLANs being interconnected, if a mistake is made. Using institutional VLAN IDs that match the CUDN VLAN numbering system and tagged connections minimises the chances of this happening.
When VLANs on a particular port are no longer required, or a port is not needed any more, it is recommended they are unconfigured by contacting Information Services. Not doing this can result in traffic passing down links unexpectedly (perhaps affecting Quality of Service or traffic levels in general), new devices being temporarily connected to the wrong VLANs or undetected loops causing packet storms. The exception to this is unused ports in 1-12 where unused ports are typically left configured as edge ports with only the voice VLAN tagged, to allow odd phones to be connected to spare ports.
Finally, while ports can be given fixed speed or duplex settings, it is recommended they remain left to autonegotiate these parameters. Disabling autonegotiation can also affect other link parameters and increases the potential for mismatched settings. Most network devices now correctly autonegotiate link settings over both fibre and copper interfaces; the odd exception is often fibre-to-copper media converters.
The status of the PoP switch can be verified by using the PING command. Note, however, that the IP addresses of PoP switches are on special 'management VLANs' that are separate from the institutional VLANs. As such, traffic to and from them must be relayed through the upstream CUDN router; just because the PoP is not responding might not mean that it is not operational but the upstream connection is broken in some way and traffic cannot be routed between the two VLANs.
The CUDN Network Status page also lists each institution's PoP and its operational status. The names of the PoPs are of the form pop-<inst>.net..private.cam.ac.uk where <inst> usually corresponds to an institution's domain name under cam.ac.uk, possibly followed by a site identifier, where an institution has multiple PoPs. The name should also be shown on the identification label on the front.
A number of global settings are configured on the PoP switch to protect the network and improve reliability. These settings should not have any effect on the network during normal operation but are described here for reference.
To avoid broadcast storms flooding the network and consuming bandwidth away from other traffic, all interfaces (regardless of type) have a limit on ingress of broadcast traffic of 5% of total operational bandwidth (e.g. for a 1Gbit/s interface, a maximum of 50Mbit/s). Traffic over this limit will be dropped; non-broadcast traffic (unicast or multicast) will continue to be forwarded.
This level has been chosen as it is very unlikely that the normal operation of the network will require this amount of broadcast traffic; situations where this is exceeded are most likely to be loops or problematic clients. The switch records an event in its internal log that this has happened.
To improve the reliability and security of the voice network, DHCP snooping and ARP protection are enabled on the voice and Lapwing VLAN(s) only. These two features validate DHCP and ARP traffic to make it harder for rogue devices to affect the operation of the network, reducing the possibility of man-in-the-middle attacks through ARP poisoning:
- DHCP snooping prevents rogue DHCP servers from operating on switch ports which are not nominated as trusted by blocking server packets from being accepted by those ports. In addition, the DHCP traffic flowing through the switch is snooped to build a binding database of MAC/IP/interface mappings.
- ARP protection uses the binding database built by DHCP snooping to block ARP packets advertising MAC/IP mappings which do not match those confirmed by the DHCP server, unless the port on which they are received is nominated as trusted.
Ordinarily, the trusted ports for DHCP snooping and ARP protection are the same and only include the CUDN uplinks. However, where there are devices with IP addresses not allocated via DHCP (e.g. a VG224 analogue telephony gateway with a statically-configured IP address), that port will be trusted for ARP protection but not for DHCP snooping (as it does not run a DHCP server, but will have a device advertising itself using ARP packets with MAC/IP mappings that were not snooped via DHCP).
If a VG224 is connected inside an institutional network (and not directly to the PoP), the switch port connected to that part of the network must be trusted for ARP protection. As such, ARP protection will be prevented from protecting devices on other ports from rogue devices on the trusted port. When adding or removing such devices from inside an institutional network, please contact Network Support to ensure the setting is correct for the corresponding port.
To protect against this situation, and rogue DHCP servers on the voice VLAN(s) in general, it is recommended an institution deploys DHCP snooping and ARP protection internally, providing the security of this feature all the way to the edge of the network.
The PoP switch periodically (every 15 minutes) backs up a copy of the DHCP binding database to an external server. If the switch crashes or restarts, it will reload the database to minimise the number of devices whose mappings will not be known and blocked.
As stated above, edge and ATA-24 ports have a limit of 5 MAC addresses with a 15 minute inactivity timer. This helps prevent problems caused by devices with faulty network interface adapters emitting many different, and invalid, MAC addresses and filling the internal switch MAC address forwarding table, causing flooding of traffic.
This value was chosen as a sensible limit for a port with a computer and phone, leaving some spare addresses for additional MAC addresses on virtual machines (for example). If the limit is reached, an event is recorded in the switch's log.
If multiple devices are to be attached to an interface on the PoP switch, it is important to ensure that interface is configured as a trunk type, regardless of whether a large, rackmount switch or desktop mini-switch is used.
Aside from the client configurations listed above, a number of additional services can be enabled on the PoP switch upon request.
Port 22 is available for the client institution to attach a traffic capture probe to monitor traffic passing through the PoP switch. This could be as a temporary measure, to investigate a problem, or as a more permanent installation to log inbound/outbound traffic.
If the institution requires this service, it should contact the UIS with the parameters for the traffic to be captured:
- the source port(s) whose traffic is to be copied to the monitor port (this can be 'all')
- the VLAN(s) to be monitored (this can be 'all VLANs')
- whether inbound, outbound or both directions are to be captured
The institution cannot request monitoring of VLANs or ports used by the CUDN itself (e.g. the uplink ports or the management VLANs). It should also be noted that monitoring multiple ports or a full-duplex link may result in more traffic being copied to the monitor port than can be handled by it (e.g. a full-duplex 1Gbit/s link is capable of carrying 2Gbit/s) - in this case, traffic will be discarded by the monitor port.
If an institution has multiple internal core switches and wishes for a redundant connection to the PoP switch, a pair of ports can be set up as a backup pair. In this situation, the PoP switch will ensure that traffic passes over only one of the pair until that fails, in which case it will select the other, until the primary returns (with a delay to avoid flapping). In this case, both ports should be set up identically, with respect to VLANs, QoS, etc. at each end of the link. Port aggregation (such as LACP) must NOT be configured.
Multiple pairs of ports can be set up in this way, if requested, but the ports involved in a particular redundant pair cannot participate in another.
This mechanism provides much of the benefit of Spanning Tree but without the need to run it between the CUDN and the client institution network. This is done by design as the CUDN uses a proprietary version of the Spanning Tree protocol (Cisco Rapid-PVST+) which has caused interoperability issues with institutional networks (both Cisco and non-Cisco).
If an institution requires a VLAN to be kept locally within a site but fed through the PoP (e.g. an 'internal' VLAN behind a firewall they want fed back through the PoP to be served as an untagged/access VLAN on phones attached to the PoP, or another switch attached to a separate port on the PoP), it can request this.
There is no charge for this service but:
- institutions are limited to a maximum of two VLANs per site
- the VLAN tag will be chosen by Information Services and will be allocated according to the CUDN VLAN number allocation system; if this tag number is incompatible with the internal number used within the institution, it can be fed into the PoP on a dedicated port where it is untagged/native
- the VLAN cannot pass onto the CUDN core
The institution should contact the UIS stating which ports on the PoP switch the VLAN is desired on and whether it should be tagged or untagged.
If the VLAN is required elsewhere on the CUDN, the full (chargeable) ***CUDN VLAN Service can be ordered from the UIS.
If an institution requires a VLAN to cross the CUDN, it can request the CUDN VLAN Service with optional Q-in-Q tunnelling. This is a chargeable service and is described separately.
If an institution wishes to monitor the PoP switch for status or statistics, read-only SNMP access can set up to allow this. Requests should be made to the UIS stating:
- the desired community string (which cannot be 'public', 'private' or any names reserved by the UIS)
- an institutional address (or network/mask) from which queries will be sent
Access will be granted to the standard system and interfaces areas of the MIB (Management Information Base). It should be noted that the PoP switches reside on special 'management' VLANs which are separate from the normal institutional VLANs so requests will be relayed through an upstream CUDN router and will not be available if that connection is unavailable.
The UIS will, from time to time, make changes to the PoP switch software and configuration to maintain its operation, keep up to date with security and resolve problems. In most cases, these can be made without affecting service to the client institution. However, if downtime is required (e.g. an operating code update), the client institution will be given suitable warning, where possible.
In the event of a hardware fault or other situation where the PoP equipment needs to be physically accessed, the UIS will need to visit the site and carry out the appropriate work. For this reason it is requested that:
- items are not placed on top of the PoP, making removal or access difficult
- cables between other devices in the same cabinet that hang in front of the PoP can be moved out of the way, enabling the PoP to be removed without affecting other connections
- cables connected to the PoP use different colours to indicate different configurations (port type and VLAN[s]), making it easier to identify the cables, if the equipment is swapped
- access to the rear of the switch is maintained, especially for the connection of a serial console cable for out of band access
- the PoP cabinet area is kept clean and generally tidy
Failure to observe these guidelines may render the UIS unable to perform the desired maintenance or cause it to take excessive time.
For scheduled power outages where the network within the institution will be unavailable for a period in excess of approximately an hour and a half (the run-time of the UPS), it would be helpful if local staff could power off the protected side of the UPS. This helps preserve the operational life of the UPS batteries by avoiding unnecessary discharge by powering the PoP when it isn't providing any useful service:
- Shortly before the power outage, the local staff can shut down the UPS by pushing the 0 button. This will shut down the primary supply to the PoP switch and cause it to switch over (without interruption to service) to the Redundant Power Supply (RPS). The switch will then continue to run on the RPS supply (which is most likely to be connected directly to the mains).
- When mains power is lost, the PoP will die as there is no UPS to protect it.
- When mains power returns, the switch will power up using the mains supply fed via the RPS.
- At some point later, the UPS can be powered up using the 1/TEST button to restore power to the internal supply. Note that the switch may continue to run from the RPS and not revert to the internal power supply, as described above.
If in doubt, the UPS can be left alone without issue.
All the CUDN PoP switches are expected to be replaced during the academic year 2016–2017 (starting in late 2016). The service is expected to offer a number of developments:
- There will be a number of different models available:
- ~24x copper 1GE downlinks and 2x 1GE uplinks (similar to the current Cisco Catalyst 3560G-24PS)
- ~48x copper 1GE + 2x 1GE SFP / 10GE SFP+ downlinks and 2x 10GE uplinks
- ~12x fibre 1GE SFP / 10GE SFP+ downlinks and 2x 10GE uplinks
- The UPS will be optional (and incur an additional cost).
- The switch will have integral 2x PSUs and not require and separate RPS unit.
- The PoP switch itself will be optional and institutions can opt to have a directly-routed BGP connection instead.
- There may be option to have two PoPs, physically located at separate locations and interconnected with 2x fibre links to support redundant connections. This will incur an additional cost (to provide the additional switch, but not as much as a second PoP, as there will be no need to additional router ports).
- The configuration of the ports is likely to change to support spanning tree and other changes to better protect the backbone of the network.
If institutions have any queries regarding the upgrade, they should contact UIS Network Support. However, the details of the new switches are not currently available — an announcement will be made when this is determined.
Last modified: 4th August 2016