skip to primary navigationskip to content
 

Bulk email and rate limiting

Introduction

The Information Services Committee has issued some guidelines on the use of bulk email in Cambridge, which state that you should seek advice if you need to send a large mailshot. This page explains why in more detail.

Both ppsw.cam.ac.uk and smtp.hermes.cam.ac.uk have a system to limit the rate at which email can be sent. This is to protect us against the bad effects of an insecure computer on the CUDN or a stolen Hermes username and password being exploited by a spammer. If a flood of junk email is sent through the central email systems, from a compromised machine or Hermes account, the whole University may be blacklisted. This has happened in the past and it is very inconvenient, so we would like to reduce the chance of it happening again.

At the same time, we do not want to restrict the legitimate use of email. Fortunately we can set generous rate limits that do not affect normal email use, and still significantly throttle unexpected floods. Therefore the rate limiting system should not be noticed by most users.

If you are running a mailing list, and you do not need to customize the messages for each recipient, then you might consider using lists.cam.ac.uk. It has a number of useful features, such as automatically removing invalid email addresses, as well as being unaffected by rate limiting. See the section on Mailing lists for details.

However the mailing list service is not always suitable, so it is sometimes necessary for a user or computer to send large amounts of email. The rest of this page explains how we accommodate this.

Configuration requirements

There is a separate page which tells you how to choose your outgoing SMTPserver, which is basically either smtp.hermes.cam.ac.uk for end users, or ppsw.cam.ac.uk for servers.

If a machine is known to be a mail server or web server which routinely sends large volumes of mail via ppsw.cam.ac.uk – peak rates of more than 60 messages in an hour – we will typically add a special rate limit for that server instead of our default policy.

However, we will not add NAT firewalls to the exception list because that would weaken the security model. We want to reduce the likelihood that a compromised machine is able to send lots of email, which means minimizing the number of machines on the exception list. Computers that need to send lots of email must have their own IP addresses.

Servers that send lots of mail must be able to handle SMTP 450 temporary errors by queueing and retrying delivery. You should not point a busy stub SMTP client directly at ppsw.cam.ac.uk because it may lose mail; instead, you should run MTA software (such as Exim or Postfix) on the server configured to relay via ppsw.cam.ac.uk , and point your stub SMTP client at this local MTA.

What happens to fast senders?

If a system sends email faster than its permitted rate the messages are quarantined pending review by a member of UIS staff. If the messages are legitimate then they are released for normal delivery. We will normally review quarantined messages in office hours, so unexpected out-of-hours mailshots may be delayed for some time.

If there appears to be a problem with the quarantined messages (e.g. they are due to a compromised account or computer) we will contact the relevant people to restore security and prevent further spam from being sent.

A different policy applies to email servers within the University which are known to correctly retry deferred email. Instead of quarantine, we use SMTP 450 temporary rejection reply codes to restrict the server's sending rate to no more than its maximum. We aim to set the limits so they do not affect legitimate email. In the event that a server hits its limit we will adjust the limit if it is too low, or contact the relevant Computer Officer if the flow of email looks suspicious.

How rates are measured

On smtp.hermes.cam.ac.uk the rate limiting system uses the authenticated username to identify the sender, so the rate it measures for a particular user includes all messages sent by that user from any computer system, including webmail. This is because we are defending against the theft of a username and password, rather than the compromise of any particular computer. Every user has the same limit.

On ppsw.cam.ac.uk the rate limiting system uses the IP address of the client computer, because in this case we are defending against security problems on individual machines. There is a default limit per IP address, and an exception list of computers that are allowed to send at higher rates.

Note that this means that all the machines behind a NAT gateway that send via ppsw.cam.ac.uk are contributing to the same rate measurement. Therefore, machines that send a lot of email should have their own IP addresses (Hermes users behind a NAT will have their own individual rate measurements based on their username.)

Measured rates are averaged over a period of an hour, which allows the limiter to accommodate short high-speed bursts. The maximum number of messages per hour is at least 60 (the exact number is subject to change); this is also the maximum number of messages in a high speed burst (the mathematical model is an exponentially-weighted moving average, so your rate an hour ago contributes at most 37% to your current measured rate, and any activity more than four hours ago is effectively forgotten). There is also a daily limit which is less than 24 times the hourly limit; this acts as a back-stop to protect against floods that last several hours (e.g. over a weekend).