skip to primary navigationskip to content
 

Delays in email delivery between PPSW and Exchange Online tenancies

last modified Jan 22, 2020 12:17 PM
Several institutions reported to us that messages to their Exchange Online tenancies were being blocked following routine maintenance to the central email switch (PPSW) on Monday 20 January. We believe we've discovered the cause and implemented a workaround that has cleared the backlog and will prevent this from happening again.

PPSW is a pool of servers at two separate locations. The active set of machines can change – either while we carry out planned maintenance (usually once a term) or in response to hardware failure. It changed at 07:30 on Monday morning while we carried out routine patching.

This had an unanticipated effect in relation to Exchange Online tenancies. Exchange Online applies rate limits to new sources of email, also known as 'grey listing'. This can cause email to be delayed for some hours or days until a new source of email is accepted.

When this happens, the Exchange Online SMTP service reports temporary errors of the form:

451 4.7.500 Server busy. Please try again later from [131.111.8.130]. (AS77713324)

This causes email to be queued on PPSW.

This grey listing feature has impacted the main UIS Exchange Online tenancy in previous years. We solved this by adding a connector, so that the UIS Exchange Online tenancy trusted the IP addresses reserved for PPSW (131.111.8.128/27). You can read more about connectors in Microsoft's documentation.

Several colleges and departments that run private Exchange Online tenancies were reporting delays caused by grey listing on Monday afternoon. We advised them to set up a connector on a case-by-case basis.

On Tuesday afternoon, we discovered that all Exchange Online tenancies (including mail domains at other institutions) were slowing down email delivery from the new PPSW systems. This appears to be the result of a change occurring with Microsoft. On Tuesday evening, we set up special routing on PPSW (using the Exim "fallback_hosts" feature) to work around this blockage.

Messages that were stuck on PPSW are now redirected to one of the previous live systems (PPSW-32), which can deliver email to Exchange Online without any delays. The backlog of email on PPSW cleared by 21:00, and any new messages are being rerouted after a single failed delivery attempt. This special routing will remain in place for a few days until Exchange Online accepts email directly from the current live PPSW systems. We're planning to use this approach in future while Exchange Online retrains itself.

We apologise for any inconvenience caused by delays in email delivery. If you have any queries or are still experiencing issues, please contact the .

UIS Service Status

Phone padded  Service status line: (01223) 463085
Website  Sign up for SMS/email status alerts
Website  Read major IT incident reports

UIS bITe-size bulletin


A regular newsletter aimed at the University's IT community, highlighting service and project news from UIS.

Sign up >  |  Back issues

RSS Feed Latest news

Microsoft Teams deployment update

Feb 28, 2020

UIS’ recent Collaboration Tools Strategy identified Microsoft 365 as a solution that met the needs of many University users and UIS has embarked on a project to deploy and encourage adoption of the collaboration tool, Teams.

The Network Access Tokens system to gain new features in March

Feb 28, 2020

We’re launching a new version of the UIS Network Access Tokens system on Tuesday 10 March that includes several often-requested features and improves security. Users will be able to create a separate token for each device, and IT staff will be able to create tokens for shared institutional devices.

View all news