Filter by topic and date
IETF email service issues resolved as of 4 March 2025
- Robert Sparks Senior Director of Information Technology
4 Mar 2025
The IETF email processing system delays reported on 3 March 2025 have been resolved. All messages in the queue were delivered by 0700 UTC on 4 March and the system is now keeping up.
The mail processing system encountered reduced disk I/O capacity due to an incident in the underlying filesystem that took some time to recover.
During this period of reduced I/O capacity, email submission to lists was very high—driven by the Internet-Draft submission deadline at 2359 UTC Monday, 3 March. This led to very long queue time for messages going to mailing lists. We are looking into ways to improve the baseline I/O capacity of the system to provide better resiliency for the future.
We believe this issue is currently resolved so will not be updating this post further.
As previously shared:
Following the email processing infrastructure transition last month, the new system was experiencing delay—sometimes over 4 hours—in delivering messages with the increased load resulting from the Internet-Draft submission deadline for the upcoming IETF 122 Bangkok meeting. We are working on short-term and long-term fixes.
Email messages were being delivered almost immediately to the IETF mailarchive.ietf.org system, which can be searched for specific messages. While it does not reflect delivery delays, general system status can be found on the IETF Status page.
One factor contributing to these delays was the number of messages being sent automatically due to the Internet-Draft submission deadline. This deadline for documents to be considered at an IETF meeting generally results in a burst of Internet-Draft (I-D) submissions. An announcement of each submission is distributed to a variety of lists, including several with large subscriber lists, such as I-D Announce. A single submission may result in many thousands, and sometimes many-10s-of-thousands of email messages being generated. This burst of I-D submissions has put additional strain on the new email processing system.
We looked into and believe we now have an understanding of the root causes of the congestion and how to help the delivery queues drain more quickly. We are also considering longer-term changes to avoid the backup in the future.