info

Current Computing Status:

Service Info

This IT Status feed has been decommissioned as of 02/13/2017.  We now have a new Services Status on our ServiceNow platform.  Please visit 4help.vt.edu to view our current Services Status. view Status History All Active Statuses

Recurring and systemic email issues

Information about recurring and systemic university email issues.  Please click VIEW to read the entire post! 

Updated:  January 17, 2014 

Due to issues we were having in mid-December, a ticket was opened with Google to address university email problems before the winter break. Our mail team continues to work with Google to resolve the issues.

 University email system administrators are also researching additional problems identified late Monday, January 6th:

  1.  Email being rejected that should not be.  UPDATE:  This seems to be resolved but the mail team is still monitoring.
  2.  Bounced email sent to a user who never sent the email.  This seems to be more visible now that we are delivering SPAM emails from Google.
  3.  Exchange Email delivery delays due to heavy loads from Activesync. This is an on-going but sporatic issue with logs filling up.
  4.  SPOOFING issues.
  5.  Receiving the message:  "This message was not sent to Spam based on your organizations's request"
  6.  January 10: Extremely delayed delivery of emails sent mid-December.  This is resolved.
  7.  January 10: Receiving very large numbers of what look to be BOUNCED messages.
  8.  January 15:  It appears that sites that use www.senderbase.org are now blocking emails from Virginia Tech. We are also being blocked by rr.com.

What we know so far:

1. Mail rejected with 5xx error messages:   Mail sent to VT Exchange users was being rejected by the VT Exchange servers if it was forwarded by certain VT mail relays (mrX.cc.vt.edu).  This resulted in "<<< 530 5.7.1 Client was not authenticated" messages. Email bounces happen often for a variety of reasons. If a sender receives a bounce (of a legitimate message) they can either resend the message, or call the recipient. This problem is believed to have been fixed sometime between 3 and 4 p.m., Monday, 1/6/14.

2.  Google is/was rejecting some mail as spam.  Senders received such messages as:

   “while talking to aspmx.l.google.com.”:

  • 550-5.7.1 [...] Our system has detected
  • 550-5.7.1 that this message is likely unsolicited mail.
  • 554 5.0.0 Service unavailable

 This is being addressed with Google.

3.  Exchange Delivery is experiencing intermittent up to 30 minute delays due to heavy load issues from ActiveSync.  THIS IS RESOLVED.

4.  SPOOFING:  Some people are receiving what looks like email messages from themselves.  This is called "spoofing" and is actually being sent by spammers who are impersonating the address.  Your email account is NOT compromised -- spammers do this to bypass spam filters on the assumption that people whitelist themselves.  

Here's a really nice SIMPLIFIED explanation from Mike Moyer, NI&S:  

Many users seem to misunderstand that the "from" address on an e-mail is actually useful or vetted somehow.  That's probably because some clients (like GMail now, and WebMail back when we ran one, Outlook/Exchange) do try to be good netizens and restrict users from sending as an arbitrary address without at least some verification that the user has access to the e-mail address they're trying to add (for the web clients users usually get sent a note with a verification link to click or the like).  So the now-common interfaces make it look like it's difficult to do.  The fact of the matter is that, by the SMTP protocol, only the domain part of an "apparent from:" e-mail address is ever verified; and that only for existence/validity. 

Think of it in postal service terms.  I can send you a letter and put anything I want for the return label: even the same name and address I'm sending to.  The postal service won't care - as long as the letter is appropriately deliverable the return address would never come in to play.  My normal mailing service (my web e-mail client) might not let me do it, but if I'm willing to write the envelopes myself (telnet to port 25 on my mail relay, use a less restrictive client or write my own), what's to stop me?  

5.  If you receive an email that states "This message was not sent to Spam based on your organizations's request" it is due to Google's suggestion to our mail team administrators to use the WHITE LIST filter to apply to mail from vt.edu.  It is reducing the amount on internal mail that gets sent to spam, but it may also cause false positives.  Please remember that we are still working to find a balance between what is spam and what is not and that it is an on-going process. 

6.  Extremely delayed delivery of emails.  We are now receiving reports of users who are receiving today (Jan. 9/Jan. 10) emails that were sent to them in mid-December.  These message had been queued and pushed further down in the sendmail queue until the queue run would abort and could never complete.  Due to Google's recommendations to add more message routers, the queues have now been drained and the emails are finally being sent.  Users may also receive bounceback mesages from emails they attempted to send in mid-December.

7.   Receiving very large numbers of what look to be BOUNCED messages.  A system disk was filled to capacity on January 9th which caused ActiveSync issues and mail to bounce.  The mail team has corrected the issue. 

8. January 15:  It appears that sites that use www.senderbase.org are now blocking emails from Virginia Tech.  We have received bounceback headers from Dell and Temple.edu who use senderbase to block organizations who send too much spam.  We are still actively blocking accounts found to be compromised and being used to send out spam.  Most of these users responded to a phish and provided their credentials to a spammer.   Headers look similar to the following for bounces from orgs blocking VT emails: 

<<< 554-ps-smtp.us.dell.com<<< 554 Connections from this sending hostname lennier.cc.vt.edu, IP address of: 198.82.162.213 are being rejected due to low SenderBase Reputation score (below -2). Your SenderBase organization: 34708. See http://www.senderbase.org/ for more information.... while talking to smtp2.ins.dell.com.

 

 

Information about other recent email problems follows in Q&A format:

 

Q1.  Why is email being caught now by Google's spam filtering?

A1.  The VT Google App email service has been capturing spam since the university transitioned to it. However, Google modified their spam filtering mechanisms in December, resulting in an increase in the number of emails identified and captured as "spam." Also, many users have not regularly checked their VT Google Apps email Spam folders prior to the recent modification. A ticket was opened with Google to review the issue right before the winter break.  We continue to work with their engineers.

 

Q2.  Why can't we obtain a spam filtering service that VT can better control?

A2.  No spam filter is perfect. Any anti-spam solution will have its version of false positives. We are looking at spam filtering alternatives for Exchange, however.

 

Q3.  I check my VT Google Apps email occasionally. Is there anything I can do?

A3.  When logged into VT Google Apps email, if you find missing legitimate emails in the Spam folder, be sure to mark them by clicking the "Not Spam" button to train Google's spam filter that they are legitimate messages.

 

Q4.  Any other advice?

A4   Suggested methods to manage spam and more information about VT Google Apps Mailbox management can be found at:   

 

Service Degraded [--]

Created: Tue, 01/07/2014 - 10:36am Updated: Tue, 12/01/2015 - 7:56pm 9402 Views