Courier IMAP and auto deleting trash after 7 days

E-mails that had been deleted for over 7 days were automatically removed from the IMAP server. E-mail date was ignored (ie the mail could have been from 2010; the actual time in “Trash” counted). This didn’t happen to other folders (Sent, Archive, Spam). This recently happened and hadn’t happened before.

I had to restore my trash folder from backups every 7 days (yay for rdiff-backup).

It took me a while to figure it out… The problem first appeared in October, right after several big changes:

  • Yosemite update
  • Airmail to Airmail 2 update (I was convinced this was the root cause, looking at my clients instead of the server)
  • Android 5.0.x
  • Android Email app sunset, changes to GMail app
  • IPv6 working decently at home after updating my RPi to Fritzbox devices.
  • Random connection errors in GMail app (this was due to misconfigured DNS in the GMail app and causing IPv6 catch all to redirect to the webserver instead of the mailserver. It didn’t happen consistently because over mobile (4G & lower) there is no IPv6 and at home is randomly falls back to IPv4 as well. IPv4 DNS was well configured.
  • Moving my ~100.000 deleted e-mails from “Deleted Items” (OS X Mail default) to “Trash” (Android & Courier default) to stop having to move them manually from one folder to the other every so often. => this was eventually the cause but I didn’t realize.

Being convinced it was most likely Airmail 2 and very maybe Android I had been looking in that direction.

Debugging was also extremely slow as I had to wait 7 days before being able to check if the changes I made helped anything.

I eventually figured out that it was not Airmail when I rebuild my whole mail database and it defaulted back to putting my deleted mails into the Archive folder instead of Trash. Archive mails were kept over 7 days, but items in Trash still removed.

That’s when I started looking at Courier IMAP config: /etc/courier/imapd (and not imapd-ssl).

There’s an option that says:

##NAME: IMAP_EMPTYTRASH:0
#
# The following setting is optional, and causes messages from the given
# folder to be automatically deleted after the given number of days.
# IMAP_EMPTYTRASH is a comma-separated list of folder:days.  The default
# setting, below, purges 7 day old messages from the Trash folder.
# Another useful setting would be:
#  
# IMAP_EMPTYTRASH=Trash:7,Sent:30
#
# This would also delete messages from the Sent folder (presumably copies
# of sent mail) after 30 days.  This is a global setting that is applied to
# every mail account, and is probably useful in a controlled, corporate
# environment.
#
# Important: the purging is controlled by CTIME, not MTIME (the file time
# as shown by ls).  It is perfectly ordinary to see stuff in Trash that's
# a year old.  That's the file modification time, MTIME, that's displayed.
# This is generally when the message was originally delivered to this
# mailbox.  Purging is controlled by a different timestamp, CTIME, which is
# changed when the file is moved to the Trash folder (and at other times too).
#
# You might want to disable this setting in certain situations - it results
# in a stat() of every file in each folder, at login and logout.
#
IMAP_EMPTYTRASH=Trash:7

Comment out that last line, and restart courier-imap(-ssl)… Simple as that.

This solved my issue.

I’m not sure when that config change happened (Debian update?) and I do not know who at Courier thought it was a good idea …. But sheesh.

Gmail (& Google Apps) & IPv6

It seems that Google’s anti spam measurement for IPv6 is basically checking if the IP has a reverse DNS. If not, reject the e-mail:

This is the mail system at host rootspirit.com.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

The mail system

<xxx@gmail.com>: host
gmail-smtp-in.l.google.com[2a00:1450:4013:c01::1b] said: 550-5.7.1
[2001:1af8:3100:a00a:21::1010 12] Our system has detected that
550-5.7.1 this message is likely unsolicited mail. To reduce the amount
of spam 550-5.7.1 sent to Gmail, this message has been blocked.
Please visit 550-5.7.1 http://support.google.com/mail/bin/answer.py?hl=en&answer=188131 for 550 5.7.1 more information. df5si15766518wjb.42 - gsmtp (in reply to end of DATA command)

There is no difference if the mail is a one word e-mail or a wall of text that crits over 9000.

The easiest way to bypass this silly check is to set up a rDNS (reverse DNS or PTR) for all your IPv6 addresses on the mail server. If you do not have this set, Google will flag all your mails over IPv6 as spam.

If you run dig -x <IPv6 address> you can find out if it’s set (answer section, after “IN PTR”):

# dig -x 2001:1af8:3100:a00a:21::1010

; < <>> DiG 9.8.4-rpz2+rl005.12-P1 < <>> -x 2001:1af8:3100:a00a:21::1010
;; global options: +cmd
;; Got answer:
;; ->>HEADER< <- opcode: QUERY, status: NOERROR, id: 30041
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 13, ADDITIONAL: 0

;; QUESTION SECTION:
;0.1.0.1.0.0.0.0.0.0.0.0.1.2.0.0.a.0.0.a.0.0.1.3.8.f.a.1.1.0.0.2.ip6.arpa. IN PTR

;; ANSWER SECTION:
0.1.0.1.0.0.0.0.0.0.0.0.1.2.0.0.a.0.0.a.0.0.1.3.8.f.a.1.1.0.0.2.ip6.arpa. 21599	IN PTR mail.rootspirit.com.

Here is a work around if you cannot set rDNS. This still allows Google to send mail to you, using IPv6. The downside is that you'll have to do this for ALL Google Apps domains you send e-mail to. Good luck.

Gmail spam “mark-as-read”

Google changed their Mailbox names (from Google Mail, to Gmail) so here is the update from my previous php script I made a few years ago.

Edit: Gmail reverted back to the previous version (imap folders were “Google Mail”, changed to “Gmail” yesterday and today). To version v0.03 won’t work anymore. I’m guessing this was a temporary change to implement their exchange/push-support.

Imap/Gmail mark as read

I’ve “updated” my previous php-app.

Since today I ended up getting this error:

Warning: imap_open(): Couldn't open stream {imap.gmail.com:993/imap/ssl}[Google Mail]/Spam in /home/yeri/.gmail.php on line 30
Certificate failure for imap.gmail.com: unable to get local issuer certificate: /C=US/ST=California/L=Mountain View/O=Google Inc/CN=imap.gmail.com - Connection failed.

Simply searching this line:

$open = imap_open ("{imap.gmail.com:993/imap/ssl/}$path"

And editing it to:

$open = imap_open ("{imap.gmail.com:993/imap/ssl/novalidate-cert}$path"

Solves the problem.

This will ignore the Google certificate. Please do double check that imap.google.com really points to Google, and it’s not some kind of DNS issue.

I’ve updated the 1st version, that ignores the certificate.