Monday, December 06, 2010

How to use dovecot expire plugin

Here I explain how to configure and use the expire plugin with dovecot.
We have a platform as below.
Fedora release 13 (Goddard)
dovecot 1.2.16:
1.In the first place, in /etc/dovecot.conf, we have to enable the use of expire plugin.
protocol pop3 {
  mail_plugins = expire
protocol lda {
  mail_plugins = expire
protocol imap {
  mail_plugins = expire
dict {
  expire = mysql:/etc/dovecot-dict-expire.conf
# it specifies that mails saved in INBOX, Trash and Spam are to be expired after 15 days
plugin {
  expire = Trash 15 Spam 15 INBOX 15
  expire_dict = proxy::expire
2.Secondly, we tell the plugin where to keep the timestamp.  In this example, we use a mysql table named mails.expires.
cat /etc/dovecot-dict-expire.conf
connect = host=localhost dbname=mails user=username password=password
map {
  pattern = shared/expire/$user/$mailbox
  table = expires
  value_field = expire_stamp
  fields {
    username = $user
    mailbox = $mailbox
The table would look like this.
mysql> show create table expires
    -> ;
| Table   | Create Table                                                                                                                                                                                                  |
| expires | CREATE TABLE `expires` (
  `username` varchar(75) NOT NULL,
  `mailbox` varchar(255) NOT NULL,
  `expire_stamp` int(11) NOT NULL,
  PRIMARY KEY (`username`,`mailbox`)
1 row in set (0.03 sec)
3.We have to make sure that mysql is supported by dovecot. 
yum install dovecot-mysql
4.After restarting dovecot, we shall soon see some entries in the table expires upon emails being saved to INBOX, Trash and Spam.  They represent the timestamp before which emails should be expunged.
mysql> select * from expires;
| username       | mailbox | expire_stamp |
| tina         | INBOX   |   1294191607 |
| louise     | INBOX   |   1294197182 |
5.The plugin is responsible to keep track of timestamp.  The actual expungement is carried out by:
/usr/sbin/dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool
Usually, we run this command in daily cron job.
What if we have emails in mbox before the plugin is enabled?  They will be expunged accordingly when the first expungement takes place.  The expire-tool finds satisfying emails according to the timestamp in mysql and that will include those emails exist there before our expire plugin is implemented. 
Note that we can use /usr/sbin/dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool --test to verify what would be executed before change is made. 
6.It is a good idea to turn on dovecot logging if you find the plugin not working.
# Log file to use for error messages, instead of sending them to syslog.
# /dev/stderr can be used to log into stderr.
log_path = /var/log/dovecot.err
For example, if you see something like 'dict: Error: Unknown dict module: mysql' in the log, then you know that dovecot-mysql is not installed.

Thursday, December 02, 2010

IMAP Migration

We helped a customer to migrate dovecot IMAP from one FC13 to another.
In the first place, we have to prepare a working sendmail and dovecot environment in the new server.
Then migration takes place as follows.
Copy accounts
Since UNIX email accounts are kept in /etc/passwd, we can setup accounts in new server by copying required entries from passwd, group, gshaow and shadow.
-rw-r--r-- 1 root root 15501 Nov 29 09:24 passwd
-rw-r--r-- 1 root root 5261 Nov 25 14:55 group
---------- 1 root root  3986 Nov 25 14:55 gshadow
---------- 1 root root  2224 Nov 25 14:55 gshadow-
---------- 1 root root 10622 Nov 29 09:24 shadow
---------- 1 root root  5331 Nov 25 14:55 shadow-
Just copying the whole file would not work as it easily leads to conflicts of system accounts in the new server.  We have to make sure that only the user accounts are being copied.
Stop sendmail and dovecot in the source machine
Before actual migration of imap folder and mbox files, we'd better stop the sendmail and dovecot daemons in the source machine to avoid further updates to the mailboxes during migration.
# service stop sendmail
# service stop dovecot
Copy ~/<user> and /var/mail/<user>
Our client is using both imap and pop3 to manage emails. 
The imap index is kept in ~/<user>/mail and pop3 mbox is in /var/mail/<user>.  After the sendmail and dovecot daemons are stopped, we can safely copy these files to the new server.  Remember to preserve the same file permissions in the new server.
It is a good idea to copy the whole home directories to the new server to avoid missing .forward, .procmailrc  or other user-specific configuration files.
Live run
After the home directories and mboxes are restored, the new server is ready for live run. 
The most straightforward way is the new server taking up the ip address of the old one as follows.
- shutdown the old server
- modify /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 (assuming that your ip is bind to eth0)
- service restart network
Then we will find mail clients connecting to the new server.
Issues that might happen
1.OE client re-downloads everything after new mail server is used
It might happen if the new server is configured to use a different uidl format from the old one.  Look for the entry "pop3_uidl_format = " in dovecot.conf in both machines to make sure they are the same.
2.Size of /var/mail
Some users tend to keep copy of messages on server and it would end up in large mbox file in /var/mail.  Regular monitoring or housekeeping on /var/mail is recommended.
3.MX priority
If we could not keep down time short, we need to setup 2nd MX such that incoming emails are queued there while the primary MX is down and pushed back after our migration completes.