Monday, December 06, 2010

How to use dovecot expire plugin

Here I explain how to configure and use the expire plugin with dovecot.
We have a platform as below.
Fedora release 13 (Goddard)
dovecot 1.2.16:
1.In the first place, in /etc/dovecot.conf, we have to enable the use of expire plugin.
protocol pop3 {
  mail_plugins = expire
protocol lda {
  mail_plugins = expire
protocol imap {
  mail_plugins = expire
dict {
  expire = mysql:/etc/dovecot-dict-expire.conf
# it specifies that mails saved in INBOX, Trash and Spam are to be expired after 15 days
plugin {
  expire = Trash 15 Spam 15 INBOX 15
  expire_dict = proxy::expire
2.Secondly, we tell the plugin where to keep the timestamp.  In this example, we use a mysql table named mails.expires.
cat /etc/dovecot-dict-expire.conf
connect = host=localhost dbname=mails user=username password=password
map {
  pattern = shared/expire/$user/$mailbox
  table = expires
  value_field = expire_stamp
  fields {
    username = $user
    mailbox = $mailbox
The table would look like this.
mysql> show create table expires
    -> ;
| Table   | Create Table                                                                                                                                                                                                  |
| expires | CREATE TABLE `expires` (
  `username` varchar(75) NOT NULL,
  `mailbox` varchar(255) NOT NULL,
  `expire_stamp` int(11) NOT NULL,
  PRIMARY KEY (`username`,`mailbox`)
1 row in set (0.03 sec)
3.We have to make sure that mysql is supported by dovecot. 
yum install dovecot-mysql
4.After restarting dovecot, we shall soon see some entries in the table expires upon emails being saved to INBOX, Trash and Spam.  They represent the timestamp before which emails should be expunged.
mysql> select * from expires;
| username       | mailbox | expire_stamp |
| tina         | INBOX   |   1294191607 |
| louise     | INBOX   |   1294197182 |
5.The plugin is responsible to keep track of timestamp.  The actual expungement is carried out by:
/usr/sbin/dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool
Usually, we run this command in daily cron job.
What if we have emails in mbox before the plugin is enabled?  They will be expunged accordingly when the first expungement takes place.  The expire-tool finds satisfying emails according to the timestamp in mysql and that will include those emails exist there before our expire plugin is implemented. 
Note that we can use /usr/sbin/dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool --test to verify what would be executed before change is made. 
6.It is a good idea to turn on dovecot logging if you find the plugin not working.
# Log file to use for error messages, instead of sending them to syslog.
# /dev/stderr can be used to log into stderr.
log_path = /var/log/dovecot.err
For example, if you see something like 'dict: Error: Unknown dict module: mysql' in the log, then you know that dovecot-mysql is not installed.

Thursday, December 02, 2010

IMAP Migration

We helped a customer to migrate dovecot IMAP from one FC13 to another.
In the first place, we have to prepare a working sendmail and dovecot environment in the new server.
Then migration takes place as follows.
Copy accounts
Since UNIX email accounts are kept in /etc/passwd, we can setup accounts in new server by copying required entries from passwd, group, gshaow and shadow.
-rw-r--r-- 1 root root 15501 Nov 29 09:24 passwd
-rw-r--r-- 1 root root 5261 Nov 25 14:55 group
---------- 1 root root  3986 Nov 25 14:55 gshadow
---------- 1 root root  2224 Nov 25 14:55 gshadow-
---------- 1 root root 10622 Nov 29 09:24 shadow
---------- 1 root root  5331 Nov 25 14:55 shadow-
Just copying the whole file would not work as it easily leads to conflicts of system accounts in the new server.  We have to make sure that only the user accounts are being copied.
Stop sendmail and dovecot in the source machine
Before actual migration of imap folder and mbox files, we'd better stop the sendmail and dovecot daemons in the source machine to avoid further updates to the mailboxes during migration.
# service stop sendmail
# service stop dovecot
Copy ~/<user> and /var/mail/<user>
Our client is using both imap and pop3 to manage emails. 
The imap index is kept in ~/<user>/mail and pop3 mbox is in /var/mail/<user>.  After the sendmail and dovecot daemons are stopped, we can safely copy these files to the new server.  Remember to preserve the same file permissions in the new server.
It is a good idea to copy the whole home directories to the new server to avoid missing .forward, .procmailrc  or other user-specific configuration files.
Live run
After the home directories and mboxes are restored, the new server is ready for live run. 
The most straightforward way is the new server taking up the ip address of the old one as follows.
- shutdown the old server
- modify /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 (assuming that your ip is bind to eth0)
- service restart network
Then we will find mail clients connecting to the new server.
Issues that might happen
1.OE client re-downloads everything after new mail server is used
It might happen if the new server is configured to use a different uidl format from the old one.  Look for the entry "pop3_uidl_format = " in dovecot.conf in both machines to make sure they are the same.
2.Size of /var/mail
Some users tend to keep copy of messages on server and it would end up in large mbox file in /var/mail.  Regular monitoring or housekeeping on /var/mail is recommended.
3.MX priority
If we could not keep down time short, we need to setup 2nd MX such that incoming emails are queued there while the primary MX is down and pushed back after our migration completes.

Wednesday, October 20, 2010

How to use Fax for Asterisk - Part 3

We also need a mechanism to retry unanswered numbers and capture the retry result.
The Asterisk callfile directive 'Maxretries:1' can be used as a simple way to retry unanswered fax numbers.  A value of 1 means that Asterisk will attempt once more.   The default retry interval is 5 min or you can specify one using the RetryTime directive. :
In the extensions.conf, we could capture failure reason in the 'failed' extension.  Asterisk will land this extension if all retries fail.
exten => failed,1,NoOp(**** HANDLE UNANSWERED CALLFILE ****)
exten => failed,n,NoOp(FAIL_REASON:${REASON})
exten => failed,n,NoOp(DNIS:${RECEIVER})
The variable ${REASON} provides certain useful information about the failure reasons as below.
0 - Failed (not busy or congested)
1 - Hung up
3 - Ring timeout
5 - Busy
8 - Congestion
We could use AGI to store the reasons and numbers to database to compile an unsuccessful fax report.

Tuesday, September 28, 2010

How to use Fax for Asterisk - Part 2

How to use Fax for Asterisk - Part 2

We send and receive faxes via the dialplan function FAXOPT and SendFax/ReceiveFax asterisk applications.


*CLI> core show application sendfax

  -= Info about application 'SendFAX' =-


Sends a specified TIFF/F file as a FAX.



 The SendFAX() application sends the specified TIFF/F file(s) as a FAX.

 The application arguments are:

    'd' - enables FAX debugging

    'f' - allow audio fallback FAX transfer on T.38 capable channels

    'z' - initiate a T.38 reinvite on the channel if the remote end does not

    's' - send progress Manager events (overrides statusevents setting in res_fa                                              x.conf)

 Use the FAXOPT function to specify session arguments prior to calling SendFAX()

 and use FAXOPT after SendFAX completes to query result status for the session.

 The SendFAX() application is provided by res_fax, which is a FAX technology agn                                              ostic module

 that utilizes FAX technology resource modules to complete a FAX transmission.


Not available


Not available

[See Also]

Not available

The FAXOPT dialplan function is used to set options to be used by SendFax or ReceiveFax and obtain the fax results afterwards.

To faxout, we can use callfile to dial the destination fax number and call SendFax in dialplan.













exten => ffa_faxout,1,NoOp(**** SENDING FAX ****)

exten => ffa_faxout,n,Wait(6)

exten => ffa_faxout,n,Set(FAXFILE=${TIFF_2_SEND})


exten => ffa_faxout,n,NoOp(**** SETTING FAXOPT ****)

exten => ffa_faxout,n,Set(FAXOPT(ecm)=yes)

exten => ffa_faxout,n,Set(FAXOPT(headerinfo)=${TAGLINE})

exten => ffa_faxout,n,Set(FAXOPT(maxrate)=14400)

exten => ffa_faxout,n,Set(FAXOPT(minrate)=2400)

; Send the fax

exten => ffa_faxout,n,NoOp(**** SENDING FAX : ${FAXFILE} ****)

exten => ffa_faxout,n,SendFAX(/tmp/${FAXFILE},dfzs)

; Hangup! Print FAXOPTs

exten => h,1,NoOp(FAXOPT(ecm) : ${FAXOPT(ecm)})

exten => h,n,NoOp(FAXOPT(filename) : ${FAXOPT(filename)})

exten => h,n,NoOp(FAXOPT(headerinfo) : ${FAXOPT(headerinfo)})

exten => h,n,NoOp(FAXOPT(localstationid) : ${FAXOPT(localstationid)})

exten => h,n,NoOp(FAXOPT(maxrate) : ${FAXOPT(maxrate)})

exten => h,n,NoOp(FAXOPT(minrate) : ${FAXOPT(minrate)})

exten => h,n,NoOp(FAXOPT(pages) : ${FAXOPT(pages)})

exten => h,n,NoOp(FAXOPT(rate) : ${FAXOPT(rate)})

exten => h,n,NoOp(FAXOPT(remotestationid) : ${FAXOPT(remotestationid)})

exten => h,n,NoOp(FAXOPT(resolution) : ${FAXOPT(resolution)})

exten => h,n,NoOp(FAXOPT(status) : ${FAXOPT(status)})

exten => h,n,NoOp(FAXOPT(statusstr) : ${FAXOPT(statusstr)})

exten => h,n,NoOp(FAXOPT(error) : ${FAXOPT(error)})

exten => h,n,NoOp(DNIS: ${RECEIVER})

exten => h,n,System( ${RECEIVER} ${FAXOPT(filename)} ${FAXOPT(status)} ${FAXOPT(statusstr)}  ${FAXOPT(error)})

Points to note.

It is not recommended to edit the callfile in /var/spool/asterisk/outgoing as it might be processed by asterisk in the middle of creation.  A better approach is to create the callfile somewhere else and 'mv' to the outgoing directory.

The resolution of tiff is important.  We can enforce the correct values by specifying them in the gs command.

gs -q -dNOPAUSE -dBATCH -sDEVICE=tiffg4 -sPAPERSIZE=a4 -g1680x2285 -sOutputFile=sample.tiff sample.pdf

We can verify our fax document using 'tiffinfo sample.tiff'

If we receive 'res_fax_digium.c: FAX handle 0: failed to queue document', it is likely due to mis-formatted tiff.

Utf8 tagline is currently not supported.

We can pass certain variables from callfile to our dialplan, eg the destination number, for generating fax status reports.  The fax status is obtained from ${FAXOPT(status)}

The is a simple shell script that merely logs whatever number we faxed with the fax status.  Alternatively, we could use AGI to insert the data to database for billing purpose.

To enable t38 fax, we have to set 't38pt_udptl=yes' in sip.conf 

If we receive error 'Audio FAX not allowed on channel', it is likely an issue with the t38_udptl value.

To monitor fax progress, we can set debug on as below.

*CLI> fax set debug on

To send faxes simultaneously (assuming that we purchased enough channels from Digium), we could put multiple callfiles in /var/spool/asterisk/outgoing and monitor this folder to add more callfiles when appropriate. 


In following parts, I would show how to control the maximum number of concurrent fax calls and automate fax retry.  


How to use Fax for Asterisk - Part 1

How to use Fax for Asterisk - Part 1

Recently, we evaluated fax for asterisk in the process to find a stable fax solution under asterisk.

Ffa is selected because it supports both DAHDI TDM fax and T38 Voip fax.  Note that t38 termination is only available when using asterisk 1.6 and above.  In asterisk 1.4, we can only use g711 faxing with ffa, a solution which is more vulnerable to network jitters and delays.  

We setup an environment as below for evaluating ffa.



Dahdi with 2-port Digium card

Then we download the free Fax for Asterisk (1-channel) from

Ffa installation involves below steps.

Create account with Digium

In the first place, we create an account with digium and order free FFA on digium store which would send us a license string afterwards.

Register free fax for asterisk

Cd /usr/src


Chmod 500 register


This program prompt us to enter the license string obtained from digium.

Determine the build

wget -O benchfax

Chmod 500 benchfax


This process examines our system to determine an optimal build and it could take over 10 minutes to complete.

Fax selector

Next, we visit and input the o/s, asterisk version and the 'build' as suggested from benchfax.  The fax selector will respond with the corresponding urls of res_fax and res_fax_digium.

Install the required components




tar xzvf res_fax-

cp res_fax- /usr/lib/asterisk/modules

tar xzvf res_fax_digium-

cp res_fax_digium- /usr/lib/asterisk/modules

Load res_fax and res_fax_digium in asterisk

Then we can load the two modules in asterisk.

*CLI> module load

*CLI> module load

To verify ffa installation, we can :

*CLI> fax show stats

FAX Statistics:


Current Sessions     : 0

Transmit Attempts    : 0

Receive Attempts     : 0

Completed FAXes      : 0

Failed FAXes         : 0

Digium G.711

Licensed Channels    : 1

Max Concurrent       : 1

Success              : 0

Switched to T.38     : 0

Canceled             : 0

No FAX               : 0

Partial              : 0

Negotiation Failed   : 0

Train Failure        : 0

Protocol Error       : 0

IO Partial           : 0

IO Fail              : 0

Digium T.38

Licensed Channels    : 1

Max Concurrent       : 1

Success              : 0

Canceled             : 0

No FAX               : 0

Partial              : 0

Negotiation Failed   : 0

Train Failure        : 0

Protocol Error       : 0

IO Partial           : 0

IO Fail              : 0

From the output, we should see that ffa now supports both g711 and t38 faxing.


Note that it is not recommended to specify 'load=>' in modules.conf as long as we have autoload=yes.


In next parts, I would demonstrate how to place fax calls and monitor results.

Saturday, August 14, 2010

Fight again brute force attacks using iptables and fail2ban

As a VOIP provider, we inevitably expose our SIP servers to brute force attempts and scans.  Here are some simple steps to scan the asterisk log using fail2ban and update iptables rules so as to reject registrations from suspicious ip addresses.
In the first place, download and install Fail2ban from  Fail2ban scans log files like /var/log/asterisk/full, /var/log/pwdfail, etc and rejects ip that makes too many unsuccessful attempts by updating the iptables firewall rules.
Install and configure Fail2ban
If you are using fedora, then install Fail2ban is simple:
yum install fail2ban
After successful installation, you should see the Fail2ban files in /etc/fail2ban.
Then we need to create a configuration for Fail2ban so that it understands attacks on Asterisk. 
Go to /etc/fail2ban/fillter.d and create a file named asterisk.conf with below contents.
# Fail2Ban configuration file
# $Revision: 250 $


# Read common prefixes. If any customizations available -- read them from
# common.local
#before = common.conf


#_daemon = asterisk

# Option:  failregex
# Notes.:  regex to match the password failures messages in the logfile. The
#          host must be matched by a group named "host". The tag "<HOST>" can
#          be used for standard IP/hostname matching and is only an alias for
#          (?:::f{4,6}:)?(?P<host>\S+)
# Values:  TEXT

failregex = NOTICE.* .*: Registration from '.*' failed for '<HOST>' - Wrong password
            NOTICE.* .*: Registration from '.*' failed for '<HOST>' - No matching peer found
            NOTICE.* .*: Registration from '.*' failed for '<HOST>' - Username/auth name mismatch
            NOTICE.* .*: Registration from '.*' failed for '<HOST>' - Device does not match ACL
            NOTICE.* <HOST> failed to authenticate as '.*'$
            NOTICE.* .*: No registration for peer '.*' \(from <HOST>\)
            NOTICE.* .*: Host <HOST> failed MD5 authentication for '.*' (.*)
            NOTICE.* .*: Failed to authenticate user .*@<HOST>.*

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
ignoreregex =
This file basically tells Fail2ban the patterns of brute force attempts.  Unsuccessful registrations are logged in /var/log/asterisk/ as 'NOTICE.*.*:Registration from' and the logfile provides the source information for Fail2ban to find suspicious ip. 
Next, we  need to put below section in /etc/fail/jail.conf:

enabled  = true
filter   = asterisk
action   = iptables-allports[name=ASTERISK, protocol=all]
logpath  = /var/log/asterisk/full
maxretry = 5
bantime = 259200

This configuration places a ban period of 3 days and notifies you via if brute force attempts are detected.
Setup exclusion in Fail2ban
It is also important to specify certain ip (like internal ip) for exclusion in the jail.conf's [DEFAULT] section.  For example,
# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip =
Configure Asterisk logging
On the asterisk side, we have to a) modify /etc/asterisk/logger.conf to use a datetime pattern recognized by Fail2ban and b) generate the required asterisk log file to match what is specified in "logpath  = /var/log/asterisk/full" in jail.conf
; Customize the display of debug message time stamps
; this example is the ISO 8601 date format (yyyy-mm-dd HH:MM:SS)
; see strftime(3) Linux manual for format specifiers
dateformat=%F %T
; Format is "filename" and then "levels" of debugging to be included:
;    debug
;    notice
;    warning
;    error
;    verbose
;    dtmf
; Special filename "console" represents the system console
; We highly recommend that you DO NOT turn on debug mode if you are simply
; running a production system.  Debug mode turns on a LOT of extra messages,
; most of which you are unlikely to understand without an understanding of
; the underlying code.  Do NOT report debug messages as code issues, unless
; you have a specific issue that you are attempting to debug.  They are
; messages for just that -- debugging -- and do not rise to the level of
; something that merit your attention as an Asterisk administrator.  Debug
; messages are also very verbose and can and do fill up logfiles quickly;
; this is another reason not to have debug mode on a production system unless
; you are in the process of debugging a specific issue.
;debug => debug
console => notice,warning,error
;console => notice,warning,error,debug
;messages => notice,warning,error
full => notice,warning,error,debug,verbose
To make the logger.conf effective, we have to type 'logger reload' in asterisk CLI.
Start and get protected
Now, we can turn on everything.
service iptables restart
service fail2ban start
Typing 'iptables -L -v', we should see something like below:
 pkts bytes target     prot opt in     out     source               destination
82337 8101K fail2ban-ASTERISK  all  --  any    any     anywhere             anywhere
 6436  426K fail2ban-SSH  tcp  --  any    any     anywhere             anywhere            tcp dpt:ssh
98669 9494K ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  any    any     anywhere             anywhere
  183 10980 ACCEPT     all  --  lo     any     anywhere             anywhere
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere            state NEW tcp dpt:ssh
 2727  419K REJECT     all  --  any    any     anywhere             anywhere            reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     all  --  any    any     anywhere             anywhere            reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT 103K packets, 9748K bytes)
 pkts bytes target     prot opt in     out     source               destination
Chain fail2ban-ASTERISK (1 references)
 pkts bytes target     prot opt in     out     source               destination
82337 8101K RETURN     all  --  any    any     anywhere             anywhere
Chain fail2ban-SSH (1 references)
 pkts bytes target     prot opt in     out     source               destination
 6436  426K RETURN     all  --  any    any     anywhere             anywhere
It is also a good idea to make iptables and fail2ban start upon reboot.
chkconfig iptables on
chkconfig fail2ban on
# chkconfig --list iptables
iptables        0:off   1:off   2:on    3:on    4:on    5:on    6:off
# chkconfig --list fail2ban
fail2ban        0:off   1:off   2:on    3:on    4:on    5:on    6:off

Thursday, July 22, 2010

What is the China 400 toll-free number?

You should already know the 800 toll-free number as a global business tool.
As you explore the huge market in China, you must also know the China 400 toll-free number because of its coverage and cost impact.
  • It is accessible by any fixed-line and mobile networks in China while the 800 toll-free is not accessible by mobile phones.
  • One single 400 toll-free can cover the whole China
  • You will be paying less to subscribe the 400 toll-free because callers will bear their own connection charge
The China 400 toll-free numbers are available in some VOIP DID providers.

Wednesday, July 21, 2010

Some examples of configuring DID in Asterisk

We have many clients using HK DID (852-5804-) and China DID (86-400-) from a HK-based DID provider.  Recently, we encountered a client using several DID providers from the US and needed us to setup an (Interactive Voice Response System) IVRS in Hong Kong to handle inquiries from his US customers.
This client is using, and  In the first place, we have to setup Asterisk, primarily in the sip.conf, to accept SIP calls from these providers.  Below highlights the parameters required in sip.conf.
; to setup the default context for all incoming calls













In the provider's configuration page, we have to input below information.

q       Manage IP

o       Input our-asterisk-ip to receive DID calls

q       Manage DID

o       Map DIDs to our-asterisk-ip













In the provider's configuration page, we need to create custom mapping because the default one will not send the called number to Asterisk.
















In the provider's configuration page, we need to update ring-to-address to did@ivrs-ip-address


Above all, we also need to check whether there are any firewall rules that block sip calls from these provider.  Note that we use G711 with some providers.  The customer intends to make certain DIDs as fax-in numbers, though the reliability will not be as good as traditional PSTN fax.  In this case, we just could not use a lossy codec like G729.