[SOLVED] Asterisk disk space problem

Recently, we see an issue with an Asterisk installation where the disk
usage keeps growing, but apparently not due to increasing call volume.

First of all, repeating 'df -h' shows that '/' is approaching its
capacity upper limit. More importantly, the free space keeps decreasing
for unknown reason.

# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 43G 39G 2.5G 94% /

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 43G 39G 2.0G 96% /

However, the 'du -sh /' provides conflicting results.

# du -sh /
13G /

Such discrepancy could be due to hidden files behind a mount point under
'/'. The 'df' command uses the file system superblock to report the
usage statistics. On the other hand, 'du' command follows the directory
tree to accumulate the usage figures and files hidden behind a mount
point are not taken into consideration.

This system is coincidentally mounting a storage device on
/var/spool/asterisk/monitor.

We can use the 'bind' option in mount to reveal the real situation.

# mount -o bind / /another-view-root
# cd /another-view-root
# du -sh /another-view-root/var/spool/asterisk/monitor

Now, we get consistent result with the 'df' command. It also shows the
wave files which became hidden when another device was mounted over them.

Furthermore, we spot a very large wave file modified recently among
these hidden files. The wave filename indicates that it was created at
least a month ago, but is still growing.

Using 'CLI> core show channels', we notice there is a active call which
is bridged to VM. This problematic call is not disconnected for unknown
reason and because all extensions under this site requires monitoring,
its corresponding wave file keeps growing.

Another device later mounts over these files and unintentionally hides
the problem until the decreasing free space alerts the administrator.

By stop this problematic call using 'CLI> channel request hangup
<channel-id>', the wave file is closed finally and subsequently deleted.