After my company decided to move its mail services to GMail, I was faced with a more complex e-mail setup. I now have four different email accounts that I need to log into and read mail from. There are several e-mail aliases on two of the accounts. So, with two different GMail accounts (one with my first name dot my last name at GMail.com and the other with my full e-mail address from my company as login [yes, the full login address is geir.isene@freecode.no@imap.gmail.com]).
I decided to plunge into it and create a full e-mail architecture that would serve me the best. This is a technical article aimed at helping those with similar challenges that I encountered in the last few days. With this I hope others will not have to spend hours upon hours wading through troubleshootings in the quest for a decent e-mail setup.
Design goals: To have all e-mail on my laptop and handle it locally for speed and convenience while at the same time keep a copy of important mail on the servers as backup and for handling when I am not at my laptop. Additionally I want the fetched e-mail stored in various folders in my local e-mail system. I want all this easily configurable so that, for example, certain mail is fetched and marked as read, but not deleted and stored in a specific folder in local mail.
With e-mail handled locally, I get faster response in e-mail handling, faster search, faster scripting, and more time for my children.
Here’s my basic setup:
- Ubuntu Linux 12.04
- The excellent i3 window manager with Conky
- Mutt (the fastest and best) e-mail client with VIM as editor (emacs may be best but VIM is the bestest)
- Ruby 1.9 for scripting (neat programming language)
First I go all the e-mail from my company mail server (which I used IMAP to read and which is now moved to GMail) with OfflineIMAP. This resulted in a some hundred thousand e-mail transfered to my local folder (~/Maildir) with all my folder structure preserved.
Here’s the first challenge; Since my company uses Courier as the IMAP server, all IMAP folders are stored in a flat directory structure with a “dot” as a directory delimiter. This means that the folder “INBOX/Geir/Personal/” is stored not as a real tree structure in the file system, but rather as the directory named “INBOX.Geir.Personal/”. With more than hundred directories, this becomes a very un-navigatable list if it wasn’t for Courier’s ability to show this as a usual hierarchy.
First I tried to point Mutt directly to my Maildir directory to handle e-mails directly from that folder. But then I was faced with the list of flat directories that makes it unmanageable. It also makes it impossible for Mutt to “subscribe” to folders as in a normal IMAP server setup.
So I decided to install an IMAP server to handle the directory structure so that the hierarchy would show up in Mutt. This would also add the benefits of easy folder creation and easier scripting with Ruby’s Net/IMAP class.
I first tried the Dovecot IMAP server, but couldntt get it to show my folders in a hierarchy structure, so I went on to install the Courier IMAP server on my laptop.
But alas, Courier would not show my directories at all. Lots of debugging and I figured out that the root directory (~/Maildir) was itself treated as the “INBOX” and all subdirectories must start with a period. Therefore I had to rename all the folders produced by OfflineIMAP – for example, “INBOX.Geir.Personal/” had to be renamed to “.Geir.Personal/” and voilá – all folders magically appeared! (Use “rename ‘s/INBOX//’ *”).
Then it was the matter of getting mail fetched from four different accounts at three different servers. I tired Fetchmail and Getmail and looked at Retchmail. But none of these provided an easy path to my design goals.
So I decided to go ahead and make my own solution. With my old imap_tools as the foundation, I created a script that not only satisfied my design goals but also provides great flexibility and an easy way to enhance it in the future (being a Ruby script makes it easy for me). The resulting script is fairly easy to use even if you don’t know Ruby. Just substitute what you need in the script and save it in your “bin” folder (set permissions with “chmod 755 mail_fetch.rb”).
Due to WordPress’ limitation in uploading files, I give you the whole script here (disregard the screwy tabs and the proportional font – again WP-limitations to html):
——————————
#!/usr/bin/ruby
# Copyright 2012, Geir Isene. Released under the GPL v. 3
# Version 0.2 (2012-04-24)
##################
# Initialize #
##################
require "ftools"
require 'net/imap'
# In .mail.conf, set the appropriate variables like this:
#
# I_server1 = ""
# I_user1 = ""
# I_pass1 = ""
#
# Iserver0 would be "localhost" for mail delivery to local IMAP server
# Use as many servers you need with I_serverX, I_userX and I_passX
load '~/.mail.conf'
$count = 0
###########################################
# Define main Fetch & Filter function #
###########################################
def matching (match, match_in, to_box, del)
res = []
message = ""
to_box = "INBOX." + to_box
if match_in =~ /s/ then res = res + $imap_from.search(["UNSEEN", "SUBJECT", match]) end
if match_in =~ /b/ then res = res + $imap_from.search(["UNSEEN", "BODY", match]) end
if match_in =~ /f/ then res = res + $imap_from.search(["UNSEEN", "FROM", match]) end
if match_in =~ /t/ then res = res + $imap_from.search(["UNSEEN", "TO", match]) end
if match_in =~ /c/ then res = res + $imap_from.search(["UNSEEN", "CC", match]) end
if match_in =~ /h/ then res = res + $imap_from.search(["UNSEEN", "HEADER", match]) end
if match_in =~ /a/ then res = res + $imap_from.search(["UNSEEN", "TEXT", match]) end
res.uniq!
res.each do |message_id|
message = $imap_from.fetch(message_id,'RFC822')[0].attr['RFC822']
$imap_to.append(to_box, message)
if del == 1
$imap_from.store(message_id, "+FLAGS", [:Deleted])
else
$imap_from.store(message_id, "+FLAGS", [:Seen])
end
$count = $count + 1
end
end
#######################################
# Log into the target IMAP server #
#######################################
$imap_to = Net::IMAP.new(I_server0, port="143")
$imap_to.login(I_user0, I_pass0)
$imap_to.select("INBOX")
###############################################################
# Log into each "from"-server. Start Fetching & Filtering #
###############################################################
#
# The syntax for matching is:
# matching("string-to match", "match-against", "send-to-mailbox", "delete mail?)
# Set "delete mail?" to 1 if you want mail deleted from the source server.
#
# The following options are available to match against:
# "s" for SUBJECT, "b" for BODY, "t" for TO, "f" for FROM, "c" for CC
# to match any part of the mail (header or body): "a" for ALL
# From I_server1 (private GMail)
$imap_from = Net::IMAP.new(I_server1, port="993", usessl="true")
$imap_from.login(I_user1, I_pass1)
$imap_from.select("INBOX")
matching( "", "s", "Geir", 0 ) # Catch rest
# Expunge mails that are set to be deleted and then disconnect
$imap_from.expunge
$imap_from.disconnect
# From I_server2 (FreeCode GMail)
$imap_from = Net::IMAP.new(I_server2, port="993", usessl="true")
$imap_from.login(I_user2, I_pass2)
$imap_from.select("INBOX")
matching( "efn-listen", "tc", "Lists.EFN", 1 )
matching( "efn-agenda", "tc", "Lists.EFN-agenda", 1 )
matching( "styre@mailman.efn.no", "tc", "Lists.EFN-styret", 1 )
matching( "", "s", "FreeCode", 0 ) # Catch rest
# Expunge mails that are set to be deleted and then disconnect
$imap_from.expunge
$imap_from.disconnect
# From I_server3 (FreeCode OLD)
$imap_from = Net::IMAP.new(I_server3, port="993", usessl="true")
$imap_from.login(I_user3, I_pass3)
$imap_from.select("INBOX")
matching( "ivy-subscribers", "tc", "Lists.IVY", 1 )
matching( "FreezoneOrg@yahoogroups.com","tc", "Lists.FZa", 1 )
matching( "ifachat@yahoogroups.com", "tc", "Lists.FZa", 1 )
matching( "koha", "tc", "Lists.Koha", 1 )
matching( "nuug.no", "tc", "Lists.NUUG", 1 )
matching( "linuxiskolen@skolelinux.no", "tc", "Lists.SLX", 1 )
matching( "spirituellkultur", "tc", "Lists.AltMus", 1 )
matching( "hhc@lists.brouhaha.com", "tc", "Lists.HHC", 1 )
matching( "", "s", "Geir", 0 ) # Catch rest
# Expunge mails that are set to be deleted and then disconnect
$imap_from.expunge
$imap_from.disconnect
# From I_server4 (FreeCode Int)
$imap_from = Net::IMAP.new(I_server4, port="993", usessl="true")
$imap_from.login(I_user4, I_pass4)
$imap_from.select("INBOX")
matching( "", "s", "FreeCode", 0 ) # Catch rest
# Expunge mails that are set to be deleted and then disconnect
$imap_from.expunge
$imap_from.disconnect
##############################################################
# Check for new mails in folders and write result to file #
##############################################################
mailboxes = [
[ "FC : ", "FreeCode" ],
[ "Geir: ", "Geir" ]
]
open('/home/noosisegei/.mail', 'w') do |f|
mailboxes.each do |a|
f.write( a[0] + $imap_to.status("INBOX." + a[1], "UNSEEN")["UNSEEN"].to_s + "\n" )
end
end
# Copy file to another file to ensure no blinking in Conky
# Read the file from Conky to display new email in each folder
File.copy('/home/noosisegei/.mail','/home/noosisegei/.mail2')
#######################################################################
# Disconnect from target server & Display number of mails filtered #
#######################################################################
$imap_to.disconnect
puts "#{$count} mails filtered"
——————————
To make the script run every minute, add it as a cron job. Use “crontab -e” to add:
* * * * * /home//bin/mail_fetch.rb >/dev/null 2>&1
Just substitute “” with your own username. It outputs the number of new mails in certain folders by writing it to a file named “~/.mail2 so that you can use this to display new mails in Conky (see my conkyrc for how this is done).
I hope this will shave a few hours off someone’s schedule.
Update: Dovecot is not compatible with the above script as it fails to deliver the $imap.status message (“UNSEEN”) from the mail repository. So, Courier is the IMAP server to use with this script.
Geir Isene - uncut 
I was shocked when I discover an OT with mutt and vim abilities! I agree both are the best of the best 😉
As for the mail setup we migrate to Google Apps too but I used Gmail builtin IMAP/POP3 fetcher so I transfer all other account’s email to my main Gmail account with proper labeling and I keep the ability to read all that from mutt, works for me if I’m on my smartphone, browser or more commonly my Linux terminal. 🙂
🙂
I wanted the added benefit of local mail handling – but the GMail fetcher is a half-way-there solution.
I should add that almost everything I write is in VIM – and I am at least twice as fast handling e-mail with Mutt than with any other e-mail client I’ve tried (and I do try a lot of stuff – from MUAs, WMs, editors, programming languages, shells, etc.
This ranks right up there with the posts on the HP Calculator.
It isn’t just the geekiness expressed by the words and the subject matter, it’s the fervency, the utter “edge of orgasm-ness” with which the words are written.
Takes your breath away.
That was easy.