I'll dive into the C# stuff as I actually get a clue what I'm talking about, in the mean time I'm going to start going over a number of the tools I've used over the years that I really like and want to share some information on them.
To start this off I'd like to talk to you about
postfix.
Postfix is incredibly powerful mail server
designed by
Wietse Venema. You can read up on it's history on their site, but it is a much nicer tool to use that
sendmail in every way I've ever cared about.
I ran my
ISP using
sendmail prior to 2002, and it worked well enough. Upgrades were a pain in the neck, and security exploits were disturbingly common. Making configuration changes was horrifically complicated.
Sendmail is extremely powerful, but it made things far too complicated. I like to make things I can hand to someone else so that I don't have to maintain it for all time -
sendmail configuration complexities made this very hard to do.
When we first wrote
mailarmory we put it together using
milters, which are a very cool idea. The very first iteration we tried out used
mimedefang from Roaring Penguin Software. It was an excellent tool, but it couldn't alter the weaknesses of
sendmail. If anything the situation was made much worse with the addition of the complexities of
milters.
For the uninitiated,
milters allow you enormous amounts of control over everything that happens in an SMTP discussion. This could let you do something like rejecting all mail at the door that has the word "monkey" in it, and send a response like "550 Server does not permit the discussion of monkeys"
I really don't want to trash talk
sendmail - it's a great application, and many people have had great luck using it. But
postfix was designed by someone who took a really good look at the weaknesses of
sendmail, and built it better. One of the first things I'd like to draw attention to is the compartmentalized nature of
postfix.
Sendmail is basically one executable that does everything remotely related to mail. By contrast
postfix has tons of smaller daemons that each just do their little part. This compartmentalization has allowed each one of these little daemons to be optimized and very well secured. It's hard to make huge applications immune to exploitation, it's relatively easy to secure simple programs though.
One of the big differences between
sendmail and
postfix is how the message queues themselves are handled.
Postfix now allows control over the "before queue" filtering via the recently added
Postfix Milters or by using the
Postfix Policy Daemon's, but back in 2001 neither of these existed, and the only option for
postfix was
after queue filtering.
This distinction is really important - Once you queue a message that means you have accepted responsibility for that message for better or worse. So which is better - before content filtering, or after content filtering? The answer isn't as clear as you might think, and it is because filtering is really expensive processing wise, and anything that makes your smtp conversation take longer kills your performance. So ideally you would want all content filtering to happen before the queue, and you would want it to take no time at all so that your server can handle an unlimited amount of email. In the
sendmail model we originally employed it worked more or less like this, with the downside that there was a rather serious performance hit by adding the content filtering in. There was also a really nasty side effect of this approach - you are handing
spammers the ability to DOS your server. By putting so much processing near the edge you are giving them a tool to use to grind your servers to a halt. This hit us pretty hard in the early days as we were learning the ropes. (FYI - morally bankrupt != stupid, many
spammers are quite intelligent).
Let me try and show an example of why this can hurt so badly - if a single SMTP process can accept one message per second, then to accept 10 messages per second you would need 10 SMTP processes. If you are doing before-queue filtering and it now would take 5 seconds to accept that message, you would now need 50 SMTP process. Now figure on the ugly fact that a server under load does everything slower, so that 5 seconds becomes 10 seconds, which means it would need 100 SMTP processes to keep up, which slows it further and .... you are having a really bad time of it. The thing to keep in mind is that basically when things start going bad, they go really bad really fast, and your server can handle less and less messages per second the more that tries to come in.
After queue filtering doesn't entirely solve this problem, but it gives some great tools that help quite a bit - most particularly it let's you control the rate that you send messages to your filter engine, while still allowing you to accept messages from the outside world. This is still not an ideal situation as the unprocessed mail backs up in your queues, so you get some mail delays, but they are much more minor than what you get with before content filtering.
So if both filter methods would let you handle say 10 messages/second, they both behave very differently when the number of incoming messages goes higher than 10.
Before queue
15 msgs/sec incoming -> only can handle 5 msgs/sec
20 msgs/sec incoming -> only can handle 1 msg /sec
100 msgs/sec incoming -> server probably crashes, at the least it will fail incoming SMTP requests
After queue:
15 msgs/sec incoming -> handles 10msgs/sec
20 msgs/sec incoming -> handles 10msgs/sec
100 msgs/sec incoming -> handles 10msgs/sec
The numbers above are made up, but the impact described is very real, the lessons I'm describing there were learned the hard way.
But - this isn't to say before queue filtering is not a really cool and powerful thing, remember it lets us stop a message before we accept it into our queue, if we block it at the door we don't have to bounce it if we later decide we don't want it. The key lesson you should walk away with is - Keep all of your before queue filtering extremely light weight. Do your heavy lifting in after queue filters.
Which brings me back to the point of this blog -> postfix puts the controls for all of this in your hands in a much easier to understand way than sendmail, and it also gives you a whole lot more options for doing whatever you want to do. So if you do things one way at first, it is often fairly straightforward to change your mind and do them a different way later.
There is a philosophy I picked up from perl that I encourage you to think about "The simple things should be easy, and the hard things should be possible". I believe postfix nails this, and it's one of the reasons I strongly endorse it.
I'd like to close on a note for you to consider: when our environment ran sendmail, I believe we had to do emergency sendmail upgrades due to late breaking remote root exploits about a dozen or so times. Since we moved to postfix, we have not made a single security related upgrade ever. Every upgrade we've done (and they are pretty easy) has been done because postfix added some cool new feature we wanted to take advantage of.
If you have any questions about anything I am writing about speak up - I know I get rambly on explaining all of this, and if you'd like to know more (or some actual details) about how to do things with postfix I'd be happy to share them as well.
Take care,
Neil
