Announcing a new beta of Deploying Rails, or why e-mail is dead

Yesterday we released the second beta of the Deploying Rails Applications book. It has a new chapter on Windows-based deployment, and includes updates now that Capistrano 2.0 is out.

Our new online system automatically sent e-mails to all existing owners, telling them about the beta.

And all but a few hundred of these e-mails were deleted by our commercial STMP service—the vast majority never reached their intended recipients. Apparently all the commercial e-mail hosting companies are now afraid of being tagged as "bulk-mailers," and will detect folks sending large numbers of emails in a given period.

All of which sucks.

Something our customers have asked for is better notification of updates to PDFs. The new site delivers. When you log in, it tells you if any of your PDFs have been updated. You can subscribe to your personal RSS feed and receive a notification that way. But most people seem to want to receive e-mail notifications. And now, just when we get into a position to be able to do that automatically, we seem to be blocked.

So, a question. We're not a big company. We don't have the resources to be able to manage the intricacies of mass emailing in-house. And we want something automated—we need to be able to send potentially many thousands of slightly-customized emails out when a PDF gets updated. And we need to be able to do it programmatically. Does anyone know of a decent solution?

Thanks.

Comments

[Rik Hemsley]

Is there a reason you don't set up a little Linux box and relay through its MTA?

[Skrud]

You could do a simple hack, and split up your mass e-mail into blocks (of say, 100 e-mails each, or whatever your smtp server allows) and then wait an interval (like 15 minutes) between each block. That's what the phplist mailing list application does, and chances are the smtp provider will never notice. Some of your users might not get their e-mail notification until the next day, though.

[Jim Menard]

How about personalized RSS feeds? My feed could let me know when there is a new version of a PDF I own.

[Jeff Lewis]

Feedburner email subscriptions would add another nice leaky abstraction to the process and might even "solve" the problem.

[Joe Van Dyk]

I use postfix on solaris to send out maybe 20k bulk emails a day.

[Dave Thomas]

Rik:

Yes--if you send 100 similar emails to Yahoo mail users, or GMail users, your box gets blacklisted. That's what we used to do.

Dave

[Dave Thomas]

Jim asks: "How about personalized RSS feeds? My feed could let me know when there is a new version of a PDF I own."

We have them: just create an account on our system and you'll be invited to subscribe. Not many people are using them at the moment.

Dave

[Andy Stewart]

Campaign Monitor lets you send bulk email programatically[1]. They have special arrangements[2] in place with many ISPs and other organisations to allow through emails sent with their service.

[1] http://www.campaignmonitor.com/api
[2] http://www.campaignmonitor.com/features/help-tips-and-free-support.aspx

They have a pretty good reputation.

[Jay Thrash]

Why not hire one of the email spammers to do it for you? After all, they have the most invested in knowing all the "tricks of the trade" to keep their mail from getting marked as spam and to keep their servers off the blacklists . ;)

Joking aside, we're quickly approaching the point where that might actually become a real solution. :( It's ridiculous that email has become so polluted that we can't even communicate with our own customers without getting blacklisted ourselves. We got blacklisted after sending just 50 emails and it took over two weeks to finally get in the clear again.

[Nick Coyne]

Don't look any further than campaignmonitor. Been using it for over a year. It has great features and their responsiveness is awesome.

[HappyCoder]

For a non-server solution I'd recommend using advanced mac mailer. It uses Apple mail framework that does best to deliver messages in a way so that it will not be considered as spam.

[Brad Schneider]

I created an account and tried to add the the RSS feed to MyYahoo and it isn't liking it. I get "We couldn't find the RSS file you asked for http://www.pragprog.com/feed/bookshelf/f/a0/2846-f5eb6137.rss" I'm not sure what exactly Yahoo doesn't like, but I thought I'd let you know.

As far as the email goes, you might want to look at something like ExactTarget, or some other targeted email specialist. They specialize in not getting blacklisted, which can be a pretty tricky game.

[Gregory Benedict]

I used to work for ExactTarget to help customers integrate with their API. Much like Campaign Monitor (I use it for small projects) they have managed relationships with hundreds of ISPs.

Where they differ is in send volume and deliverability rates. They send over 700M opt-in emails per month and expect to do 1B+ in December. A high rate of success with that volume is very hard to do, and they do it better than any one.

Let me know if you have questions on how things work and I can fill you in.

[Tom]

I'm in the process of setting up a web site with emailing for account setup etc. This is a pain - some GMail mail isn't getting through, but most is....

There's a Pragmatic book in this experience, just waiting to be written.

[Jim Menard]

Dave, thanks for the pointer to the RSS feed. I've just subscribed.

[Michael Koziarski]

The major email providers have rate-limits they want observed, and special bulk-hosts that you can deliver to instead of their primary and secondary MX hosts. There's a class of consultants called 'deliverability consultants' who can help you set up your systems and email to honour these 'rules'.

Campaign monitor and other bulk-delivery systems play nicely with these rules which is why their deliveries are so successful.

On the whole email is horribly broken, but any commercial smtp provider should be able to make this stuff work...

[Rik Hemsley]

Silly idea: Tell those who might be interested to set up a Google Alert for a specific string, then whenever you want to announce something, stick it on a web page that Google will index.

I'll get my coat.

[James Wilson]

Campaign Monitor does NOT let you send mail programmatically.

Their API lets you manage subscriptions, and get some modicum of basic reporting numbers, but that's it.

In other words, your application can't trigger any form of emails to be sent via their API.

A human has to log in to the web administration application and do the send by hand.

Can't hurt to ask them to add the feature, we asked for it years ago, more people pestering them may make it happen.

[steve]

Dave, here's the solution.

The personalised RSS feeds are the answer to your problem. It's the account creation that sucks.

Fix the account creation. Have your personalised RSS feeds on something like:

rss://pragdave.blogs.pragprog.com/AUTOGENERATEDHASH/

Been a while since we last chatted on IRC!

[steve]

To be clearer, the AUTOGENERATEDHASH is obtained by clicking on a "get an RSS feed" page, with every visitor to the page getting another hash.

Putting the RSS feed into their reader means they'll be using the same hash, not constantly generating new ones.

[Dave Thomas]

Steve:

I'm probably missing something. I'm talking about announcements sent from our online store, not this blog.

[Keith McHaggisBreath]

Hi Dave,

Why not setup a mailing list and let your customers subscribe to receive updates ?

You could try yahoo/google groups, host your own mailman server or maybe find a hosted mailman provider.

Keith

[Devdas Bhagat]

I work in the email space, though I don't directly deal with deliverability.

If you regularly send out email to the list, then talk to your provider about it. They can probably help.

Alternatively, talk to a list provider. One I can recommend is Constant Contact (http://www.constantcontact.com/index.jsp).

If you need deliverability advice, I can suggest WTTW (http://wordtothewise.com/).

Note that other than knowing good people at both these places, I have no commercial interest in either recommendation.

[Ken Hudson]

Hi Dave,

Your question is a good one and one I've been trying to address, too. Regarding your statement "We're not a big company", I would really like to see a mailing list or ?? where people who aren't employed by big companies could get together and share solutions to these types of problems.

[Khushil Dep]

Hi Dave,

Why not just set your own mail servers up and send from them? It's a trivial task to setup these servers and only slightly less trivial to write an app to deliver these e-mails spaced out within a time-frame to avoid 'blacklisting' by hosts which don't like mass mails. If you want to be really clever just get a nice virtualised solution whith multiple mail servers with IP's from different IP ranges. I'm surprised you use a 'commercial' smtp solution when owning your own is not a great oulay - technically nor financially.

Khushil