We successfully uploaded the new control panel design and navigation last night. Below is a screen shot of the new design (please note that some sections only apply to certain types of customers, like Resellers Tools for example):

If you have any questions or if you experience an issue with the new control panel, please contact our Customer Care team and we’ll help in any way we can.

Thanks to Manny for putting this design together and Brian H. (who started full-time this week--YES!) for implementation.

We have a lot more new control panel feature releases coming in the next two weeks...

A couple of weeks ago we blogged about some of the improvements we’re in the process of making to our anti-spam system. Today we released the first round of those improvements.

In this release, our safelisting and blacklisting systems got an overhaul. Previously, when users added addresses to their safelist, mail from that sender could still get blocked. This was because the safelist rules were used in the spam content filter part of our system (SpamAssassin), but not in the Postfix system that attempts to reject spammer SMTP connections. Also, mail from blacklisted senders would get tagged as spam and delivered to the spam folder, rather than getting blocked outright by Postfix.

Now all of this takes place within Postfix. Email from senders that are on the recipient's safelist will never get rejected by Postfix, and email from senders that are on the recipient's blacklist will always get rejected by Postfix.

Thanks to Mike T., our new anti-spam guru, for knocking this out.

Now we begin work on the next phase... letting customers add IP addresses to their safelists and blacklists. Stay tuned!

We are uploading a lot of new control panel features and design enhancements tonight. I will blog more about the improvements over the coming days but right now I just wanted to let everyone know that we have to bring the control panel offline for a little while tonight, starting at 8:00 p.m. EDT. The API will not be affected. We don’t anticipate having to take it offline for very long—we’re thinking an hour at most.

As a follow-up to my blog post from last week, everything I said we were going to do is now either complete or underway. We’ve reconfigured all of the servers that were having problems at the beginning of last week, we’ve put new servers online, and we’ve placed the order for the 50 new servers we’re going to put online in order to build a plethora of excess capacity into our email hosting system.

To all of our customers who have experienced issues with our email hosting system in the last several days—I sincerely apologize.

June has been a bumpy month for us from an infrastructure perspective. I don’t want to make excuses; I just want to give you the facts and let you know what we’re doing to remedy the issues we’ve been facing.

Here is what has gone wrong:

1. We installed the latest version of the Red Hat operating system on our newest round of servers. That operating system installation has not performed up to par in our configuration. This has caused several mail queues, resulting in email delivery delays, for anywhere between 1-8% of our customers during peak hours of the business day. It has also caused several dropped IMAP connections which in turn may also log affected users out of webmail. Not good, I know.

2. The solution would naturally be to move users off of the servers with the bad OS installation, reinstall the older OS, and then move users back onto the better performing servers. But unfortunately we mis-timed the ordering of those new servers and just when we don’t need a delay in hardware delivery times, we’ve experienced pretty significant ones. When it rains, it pours.

Here is what we’re doing to fix the problems AND get ahead of the curve:

1. We’re putting a new round of new servers online tonight and another round later this week. We’ll be moving users off of the servers with the bad OS installations so that we can reinstall the OS and get them back to performing the way they should. We should see improvements beginning tomorrow but we may not be all the way out of the woods quite yet.

2. We’re putting together more efficient ordering processes with our infrastructure providers. We will not mis-time our orders in the future and any hardware delivery delays shouldn’t affect our customers because we’re not going to try to cut it so close anymore.

3. We’re putting together a more efficient server provisioning process. Right now it takes our engineering team hours to get a server online. In a couple of weeks it will take a matter of minutes—tops.

4. After we add the servers we need and get the servers with the bad OS reconfigured, we’re going to add an ADDITIONAL 50% capacity to our system—on top of what we actually need. And we’re not going to stop there. In fact, we’ve committed to adding at least 10 new servers to our server farm every month for the next 24 months. To put things in perspective, we have approximately 100 servers in our farm right now. Within the next month we’ll have 160 online and we’ll add at least 10 per month from there, or more, if our customer base grows faster than it is growing right now.

We’re pouring significant capital and human resources into the infrastructure hosting side of our business. We’re adding servers, investing in the best possible managed services around those servers, and internally, we’re innovating like crazy, doing our best to build the most intelligent, best performing infrastructure we can.

If you’re an affected customer, please do not take these past few days as an example of the quality of service we deliver. We aim to be the best. Sometimes we fail. And for that, I’m sorry.

Better days will be here soon.

Thank you for your patience,

Pat

This past week we started offering 5 Gigabyte mailboxes for $3 per mailbox, per month with a 12-month prepayment. That is just $1 more than what 1 Gigabyte mailboxes cost. If you would like to upgrade to 5 Gigabyte mailboxes, you can do so by contacting Customer Care (it may take a couple of days to fulfill your request as we’ve got lots of businesses requesting upgrades already). We’re working on making automatic upgrades available via the control panel, but that isn’t quite possible yet. Coming soon!

I just wanted to quickly announce that we hit quite a major company milestone here at Webmail this past Wednesday. While it took us several years get the first 10,000 businesses to sign up for our email hosting services, it took just six months to sign the next 10,000. In other words, we now host email for more than 20,000 businesses worldwide!

I won’t go into too much more detail than that. If you’re interested, here is the press release we issued this morning.

Special thanks go out to all of our employees who continue to work so hard as well as our customers that continue to believe in us and appreciate the services we provide.