Welcome, Guest. [ Log In ]
Question   v4.11 November 2002
Search KBase


Top 5 in this Area:
1. v4.1 January 2002
2. v4.10 October 2002
3. v4.3 March 2002
4. v4.4 April 2002
5. v4.2 February 2002

 
v4.11 November 2002
0.  Introduction.
1.  Outages, aiyah.
2.  Webmail2 beta test!
3.  Woody migration continues..
4.  DHSOTM


########################################################################
0.  Introduction.

Welcome back once again to the ever-later edition of the Happy DreamHost
newsletter! I hope you'll excuse the tardiness.. probably in all the
holiday festivities you didn't even notice. Which is fine. I understand
if taking time off to visit friends and relatives is more important to
you than a stinky little newsletter not good enough to get printed up on
real paper. I realize only 9 of you even read past the first paragraph,
and even fewer of you would even notice if you never got a DreamHost
newsletter again. And that's fine with us, we don't mind! Not even a
little bit, nope. Sniff.

Well, I hope a few more of you than usual find it in your heart to take
a minute out in between bites of leftover turkey/tofurkey (ew,
"tofurkey" looks so dirty when you write it out! I think I'm going to
use "bean-curd bird" instead from now on.) to read what looks
to be a super-meager newsletter this month. Oo, I have an idea! To
entice you to read through this whole newsletter I'm going to hide
somewhere in each section my own special recipes, Martha Stewart-style,
on what to do with your turkey left-overs! 

DreamHost: "Don't like the webhosting? Try the turkey!"


########################################################################
1.  Outages, aiyah.

From a stability standpoint, November 2002 was probably the worst month
in DreamHost's history. For almost a full week close to 5% of our
customers experienced numerous website and email problems, and then for
another two days afterwards most of our customers had strange CGI issues
coupled with more website troubles. I thought it would be a good idea to
give a "post-mortem" on that week here in the newsletter so everybody
has a clear understanding of what went wrong and what we've done to
prevent these problems from occuring again.

The main problems first began a few days after we deployed our new disk
filer. It turns out, due to a mistake on our part, a few of the disks
in the filer had not been completely initialized. Things were working
fine though, and we began to move users off of the overloaded filer. The
trouble arose when we needed to expand the size of the volume on the new
filer. It failed to correctly detect the disks that hadn't been
completely blanked, which led to raid and filesystem corruption. Even
with the help of engineers at the manufacturer, it took a long time to
ascertain what the problem was and how we could recover from it. Then,
once we knew what to do it _still_ took a long time for the recovery to
(Scrounge up some shredded cheese and have tacokeys!) complete, just due
to the immense amount of data that needed to be copied over the network.
For a while a number of sites were up with a "read-only" filesystem
which allowed for the displaying of the website but wouldn't allow
updating and also stopped email from functioning.

To make sure this sort of thing never happens again, in the future all
new disks will be explicitly initialized (and re-initialized) before
volumes are expanded. More importantly, we're also decreasing the
typical size of our volumes so that expansions will rarely be necessary.
Yeah, I know that seems counter-intuitive, but trust us! We've really
learned a lot from this unpleasant experience, and suffice it to say
our procedures for dealing with filers are about fifty times better than
before. This sort of problem should never happen again.

Just as we had finally fixed the filer situation, a new problem
cropped up. We had just put a new program on about 10% of our hosting  servers that were experiencing heavy load to restart the apache
instances when the load started to get too high. We've used this program
before as a temporary bandaid for overloaded machines as we upgrade them
or move users off and hadn't caused any problems. Unfortunately on these
machines, it did. The reason was these machines had some of their
"ulimits" set far too low. When the ulimits are too low, CGI gets
effectively "choked" shortly after an apache is restarted. The thing
was, these low ulimits only appeared when an apache was restarted via a
cron job (like the new program was), and not when restarted normally.
Since these apaches had never been restarted via cron before, the bad
ulimits had never gone into effect.. until then!

Unfortunately, the problem was misdiagnosed and the attempted solution
only succeeded in spreading the problem to the REST of our shared
hosting servers. It's a bit hairy what we did, but suffice it to say
there were big problems until about two days into this whole fiasco we
finally realized in a flash where those bad ulimits were coming from and
were able to fix the situation everywhere, for good. It was a nightmare
at DreamHost that week, and we'd like to say we really appreciate all
our customers who were more than understanding and very patient with us
during our times of crisis!


########################################################################
2.  Webmail2 beta test!

Like we promised last month, a new webmail system is here!  We really
like squirrelmail and it's great, but even after lots and lots of TLC
and load balancing, that it just can't scale, so we found something that 
did. But, it's still in beta, so a few words of warning before I give you
the url...

It's not as feature-filled as squirrelmail! It may not even be faster
for you.. but, what it is (or should be, we think) is stable and
scalable. That is, it's not going to bring down the server running it, so
it should work (Two words: turkey jerky.) more of the time, and it should
also work even if our IMAP mail servers themselves are overloaded and
acting a bit slow.

Right now the only way to access the new webmail system is at:

http://webmail2.dreamhost.com/

We haven't set it up at http://webmail2.yourdomain.com/ for everybody
yet, but we plan to pretty soon. There'll be an announcement when that's
out.. so watch for it!

Oh and by the way, the new webmail system is actually one created by a
Happy DreamHost Customer coincidentally! Check out his site at
http://www.endymion.com/ and thank him for helping improve your webmail!

Remember, we're keeping squirrelmail up for the forseeable future still
at http://webmail.yourdomain.com/ , and if it works okay for you, feel
free to keep using it! But now at least you've got a second alternative
for web-mail if squirrelmail is acting a but, uh, squirrely.


########################################################################
3.  Woody migration continues..

The other thing we'd be working on for November I mentioned last
newsletter was the migration of our shared hosting servers from Debian
Potato to Woody. Well, we haven't (And for the kids: peanut-butter and
turkey sandwiches!) completed that yet, but we're slowly making
progress. In light of all the problems we had a few weeks ago, we're
being as conservative as possible for this upgrade! So anyway, we're
about half way through now, and I thought you should get an update! 


########################################################################
4.  DHSOTM

(See https://panel.dreamhost.com/?tree=home.dhsotm for more info!)

This month's big winner in the DHSOTM contest is Jonathan Postal's
photgraphy site at:

http://www.jonathanpostal.com/

I honestly  think I've never seen a better looking site on the Internet.
Period. And the pictures ain't bad neither. Go check it out right now
and you'll see it's no surprise Jonathan won this month with the highest
average site rating ever!

(Cool off after reading the newsletter with a nice cold "turksicle"!)

########################################################################

Josh!

P.S. Do you have an entertainment related site?  We want to hear about
it! Are there any areas of your site/business that DreamHost has helped
you improve? What factors did you examine before choosing DreamHost to
host your site? What success have you achieved?  Email us at
stories@dreamhost.com and your site/business might be featured on our
web site or you could be interviewed by a reporter for a
magazine/newspaper.

Last updated: Dec 02, 2002.

User Post (2002-12-10 20:03:26 by calebcro)
well written newsletter- thanks for the inside scoop. salute for admitting the mistakes- hey hairballs happen!

nice refresher from the corporate plastic of my last host server. they would never have been upfront. you're mistakes are working FOR you with this
customer.

i like it here!

caleb crosby