Welcome, Guest. [ Log In ]
Question   v1.10 October 1999
Search KBase

Top 5 in this Area:
1. v1.11 November 1999
2. v1.10 October 1999
3. v1.8 August 1999
4. v1.4 April 1999
5. v1.9 September 1999

v1.10 October 1999
Table of Contents
0. Introduction.
1. The Crack.
2. New Security Measures.
3. Building Better Backups.
4. New Password Policy.
5. Still win an iBook!
6. NetSol Pay in advance. 
7. DHSOTM October '99
8. Random Quotes(s) From newdream.net
9. New Features Delayed..
10. Dreamhost 2.0.. November 1st!

0. Introduction.

Welcome back to another rip-roaring episode of the modern classic, "Dreamhost
Newsletter"! This time around, it may not be as fun as it usually is (and 
if you're a new customer and don't know how fun it usually is, I'd recommend
visiting https://secure.newdream.net/admin/newsletters/ for the full archive!)
because we have some serious issues to discuss regarding the recent security
compromises we experienced a little over a week ago (we've had to delay the
newsletter as we dealt with related problems).

First off, we'd like to thank everyone for their understanding and patience
throughout this whole fiasco; all the encouraging correspondence has made the
stresses bearable. We've learned a lot from this and we're going to be a much 
better web host because of it! If you don't know what we're talking about.. 
please read on!

1. The Crack.

Friday, September 24th, at around 7:30pm PDT, we noticed one of our server 
machines acting strangely.  It wasn't letting us log in, and the open shell I had
wasn't letting me run basic UNIX commands like 'ls' or 'cd'.  Confused, and more
than a little worried, I switched to another shell window logged into another
server with a process monitor running.  That window confirmed my suspicions.
From there, I could see 8 or so 'rm' processes running as root.  We had been cracked
and the intruders were deleting all of our files!  

Acting quickly, I called our colocation facility and had them immediately shut
down all of the affected machines.  That action helped minimize the overall 
damage.  We wouldn't actually know how extensive that damage was for
several hours, though.  The only way to be sure a cracker is out of a system
is to do a full reinstall of the operating system from a source known to be 'clean'.
Reinstalling Windows (or your OS of choice) on your home PC is a task in itself.  
Imagine doing it on a linux server housing as many as 800-1000 active websites!

From parts we had at our office, we were able to build 4 entire new server
machines.  With those, we began to set up 4 brand-new hosting systems.  In 
this sort of situation, our primary directive is to get the websites we host back up 
as quickly as possible.  Remarkably, we were able to get all 4 machines back up
and running within about 9 hours.  At that time, we still didn't know how the
crackers had compromised our security, so we had to take extreme precautions
and disable all logins until we could survey the damage and make sure we could
prevent the same thing from happening again.

As soon as the crisis began, we called in as much of our staff as we could to help
out wherever possible.  We devoted as many as 4 people just to answer the flood
of support messages.  Most of all, we wanted to keep all of you informed about 
what was happening.  We are dedicated to providing good support, and this
time was no different.  I must send out a hearty "thank you" to everyone on our
team that helped out.  Everyone's dedication and hard work turned a crisis into
merely a "terrible situation" very quickly.  We really couldn't have done it without
every person we have working with us.

As it turned out, the damage was pretty severe on the 4 'infected' machines, and
minimal on the rest.  The arduous task of restoring data from the backups began.
At the time of the crack, our backup system was two-tiered.  Each server machine
has a locally mounted backup drive for quick archiving of user data.  We were able
to recover most of the lost data from those.  Unfortunately, the crackers had started
to delete those drives as well, so we couldn't restore everything from there.  The 
rest had to come from the second tier: the tape backups.  We currently do full daily
tape backups of every machine, with a weekly rotation.  That means we have up
to one week's worth of data at any point in time.  From the tapes, we were able to
recover most of the rest of the data on every machine except one.  One of the
machines had a bad tape drive.  It had been bad for about a week, and we hadn't
noticed.  There's a lot of details to keep track of in a system the size of ours, and
occasionally things go unnoticed for awhile.  I don't like it when that happens,
but it does.  In this case, the results were disastrous.

The secondary damage resulted from files being deleted over
nfs on mounted remote filesystems.  We have to share files between machines
occasionally when we move somebody from one to another.  We have now 
reconsidered that method, however.  In the future, we'll be mounting remote
filesystems read only to prevent situations like this.  

Most of us worked almost continuously for the next 2 or 3 days after the attack.
Our time was spent fixing the thousands of little things that broke when we
quickly reinstalled 4 server systems,  providing much needed technical support,
and recovering lost data.  As soon as we realized some of the data may be
unrecoverable from our backups, we located some data recovery specialists, got 
some price quotes, and put one of them to work.  We've been able to recover
more data that way, but we're still working on it.  That partially explains the
tardiness of this newsletter.

2. New Security Measures.

We never should have let this crack happen.  That said, we plan to never let
something of this nature happen again.  On top of that, we've already begun
to build out our backup system.  In the next couple of weeks, the work we've
been putting into our system will be completed.  The resulting system will
be much more robust.  We had been planning to put forth most of these system 
changes already, but this situation has forced us to change around the timetables
a bit.

First off, we've done and will continue to do a lot to improve security.  We have 
begun implementing a system to keep track of what versions of software we have 
running on all of our systems to make it easier to keep up with and patch security 
holes as soon as we find out about them.  The system will keep a database of
all installed software and regularly poll linux software websites for new versions
of things.  In addition, security advisory reports will be automatically scanned
(as well as read by a human) for problems.  We have already installed updated
versions of all of the major software components on our systems.  Our own 
software will be packaged for easier distribution across our systems.  That will
make our response time to any security holes in our own software as quick
as possible, as well.

In addition to keeping up with software updates, we also run a sort of software
security system known as tripwire.  The tripwire system watches all files on the
system for signs of modification.  A cracker that successfully enters a system
must also evade detection by modifying some key files.  One problem with 
tripwire in an environment like ours is the difficulty keeping up with all of the
file changes across many systems.  We've already improved our tripwire system 
so it automatically purges old data on a regular basis .  That'll keep the database 
of changed files smaller so it's easier to monitor.  We have also designed and are 
in the process of implementing an automated tripwire database scanner.  It will
compare the tripwire information with the known information about our installed
software to try to reconcile and hide redundant information.  That will make our
jobs much easier and give us more of an edge on discovering break-in attempts.

One of the most difficult tasks we face as security administrators is how easy it
is for a would-be cracker to gain access to our system.  All they have to do is 
go to our convenient website and signup!  We have decided to increase security
on our accounts including shell access by requiring more information from any 
users with that access.  Without shell access, a cracked account does not offer 
much to the cracker.  We will continue to set up ftp access immediately, but we 
will personally verify and approve the account information very carefully before
setting up shell access.  Unfortunately, that will increase the time for setting up
new accounts, but I'm sure you'll agree this is a necessary measure.

We also have to watch out for especially sneaky crackers!  We won't be able to 
verify all user information 100%, so there is always a chance that one might get in.
At that point, our process monitoring system will come into play.  We have already
started using a system to watch all user activity and record it into a database.  From
there, we can program the system to act as it sees fit.  We plan to extend that system 
to look out for suspicious processes and warn us via pager 24 hours a day.  In extreme
situations, the system may even be able to shut itself down, but we will have to be
very careful before letting something like that go live!  This sort of system will
not be complete protection either, but it will serve as one more added layer.  We 
have also been surveying existing security software, and have found some
interesting packages.  More on that later!

3. Building Better Backups.

The second part of the bigger, better, stronger Dreamhost is an improved backup
system.  I mentioned the two tiers to our existing system earlier; local disk 
archives and daily tape backups.  In this situation, our local disk backups were
not reliable enough because they could be deleted just as easilly as the original
data.  From now on, we will not be leaving the backup drives mounted in between 
backups.  If the disks had not been mounted, I believe the crackers would have 
been unable to detect and erase them.  We will also improve our tape backup
system by automating the monitoring more.  If we could have detected the failed
tape drive earlier, the overall damage would have been significantly lessened.

In addition to what we already have, we plan to implement a couple of measures 
for extra redundancy.  We have already ordered a whole extra set of tapes
to use for offsite backups.  This situation has made us painfully aware that anything
can happen and we cannot rely on any single element in the system.  We must
provide fault tolerance wherever possible.  Offsite backups will protect us from
the slight possibility that our colocation provider might be robbed, blow up, or fall
off the face of the planet.  On top of that, it's just a good idea.

The second new component of the backup system we're adding is a second level
of disk backups.  We're adding what we're calling a 'data harvester'.  It's basically
a very large, and very redundant file server.  The machine itself will be treated as 
extremely sensitive and secure.  We'll be able to keep several weeks worth of all
user data on the harvester, which will also be fully RAID 5 to safeguard against 
failed drives.  We'll protect the machine itself with our lives (well, almost!).  As an 
added feature, we are also working on a web-based front-end to this harvester.  
That front-end will allow you to personally look through and recover your own 
files without contacting us.  I'm excited about that because I don't know of any 
other web host that offers it at this time.

4. New Password Policy.

One thing we required after the break was that everyone change their current 
password. You were unable to log in until you did, and you could change it 
through our handy-dandy interface at the web panel. Not only did you need to 
change your password, you needed to make it a good one. We implemented a very 
rigorous password-checking algorithm which makes sure your password is longer 
than 5 characters, is not all numbers, and includes no English word (forward OR
backwards) or part of your personal information or username in it!

Some people have complained that our dictionary is a bit broad when it comes
to English words.. passwords have been rejected with errors like "Password
can be derived from the english word 'ngo'" and the like. Well, better safe
than sorry, right? Anyway, here is one good way to come up with a good password
that you'll still be able to remember:

Think of a short sentence, and take the first letter from each word. Make it
a sentence with some sort of number in it too, like maybe:
"Dreamhost is #1 with me!"
That sentence would help you remember the password "Di#1wm!" which, believe it 
or not, would pass our new stringent requirements (but hey, don't use that 
password now that it's been distributed in our newsletter, okay?).

5. Still win an iBook!

That's right, the contest is still on! It runs until October 31st, (or Halloween, 
whichever comes sooner) and the lucky winner will be the first on his/her/its
block to own a shiny, new, iMac-in-a-can, DreamHost-blue, laptop iBook computer! 
The rules are still up at http://www.dreamhost.com/contest/rules.html and we'll 
just summarize them here for you once again..

You get ten entries in the drawing just for receiving this newsletter (that is,
by already being a Happy DreamHost Customer), and for each person you refer to
us who signs up before October 31st you get another 100 entries! You can also get
50 entries for every new service you add to your account before that date, and
one entry (no purchase necessary) by mailing us an index card with your info on it.

Yay fun! This contest is such a success so far, we might just have to do a similar
thing again soon. Any ideas on what we should give away or do? If so, send them to
us through the Support Page (https://secure.newdream.net/admin/support.cgi) and make
sure to make the subject "Suggestion". Thanks a lot you krazy kontest kids!

6. NetSol Pay in advance. 

Well it's finally happened.. Network "Solutions" is now only allowing registration 
of domain names via their web interface, and only if they are fully paid for in 
advance via credit card. This does "strike back" a bit at domain squatters (people 
who register hundreds of common domain names automatically without paying just to 
try to sell to people who would like the names for their business), but also 
happened to render our automatic domain registration script useless (since it is
impossible to register domains through internic now in any way other than through 
their site).

Don't worry, we're coming back (like Return of the Jedi) with our own "strike back"
(like The Empire) at Network Solutions! Later this month we'll be pleased to announce
you will be able to register your own .com, .org, or .net domain directly with 
DreamHost, and for a price much lower than the $70/2 years Network Solutions (and
register.com) charges! Finally, all your domain name and hosting needs under one
roof, and a technically competent roof at that (A New Hope appears!).

7. DHSOTM October '99


Surely one time or another you've all gotten one of those chain e-mails about stupid 
but true laws still in the record books? You probably read it, got a good chuckle, 
and promptly deleted it.

It's a good thing you did.. because now you can get all your dumb law jonesin' 
fulfilled in one place, organized by country, state, and city, conveniently accessible 
through your web browser! And for providing this invaluable service to mankind, we are 
proud to bestow this month's DHSOTM award to dumblaws.com! 

There was a bit of a low turn out this month for the DHSOTM contest, and we were sad 
until we realized why! The area at the web panel was completely out of date.. it didn't 
have the last three winners, and didn't explain the new way to submit your site 
(through the support area.. make sure you pick DHSOTM as the category). That's been 
fixed up now, and will be kept up to date from now on, so please send your sites our way!

8. Random Quotes(s) From newdream.net

One of the lessons of history is that nothing is often a good thing to do and always a 
clever thing to say. 
    -- Will Durant 


bureaucracy, n: A method for transforming energy into solid waste. 


Remember: Silly is a state of Mind, Stupid is a way of Life. 
    -- Dave Butler 


"It's a summons." "What's a summons?" "It means summon's in trouble." 
    -- Rocky and Bullwinkle 

9. New Features Delayed..

We were planning on having a bit more things done by this point than have actually 
gotten done, and we apologize for that. Mostly the delays are due to all the time
and energy we've had to spend dealing with the afore-mentioned crack. It's set all
our development timelines back a couple of weeks. We appreciate everyone's 
understanding with the problems of late, and we'll make it up to you all with a
whole bunch of great new stuff next month. Enough stuff to keep DreamHost above
and beyond the cutting edge in Webhosting technology!

10. Dreamhost 2.0.. November 1st!

That's the target date.. stay tuned. It's going to be great.. we might even start
having to be "Ecstatic Dreamhost Employees" from then on!

Other sites to check out from the DHSOTM contest for October:


Last updated: Jan 27, 2001.