of trials and torrents
someone hacked our servers to distribute torrents of
pirated material; we must pick up the tab.
or: why heinlein is right
the net is a tough place to live in -- even more so
when you try to make a living there. it is also a playing ground for a lot of
unsavory characters that - given half a chance - will take anyone for a ride. it
appears that we gave those people more than half a chance. the damage done to us
thankfully doesn't appear to be fatal, but we do feel it's impact.
here is what happened -- but first, some (perhaps
irrelevant or even boring) information about the way we work: we do all our
business using the internet. we use it as an enabling factor for some services,
and a facilitator for others. our web site is maintained at what we
affectionately call "the barn" - our offices. they are connected to the internet
through a broadband connection; yet we do not host the site itself, as we deemed
this to be too dangerous. should someone break into our production (or even
worse: development) environment, we'd go out like a candle. so, rather than
hosting our site ourselves, we turned to one of our country's biggest, and most
experienced hosting provider. we synchronize our production machines regularly
with the host, and keep this once-off setup for security reasons. a special
process ensures that no production data is ever left on the host. we figured
that this should keep us relatively safe. there simply are too many shady
characters on the internet that know much more about security than we do. by
separating our web facing server side from our mission-critical production side
we thought that we were safe, and nothing bad could happen to us that we
couldn't fix with a simple re-synch. the web-facing side was secure simply
because it's being run by pros.
it turns our that although we were right about the
first half of our reasoning (our production machines are still relatively safe),
our hosting side wasn't. worse, a simple re-synch did not fix the problem; real
damage was done. here's how: a few days ago we received a routine warning from
our provider that our allocated disk quota for the web host was exceeded. this
rather surprised us, as we know that our maximum disk quota is a factor of 10
above our normal use. we only pay for the space we actually use, but have a
reserve in case we want or need additional space (this happens when we update
web designs, as we duplicate the whole web content, and then test it before
switching it on-line). we also use this reserve to test features of new
products.
another reason for the reserve disk space is that we
use it for file transfers to and from our 'pro clients' (i.e. those we work for
on a contractual basis). for example, one client sends us a film clip to
process. they upload it to the hosted web server, and we then download it to our
production servers. since many of our customers (artists, usually) are even less
tech-savvy than we are, we simplified the process for uploading data for them.
since these clips can be rather large, we require a high reserve quota. since we
have lots of download from both clients and internet users getting iMovie
plug-ins, we also require a high bandwidth internet link.
and this is where we got hit. suddenly, in a corner
of our web server, we detected a new folder containing gigabytes of unknown
data. data that turned out to be pirated, copyrighted software. worse, there
were lots of mysterious small files that were soon identified as torrent seeds.
someone was illegally using our server to distribute stolen data. not only were
they ripping off the copyright owners (as software developer/content provider
ourselves we feel strongly about these things) - no, they were also leaving us
holding the bag to pay for storage, and transmission costs. since our hosted
server intentionally uses a high-bandwidth link to the internet backbone,
transmission cost can become substantial. in the case of these torrents, they
were. initial assessment is that the cost for the additional services will lose
us about one month of profits. i guess we were lucky that we detected the break
in relatively early on (only a few days had passed since the break-in). had this
been going on for a few weeks, the resulting cost could well have killed us as a
company.
exploiting a flaw in the hosting provider's OS
(Unix), these crooks escalated their access privileges, disallowing us to remove
the pirated files ourselves. as i write this, our provider is restoring the
server. we really hope the bill for that won't be too high (they'll hopefully
simply nuke the server, re-install, and then tell us to re-synch). we do not
know if we'll get into hot water because our server was hijacked and used to
illegally distribute copyrighted material. we *really* hope we don't, as we are
ill prepared to fight a legal battle over this.
so, is there an upside to all this? perhaps -
although only a small one: we learned a few things.
firstly, i guess we
where a bit naive and arrogant believing that we where safe. we thought that
this would not happen to us. we only looked at our side of the problem,
forgetting the (hosted) web-facing side. we also thought that since we are a
small developer, with only a tiny web presence, no-one would be interested in
us. we were proven wrong. nobody on the web, it seems, is too small or too
uninteresting not to take advantage of. for these people, no low is too low.
"drive someone out of business? i don't care - as long as I can torrent my
software."
next up - an obvious lesson: read your provider's
contracts. if you have a leverage factor attached to cost (in our case disk
usage and bandwidth), make sure that you monitor it. we did not watch it closely
enough, and got hit. we entered into those contracts fully knowing what they
meant. those contracts were (and still are) good, and we will fulfill them. we
will, however, take precautions that in the future we detect sudden changes in
disk usage or bandwidth more rapidly. our provider was quite nice about this,
has gone a few extra miles, and even indicated a willingness to re-negotiate
some terms of our contract to help in similar situations.
the episode also drives home another point: as long
as you are connected to the net, you are constantly threatened by miscreants.
we'll have to beef up our security, and probably regularly audit our security
measures. how much this will cost we don't know. but we simply can't afford a
few more break-ins like this. it also shows how the shadowy so-called
'sub-culture' of illegal file sharing has tangible effects on small companies.
to quote Heinlein: "there ain't no such thing as a free lunch." in addition to
the obvious theft of copyrighted material, bandwidth and storage also have to be
paid for. most illegal torrents nowadays exploit, and maybe even ruin, some
unwitting small company like us that did or could not invest enough into it's
security.
we also learned something nice, though: that we can
depend upon our clients. you see, because of all this we are also in a tight
spot now that we can't easily allow our clients to transfer data any more. we
will surely find a new way, but that will take time, and effort. luckily, most
of our clients understand the predicament we are in, and are willing to jump
through a few more hoops to work with us. for this we thank them and
you.
Posted: Wed - April 5, 2006 at 06:17 PM