of trials and torrents
someone hacked our servers to distribute torrents of pirated material; we must pick up the tab.
or: why heinlein is right
the net is a tough place to live in -- even more so when you try to make a living there. it is also a playing ground for a lot of unsavory characters that - given half a chance - will take anyone for a ride. it appears that we gave those people more than half a chance. the damage done to us thankfully doesn't appear to be fatal, but we do feel it's impact.
here is what happened -- but first, some (perhaps irrelevant or even boring) information about the way we work: we do all our business using the internet. we use it as an enabling factor for some services, and a facilitator for others. our web site is maintained at what we affectionately call "the barn" - our offices. they are connected to the internet through a broadband connection; yet we do not host the site itself, as we deemed this to be too dangerous. should someone break into our production (or even worse: development) environment, we'd go out like a candle. so, rather than hosting our site ourselves, we turned to one of our country's biggest, and most experienced hosting provider. we synchronize our production machines regularly with the host, and keep this once-off setup for security reasons. a special process ensures that no production data is ever left on the host. we figured that this should keep us relatively safe. there simply are too many shady characters on the internet that know much more about security than we do. by separating our web facing server side from our mission-critical production side we thought that we were safe, and nothing bad could happen to us that we couldn't fix with a simple re-synch. the web-facing side was secure simply because it's being run by pros.
it turns our that although we were right about the first half of our reasoning (our production machines are still relatively safe), our hosting side wasn't. worse, a simple re-synch did not fix the problem; real damage was done. here's how: a few days ago we received a routine warning from our provider that our allocated disk quota for the web host was exceeded. this rather surprised us, as we know that our maximum disk quota is a factor of 10 above our normal use. we only pay for the space we actually use, but have a reserve in case we want or need additional space (this happens when we update web designs, as we duplicate the whole web content, and then test it before switching it on-line). we also use this reserve to test features of new products.
another reason for the reserve disk space is that we use it for file transfers to and from our 'pro clients' (i.e. those we work for on a contractual basis). for example, one client sends us a film clip to process. they upload it to the hosted web server, and we then download it to our production servers. since many of our customers (artists, usually) are even less tech-savvy than we are, we simplified the process for uploading data for them. since these clips can be rather large, we require a high reserve quota. since we have lots of download from both clients and internet users getting iMovie plug-ins, we also require a high bandwidth internet link.
and this is where we got hit. suddenly, in a corner of our web server, we detected a new folder containing gigabytes of unknown data. data that turned out to be pirated, copyrighted software. worse, there were lots of mysterious small files that were soon identified as torrent seeds. someone was illegally using our server to distribute stolen data. not only were they ripping off the copyright owners (as software developer/content provider ourselves we feel strongly about these things) - no, they were also leaving us holding the bag to pay for storage, and transmission costs. since our hosted server intentionally uses a high-bandwidth link to the internet backbone, transmission cost can become substantial. in the case of these torrents, they were. initial assessment is that the cost for the additional services will lose us about one month of profits. i guess we were lucky that we detected the break in relatively early on (only a few days had passed since the break-in). had this been going on for a few weeks, the resulting cost could well have killed us as a company.
exploiting a flaw in the hosting provider's OS (Unix), these crooks escalated their access privileges, disallowing us to remove the pirated files ourselves. as i write this, our provider is restoring the server. we really hope the bill for that won't be too high (they'll hopefully simply nuke the server, re-install, and then tell us to re-synch). we do not know if we'll get into hot water because our server was hijacked and used to illegally distribute copyrighted material. we *really* hope we don't, as we are ill prepared to fight a legal battle over this.
so, is there an upside to all this? perhaps - although only a small one: we learned a few things. firstly, i guess we where a bit naive and arrogant believing that we where safe. we thought that this would not happen to us. we only looked at our side of the problem, forgetting the (hosted) web-facing side. we also thought that since we are a small developer, with only a tiny web presence, no-one would be interested in us. we were proven wrong. nobody on the web, it seems, is too small or too uninteresting not to take advantage of. for these people, no low is too low. "drive someone out of business? i don't care - as long as I can torrent my software."
next up - an obvious lesson: read your provider's contracts. if you have a leverage factor attached to cost (in our case disk usage and bandwidth), make sure that you monitor it. we did not watch it closely enough, and got hit. we entered into those contracts fully knowing what they meant. those contracts were (and still are) good, and we will fulfill them. we will, however, take precautions that in the future we detect sudden changes in disk usage or bandwidth more rapidly. our provider was quite nice about this, has gone a few extra miles, and even indicated a willingness to re-negotiate some terms of our contract to help in similar situations.
the episode also drives home another point: as long as you are connected to the net, you are constantly threatened by miscreants. we'll have to beef up our security, and probably regularly audit our security measures. how much this will cost we don't know. but we simply can't afford a few more break-ins like this. it also shows how the shadowy so-called 'sub-culture' of illegal file sharing has tangible effects on small companies. to quote Heinlein: "there ain't no such thing as a free lunch." in addition to the obvious theft of copyrighted material, bandwidth and storage also have to be paid for. most illegal torrents nowadays exploit, and maybe even ruin, some unwitting small company like us that did or could not invest enough into it's security.
we also learned something nice, though: that we can depend upon our clients. you see, because of all this we are also in a tight spot now that we can't easily allow our clients to transfer data any more. we will surely find a new way, but that will take time, and effort. luckily, most of our clients understand the predicament we are in, and are willing to jump through a few more hoops to work with us. for this we thank them and you.
Posted: Wed - April 5, 2006 at 06:17 PM