It’s been almost three years since business owners Carl Woerndle and his brother Alex were caught up in a cyber attack so damaging, it destroyed their once prospering technology business, Distribute.IT.
At this year’s CIO Summit in Perth, Woerndle – now a cyber security advisor at Deloitte – gave a warts and all account of how he and other staff at his former company dealt with the crisis and the fallout.
The brothers founded Distribute.IT in 2002, a typical startup working out of a spare room with, as Carl puts it, “no money, lots of good ideas and lots of enthusiasm".
At the time, the fledgling firm was granted one of five domain registrar licences available in the marketplace following moves by the government to place more control around domain name management, and remove Melbourne IT’s monopoly position.
The company adopted a channel sales strategy, appointing resellers to on-sell its services unlike its competitors, which were focusing on retail business.
Over the next nine years, the firm branched off into Web server hosting, in 2006 building its infrastructure around virtual server technology. By 2011, it was operating three data centres and was an Australian distributor for Verisign.
Distribute.IT had secured a 10 per cent market share for Australian domain names with 250,000 under management, and more than 200,000 clients on its books.
“We were growing at 4 per cent a month,” says Woerndle. “We also had over 30,000 hosting clients on that infrastructure ... and 3,000 active resellers on our database at that time.”
The company had more than 50 global domain name accreditations and opened an office in Jakarta, Indonesia as a launching pad to sell through Asia. It was looking at a couple of acquisitions and a possible IPO in 2014.
The initial breach – week one
At 5pm on Friday June 3, 2011, Woerndle received a call from his CIO alerting him to a breach in the company’s network.
“Being a hosted platform, we had 16,000 IP addresses in our network, we had ports all over the place for people to transact their business in and out, it [was] a very open network,” says Woerndle.
“We were very used to hacking attacks, and attempts to infiltrate our network were actually a daily occurrence for us.”
Distribute.IT had experienced six major distributed denial of service (DDoS) attacks up until June that year and had written its own proprietary software to detect and “shift out” those IP addresses when they were hacked.
“We had about 30,000 clients and a minimum of two per day were targeted on our network. Most of those were limited to the websites themselves such as a vulnerability on the site or a particular server.”
But this attack was different. The hacker had managed to bypass the company’s entire security protocol, get behind its firewall, and gain access to its master user access information.
“We had somebody inside our network; I’m watching them, they are not doing anything, moving very freely around our network structure not stealing anything, just a bit of a fishing exercise,” he says.
Distribute.IT had a secondary problem. Under its agreements with the regulator – .au Domain Administration (auDA) – it was mandatory for the company to report breaches of this nature.
That evening, the company’s team of 12 IT staff gathered at its Melbourne data centre to create a plan to deal with the breach.
“We took a proactive and aggressive approach to dealing with the issue – we had a lot spare equipment lying around and some plans around building a new network structure and thought it was a good opportunity to kill two birds with one stone.
“Over the next 48 hours, we pulled out entire network off the rack – three data centres – at one point everything was sitting on the ground.”
This created a complete blackout for its customer base before the network was rebuilt and back online on the Monday.
“We contacted the regulator and advised them of the breach and they made us reset all 250,000 passwords and user information, and advise every single one of those customers of the new details before they would let us connect back to the registry again,” says Woerndle.
To do this, the team spent the next couple of days writing proprietary software to distribute the information to its clients. Distribute.IT was able to connect back to AusRegistry by Thursday that week, six days since the breach.
“We were very cautious about the way we opened up access back into our network again. We made a lot of our customers send through requests for IP addresses, names of people for access, ports numbers that they wanted access to – we opened these up very slowly one by one,” he says.
Carl and his brother and most of Distribute.IT’s staff had worked back-to-back 72 hour shifts during the week following the breach with less than four hours sleep in between. They were exhausted but felt comfortable with the company’s position by the following Friday night.
At 4:30pm on Saturday June 11, Distribute.IT’s network monitoring system went crazy. The IT team was watching servers go offline every few seconds.
The hacker had regained access to the company’s network, meaning that all the work completed the week prior was a total waste of time. The front door was wide open and this time the attack was completely malicious.
“The first thing they did was replace our primary website with a blank page that says ‘you have been hacked’ using the moniker ‘evil’, put all the passwords from our network down on that page – all the passwords we had reset two hours before the attack,” says Woerndle.
The hacker accessed the company’s firewall, changed the settings, and locked staff out of the network.
The hacker also embedded a program inside its network and ran a command called 'rm-rf', which removes the file names and leaves scrambled data on the drives, making them useless.
“They had a cascading program that ran through our network to destroy as many servers as they could until they got control of it,” says Woerndle.
This attack targeted Distribute.IT’s primary trading platforms and hosting systems, shared Web servers, and all the corresponding backup systems, removing its ability to trade.
“They went as far through the system destroying what they could until we could get control of it. They only way we could get control of our network was to go to the data centre, pull the plug out of the wall and turn the power off.”
The 12-man IT team were called back in an attempt to recover services. The company also contacted the Australian Federal Police (AFP) for assistance.
“What we’d done the week before, we did again – we rebuilt our entire infrastructure from the ground up. We were into our third 72-hour block [working on the problem] and by this time, we were completely and utterly exhausted.”
The network was switched on again on the evening of Monday June 13. But with its primary websites and VoIP systems down and client databases compromised, Distribute.IT had no way of communicating with its clients.
“Even if we did, we didn’t know who they were anyway,” says Woerndle.
The company had no choice but to create a blog site and post regular updates for customers as the recovery progressed. The site included an email address and Distribute.IT’s small IT team were dealing with 20,000 emails each day. The company was fielding an average of 300 emails daily before the attack.
Earlier that week competitors were running fire sale offers for Distribute.IT clients.
“We were pretty happy with that as you can imagine,” says Woerndle in a sarcastic tone.
Again, the company completed its regulatory reporting, went through the same domain reset process, and met with the AFP for the first time.
“By the 14th (Tuesday), I am through three shifts of 72 hours, I’m completely exhausted. Around the period, even though it was closer to 90 per cent of our servers [and] we were able to get back up and running again, we still had critical infrastructure that was down.
“All our core trading platforms and backups, a large number of shared Web servers and those backups were still completely compromised.”
Distribute.IT started to lose clients as people came to the company’s data centres to pick up their equipment. Trust and brand equity that had been built up over nine years was starting to erode.
By Wednesday, three IT staff had resigned, and two others went AWOL. Distribute.IT’s CIO suffered particularly badly.
“Everything fell onto this poor fellow. The AFP wanted him, we wanted to know what we could do to help. He had to come up with a game plan and was under immense pressure,” says Woerndle.
The CIO was under so much strain that on Wednesday, he collapsed on the floor of the data centre and was taken home by other staff for medical treatment.
“The game plan changed for us again on the 16th (Thursday) because it was the first day we hit the mainstream press. The customer churn during this period was starting to accelerate … there was a lot of pressure and misinformation running around the place,” he says.
The media coverage hit its peak on the 18th and 19th of June and journalists were camped outside Distribute.IT’s door.
“Every time you wanted to go out for a toilet break, they would stick a microphone under your face for an interview. I say this with most respect for IT guys but our IT guys weren’t used to dealing with anybody at all; they sat in the backroom very quietly, they certainly weren’t equipped to deal with the media.
“One guy was under so much pressure he didn’t want to go outside – he had a bucket under his desk, he was taking toilet breaks under his desk.”
Knowledge of the hack became so widespread that the company had an email from hacking group Anonymous saying ‘it wasn’t us’.
“We had phone calls from every major bank in Australia concerned about credit card leakage, PCI compliance, we had the privacy commissioner on the phone – people we hadn’t even heard of who wanted to get a piece of the action so to speak. We even had a call from [Julia] Gillard’s office asking how they could help.”
Distribute.IT had to announce that it had lost four of its major Web servers used by 4,000 hosted clients. Around 3,000 resellers were also affected.
“We are talking about our primary servers and up to four levels of backups completely destroyed by the hacker. When we were having our board meetings, we were starting to think about litigation, are we covered for insurances?”
The recovery effort was slow and by the weekend, 12 IT staff became 6. The CIO turned up 48 hours after his breakdown, which Woerndle described as a ‘godsend'.
“It shows how heavily reliant we were on that particular individual. During the weekend we had good support from the regulators, but felt we were two to three weeks out from recovering our core trading platform.”
On the Monday morning (June 20), Distribute.IT received a breach notice from the regulators indicating that they felt the company was no longer able to perform its duties as a domain name registrar.
“They give you a timeframe in which to rectify the situation or they enact a contingency plan. A contingency plan is pretty easy in the domain name industry. They take all 250,000 domains off you and give them to one of your competitors.
“We had a secondary problem with this – our resellers – which weren’t recognised in the domain industry. So if we had of lost our domains, our resellers would have lost their domains to our competitors, wiping out another 3,000 domain businesses.”
And the timeframe to restore services? 24 hours.
“My brother and I knew at this point that our business was gone.”
The men contacted NetRegistry, which had expressed interest in buying the business before the disaster. A full acquisition of its assets by NetRegistry was negotiated within 24 hours.
“Next morning, the people from NetRegistry came down to Melbourne, we finished signing off contracts and they handed us a cheque for the business which was significantly less than the offer they had made us three months prior.
“We handed over the key to the business. My brother and I took our little cheque and handed it over to the bank. It unfortunately wasn’t enough to cover our obligations so we had to go through a full liquidation and we were out of business.
“Between Alex and I, we had $6 in our pocket – we went down to the local coffee shop – bought a couple of coffees, sat down, didn’t say anything to each other for two hours and thought: 'What did we do wrong?'”
The aftermath
During the attack, Distribute.IT received an anonymous email from someone claiming to know who was responsible for the attack, including the person’s name, address and phone number.
The email was sent to the AFP. Three weeks later the AFP arrested an unemployed truck driver from NSW, who it had put under surveillance after receiving the email.
“While he was trying to cover his tracks with our hack, he was also preparing to do something else. They raided him on a Friday night, and within an hour of him pushing the button and doing what he did to us to another 360 networks across Australia.
“They arrested him for that and he ended up doing two and a half years [in gaol] for that hack. But he was never formally charged with our [hack]. He was an IT nutter, sat at a computer for 20 hours a day, hated the world, applied for IT jobs; not with us, we were his random victim.
"He learnt his trade from [videos] on YouTube and from [other hackers] on chat sites."
So how did he get in? He hacked Distribute.IT’s VoIP system through a simple software vulnerability, and its website was taken down through a vulnerability in the PHP [programming language].
But it was the third breach that was key to this hack.
“We focused on efforts on the network itself, rebuilding the network, putting the security around it. What we missed during this period was what came from outside.
“The [hacker] had been in contact with the company and made a couple of relationships internally – a bit of social engineering. He somehow managed to isolate a vulnerable staff member in IT.
“He had conversations with that staff member and they exchanged emails. That staff member was second in charge in our IT area. As part of that exchange, he managed to put some malware on his [the staff member’s] personal laptop at home.
“During the second week [of the hack], he left himself a keyhole into the laptop and used that keyhole access to get control of the laptop and then the secure VPN back to the network.”
Woerndle says the way in which you manage the early stages of a hacking incident will have a big bearing on the outcome. Distribute.IT’s decision to take down its network after the first breach alerted the hacker.
“So whatever he had in mind, he had to initiate it at that point or he would have run out of time. In retrospect, what I should have done was the complete opposite; it’s one of the hardest human emotions to watch what goes on.
“That’s the point in time where you get forensics involved, have a look around the network, see where those entry points were and build up a real case against the perpetrator,” said Woerndle.
So how have the brothers recovered personally and professionally from this incident?
“It was a perfect storm of events because we had plans to open up in China and were preparing for an acquisition. So after nine years, in early 2011, was the first time we put our homes up against our business.
“When this event took place, Alex and I lost our homes. The single hardest thing to this day was me going back home a day after the end of it – I hadn’t seen my family in three weeks – and telling them we had lost everything,” he says.
“We spent nine years on that business, Alex and I worked seven days a week on that business. We took four weeks holiday during that whole period, we put our heart and soul into that.”
“It took Alex and I six to 12 months to get over it. To my brother’s credit, it was him who came along one day and said, ‘You know what, nobody ever talks about this stuff. There are a lot of lessons there for other people, maybe we should talk about it.’
“I remember looking at him thinking, ‘Are you mad?’”
Carl has recovered and still has an entrepreneurial spirit. He has a few “software plays in the background” that he is trying to develop.
“It’s a long journey back,” he says.
Speaking to others about his experience has certainly been a positive step.
Source: Byron Connolly (CIO) on 12 May, 2014 11:38
0 comments:
Post a Comment