In this series of posts I’m revisiting an answer to a question that appeared on Server Fault way back in 2011. I’m pleased to say that it’s been viewed over 100,000 times, and I like to think its helped a few of them.

But it’s time to look again. Since I wrote that post, there have been some huge intrusions, such as the well known Ashely Madison, Anthem Medical Data and JP Morgan breaches that affected millions of people.

[source: wikipedia list of data breaches]

I'm not writing for the people who operate at the kind of scale of those sites here. I'm writing for the scores of people who manage small and medium enterprise systems which handle data and are if anything, despite being a smaller target than the kinds of organisations I mention above, are likely to be under just as much pressure as they don't have the kinds of infosec budgets of a Sony or an Experian and could well be seen as a softer target. My original post text from Serverfault is quoted below, with my thoughts, revisions and edits added as new text.

(Part 2 and Part 3 are available)

Don't Panic

First things first, there are no "quick fixes" other than restoring your system from a backup taken prior to the intrusion, and this has at least two problems.

1. It's difficult to pinpoint when the intrusion happened.
2. It doesn't help you close the "hole" that allowed them to break in last time, nor deal with the consequences of any "data theft" that may also have taken place.

This question keeps being asked repeatedly by the victims of hackers breaking into their web server. The answers very rarely change, but people keep asking the question. I'm not sure why. Perhaps people just don't like the answers they've seen when searching for help, or they can't find someone they trust to give them advice. Or perhaps people read an answer to this question and focus too much on the 5% of why their case is special and different from the answers they can find online and miss the 95% of the question and answer where their case is near enough the same as the one they read online.

That brings me to the first important nugget of information. I really do appreciate that you are a special unique snowflake. I appreciate that your website is too, as it’s a reflection of you and your business or at the very least, your hard work on behalf of an employer. But to someone on the outside looking in, whether a computer security person looking at the problem to try and help you or even the attacker himself, it is very likely that your problem will be at least 95% identical to every other case they’ve ever looked at.

Don’t take the attack personally, and don’t take the recommendations that follow here or that you get from other people personally. If you are reading this after just becoming the victim of a website hack then I really am sorry, and I really hope you can find something helpful here, but this is not the time to let your ego get in the way of what you need to do.

I still stand by this first part, more or less, though I will admit to borrowing the first two words from Douglas Adams.

• It's important to reflect before you act. Before doing anything, you need to make sure that everyone concerned knows what their responsibilities are to the business, its customers, and the legal frameworks in the countries you operate in.You may have obligations to notify customers that they're victims of information theft, to notify legal/government departments, etc. and it's important that you don't act in haste and potentially lost information about the extent of an attack or destroy evidence of who was responsible, how they did it, etc.
• It's important to appoint someone to manage the incident. It is their job to co-ordinate the technical, legal and operational responses of the business to the incident, which means that they cannot get bogged down in details from any one of those areas.
• Therefore: The incident manager should not be the technical lead in the investigation. It's tempting in a smaller business to put the "IT manager" in charge of the incident (that's fine) and then also expect the IT Manager to be her own technical lead, which is a gross error.The technical role should be delegated, possibly even outsourced to a specialist, and the manager needs to be taking a step back from worrying about technical trivia and instead concentrating on what the business needs to do to protect itself and its customers.
• Don't take the attack personally. It's just business, even when it isn't. This is all about making good decisions. It's ok to be upset but it's not ok to make bad decisions because you're upset.Ability to step back from the emotional impact of things like this and work dispassionately should absolutely play a part in your choice of incident manager, technical lead, and so-on, in case you were wondering.

You have just found out that your server(s) got hacked. Now what?

Do not panic. Absolutely do not act in haste, and absolutely do not try and pretend things never happened and not act at all.

First: understand that the disaster has already happened. This is not the time for denial; it is the time to accept what has happened, to be realistic about it, and to take steps to manage the consequences of the impact.Some of these steps are going to hurt, and (unless your website holds a copy of my details) I really don’t care if you ignore all or some of these steps, that’s up to you. But following them properly will make things better in the end. The medicine might taste awful but sometimes you have to overlook that if you really want the cure to work.Stop the problem from becoming worse than it already is:

1. The first thing you should do is disconnect the affected systems from the Internet. Whatever other problems you have, leaving the system connected to the web will only allow the attack to continue. I mean this quite literally; get someone to physically visit the server and unplug network cables if that is what it takes, but disconnect the victim from its muggers before you try to do anything else.
2. Change all your passwords for all accounts on all computers that are on the same network as the compromised systems. No really. All accounts. All computers. Yes, you're right, this might be overkill; on the other hand, it might not. You don't know either way, do you?
3. Check your other systems. Pay special attention to other Internet facing services, and to those that hold financial or other commercially sensitive data.
4. If the system holds anyone's personal data, immediately inform the person responsible for data protection (if that's not you) and URGE a full disclosure. I know this one is tough. I know this one is going to hurt. I know that many businesses want to sweep this kind of problem under the carpet but the business is going to have to deal with it - and needs to do so with an eye on any and all relevant privacy laws.

However annoyed your customers might be to have you tell them about a problem, they'll be far more annoyed if you don't tell them, and they only find out for themselves after someone charges $8,000 worth of goods using the credit card details they stole from your site.Remember what I said previously? The bad thing has already happened. The only question now is how well you deal with it.

Disclosure is where it's at now.

If I'm going to criticise my 2011 self for anything, it's for failing to take disclosure seriously enough. Well that and waffling. But in 2018, disclosure has become a huge part of the response to a security breach and it's completely right that it should be.

Intrusions into servers themselves while still a problem that needs to be addressed operationally, are almost unimportant these days, because data and services are now seen as the valuable commodities, more so than servers themselves(1), and these days it should be fairly trivial to rebuild a server with tools like puppet, Microsoft System Center, Chocolatey and Kubernates. We should be treat servers as cattle, not as pets, we should be worrying about the services running on these servers and “scaling out” so that the loss of one server isn’t an issue as such, and this completely changes how we can approach an intrusion into any one server - of course it also increases the complexity of determining whether or not your entire farm is rooted, but there ain’t no such thing as a free lunch.

There are a number of positions that need to be considered in disclosure and you need to be aware of how these are perceived throughout the world. If you’re a US company operating in the EU and holding data on EU B2C customers on systems that are local to you then you may have a very complex set of laws to consider, for example, and it may be difficult to know how to balance these requirements.

One thing that might help, ironically, is the rise of cloud. If you can store and process personal data in the same legal zone as the customers who own that personal data then you will probably greatly reduce your legal complexity. I’m predominantly referring to the UK interpretation of the disclosure requirements for GDPR in my examples here because I’m a UK citizen living and working in the UK/EU for an UK/EU employer who tasks me with securely storing and processing the data of UK/EU residents.

This is not any kind of legal advice or opinion and you absolutely should check the legal requirements for the country you’re based in and the countries your company operates in.

Legally, you may have an obligation to disclose any data breach that occurs on your site. In the EU we are obligated under Article 33 of the GDPR regulations to notify our local “supervisory authority”, which is the information commissioner (ICO) in the UK and under Article 34 we’re also obligated to notify “data subjects” about any breach that’s likely to have resulted in any of their “personal data” being shared with others.

If you read any of these articles you’re probably still not any wiser about when and whom to notify - what are the thresholds for notification? To that end, I’ll point you at this discussion on the Varonis blog which tries to break down what the thresholds mean and what actions you need to take.

But there’s another angle. What is the ethical impact of your disclosure policy? If you feel that an incident doesn’t quite meet the legal requirements where you are to require you to disclose an intrusion/data theft, then what about ethical obligations? What about customer confidence, which can be critically impacted when you’re caught acting unethically around an intrusion? At this point I would suggest that disclosure of data loss or theft is the most important and complex part of your playbook should you suffer a security incident. Get it wrong and you may find yourself in legal difficulties and you will certainly lose credibility with your customers. Get it right… and while it’s still never going to be fun to admit you suffered an intrusion, you can at least win respect from current and potential future customers by showing professional resolve and an ability to learn from your mistakes.

 

---