Imagine someone withdrew money from a company's account at night. The next morning panic breaks out, leading to yet more problems. The IT department can reinstall a compromised system from scratch or restore it from backup. Reinstalling from scratch will wipe out all traces left by the attackers, and external investigators will have to search for clues in other systems. Restoring from backup carries the risk of accidentally reinstating a compromised image. In this paper, we will describe common mistakes that experts make when responding to security incidents.
Mistake 1. Absence of a response plan
Money was stolen from a company. The company has no security tools apart from antivirus software. Company management does not agree with shutting down the server and blocking user accounts because they don’t understand how it will affect the rest of their business. It takes the company days to establish who the compromised computer belongs to and what their role in the company is. Meanwhile, hackers are emptying ATMs, withdrawing money, or stealing data. Numerous organizational problems hinder incident response. Things are quite different when a company has logs, asset management, and an understanding of business processes. This allows the company to detect hackers before they do significant harm, and address the issue immediately without wasting precious time. This greatly improves incident investigation and helps prevent serious consequences.
The goal of having a response plan is to avoid being caught off-guard and to know in advance what needs to be done to minimize damage. It is important to assess the technical means used to collect and store data needed to investigate the incident. It is also important to evaluate the competence of the internal incident investigation team. If the team does not have enough experience, the company has to bring in external investigators.
A response plan also determines who does what in case of an incident. The IT department collects incident-related data. The company's management, legal department, and PR are responsible for external communications about the incident. They need to know who they should notify (customers, regulators, authorities) depending on the incident (customer data theft, money theft, etc.).
In some sectors (such as banking and critical infrastructure), security incidents must be reported to regulators or authorities. Other companies must contact their insurance providers immediately, as any delay could invalidate their insurance coverage. All this must be agreed upon in advance with company lawyers and incident investigators.
To avoid bothering half of the company every time a virus is detected, incidents must be classified by severity and the importance of the asset. For example, spyware on an accountant's computer is a top-level incident. There should be an incident response protocol with checks for all the actions an attacker could possibly have carried out on the infected machine, such as transferring money, changing account details, accessing online banking, installing backdoors, copying electronic signatures, etc. The next task is to carry out a full investigation to establish how far the hackers got inside the network. This will help in choosing appropriate countermeasures. In our example with the accountant's computer, such measures may include resetting remote banking system access keys, blocking account transactions, replacing the computer entirely, or blocking network access. Similar response plans must be prepared for the computers of CEOs, top managers, and regular employees, and for other important network assets.
After discovering suspicious activity, experts often spend days establishing where the affected computer is located, who it belongs to and what role it plays in business processes, and whether there are any restrictions on turning it off or collecting data from it. IT and infosec teams should have this information ready in advance. If the compromised system is in the accounting department and is used for payments, it cannot be disabled, because in that case business operations will come to a halt. For mission-critical systems, the IT department must have backup computers ready, for example. Backup computers are usually not available, and so the investigation is delayed. Meanwhile, attackers may be withdrawing money or stealing data.
A response plan must be regularly tested by conducting penetration tests. Pentesters get into an accountant's computer and simulate various actions an attacker could perform. Any actions that the pentesters manage to carry out successfully must be included in the response protocol, and in the event of a real incident the investigation team must check whether the intruders managed to carry out any of these actions. A company can also conduct tabletop exercises to play out attack scenarios simulating computer compromise, ransomware attack, or money theft. Participants must test a response plan: identify all owners of the compromised systems, think of actions attackers can carry out and in what order, and check whether all the concerned parties are involved in the investigation. By mimicking real-life situations, such training helps avoid serious consequences and identify security flaws that must be fixed.
For typical incidents, such as phishing or a ransomware attack, a step-by-step instruction can be prepared. If a suspicious activity cannot be categorized, it is important to understand:
Where exactly in the network assets were affected
What attackers can do with these assets, and what can be done to protect them
Who is responsible for these assets
How to communicate the threat inside and outside the company
How to isolate the threat
A response plan is indispensable, because an incident tests a company's resilience and may cause significant damage and even complete failure of the business. However, no company is 100% secure: sooner or later incidents will happen to everyone, and it's better to stay prepared. A well-thought-out plan gives a clear and precise algorithm to avoid chaos in a crisis.
Mistake 2. Underinvestigated incidents
A company reinstalls the OS on a compromised computer, ticks the "incident closed" box, and a week later has a large sum of money stolen from its account. This happens quite often to companies that take half measures. It is important to understand how attackers got into the network, reconstruct the attack chain, and take measures to contain and eliminate the threat. Otherwise, some hosts may remain infected, and hackers can use them to continue the attack.
Mistake 3. Absence of event collection infrastructure
Attackers could have penetrated a company's network a long time ago, and no traces of the attack remain. Like any operating system, Windows collects events but it stores data locally and for a limited time, often only until the OS is restarted. Network devices, have limited memory for event storage. Considering this, we can't find out whether a computer connected to a malicious server three months ago.
Of all the companies we have studied, only around 15% have event collection infrastructure in place. Companies with SIEM systems often have them collecting dust. No one in these companies checks whether their SIEM system collects data correctly and in the required format. Default settings are often kept. Data can be stored for a day or a week, with no use at all.
IT departments use monitoring and log collection tools. These systems are also effective and can be used in investigation, so don’t forget about them.
At the very minimum, event collection must include collecting data from operating systems and network equipment (such as firewalls) and storing it for at least a year. This will allow investigators to determine which hosts connected to what.
This information can also be used to detect new incidents. By conducting a retrospective analysis, we can find out whether a company was attacked in the past. With this information we can act promptly, rather than a year later when confidential information pops up on the darknet at the worst possible time, for example, right before a company goes public.
Without event collection infrastructure, it is impossible to get data from the past. It also saves time as you don't have to analyze dozens, hundreds, and sometimes thousands of computers. Properly configured specialized infosec solutions are a must-have. Event collection tools, such as a correctly configured SIEM system, can identify an incident, reconstruct the attack chain, and find the attack entry points. A SIEM system pieces together the big picture from separate micro events.
Mistake 4. Absence of asset information
At best, companies give incident investigators a description of the business system components, who is responsible for this system, and what its purpose is. The situation is usually worse. Most companies either don't have asset management at all or don't update their asset information. Network configuration can be described in a three-year-old document, although the network may have grown twice since the document was created. This makes quick and effective incident response impossible.
Asset management does not require large investments. Take a look at hackers—they don't need complex tools to analyze a company's infrastructure and get a clear idea of how it works. As a result, they often know a company's infrastructure better than its own IT department do. So, a company does not need complex tools either. First and foremost, it needs to understand its own business processes.
Expensive tools can make the task easier, but it’s about understanding own processes. A specialized SIEM system is a good choice, but even a simple spreadsheet will do the job. The main thing is to understand your processes thoroughly.
Mistake 5. Absence of documentation
Here we mean documentation of the interactions between departments. It can determine whether accountants can have access to the IT specialists' computers, or whether employees can send each other work documents via personal email. Technical documentation describing how systems interact with each other is also important. Example: according to a company's documentation, a component works with only one module, but in fact it works with three modules. If the IT specialist who knew about this left the company and then some time later an incident happens, the investigators will be chasing clues in the dark. Such practices often come to light in conversations with employees: for example, they may reveal having no other means to share documents except personal email. This eats up a lot of time. The situation is even worse when no employee can explain the practices in place.
Mistake 6. Destroying evidence
A common mistake is to investigate an incident without having the proper qualifications. When taking a snapshot of a drive, one can accidentally wipe evidence from the drive. Sometimes, after identifying the key host used to conduct an attack, we learn that its owners have reinstalled the operating system and destroyed artifacts critical to the investigation. In some cases, a computer must not be turned off when a memory dump is going to be taken, otherwise the evidence will be destroyed. A SIEM system does not always have all the necessary data. Without historical copies, it is often impossible to investigate the attack chain. That is why it is so important to have a strict plan outlining how to deal with attacked hosts. All the actions from the beginning of the investigation must be approved by professionals. In urgent situations, PT Expert Security Center quickly starts the investigation, often before any official agreements have been formalized.
Mistake 7. Provoking attackers
Employees without enough experience and knowledge may try to block everything and anything or threaten attackers in correspondence, without realizing the gravity of the situation. Hackers may retaliate and cause additional harm to the company. For example, they can release ransomware into the entire infrastructure just for fun, or disable a critical service. It is necessary to anticipate how the hacker will react if detected, and to be prepared.
Mistake 8. Not learning lessons
Some companies don't learn their lessons from security incidents. Even after a thorough investigation of an attack, some companies do not follow our recommendations on how to improve their security and fix vulnerabilities. Let's say, in summer we eliminate consequences of a breach, remove malware, reset passwords, and clean up a company's network. In autumn hackers strike again. They feel right at home in the company network. One government institution was hacked twice in such a manner. Still, most companies do take protective measures after an incident, even if not at once. If a network is under constant attack, the security department doesn't have an extra month to fix the critical vulnerability used by attackers to hack the network in the first place. The next incident will happen much sooner.
Mistake 9. Alerting attackers
If infosec staff communicate via a mail system that has been compromised, attackers can monitor all measures they are taking. The hackers will either cover their tracks and lay low, which will complicate the investigation, or go on a rampage (see Mistake 7), in which case the company will have far bigger issues to deal with than just a security incident investigation. That's exactly what happened when an administrator corresponded with us via the desktop version of Telegram from a compromised computer. Hackers read the correspondence and stayed one step ahead of us, which complicated the investigation even further until we discovered that the administrator's computer had been compromised and we switched to a different communication channel.
It is necessary to have backup communication channels outside a company's infrastructure in case of an incident. For example, use Telegram or WhatsApp on trusted devices. If your entire network is compromised, you no longer control it. Think carefully about each step of your investigation.
Mistake 10. Zombie incidents
Even after investigations are completed and consequences are eliminated, incidents can come back to haunt you. Example: a compromised six-month-old image is reinstated from a backup copy, and attackers suddenly obtain access to the internal network again. This is a rare occurrence, but with serious consequences. It is important to keep an eye not only on current the infrastructure, but the archiving and backup systems as well. It's a good idea to monitor systems for signs of repeated compromise some time after an incident was investigated. A zombie incident may be caused not only by backup systems, but by the reappearance of an asset that was absent during the investigation. For example, an employee was on vacation, and their PC was not checked. Or a computer was being repaired. Once the infected computer is reconnected to the network, the hackers regain backdoor access to the company's systems. Only a well-designed monitoring system and compliance with post-investigation recommendations can help the company quickly respond to such threats.
Incident investigation may seem easy. All you have to do is get to the bottom of an incident, reconstruct the attack chain, find the source, and block the threat. But if a company does not have the minimum infrastructure necessary to identify and mitigate incidents, or responds to incidents without proper experience, it is not always possible to remove a threat. That's why it's best to prepare in advance for encounters with attackers.
Elmar Nabigaev
Deputy Head of PT Expert Security Center, Positive Technologies
