Hunting the (Cyber)Hunters

From Cyber Threat Hunting by Nadhem AlFardan

Cyber threats are constantly multiplying and evolving, and a good defense isn’t good enough anymore. The safest approach to cyber threats is to hunt them down before they do damage, rather than reactively waiting for them to be uncovered.

Read on to learn more.

Cybersecurity Threat Landscape

Today’s cyber threat landscape is complex, constantly evolving, and diverse. Threat actors, ranging from organized cybercrime to state-sponsored groups, actively improve existing attack techniques and tools and create new ones to reliably establish and quickly move through the Cyber Kill Chain, starting from reconnaissance to actions on objectives.
The Cyber Kill Chain developed by Lockheed Martin, shown in Figure 1, describes the set of stages that adversaries typically go through to achieve their final objective(s). The Cyber Kill Chain consists of seven stages.

1. Reconnaissance: the attacker assesses the situation to identify potential attack targets and tactics. For example, an attacker harvests social media accounts or performs an active vulnerability scan on publicly accessible applications.
2. Weaponization: the attacker develops the code to exploit vulnerabilities or weaknesses that the reconnaissance stage uncovered. For example, preparing a phishing email, formulating a SQL injection code, or preparing malware code.
3. Delivery: the attacker uses the delivery vectors to send the weaponized payload. For example, an attacker uses email to deliver malware code.
4. Exploitation: the attacker executes the code she created in the weaponization stage.
5. Installation: the attacker creates a channel that allows her to reach the compromised system.
6. Command and Control: the attacker establishes a command-and-control channel (C2) with an external server. For example, an attacker uses Twitter as a covert command and control channel to communicate with compromised systems.
7. Actions on Objective: the attacker fulfills the objective(s) of the attack. For example, an attacker encrypts files on the endpoint in the case of a ransomware attacker.

Figure 1 Lockheed Martin Cyber Kill Chain

A popular meme in cyber security, credited to Dmitri Alperovitch, states:

“there are only two types of companies: those that know they’ve been compromised, and those that don’t know.”

Threat hunting allows organizations to take a proactive approach in which they assume that they have been hacked and can uncover evidence of that.
We now have some idea about the complexity of the security threat landscape; let us dig into essential concepts of threat hunting and describe its relevance and importance.

What does this book teach?

This book will teach you how to use data platforms and build security capabilities and tools to execute cyber threat hunts, making use of standard searches, threat intelligence, statistics and machine learning.

Why is it important now?

The cyber security threat landscape is continuously changing. With many new breaches reported/discovered that have gone undetected for a long time, cyber threat hunting has become a critical proactive service that many CIOs, CISOs, cybersecurity specialists and other security practitioners are keen to start or enhance, by establishing a cyber threat hunt framework, process, playbooks and the successful execution of effective cyber hunts using advanced analytic capabilities

Why hunt?

There is no perfect cybercrime. Adversaries leave clues and a trail of evidence when executing one or more of the cyber kill chain stages.
Advanced adversaries have shifted from using noisy attacks that trigger security alarms to more stealthy ones that leave a small footprint and trigger minimal alerts, if any, going unnoticed by automated detection tools. According to a SANS published report, “the evolution of threats such as file-less malware, ransomware, zero days and advanced malware, combined with security tools getting bypassed, poses an extensional risk to enterprises.”
The increased threat actors’ sophistication in operating in covert nature and their ability to launch attacks with minimal chances of detection are driving organizations to think beyond their standard detection tools. The change in the adversary behavior requires defenders to establish proactive capabilities such as threat hunting and deploy advanced analytics using statistics and machine learning. For example, hunters can regularly search for potential data exfiltration activities through Domain Name Service (DNS) by applying volume-based statistical analytics without waiting or relying on network security tools such as intrusion detection systems to generate security alerts.
Organizations rely on the threat hunter’s skills to uncover the above threats during threat hunt expeditions, resulting in reduced dwell time and increased cyber resilience. The dwell time is the time between an attacker’s initial penetration of an organization’s environment (threat first successful execution time) and the point at which the organization finds out the attacker (threat detection time).
In addition to reducing the dwell time, running threat hunting expeditions introduces other security benefits to the organization, such as:

• Identifying gaps in security prevention and detection capabilities.
• Tuning existing security monitoring use cases.
• Identifying new security monitoring use cases.
• Identifying vulnerabilities that assessment activities did not uncover.
• Identifying misconfiguration in systems and applications, which might impact security, operation, and compliance.

To capture the above list of benefits, organizations need to establish and operate a robust threat hunting process that clearly describes the threat hunting expeditions’ inputs and outputs. The book helps you establish a robust threat hunting program using practical examples and providing templates.
Now that we established the need for a proactive approach to uncover cyber security threats let us describe how to structure a threat hunt.

Structuring threat hunting

Threat hunting takes a hypothesis-driven investigation approach. A hypothesis is a proposition that is consistent with known data but has been neither verified nor shown to be false. A good hypothesis should be relevant to the organization environment and testable in terms of the availability of data and tools. Taking a hypothesis-based approach is referred to as structured threat hunting.
On the other hand, unstructured threat hunting refers to activities in which hunters analyze data at their disposal to search for anomalies without a pre-defined hypothesis. For example, the hunter might process and visualize data to look for unexpected changes in patterns such as noticeable spikes or dips. Finding such changes can lead the hunter to investigate further to uncover undetected threats. In this book, we focus on structured threat hunting, but we do not discourage you from exploring data without a formal hypothesis from time to time.

Coming up with a hypothesis

The threat landscape associated with the environment you try to protect should drive what hypothesis (e.g. an attacker has gained access to an organization’s endpoints via PowerShell) to create and execute. Different sources concerning threats and their relevance to the environment can assist you in understanding the threat landscape. Threat hunters translate this understanding to hypotheses.

Testing the hypothesis

It is the job of the threat hunter to test the hypothesis using the best resources available at the hunter’s disposal. Testing the hypothesis can start by defining a manageable list of activities that can uncover the first set of evidence or indicators concerning the hypothesis or guide the hunters to subsequent searches. For example, the following activities are relevant to the previously stated hypothesis.
Hunting for suspicious PowerShell activities could reveal the existence of the compromise, proving the hypothesis. The successful execution of the following may uncover evidence of compromise (or not).

Executing the threat hunt

Executing a threat hunt might take an hour or might go for a week, depending on multiple factors. Failing to prove the hypothesis does not necessarily mean that a threat does not exist. It means that the hunter could not uncover the threat with the skillset, data, and tools available.
The book focuses on structured hunting, in which the threat hunter, working with other security team members to define and prove a hypothesis, targets adversaries’ Tactics, Techniques, and Procedures (TTPs).
The organization’s threat hunting maturity level should improve over time. There are many lessons the hunter will learn from the hunt expeditions. The book provides practical lessons on how to plan, build and operate an effective threat hunting program.
Now we have a good idea of what threat hunting is; let us compare it with threat detection, a fundamental security monitoring service, and draw differences and highlight similarities.

Threat hunting vs threat detection

Detection is tool-driven, while hunting is human-driven. In hunting, the hunter takes center stage, compared to tools having that role in the world of detection. Threat hunting relies heavily on the experience of the threat hunter for defining the hypothesis, looking for evidence in a vast amount of data, and continuously pivoting in search of the evidence of compromise. Threat hunting does not replace threat detection technologies; they are complementary.
Threat detection refers to the reactive approach in which Security Operation Center (SOC) analysts respond to security alerts generated by tools. For example, SOC analysts would triage and investigate a security event generated by an Endpoint Exposure and Response (EDR) tool or a security alert generated by a Security Event and Information Management (SIEM) system.
SOC analysts attend to security alerts detected and reported by security tools and perform triage and investigation of security incidents. Figure 2 shows at a high level the threat detection process, in which SOC analysts would primarily perform cyber threat farming. Like farmers, SOC analysts generally wait for alerts (ripe crops) to show up on a dashboard to triage and respond to (harvest and process.) On the other hand, hunting takes a proactive approach. Hunters take the lead by going out in the hunting field to conduct expeditions, equipped with the right mindset, experience, situational awareness, and the right set of tools they require for an expedition.

Figure 2 Threat Detection High-Level Process

Detection is an essential SOC service. Addressing deficiencies in the security monitoring service should be a top priority while establishing or outsourcing a threat hunting capability. Organizations should not consider establishing a threat hunting program to offload the work from the security monitoring team to threat hunters.
Detection and hunting should work together to deliver a better coverage of the cyber threat landscape. Detection and hunting interact and, in some instances, overlap. There will always be cases where detection is an input to a threat hunt and vice versa. For example, a threat hunter might build a hypothesis that considers a widespread system compromise based on few suspicious activities detected on one or more endpoints and observed by the security monitoring team.
Detection and hunting can use the same or different analytic techniques to detect or hunt for malicious activities. For example, user behavior analytic tools deploy statistical analysis and machine learning to detect and report anomalous user behavior to the security monitoring team. Hunters can make use of similar techniques for cyber threat hunting. Although hunters would not lead the development of machine learning models, they must understand and apprehend the capabilities and limitations of the different analytic techniques.
Threat hunters are highly skilled resources. Let us have a look at the set of skills that threat hunters possess.

The background of a threat hunter

A threat hunter is a cyber security specialist who proactively and interactively seeks to uncover attacks or threats that evaded detection technologies deployed in various places in the network.
Successful threat hunters are curious, prepared to tackle new challenges, and equipped with a good understanding of their hunting field. As a threat hunter, you will face challenges such as the unavailability of data, slow searches, improper event parsing, old technologies, incomplete or not access systems. The hunter should raise these challenges during and after a hunt expedition. Some of these challenges might get addressed in a reasonable time, while others might take a long time or might not get addressed at all, especially ones that involve financial investments. These challenges should not prevent the hunters from finding new ways to enhance the effectiveness of the threat hunts by looking at other data and systems and tune the techniques the hunter deploys. Hunters are resourceful.
An offensive mindset gives the hunter an advantage in creating effective threat hunt plays and executing threat hunt expeditions.
During a hunt expedition, not being able to prove the hypothesis should not discourage a hunter. It is a common outcome that can be due to various reasons, including:

• The attack or the threat described in the hypothesis has not taken place.
• The Hunter might not yet have the full context about the environment. For example, running a threat hunt against a newly deployed set of systems and applications might prove to be challenging when running the hunt.
• The Hunter might not yet have the skill set required to uncover sophisticated attacks against technologies that the hunter is not very familiar with. For example, running a threat hunt expedition against a private Kubernetes environment while the hunter is unfamiliar with containerized deployments.
• Lack of data required for the hunter to perform a thorough investigation.
• The use of inappropriate techniques to uncover sophisticated attacks. For example, running basic searches to uncover advanced persistent threats (APTs).

As a threat hunter, you cannot be expected to know everything. Successful threat hunters spend an ample amount of time researching and, in many cases, trying new Tactics, Techniques, and Procedures (TTPs.) Cyber security is a dynamic landscape, and having valuable research time enhances the chances of uncovering advanced TTPs.
As a threat hunter, understanding the threat hunting process is essential. Let us take a look at the threat hunting process.

Threat hunting process

Defining a process helps threat hunters establish, conduct, and continuously improve the overall threat hunting practice and the individual threat hunt plays, increasing, over time, the probability of uncovering threats. Not only does it help improve the quality of threat hunts, but the process also incorporates other values that threat hunting introduces to the organization, such as updating existing or developing new detection and threat intelligence content.
Figure 3 shows in a high-level the threat hunting process, which starts by formalizing a hypothesis, followed by trying to prove the hypothesis. If the hunter could not prove the hypothesis, then try to improve it by updating the hypothesis details and searching again for the threat. If proven, then the threat has been uncovered. The hunter does not stop there; expand the scope and search for indicators on other systems to understand the attack’s magnitude and spread. The hunter would then engage the incident response team and document and share new content that would be helpful to the security monitoring and threat intelligence team.

Figure 3 Threat Hunting High-Level Process

What do you need to know to use this book?

This book has been written primarily for security, network, and systems professionals who are familiar with security tools and Python. The minimally-qualified reader should have the following skills:

• Security: Deploying and managing security tools such as network firewalls, endpoint security software and intrusion detection tools.
• Networking: good understanding of the TCP/IP stack and how network connections are established and terminated.
• Operating system: general Linux and Windows knowledge, including how the filesystem, event logging, and registry (in the case of Microsoft Windows) work.
• Statistics: basic knowledge on how to calculate mean, median and range for a data set.
• Machine learning: high-level understanding of what machine learning is.
• Programming: Basic Python programming knowledge.

Being able to run searches on tools such as Splunk, Elastic or similar would be useful, but optional.

Learn more about the book here.