Analysis of the Linux backdoor used in freenode IRC network compromise

This is a local mirror of a blog written by me and originally published by NCC Group.

This project was a highlight of my time at NCC Group, and I enjoyed collaborating with Pete Beck during analysis.

The page was referenced by in a document “Analysis of the VENOM Linux rootkit” written by EGI CSIRT. Their report contains details we chose not to publish at the time.

Background

freenode is a large IRC network providing services to Free and Open Source Software communities, and in September the freenode staff team blogged about a potential compromise of an IRC server. NCC Group’s Cyber Defence Operations team provided pro bono digital forensic and reverse engineering services to assist the freenode infrastructure team with their incident response activities.

In this post we discuss a subset of the information we documented about one of the components involved in the compromise, specifically a Linux backdoor with some interesting functionality and features.

One difficulty all attackers face after compromising a system is how to retain control over a long period of time in a stealthy manner.

Backdoor tools which listen for incoming connections can be easily identified by a port scan or by listing open sockets.

Tools which periodically connect outbound to a server are usually limited to a small number of addresses or a predictable domain generation algorithm.

The backdoor discussed in this post avoids these issues by using a novel method for recognising specially generated incoming packets, bypassing most typical host firewalls and enabling the attacker to change IP address without losing access.

Mechanism of operation

The backdoor has a number of components which provide the attacker root shell functionality or remote access to any file. However, the most interesting feature is that it is triggered by “magic” TCP packets which contain a certain combination of header values.

The overall backdoor package found on the server consists of:

• A kernel module, which listens for magic packets and triggers the user-mode helper.
• A user-mode helper, which connects outbound from the server and contains code for remote shell or file access functionality.
• A script, ensuring everything is loaded at boot.
• A second user-mode binary, responsible for various housekeeping activities.

Overview

A successful interaction with a compromised system is illustrated below. Three magic packets are required to trigger the backdoor; this initiates an outbound connection from the compromised system to the sender of the magic packets.

A full TCP connection is not required to activate the backdoor - three individual packets for the port knocking sequence will successfully start the sequence below.

Data transmitted between the attacker and compromised server is encrypted by RC4 at all times.

In the sample analysed by NCC Group the same fixed string is used to initialise RC4 for both transmit and receive.

This means that decryption of the attacker’s communication is possible once the RC4 key has been retrieved from a compromised system. However, it is likely that this RC4 key will be different each time the backdoor is placed on a new server.

Triggering the backdoor

The headers of all incoming TCP packets are analysed to look for a specific combination of values - we refer to these as magic packets.

Specifically the kernel driver will add the source port and sequence number together and compare them to a fixed value. We are not disclosing the value at present to allow freenode further time to continue their investigations.

Once three TCP packets have been received with the correct combination of source port and sequence number the user-mode binary is called with the source IP (from the magic packet) and a destination port. This port number is derived from the window size in the third magic packet, minus 8192.

An overview of the TCP headers is shown below with the trigger and callback ports highlighted.

Persistence

On the server analysed by NCC Group persistence for the rootkit was achieved by adding to the script /lib/lsb/init-functions.

This file is included by every Linux Standard Base (LSB) compliant init script and provides a number of essential functions. The additions were responsible for running the attacker’s shell script, which is described below.

Main components

Kernel driver

The name of the kernel module when loaded is ipt_ip_udp.

This kernel driver uses a netfilter firewall hook to inspect packets arriving at the compromised system. This hook is configured in the PREROUTING chain with the highest priority so will evade normal packet filtering in the INPUT chain. This means that even if a normal firewall rule would drop the magic packet, it will still be seen by the backdoor kernel driver.

There are no checks for the destination port in the kernel driver therefore magic packets can be sent to any port.

The three packets do not need to be identical - the source port and sequence number can be randomised per-packet as long as these fields add to the correct value. In addition it is possible to generate the three magic packets very slowly with a large interval between each packet.

The combination of magic values, an arbitrary destination port and the potential for long delays between packets makes it difficult to implement an effective IDS signature for this traffic.

A proof-of-concept trigger script is shown below which uses Scapy to do IDS avoidance, please note that as previously discussed we have omitted the magic value.

#! /usr/bin/env python

import random
from scapy.all import IP,TCP,send
import argparse

########
# Author: NCC Group, Cyber Defence Operations (CDO)
#   Date: September 2014
#
# Sends three TCP packets with the correct header values to trigger
# a connect back on the specified port.
#
# This script implements simple randomness for the "magic" header
# values to demonstrate how difficult it would be to signature in IDS.
########

parser = argparse.ArgumentParser(description='Trigger the backdoor')
parser.add_argument('TARGET_IP', metavar='TARGET_IP', type=str,
                   help='IP to send magic packets to')

args = parser.parse_args()

# Target port can be any valid port.  iptables filtering does not apply
# if it's in the normal INPUT chain.  This could be randomised per-packet.
TARGET_PORT = 80

CALLBACK_PORT = 1234

for n in range(1, 4):
    # To trigger backdoor the source port + sequence must add up to <Magic Value>
    # Backdoor will connect to us on window - 8192
    source_port = random.randint(1024, 2048)
    sequence = <Magic Value> - source_port # Value not disclosed at this time
    packet = IP(dst=args.TARGET_IP)/TCP(dport=TARGET_PORT,sport=source_port,seq=sequence,window=8192 + CALLBACK_PORT)

    print "[+] Sending magic packet {} of 3 to {}:{} (sport: {}, seq: {})".format(n, args.TARGET_IP, TARGET_PORT, source_port, sequence)
    send(packet)

When three “magic” packets have been successfully received the kernel module will trigger the user-mode helper binary.

User-mode helper

This binary opens a connection to the attacker in order to provide access to the system. It provides backdoor functionality with following features, all executed as root (the desired function is selected by the attacker after an outbound connection has been established).

• Execution of a shell - input and output redirected to the network socket.
• Arbitrary file read - output to network socket.
• Arbitrary file write - input from network socket.

This file was located on the system as /bin/dh, this path is hard coded into the kernel module but could easily be changed. However, it changes the name displayed in process listings to hald-runner by overwriting the value stored at argv[0].

The helper initiates a connection to the IP address and port specified as command-line arguments. This is likely to succeed on many servers as firewall rules often permit outbound traffic with fewer restrictions.

Once connected the helper expects a password which is hashed using DES crypt() and compared to a built-in value.

When the correct password has been provided the helper sends a fixed four byte value indicating success. The attacker then sends four bytes which relate to the commands offered by the backdoor, the most useful of which is a root shell. This shell is executed with a number of environment variables that reduce how much logging information is stored, for example HISTFILE is unset.

Detection rules

The following rule can be used with IDS solutions that accept Snort format signatures. Due to the construction of the magic packet it is impossible to create a basic IDS rule; however this could be achieved with scripts in either Suricata or Bro.

The rule provided signatures fixed strings from the kernel module after encryption by RC4. These relate to the “hello” packet and the packet sent after successful authentication.

alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg: "Unknown Linux Backdoor"; content: "|b5a46fce2166|"; depth: 6;flow:established, to_server; sid:1;)

Caveats:

There is a risk of a false positive as only six bytes are being matched in any TCP stream.

It is important to note that the IDS rules shown above will only detect the variant we have currently analysed. It would be trivial for the attackers to change the RC4 encryption details; therefore we recommend the usage of host based scanning instead.

Conclusion

Whilst the handshake and data security mechanisms are arguably well designed the persistence mechanism isn’t in any sense stealthy. This particular rootkit would be easily detectible using tools as Tripwire and Rootkit Hunter.

Attribution to a particular group is not possible based on the artefacts provided to NCC Group.

While the techniques used are well engineered they are certainly not unique. For example netfilter hooks were discussed in the context of rootkits back in a 2003 Phrack article titled “Kernel Rootkit Experiences”. Similarly port knocking and RC4 encryption for concealment and transport security are not highly sophisticated yet are sound approaches if developing a rootkit.