The Standoff at PHDays VI: New Format for a New Reality

Positive Hack Days VI has come and gone, making it the perfect time to take a look back as well as see what is in store for next year. The theme of these PHDays was “The Standoff” – an idea that event organizers had wanted to explore for years, and in May came to fruition as PHDays VI СityF: The Standoff. No mere hacker game, this was a two-day battle of the best in cybersecurity.

Last year at PHDays V, event organizers tried to make the cybersecurity competition more like the real world. Per the event scenario, each capture-the-flag team represented a group in an imaginary country. The CTF teams accepted assignments (i.e., for hacking into different systems) through a DarkNet hacker marketplace. This year, the creators did one better by spicing up the hacker-heavy games with new participants: defenders and security operations center (SOC) specialists. This made the game much more lifelike and diverse – instead of only participants accustomed to being on “offense”, other specialists who build cybersecurity systems and investigate incidents were now represented as well.

“In terms of those participating, CTF games tend to be hacker-only. But what about the people responsible for security of real-world sites such as integrators, SOCs, infosec experts, and others? This enormous part of the IT picture was being overlooked. So at PHDays VI CityF: The Standoff, we wanted for as many people as possible to see this other side of security. We really wanted to experiment with this format where the experts and SOCs who defend systems, as well as the hackers who try to penetrate them, are each doing what they do best,” commented Boris Simis, Deputy CEO for Business Development at Positive Technologies.

Alexey Lukatsky, Security Consultant at Cisco, praised the event as “a milestone in how cybersecurity events simulate real-world conditions”. He continued, “CityF was different from run-of-the-mill wargames and scenario-driven CTF events because there were two sides to the battle. The effect is something like a red team vs. blue team approach, where one team is attacking and the other team is defending. The battle at CityF centered around a mini-city. The red and blue teams consisted of security experts with an impressive depth of experience that was exciting to watch.”

Rome wasn't built in a day

All events unfolded in CityF, whose virtual infrastructure is comparable to that of any large modern city in the real world. With a bank, phone company, electric company, office complex, and smart home, the city even had its own internet with special news and entertainment sites, as well as social networks.

Building the virtual city took six months. Thanks to the joint efforts of organizers and partners, all the models and stands were up and running in record time. Securing this infrastructure was staggeringly complex – perhaps not surprising, considering that the models and stands were designed to simulate real-world conditions.

Mikhail Levin, Product Promotion Manager at Positive Technologies and member of the PHDays organizing committee, noted: “In terms of computational power, this was a full-fledged city. Building it required colossal amounts of servers, networking equipment, and software. We built the city but we could not have done it without the help of our partners, Cisco and Check Point, which provided the necessary equipment and actively assisted with installation and setup.”

A number of new solutions were deployed: Cisco APIC (Cisco Application Policy Infrastructure Controller), Cisco Nexus 9000 switches, Cisco ASA 5585 firewalls, and Check Point Next Generation Firewall.

“Our relationship with Positive Technologies has been a long and rewarding one, both professionally and personally. We were delighted to help yet again at PHDays this year with the technical infrastructure. This task was more complex than in past years because of the much larger amount of networking equipment and servers needed for the simulation. But we rose to the occasion! It wasn't about money or ulterior motives for us, but simply helping good people to do a good thing,” said Alexey Lukatsky.

Besides industry titans, creating the virtual infrastructure of CityF required the efforts of startups as well. Lomoon provided its e-banking system to the CityF bank, for example, while most of the smart home stand was prepared by Advantech and ProSoft.

And last but certainly not least, the standoff participants – both attackers and defenders – also prepared in earnest. According to the event rules, the defending teams were provided with access to the infrastructure beforehand to configure protection in any way they found appropriate. They used the time-tested tools of their trade: application firewalls, network perimeter protection tools, attack detection and prevention systems, correlation analysis methods, and even SIEM. The range of vendors spanned the gamut: HP ArcSight, IBM QRadar SIEM, Microsoft Operations Management Suite, Qualys, Bot-Trek TDS, Security Onion, Balabit Shell Control Box, Windows Server Update Services, and IDS/IPS solutions were all in use.

Some of the defender and SOC teams could not resist using more creative methods of thwarting opponents. The False Positive team used several of their own in-house tools for incident investigation, for example, and You Shall Not Pass made use of a decidedly outdated Motorola С118 cell phone and an Ubuntu virtual machine for GSM network monitoring.

The defenders had hunkered down, but the attackers by contrast hurled themselves into battle practically barehanded. Armed with only laptops and standard hacking tools, they had Burp Suite for web application attacks, Nmap for IP network scanning, Wireshark for traffic analysis and capture, Cain & Abel for password recovery, and Metasploit for exploit development and debugging.

Keeping it real

The Standoff was challenging for the organizers, but equally so for the participants, for whom the rules and scenario were absolutely new. Instead of abstract tasks, the conditions in CityF were based on the real world. As noted by Timur Yunusov, Head of Banking Security at Positive Technologies and member of the PHDays organizing committee, “Despite all the advantages of the classic CTFs that we know and love, they have drawbacks too. CTFs can turn into a contest of riddles and solving artificial tasks, rather than simulating reality.” The goal of the event organizers was to show how live systems are penetrated and protected in practice (and in a way that is understandable to non-hackers). The tasks for hackers included stealing money from the city bank, providing themselves with unlimited cell phone use, causing an accident at a power dam, and turning off the lights in a smart home. Meanwhile, the defender and SOC teams had to withstand the onslaught. Or in other words, just like real life.

While some glitches did arise during the contest, they were resolved and participants were generally thrilled with the experience gained at the competition.

“There were some organizational hiccups due to the ambitious scale of the event, as well as the new rules, but things got ironed out and we're charged up for the year ahead. Questions about the game rules and scoring were ultimately resolved at the awards ceremony,” said Ivan Melikhin, Technical Director at Informzaschita, which sent two teams to CityF (izo:SOC and weIZart, consisting of defenders and SOC, respectively).

Still, some of the participants were left wanting more, specifically to compete with their peers, not just with hackers. And the traditional CTF format found supporters as well.

“Impressions were both positive and negative: the concept is interesting and practical tasks were awaiting the participants. But player interactions and scoring/penalties (especially for defenders) could have been better thought out in advance,” suggested Rdot team member Omar Ganiev. filthy thr33 team member Kirill Shilimanov concurred:“It was a bit of a mixed bag for me. The first day was effectively wasted for the attackers, since there was no access to services at all. When the services opened up and the attacks started flying, the fun really started. Kudos to the organizers for setting up services that were complicated and interesting.”

The 30-hour onslaught

The standoff lasted approximately 30 hours. During that time, five teams of defenders and three SOC teams defended five targets. The judges recorded between 3,000 and 20,000 security events at each target. A total of around 200 serious attacks were made, most of which had significant outcomes.

In 99% of cases, attacks were concentrated at the perimeter of the targets. And just like in real life, web attacks formed the most common vector. This was no surprise for the defenders, who had anticipated this approach and planned accordingly.

“In protecting the office infrastructure, we paid special attention to the web servers. This decision paid off: the hackers used a wide range of penetration testing tools, and although the IPS handled exploits aimed at the operating system, sophisticated attacks on web servers and application logic meant that we had to resort to manual analysis of WAF, web server, and operating system logs,” said Dmitry Berezin, Information Security Expert at Croc and Green team member.

Perhaps more surprising for the defenders was what the attackers did not use. Although popular in practice, social engineering was all but ignored at PHDays. Only one team took advantage of opponent carelessness to photograph usernames and passwords for the defenders' forum. No serious incident resulted. “We were prepared to see social engineering but the attackers barely even attempted to use it,” lamented Vladimir Dryukov, Solar JSOC Director at Solar Security and False Positive team member.

As the defenders admitted later, they were prepared for the worst. Armed to the teeth and placing traps, they waited for attackers to exploit vulnerabilities in apps, web apps, operating systems and services, as well as configuration errors. But things turned out rather differently.

“Our team had everything protected: workstations, servers, corporate email, domain, e-banking, video conferencing, document management, and instant messaging. But many of our defenses weren't even needed, since the hackers didn't make it to the internal network. We did not see any Golden Ticket or Pass-the-Hash attacks on Kerberos authentication or any trojan/backdoor attacks. Nor did the hackers step into any of the honeypots that we had set. And none of them even tried to hack our vulnerable proFTPD server,” shared Inna Sergienko, Head of Department at AST Group and AST team member.

The False Positive team boasted that attackers were able to plant only one flag on the infrastructure protected by them: “The organizers added seven new services on the perimeter simultaneously, and the defenders and ourselves were a bit late with the security profile for the last system because of the time needed for the first six. But the attackers' victory was short-lived, because a few minutes later we restored the system's state and neutralized the threat.”

Collaboration between the defenders and SOC teams showed great value. In the evaluation of the judges, all the SOC teams had the most complete picture of events at the targets, while the defenders were busy responding quickly to incidents. For instance, when the industrial system defenders shut down protection systems (as required by the game scenario), the relevant SOC team monitored attacker actions from start to finish. In real life, this level of awareness would allow thwarting the attacks even without the help of protection tools.

“The Informzaschita team protected a power dam as well as two substations (500 and 10 kV). According to the scenario, on the evening of the first day we began to weaken the defenses, and by the end of the day, almost all the protection systems were deactivated. Only the SOC was performing security monitoring. When the target was under protection, not a single infrastructure attack was successful. All the breaches and flooding occurred when the infrastructure was not protected,” boasted Ivan Melekhin.

So what were the attackers able to accomplish?

• Steal account credentials, including several domain user accounts
• Attack industrial control systems (discharge water, disable and burn power lines)
• Penetrate the process control network of an automated control system by exploiting corporate network vulnerabilities
• Perform network attacks on smart home systems (disable equipment via the internet)
• Steal money from the bank (approximately 22,000 rubles) and obtain card information
• Steal, and in some cases delete, backups of system files, disks, and archives belonging to the CorpF office
• Perform GSM/SS7 attacks and steal money via fake USSD requests
• Attack both the attacking and defending teams (social engineering was used to steal the password of the defenders' forum, while the defending Vulners team broke into computers of the hackers)
• Deface a number of websites, including the CorpF office website
• Detect an insider (CorpF office employee)

Results and reflections

PHDays is an excellent example of how IT security professionals can ensure exceptional security for sensitive infrastructure without interfering with operations. The ultimate goal of the hackers – to capture the city's domain and win the contest – was not achieved by any of the teams. This result was a surprise for the organizers, who had expected a hacker victory. The jury could not name clear winners, so awards went to the hacker teams that distinguished themselves during the game. Defender and SOC teams received awards in various categories.

Alexey Kachalin, Deputy Director for Business Development in Russia at Positive Technologies and member of the PHDays organizing committee, was enthusiastic about the results: “Everyone won, both the organizers and the participants. The event was one-of-a-kind and it's difficult to devise precise rules for such a massive game until it has actually been played out. We hope that this year's participants will return next year and help us to prepare the competition. We'll be listening to both attacking and defending teams to design rules and a format that allows everyone to demonstrate their strengths.”

Kachalin's positive assessment of PHDays VI was shared by Viktoria Alekseeva, PHDays Forum Director: “PHDays is fundamentally about people and the enthusiasm that makes this event tick. Over 100 people worked for the entire year to make the forum one truly to remember. Every year we, the organizers, are outdoing ourselves and setting new records. The event was a great success by any measure, with 4,200 participants in all! Thank you to everyone who helped us to organize PHDays!”

While the slogan and theme for next year's event have not been decided yet, the organizers are excited about expanding on the “standoff” concept. Mention has been made of gameplay twists involving more action, social engineering, and contingent events: employee firings, business process changes, and daytime and nighttime scenarios. And of course, CityF will continue to grow and become even more “populated”.

“The world has changed so much in recent years, with cybersecurity becoming a part of everyday technologies. Connected infrastructure and the Internet of Things have made the threat picture more complex, attacks more sophisticated, and the potential damage more significant. Simply protecting systems in the old way won't cut it anymore. To keep pace and improve our protection capabilities, we ourselves must work harder than ever. That's why PHDays have changed to match. At the conference this year, we have given industry professionals the opportunity to take part in this standoff and acquire real-world experience in defending vital infrastructure. Seeing their excitement after 30 hours of non-stop battle is the best reward for us. We'll continue to push the envelope next year, by bringing professional penetration testers and other IT pros into the game,” outlined Yury Maximov, CEO of Positive Technologies.

Many of the partners and teams have already confirmed their intention to take part in next year's sequel. Alexey Lukatsky, for example, awaits the chance for Cisco to contribute to PHDays VII: “The game format has enormous potential and CityF has set a very high bar for CTF games going forward. So barring major geopolitical deteriorations, Cisco will be delighted to be the technology partner for PHDays again next year. We think we could also offer speakers and – once we discuss and hammer things out inside the company – possibly even defenders for a segment of CityF.”

While it is too early to say much about PHDays VII, planning is already underway. So, attackers and defenders, don't let your guard down – the biggest challenges are yet to come!