An interview I gave regarding the state of mainframe security. Pt. 1.
Well – the time has come to start doing what I love to do full time. I couldn’t be happier to announce that I’m working with RSM Partners, Ltd to help bring their amazing mainframe services, security & software business to North America. This is going to be a great challenge and a great opportunity. Super excited to work with all the amazingly talented people at RSM.
Setting up a vm on top of linux which communicates via a TAP adapter (on the 10.1.1.x network), I wanted iptables to prevent brute forcing to both the host ports (here 22 for ssh) and ports forwarded to the vm (here 443) as they are exposed to the internet. This little snippet does both by using iptables’ conntrack – simply more than 3 connections to either of those ports mentioned inside 60 seconds will lock that source IP out for 60 seconds.
The offending connections are marked in the nat table – prerouting chain, and checked (depending on whether forwarded or direct to host) in the filter table forward / input chains respectively. Logging is optional, you may choose to just DROP them once you’re confident on your ruleset.
Here’s a sample of the final ruleset I made:
:PREROUTING ACCEPT [14138:1459925]
:INPUT ACCEPT [9963:618428]
:FORWARD ACCEPT [4175:841497]
:OUTPUT ACCEPT [8303:1284925]
:POSTROUTING ACCEPT [12453:2124922]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [9184:1653115]
:BFLOG - [0:0]
:FWDLOG - [0:0]
:INPLOG - [0:0]
:OUTLOG - [0:0]
# input chain
-A INPUT -i lo -j ACCEPT
-A INPUT -m connmark --mark 0x5 -j BFLOG
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -j INPLOG
# forward chain
-A FORWARD -m connmark --mark 0x5 -j BFLOG
-A FORWARD -s 10.1.1.2/32 -j ACCEPT
-A FORWARD -d 10.1.1.2/32 -j ACCEPT
-A FORWARD -j FWDLOG
# new chain to log brute force attempts
-A BFLOG -j LOG --log-prefix "BFDROP:" --log-level 6
-A BFLOG -j DROP
# new chain to log forward attempts
-A FWDLOG -j LOG --log-prefix "FWDDROP:" --log-level 6
-A FWDLOG -j DROP
# log not accepted input attempts
-A INPLOG -j LOG --log-prefix "INPDROP:" --log-level 6
-A INPLOG -j DROP
:PREROUTING ACCEPT [2:128]
:INPUT ACCEPT [2:128]
:OUTPUT ACCEPT [37:2477]
:POSTROUTING ACCEPT [33:1984]
:ZNAT - [0:0]
# prerouting chain, nat table
-A PREROUTING -p tcp -m tcp --dport 443 -m conntrack --ctstate NEW -m recent --set --name VMPORT --mask 255.255.255.255 --rsource
-A PREROUTING -p tcp -m tcp --dport 443 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 4 --name VMPORT --mask 255.255.255.255 --rsource -j CONNMARK --set-xmark 0x5/0xffffffff
-A PREROUTING -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -m recent --set --name SSH --mask 255.255.255.255 --rsource
-A PREROUTING -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 4 --name SSH --mask 255.255.255.255 --rsource -j CONNMARK --set-xmark 0x5/0xffffffff
-A PREROUTING -j ZNAT
# postrouting for outbound nat/masquerade
-A POSTROUTING -o eth0 -j MASQUERADE
# table for inbound destination natting
-A ZNAT -p tcp -m tcp --dport 22 -j RETURN
-A ZNAT -p tcp -j DNAT --to-destination 10.1.1.2
For debugging, I cannot recommend highly enough using the TRACE target on the raw table (PREROUTING chain).
iptables -I PREROUTING -t raw -p tcp --dport 22 -j TRACE
Will show in your log, every stop along the iptables chains for every packet, including which rule or policy was acted upon it to get it to it’s final destination and shape. Don’t forget to remove it when you’re done!
Also, install the actual conntrack utility to see the connection tracking tables.
Update: The PR was accepted. Update your Metasploit installation and have a look!
Version 1 of an APF privilege escalation (Requires Creds) of a metasploit module has been submitted for inclusion. This version has no bells or whistles.
You can view it here:
Here’s the presentation I gave at sp4rkcon:
Thanks to the folks at ConFoo.ca for hosting my guest post to their blog:
Not to be contrarian, but – well – let me be contrarian. Rant coming. TL;DR – there needs to be a free version of z/os & it’s siblings sooner than later, to not do this is to potentially starve the platform out of existence as we know it.
I don’t think, for a moment, that when people (looking to learn) ask for a mainframe-in-the-cloud type experience, that they are asking for Linux One or Linux on Z, they want a z/OS-type platform on which to learn and play. Otherwise it’s just Ubuntu or SUSE, like I can run on my laptop. Except for anyone but the kernel developers and a few select others, most would never know the difference (outside of performance, perhaps). It’s certainly not the classic z/OS / VM / tpf / etc. experience that most mainframers talk about daily.
As for the other offerings, none are the same (or even really the same sport) as having your hands on a “real” z/OS (or z/VM, etc.) mainframe – the closest of which, for people not buying hardware, would be z/PDT – the System Z hypervisor that runs on Linux. By real, I mean a system on which one can provision storage, configure parmlibs, install software with SMP/E, develop load modules, configure platform software and tcp/ip and more: IPL the system, crash it, figure out how to build a stand alone dump; figure out how to read that dump, get the system up and running again; gen a system from scratch; install an upgrade with a serverpac and so on. If you don’t know what some of those things meant, good, that’s the point.
Until IBM figures out that they’re losing opportunities because of this, I fear the platform is going to get harder and harder to support and defend. Most (if not all) of the cloud – or public offerings on Z (again, not talking Linux) are for developers. Master the mainframe, z Systems cloud trial (RD&T for z “Test drive development tools”) etc.
Herein lies the rub. Where will the next generation of Storage Engineers & System Programmers come from? Who will write the DFSMS/ACS routines, or write the assembler-based system exits? Who will wade through SMP/E reading hold data and figuring out how to fix or remove a wonky PTF that didn’t apply correctly or went PE? Who will configure the VTAM / 3270 applications and the intricate work tweaking TCP/IP net filter and ATTLS? Who is going to do the detailed capacity / performance analysis and tuning of the storage, wlm, cpus and so on? To say nothing of the gargantuan task of securing these beasts.
These are skills with theoretical backgrounds in many other disciplines, but the specifics and technical difficulties pertaining to using those skills on this platform are non-trivial. People need time, mentors and opportunity to learn it. That opportunity is nearly gone – or unrealistically appraised at this point.
Sure there are a few colleges which teach these skills, and the tried and true way of apprenticeship still works if you can get it, but how prevalent is that? Moreover, why would a fresh-out-of-school person take a chance on an OS/platform that they’ve never gotten to put their hands on? In today’s world, they can get a free/inexpensive version of every. single. OS. on. the. planet. for personal use (Microsoft & VMWare development and full evaluation versions, Linux is open source and free, as are the BSDs) – except for z/OS and it’s time-tested brethren. Why is that? How does that secrecy help generate buzz and the next generation of loyal mainframers?
To ask the fresh, talented, next generation of techies to go to work in a mainframe shop – or to go to a school to learn mainframe is asking them to take a gigantic leap of faith. They have the opportunity to be hands-on with 99.999% of the tech out there before they leave high school; but somehow, someone expects that they’ll self-select into becoming a z/OS sysprog? Why would they? Not having a clear track to this pipeline is the single biggest security issue and threat to this platform there is. Companies will hire the remaining few, then outsource, then divest – unless we (and IBM) start driving interest by making the platform (the whole platform, not just the development bits) available to anyone who wants to play with it.
It’s a huge opportunity missed, and I hope it changes soon. One of the hardest things to see is, after giving a talk at a non-mainframe centric conference, people who come and ask how they can get involved directly. You can’t. Unless you go to work or school somewhere special, or are willing to lay out several thousand out of your own pocket – you just have to admire it from afar. And that’s too bad, because it’s a kick butt OS and a super-challenging ecosystem that the unbelievably sharp new technologists would sink their teeth into. They’d eat it up. Many were programming from the time they could walk and computers just. make. sense. But this computer, with it’s super configurable and somewhat non-forgiving “you better know what you’re doing or how to figure it out” practices and protocols, requires time and a steep ramp-up period to become proficient. It has to start now.
When you go to those job fairs, conferences, or just on the next marketing push – come up with a way to give away for free or cheap a copy of these OS’s that run on a hypervisor like zpdt for people to just play with, destroy, hack, but mostly learn. Will there be some negative consequences ? Maybe. But fear not, the rest of the OS providers who have gone before in this space have already figured that out, with a mix of bug bounties, licensing agreements and lawyers. But that’s a topic for a different post.
Next week at SHARE – San Jose, I’m giving a talk on ransomware on z/OS. I’ve been asked multiple times if I thought ransomware could happen on Z, is it possible: Unequivocally yes. Come see this talk and watch a live demonstration of how this might work. If you are responsible for mainframe security, work for a company with a mainframe, or just want to better understand the landscape of this particularly insidious threat, don’t miss this talk.
Ransomware is a combination of 3 basic moving parts:
- A delivery mechanism (Phishing email, infected web page, malicious program).
- This infects the user’s machine – allowing for sniffing of credentials and network traffic. It can then upload a payload to the host system.
- File cataloging and encrypting.
- Just what it sounds like – find files of interest, encrypt them in place, destroy the local copy of the key.
- Some type of Command & Control (or at least reporting) – centralized server.
- Some means of transferring the keys out to the bad guys. Also, a way for the affected users to connect and pay ransom. (This is not strictly required, but does have precedent. Steps #1 & #2 can happen regardless of the system’s ability to ‘phone home’ ).
We will also look at how to attempt to mitigate this catastrophic event, as well as ideas about how to recover from it. Items such as two-factor authentication, proper ICSF / RACF security controls, egress filtering and intrusion detection.