Category: Security

Who’s gonna run this thing?

Not to be contrarian, but – well – let me be contrarian. Rant coming. TL;DR – there needs to be a free version of z/os & it’s siblings sooner than later, to not do this is to potentially starve the platform out of existence as we know it.

I don’t think, for a moment, that when people (looking to learn) ask for a mainframe-in-the-cloud type experience, that they are asking for Linux One or Linux on Z, they want a z/OS-type platform on which to learn and play. Otherwise it’s just Ubuntu or SUSE, like I can run on my laptop. Except for anyone but the kernel developers and a few select others, most would never know the difference (outside of performance, perhaps). It’s certainly not the classic z/OS / VM / tpf / etc. experience that most mainframers talk about daily.

As for the other offerings, none are the same (or even really the same sport) as having your hands on a “real” z/OS (or z/VM, etc.) mainframe – the closest of which, for people not buying hardware, would be z/PDT – the System Z hypervisor that runs on Linux. By real, I mean a system on which one can provision storage, configure parmlibs, install software with SMP/E, develop load modules, configure platform software and tcp/ip and more: IPL the system, crash it, figure out how to build a stand alone dump; figure out how to read that dump, get the system up and running again; gen a system from scratch; install an upgrade with a serverpac and so on. If you don’t know what some of those things meant, good, that’s the point.

Until IBM figures out that they’re losing opportunities because of this, I fear the platform is going to get harder and harder to support and defend. Most (if not all) of the cloud – or public offerings on Z (again, not talking Linux) are for developers. Master the mainframe, z Systems cloud trial (RD&T for z “Test drive development tools”) etc.

Herein lies the rub. Where will the next generation of Storage Engineers & System Programmers come from? Who will write the DFSMS/ACS routines, or write the assembler-based system exits? Who will wade through SMP/E reading hold data and figuring out how to fix or remove a wonky PTF that didn’t apply correctly or went PE? Who will configure the VTAM / 3270 applications and the intricate work tweaking TCP/IP net filter and ATTLS? Who is going to do the detailed capacity / performance analysis and tuning of the storage, wlm, cpus and so on? To say nothing of the gargantuan task of securing these beasts.

These are skills with theoretical backgrounds in many other disciplines, but the specifics and technical difficulties pertaining to using those skills on this platform are non-trivial. People need time, mentors and opportunity to learn it. That opportunity is nearly gone – or unrealistically appraised at this point.

Sure there are a few colleges which teach these skills, and the tried and true way of apprenticeship still works if you can get it, but how prevalent is that? Moreover, why would a fresh-out-of-school person take a chance on an OS/platform that they’ve never gotten to put their hands on? In today’s world, they can get a free/inexpensive version of every. single. OS. on. the. planet. for personal use (Microsoft & VMWare development and full evaluation versions, Linux is open source and free, as are the BSDs) – except for z/OS and it’s time-tested brethren. Why is that? How does that secrecy help generate buzz and the next generation of loyal mainframers?

To ask the fresh, talented, next generation of techies to go to work in a mainframe shop – or to go to a school to learn mainframe is asking them to take a gigantic leap of faith. They have the opportunity to be hands-on with 99.999% of the tech out there before they leave high school; but somehow, someone expects that they’ll self-select into becoming a z/OS sysprog? Why would they? Not having a clear track to this pipeline is the single biggest security issue and threat to this platform there is. Companies will hire the remaining few, then outsource, then divest – unless we (and IBM) start driving interest by making the platform (the whole platform, not just the development bits) available to anyone who wants to play with it.

It’s a huge opportunity missed, and I hope it changes soon. One of the hardest things to see is, after giving a talk at a non-mainframe centric conference, people who come and ask how they can get involved directly. You can’t. Unless you go to work or school somewhere special, or are willing to lay out several thousand out of your own pocket – you just have to admire it from afar. And that’s too bad, because it’s a kick butt OS and a super-challenging ecosystem that the unbelievably sharp new technologists would sink their teeth into. They’d eat it up. Many were programming from the time they could walk and computers just. make. sense. But this computer, with it’s super configurable and somewhat non-forgiving “you better know what you’re doing or how to figure it out” practices and protocols, requires time and a steep ramp-up period to become proficient. It has to start now.

When you go to those job fairs, conferences, or just on the next marketing push – come up with a way to give away for free or cheap a copy of these OS’s that run on a hypervisor like zpdt for people to just play with, destroy, hack, but mostly learn. Will there be some negative consequences ? Maybe. But fear not, the rest of the OS providers who have gone before in this space have already figured that out, with a mix of bug bounties, licensing agreements and lawyers. But that’s a topic for a different post.

Is that ransomware on your mainframe?

Next week at SHARE – San Jose, I’m giving a talk on ransomware on z/OS.  I’ve been asked multiple times if I thought ransomware could happen on Z, is it possible: Unequivocally yes.  Come see this talk and watch a live demonstration of how this might work.   If you are responsible for mainframe security, work for a company with a mainframe, or just want to better understand the landscape of this particularly insidious threat, don’t miss this talk.

Ransomware is a combination of 3 basic moving parts:

  1. A delivery mechanism (Phishing email, infected web page, malicious program).
    • This infects the user’s machine – allowing for sniffing of credentials and network traffic.  It can then upload a payload to the host system.
  2. File cataloging and encrypting.
    • Just what it sounds like – find files of interest, encrypt them in place, destroy the local copy of the key.
  3. Some type of Command & Control (or at least reporting) – centralized server.
    • Some means of transferring the keys out to the bad guys. Also, a way for the affected users to connect and pay ransom. (This is not strictly required, but does have precedent.  Steps #1 & #2 can happen regardless of the system’s ability to ‘phone home’ ).

We will also look at how to attempt to mitigate this catastrophic event, as well as ideas about how to recover from it.  Items such as two-factor authentication, proper ICSF / RACF security controls, egress filtering and intrusion detection.

Mainframes – Java – Deserialization

I was asked a week or so ago whether or not I thought z/OS would be susceptible to the types of Java deserialization attacks we’ve seen (a great primer from Fox Glove Security).   Of course!, I replied.  However, I don’t like unsubstantiated claims – so I built this POC:

java

It uses the basic ysoserial payload generator found on Github.   The SerialTestPlain.java file I use to test is from a blog here:

The source is simple:

Simple enough right?    Java on the mainframe is basically Java anywhere.  The only major gotcha (which should come as no surprise to anyone) are with issues of character translation  EBCDIC<->ASCII.   In this case, the ysoserial jarfile I built on x86 and just binary copied it to OMVS and that worked out of the box.

Other times I’ve had to use an a2e / e2a custom decoder – just depends on the implementation.  Currently working to test the JBoss exploits and modify them, if need be, in MSF for z/OS.  More as that unfolds!

NOTES:

While testing this POC first on x86, I kept running into an error like this:

The above mentioned blog helped – Basically Java 1.8u72 (since last December) needs to have the most current version of ysoserial, and use the CommonsCollections5 in order to work (and it does).   Prior versions of Java work just fine with the Release Version (0.04) of ysoserial.

Also, aside from fixes that are library based (like the Adobe Commons Collections one used here), most fixes to this bug have to happen in customized code, often written by organizations.   That makes this vulnerability particularly ugly and potentially difficult to mitigate.

Things I’ve Learned (and things to come)

I started writing a list of topics I’ve learned, some in excruciating detail, some just enough to know where to look for further details (trust me, that is no small feat).

I’m writing this not only as a way of keeping me honest on those days when nothing goes right, but also as a way to incentivize those among you who, like myself, have an insatiable desire to learn – and the tenacity to “figure it out.”

In most organizations, the below is accomplished by teams of people.   Some of the items (gen’ing a system from scratch, for instance – or setting up SMP/E, SMS, etc. from scratch – might never be a part of even a very senior mainframer’s repertoire).  I wanted to see what it would take to go it alone.

My plan is to build this page out with good links to relevant data – and/or if I get really ambitious build some how-tos on the finer points, if there is interest.  Real language how and wherefores.

THINGS I’VE LEARNED  (since I started a deep technical dive into mainframes)

  • RACF
    • password construction / algorithms
    • user profile management
    • using callable services
    • TSO commands for many common elements
    • building certificates
    • importing certificates
    • user certificates
  • Storage Management
    • Configuring SMS from scratch
    • Initializing devices
    • using DFDSS to move, backup and restore files
    • using IDCAMS for catalog and VSAM file management
    • what the eff a VSAM file is
    • how to allocate datasets
    • different access methods (qsam, bdam, etc)
    • what the hell a cylinder (or track) is
    • How big a mod 3,9,27,54 EAV are
    • Initializing volumes
    • labeling and initializing tapes
    • troubleshooting space abends (D37,B37,E37)
  • System Programming
    • SMP/E updates, installation, management
    • Building jcl from scratch
    • Configruing IPL parms, parmlibs, and startup shutdown procs from scratch
    • checking system resources
    • How apf authorization works
    • building a lnklist
    • building an lpalib
    • building a multi-tier catalog system
    • taking SVC dumps
    • Getting a trace of a component
    • reading said trace
    • using IPCS
    • troubleshooting failed ipls
  • z/OS crypto
    • keychain management
    • key management
    • password configuration
  • Assembler Programming
    • What a load module is
    • What a program module is
    • how to disassemble them
    • writing assembly
    • using 4 different debuggers
    • patching programs the hard way
    • building a ZAP
    • Using Macros
    • Compiling, Linking
    • Callable service usage
    • What the hell Language Environment is
  • SDSF
    • Jes2 job management
    • Reading a job log
    • managing output
    • managing active jobs
    • reconfiguring SDSF screens
  • JES2
    • NJE
    • Job management
    • Configuration files
    • parmlib entries
  • TCP/IP
    • TN3270 configuration
    • FTP configuration
    • FTP/S configuration
    • TN3270 + ssl configuration
    • Policy Agent configuration
    • TSO / USS tcp/ip commands
  • OMVS/USS
    • creating zfs filesystems
    • dbx debugger
    • compiling and linking with xlc
    • What the hell Language Environment is
  • z/OS operations
    • Many console commands (devices, stg)
    • How to research a WTOR
    • Vtam commands
    • tcpip commands
    • device commands
  • ISPF
    • Panel customization
    • DDLIST wizardry
    • Editor fine-tuning
    • Keylist modification
    • using the editor – line & main command sets

THINGS LEFT TO LEARN

  • VTAM
  • REXX
  • memory areas and control blocks in depth
  • So much more (work in progress)
  • SMF
  • z/OMSF
  • policy agent
  • ATTLS (in depth)
  • Coding Exits
  • Hardware configuration
  • Cross memory operations (PC, SRB, etc)
  • much more

A logical first step

The first z/OS exploit module in the Metasploit Framework, landed last Friday.

This is an exploit which takes advantage of a default or poorly configured FTP server. And, it requires valid credentials.  However, given this (and it’s a very common configuration), you will be presented with a very nice Unix shell – allowing for deeper testing of the system.

This is how it begins:  attackers look for low hanging fruit. The evolution of pentesting tools for the mainframe has to start somewhere, and this is the first concrete milestone in what has been an ongoing journey. Many x86 exploits are simply taking advantage of default configurations or poorly written code. z/OS is no different – it can suffer from neglected configurations and defaults like all other OS’s. So, that’s where I started. From here, we’ll build on default configuration exploits and work up and on through binary / code exploitation. Baby steps.

At any rate – I’m very proud of how this turned out. Thanks to those who helped in the prototyping phases (SoF & others noted within the exploit) – and as always, the super helpful folks on the MSF teams. For those of you testing mainframe systems – hopefully it’ll help red teamers with an easy win and start the conversation on securing the big iron.

There are more goodies in the queue, so stay tuned!

PR # 6834 – Authorized FTP JCL exploit for z/OS