Category: Tips

Iptables brute force protection w/ nat

Setting up a vm on top of linux which communicates via a TAP adapter (on the 10.1.1.x network), I wanted iptables to prevent brute forcing to both the host ports (here 22 for ssh) and ports forwarded to the vm (here 443) as they are exposed to the internet. This little snippet does both by using iptables’ conntrack – simply more than 3 connections to either of those ports mentioned inside 60 seconds will lock that source IP out for 60 seconds.

The offending connections are marked in the nat table – prerouting chain, and checked (depending on whether forwarded or direct to host) in the filter table forward / input chains respectively. Logging is optional, you may choose to just DROP them once you’re confident on your ruleset.

Here’s a sample of the final ruleset I made:

For debugging, I cannot recommend highly enough using the TRACE target on the raw table (PREROUTING chain).

Something like:

Will show in your log, every stop along the iptables chains for every packet, including which rule or policy was acted upon it to get it to it’s final destination and shape. Don’t forget to remove it when you’re done!

Also, install the actual conntrack utility to see the connection tracking tables.

Who’s gonna run this thing?

Not to be contrarian, but – well – let me be contrarian. Rant coming. TL;DR – there needs to be a free version of z/os & it’s siblings sooner than later, to not do this is to potentially starve the platform out of existence as we know it.

I don’t think, for a moment, that when people (looking to learn) ask for a mainframe-in-the-cloud type experience, that they are asking for Linux One or Linux on Z, they want a z/OS-type platform on which to learn and play. Otherwise it’s just Ubuntu or SUSE, like I can run on my laptop. Except for anyone but the kernel developers and a few select others, most would never know the difference (outside of performance, perhaps). It’s certainly not the classic z/OS / VM / tpf / etc. experience that most mainframers talk about daily.

As for the other offerings, none are the same (or even really the same sport) as having your hands on a “real” z/OS (or z/VM, etc.) mainframe – the closest of which, for people not buying hardware, would be z/PDT – the System Z hypervisor that runs on Linux. By real, I mean a system on which one can provision storage, configure parmlibs, install software with SMP/E, develop load modules, configure platform software and tcp/ip and more: IPL the system, crash it, figure out how to build a stand alone dump; figure out how to read that dump, get the system up and running again; gen a system from scratch; install an upgrade with a serverpac and so on. If you don’t know what some of those things meant, good, that’s the point.

Until IBM figures out that they’re losing opportunities because of this, I fear the platform is going to get harder and harder to support and defend. Most (if not all) of the cloud – or public offerings on Z (again, not talking Linux) are for developers. Master the mainframe, z Systems cloud trial (RD&T for z “Test drive development tools”) etc.

Herein lies the rub. Where will the next generation of Storage Engineers & System Programmers come from? Who will write the DFSMS/ACS routines, or write the assembler-based system exits? Who will wade through SMP/E reading hold data and figuring out how to fix or remove a wonky PTF that didn’t apply correctly or went PE? Who will configure the VTAM / 3270 applications and the intricate work tweaking TCP/IP net filter and ATTLS? Who is going to do the detailed capacity / performance analysis and tuning of the storage, wlm, cpus and so on? To say nothing of the gargantuan task of securing these beasts.

These are skills with theoretical backgrounds in many other disciplines, but the specifics and technical difficulties pertaining to using those skills on this platform are non-trivial. People need time, mentors and opportunity to learn it. That opportunity is nearly gone – or unrealistically appraised at this point.

Sure there are a few colleges which teach these skills, and the tried and true way of apprenticeship still works if you can get it, but how prevalent is that? Moreover, why would a fresh-out-of-school person take a chance on an OS/platform that they’ve never gotten to put their hands on? In today’s world, they can get a free/inexpensive version of every. single. OS. on. the. planet. for personal use (Microsoft & VMWare development and full evaluation versions, Linux is open source and free, as are the BSDs) – except for z/OS and it’s time-tested brethren. Why is that? How does that secrecy help generate buzz and the next generation of loyal mainframers?

To ask the fresh, talented, next generation of techies to go to work in a mainframe shop – or to go to a school to learn mainframe is asking them to take a gigantic leap of faith. They have the opportunity to be hands-on with 99.999% of the tech out there before they leave high school; but somehow, someone expects that they’ll self-select into becoming a z/OS sysprog? Why would they? Not having a clear track to this pipeline is the single biggest security issue and threat to this platform there is. Companies will hire the remaining few, then outsource, then divest – unless we (and IBM) start driving interest by making the platform (the whole platform, not just the development bits) available to anyone who wants to play with it.

It’s a huge opportunity missed, and I hope it changes soon. One of the hardest things to see is, after giving a talk at a non-mainframe centric conference, people who come and ask how they can get involved directly. You can’t. Unless you go to work or school somewhere special, or are willing to lay out several thousand out of your own pocket – you just have to admire it from afar. And that’s too bad, because it’s a kick butt OS and a super-challenging ecosystem that the unbelievably sharp new technologists would sink their teeth into. They’d eat it up. Many were programming from the time they could walk and computers just. make. sense. But this computer, with it’s super configurable and somewhat non-forgiving “you better know what you’re doing or how to figure it out” practices and protocols, requires time and a steep ramp-up period to become proficient. It has to start now.

When you go to those job fairs, conferences, or just on the next marketing push – come up with a way to give away for free or cheap a copy of these OS’s that run on a hypervisor like zpdt for people to just play with, destroy, hack, but mostly learn. Will there be some negative consequences ? Maybe. But fear not, the rest of the OS providers who have gone before in this space have already figured that out, with a mix of bug bounties, licensing agreements and lawyers. But that’s a topic for a different post.

Is that ransomware on your mainframe?

Next week at SHARE – San Jose, I’m giving a talk on ransomware on z/OS.  I’ve been asked multiple times if I thought ransomware could happen on Z, is it possible: Unequivocally yes.  Come see this talk and watch a live demonstration of how this might work.   If you are responsible for mainframe security, work for a company with a mainframe, or just want to better understand the landscape of this particularly insidious threat, don’t miss this talk.

Ransomware is a combination of 3 basic moving parts:

  1. A delivery mechanism (Phishing email, infected web page, malicious program).
    • This infects the user’s machine – allowing for sniffing of credentials and network traffic.  It can then upload a payload to the host system.
  2. File cataloging and encrypting.
    • Just what it sounds like – find files of interest, encrypt them in place, destroy the local copy of the key.
  3. Some type of Command & Control (or at least reporting) – centralized server.
    • Some means of transferring the keys out to the bad guys. Also, a way for the affected users to connect and pay ransom. (This is not strictly required, but does have precedent.  Steps #1 & #2 can happen regardless of the system’s ability to ‘phone home’ ).

We will also look at how to attempt to mitigate this catastrophic event, as well as ideas about how to recover from it.  Items such as two-factor authentication, proper ICSF / RACF security controls, egress filtering and intrusion detection.

Things I’ve Learned (and things to come)

I started writing a list of topics I’ve learned, some in excruciating detail, some just enough to know where to look for further details (trust me, that is no small feat).

I’m writing this not only as a way of keeping me honest on those days when nothing goes right, but also as a way to incentivize those among you who, like myself, have an insatiable desire to learn – and the tenacity to “figure it out.”

In most organizations, the below is accomplished by teams of people.   Some of the items (gen’ing a system from scratch, for instance – or setting up SMP/E, SMS, etc. from scratch – might never be a part of even a very senior mainframer’s repertoire).  I wanted to see what it would take to go it alone.

My plan is to build this page out with good links to relevant data – and/or if I get really ambitious build some how-tos on the finer points, if there is interest.  Real language how and wherefores.

THINGS I’VE LEARNED  (since I started a deep technical dive into mainframes)

  • RACF
    • password construction / algorithms
    • user profile management
    • using callable services
    • TSO commands for many common elements
    • building certificates
    • importing certificates
    • user certificates
  • Storage Management
    • Configuring SMS from scratch
    • Initializing devices
    • using DFDSS to move, backup and restore files
    • using IDCAMS for catalog and VSAM file management
    • what the eff a VSAM file is
    • how to allocate datasets
    • different access methods (qsam, bdam, etc)
    • what the hell a cylinder (or track) is
    • How big a mod 3,9,27,54 EAV are
    • Initializing volumes
    • labeling and initializing tapes
    • troubleshooting space abends (D37,B37,E37)
  • System Programming
    • SMP/E updates, installation, management
    • Building jcl from scratch
    • Configruing IPL parms, parmlibs, and startup shutdown procs from scratch
    • checking system resources
    • How apf authorization works
    • building a lnklist
    • building an lpalib
    • building a multi-tier catalog system
    • taking SVC dumps
    • Getting a trace of a component
    • reading said trace
    • using IPCS
    • troubleshooting failed ipls
  • z/OS crypto
    • keychain management
    • key management
    • password configuration
  • Assembler Programming
    • What a load module is
    • What a program module is
    • how to disassemble them
    • writing assembly
    • using 4 different debuggers
    • patching programs the hard way
    • building a ZAP
    • Using Macros
    • Compiling, Linking
    • Callable service usage
    • What the hell Language Environment is
  • SDSF
    • Jes2 job management
    • Reading a job log
    • managing output
    • managing active jobs
    • reconfiguring SDSF screens
  • JES2
    • NJE
    • Job management
    • Configuration files
    • parmlib entries
  • TCP/IP
    • TN3270 configuration
    • FTP configuration
    • FTP/S configuration
    • TN3270 + ssl configuration
    • Policy Agent configuration
    • TSO / USS tcp/ip commands
  • OMVS/USS
    • creating zfs filesystems
    • dbx debugger
    • compiling and linking with xlc
    • What the hell Language Environment is
  • z/OS operations
    • Many console commands (devices, stg)
    • How to research a WTOR
    • Vtam commands
    • tcpip commands
    • device commands
  • ISPF
    • Panel customization
    • DDLIST wizardry
    • Editor fine-tuning
    • Keylist modification
    • using the editor – line & main command sets

THINGS LEFT TO LEARN

  • VTAM
  • REXX
  • memory areas and control blocks in depth
  • So much more (work in progress)
  • SMF
  • z/OMSF
  • policy agent
  • ATTLS (in depth)
  • Coding Exits
  • Hardware configuration
  • Cross memory operations (PC, SRB, etc)
  • much more

Mainframe Shell – Metasploit Framework

The public metasploit-framework now (officially) has the basic underpinnings for beginning mainframe pentesting.

As of 10/25 – there is now a shell which, along with the some core architecture files implemented a while back, will let non-EBCDIC based machines running the Metasploit Framework communicate with processes on the mainframe, doing the ASCII<->EBCDIC conversions under the covers.

What can you do with it?  Nothing.  Yet.  The next steps are payloads, then exploits.   The payloads are in the bank, written and being tested against a couple different versions of Z/OS (1.12, 1.13, 2.1 for now).  Those should show up in the framework soon.   Once there, they provide the last basic requirement for exploit development (generally rewarded with some type of shell).

Other items in various stages of development (TN3270 work done in collaboration with and much coding from @mainframed767) :

 

Bind Shell – shellcode and source

This is an addendum to the last post.  Here is shellcode (and it’s stripped down source) that achieve the same goal as the prior post.   The difference is the payload is XOR encoded and the shellcode, and it’s source, have a built in decoder stub that decodes the payload in memory then jumps to it.

If the payload decoder coding looks a bit obtuse, it’s because instructions and operands were chosen that have neither nulls “\x0” nor EBCDIC newlines “\x15” in them.

The code also includes an egghunter that finds the location of the payload in memory, in case they need to be separated.   You can read about egghunters here and here if you aren’t sure what that means.

Full source code on github
Shellcode version

Tips / Tricks – 7/2/15 (update)

Updated.  Added update to the packet capture section below, included pcap export!

ISPF editor

  • Want more real estate in your ISPF editor?   In an editing session enter EDSET in the command line, then check the line marked:
      X Remove action bars in ISPF edit and view panels
    This will remove the menu bars and give you more room.   If you haven’t already, I’d clear these two settings in the main menu ISPF settings also.

ispf_set

Unix System Services

  • These commands will help you find / list APF authorized binaries in USS.  First one finds apf auth with sticky bit (suid) set.  These could be great targets for exploits.  Second finds all apf authorized binaries (no dll’s or shared object files).  The ls -E is just a switch to ls, you can use separately to show the extended flags.  an “a” in the display (e.g.  a-s-) means the apf auth bit is set.
    • find / -type f -ext a \( -perm -2000 -o -perm -4000 \) -exec ls -l {} \;
    • find / -type f -ext a -exec ls -E {} \;|grep -v .so|grep -v .dll

Packet Captures – TCP/IP Networking

  • Getting a packet capture on a mainframe is a non-trivial event.  I’ll save you the trouble of digging up all the requisite tutorials and lay down my version here:

There are 4 major components to this effort:
1) A trace writer proc, put in your favorite PROCLIB, created with JCL such as:

2) The TCPIP packet trace functionality. Started with a command like this from the master console or SDSF – (check the link at the bottom for more filter options):

3) Then, start the writer and execute the capture by issuing the following commands:

4) Using the IPCS functionality to format the trace into a spool file.  I use JCL like this (Make sure to modify datasets to match your system):

Bonus Step 5)  If you want to export the dump to a pcap file, readable by tcpdump, wireshark and the like, substitute step 4 JCL (the inline portion after DD* only) for something like the following:

Then I like to use USS copy and sftp (make sure you use something that does strict binary transfer) to dump the file to your PC and open with Wireshark.  Enjoy!

That’s it .. simple right?

Here’s some useful links:

TCPIP Packet Tracing

How to collect packet trace on z/OS

V TCPIP,,PKT command syntax

ZOS IP Diagnosis Guide (SNIFFER CTRACE and others syntax)