All right.  So we're ready to go it 6 oÕclock.  First of I want to thank everyone for coming out so late. Your coming in for my talk so I guess thereÕs a lot of competitions here. I am unfortuantely not as funny as some of the other host right now.  So this is staying persistent in software defined networks.  My name is Gregory Pickett with hell fire security.  I am part of the cyber security operation group. Over view of today's talk. White box internet.  What is it? ItÕs not common knowledge right now.  Emerging really.  After that is stupid is as stupid does.  Next is exploiting it.  Pointing number a witnesses and how we can take advantage of them.  Next is moving forward. How do we remediate and also to mitigate those weakness and vulnerability how do they evolve. And finally wrapping up where this taking is us and where we can accomplish. What I am trying to accomplish like talk like this.  All right let start wipe box Ethernet.  Well is standard hardware. Blank slate. Running merchant silicon trident and broadcon chip sets.  Intel AMD power pc processers.  Using an open operating system that is often Linux. So I think I donÕt see that it is not linux. The idea is to move the hardware from the equation. To make it a commodity by using off the shelf components.  So from that point forward will be the software that will make the difference. It is critical for software to find working.  But it can be use without it.  Of course question, why do it.  Why are we moving to this direction?  Well the same reason why theyÕre implementing software to find that working.  Reducing cost. Increasing flexibility. Looking of course to gain more control.  From can gain the control to means can you actually remote to white box either switch SHH you can issue command to define the switch the bigger the network like you would with a regular switch traditional switch.  Can you also script for management purposes or you can load an agent on it a puppet.  Chef. To make a part Def ops right this network automation orchestration. Or you can gain that control with software to find networking. Software define networking control plane, essentializing the controller.  This makes the network flexible and of course responsive. Everything in software. [Inaudible] hardware data plain. White box even in your data ban.  The data plan because just as flexible.  Use what ever you want. You're not tie into vendors. Propriety hardware. Your  Technology.  Any sort of command you know requires cisco certified are required that sort of things to point out one particular vendor you're hooked with and stuck with them right.  All right.  Now to make all this possible to make white box essentially plug and play they adapt to oni. ItÕs firmware for bare mineral network switches.  They're a booth underneath.  Which you grab to booth the system. And then oni is there to booth the network. You grab install conference system. And in the event you like a new one you can change the software and then go grab a new one a conference install.  Different one.  Change whenever you like.  Onnie comes pre-installed on network switches part of the firmware and automates that switch deployment. So white box internet.  The blank slate and Oni that boot orderÑ what could go wrong? There are number of weakness in operating system. Onee is linux based. Privilege account  The has no password and ONEE will not foresee you to change the password. It remain still. Management services. Use tolnet. Insecure by design. HSS is weak too. Insulation mode, key is only 18 bits in each computer.  Hyper mode 26 bits not looking good so far. The installer has weaknesses as well. It uses a set of predicable URLs as well as define in the standard. One year off fist and another year after that basically goes through series of URLS better define by the standard by the process is does cares out the install and where is goes it is very defined in the standard for stalk you get or you get the exact URL from DHCP. After the assignment.  After that build the URL from the DHCP response, after that look to neighbor for insulation and from that that fail go to protective waterfall so is very predictable where onee is going to be looking. And when we arrive to a particular location it goes then and looks for a particular file. Then it goes to a series of particular filenames better define by the standard.  So as an attacker we know head of time where we are going to be looking.  And you know ahead of time what you are asking for. So you can hand onee a compromised installation and onee will install and that's possible.  Because there is no encryption no authentication. So should you hand Onee a compromised installation any sort of the installation.  Once that compromised is running OneeÕs partition is exposed. Exposed that could be modify. Be itself be compromised. And those secure bootÑ there is really nothing could stop that compromise installation.  And it can compromise Onee and will continue to operate over and over again and doing what ever the intruders told it to do. that means what does this mean of course? With all these weaknesses.  There a lot of opportunity to blow Onee up. It is not very well protected.  So you can blow it up.  And first stop of course is to compromise it directly.  With the root password starts with blank. [inaudible] And you could of course log in. We could sniff the traffic. Apply a man in middle modify. try to close the telnet, or easily access the SSH but that's not likely. Onee just throwing a number up 0.0001% Its job is to install conference system all 45 seconds. Once it is done with that it does a reboot variable.  [inaudible] Reboot the switch and then the conference system and will run for the rest of the time. Like a 99.9% of the time.  Next slide is can be compromise the installation rouge DHCP server. From the IPv6 neighbor  Or maybe spoof a Kvp server.  Also very difficult.  So is like right place right time.  If you happen to be there at the right time you get it, you get comes insulation to it, you're there for however long but you don't gain that nirvana layer which is persistent.  Are you going to be in the right place right time again?  So is there a better way. Compromising it indirectly.  You know the conferences is good and will be up 99.9% of the time. See what you have to work with.  You're passing in our conference system.  Modify only. we have an exposed partition.  Take advantage of that.  Compromise Onee no secure box is stopping you from doing this and take advantage of ONEE. So over and over again. Onee will keep doing that you told it to do. Compromise our conference system. Now you compromise Onee and every time a new conference is installed youÕre basically back again. Now you're in the firmware, your essentially there forever.  And that's we are looking for as attackers or white head penetration we are looking for that persistent.  Network operating systems, installed by ONEE operate the switch to do all the packet forwarding and all the different features that you like to see in the switch. Onee compiled the distributions right.  There a lot of network operating systems but only a handful that have ONIE compatible distributions.  And the number is growing. And they just edit peak 8 a couple of months ago. But when I started this there was only about 4 that were really prevalent.  First is open network Linux after that is switch light QX linux and non XOS. Open network linux distribution for bare metals with is switches. And of course as they are all are.  Debian in Linux is very popular. Bare bones no features.  It will run the switch but you are going to have a really hard time finding the switch configuring the network because it is really just a reference.  Is a started that could open project came up with. They want you to use this to develop your own network operating system. Obviously there to promote ONNE so you are getting them a starter. So they are telling you to start a new connection that will run with Onee.  And that's what switch light did.  I looked at version 2.6.  Is a package open network clinic.  That took that starter.  They added SO rest which operates allot like OBS 2 B to define the switch. [inaudible] Open flow agent for loading the flow tables like configure the network. It is not a standalone. Is really is a part of a total solution called big cloud fabric. The idea is to plug the ideas in the network and big cloud fabric will take over. You are actively discourage from managing the switches. And for all instance and purposes at least while look at it to ban the network switch fabric we will see the how well that turns out.  And of course the whole the total solution is maintained by big switch networks.  We should next look at 2.5.4 is important to know that I'm looking at the latest versions were so I want you to know what those versions were.  Based on debeian Linux as well. You install puppet chief antible cicada network environment manage the rest of your infrastructure. And is maintaining by cumulous networks. Finally we have non SOS this is version 3.3.4.  Based on enterprise base5. There is a newer version 3.4 but it turns out all the problems that 3.3 has they're still there on 3.4 just thought you should know that.  You would install puppet chef. Antible to make it part of a ops amounted or e switch or to make a part of an SDM environment it is very flexible. Maintain by technologies. So we of course this is what I do here. Introduce you to these and then talk about really the weaknesses. No encryption no authentication on switch light indigo.  Nor V switch. With indigo its a matter of spoofing the controller. E-switch you just talk to it. It will listen to whatever you tell and will do what you tell it to do. Outdated open SSL this is done because honestly I need to fill the slide a little bit.  I didn't want a lot of blank spaces.  So they're running a bit behind open SSL and when I saw it I think did check that and hard bleed is not a problem.  As someone was looking at these they think.  How hard these environments are I donÕt like to see old software I donÕt think any of us do. So with no encryption or authentication on these of course these are normal to topology flow or modification through unauthorized access. Add access, remove access, hide traffic change traffic.  I mentioned in the press a bit about east dropping and that's of course what would likely happens on a switch light network on non SLS not running an e switch.  But there are bigger problems and this is what is going to end up leading to the persistent.  Start up with something rather simple. Default switch accounts.  Switch light has admin. Cumulus has cumulous and non xos has admin which is low priveledge or safe accounts. These is big deal for two reasons. First that you have a limited ability to add other uses.  You have stuck with these guys. [inaudible] But I'm going to show you some command injection.  Get around these limitations as I mentioned they try to put on.  Show that in a bit here and then the second reason is a big deal because these accounts are the only obsticle and only obstacle that would get you on the switches which didnÕt sound so good right? This would tap your key log in and the switch is yours and then the network.  All right and this is where all that begins.  Easiest gate to show. Switch light uses a wrapper type enable debug bash you are at the shell. QX Linus you are actually connect to the operating system and so you donÕt need to do any escape you already at the shell.  SOS well it has a very done shell however puppets there and puppet do dirty work for you.  So puppet will actually open up back door which is not confrontation but will find it if you pack the firmware always helpful and when you have that shell you get instant elevation.  You immediately become ready and switch light turns out admin is UID is 0. But when you have that shell are you immediately out wreck. So cumulous you have unrestricted sudo access so you are a rouge equipment which is also not good. And SOS you start the back door with admin so running under admin privileges and when you exit back door and you take a look at  your UID it turn out that you are UID 0 also ready privileges that one password get that out of the way we know that to compromising a work station is trivial.  Key logging is trivial needs to full control of your network. First the switch and then the network with authorized Access high traffic change traffic. All three app systems.  And then of course compromise from the frame work on authorized x which gives you access to the flash. Which gives you access to onee you can do whatever you want on it. You can modify your firmware work compromise.  So your network is one key logger away.  I'm going to show you this here.  Last year I did open source. Easy to get access to. People always ask about vendor products because you pay money good money for good vendor products and you want to see if the money you paid is worth it.  This year I decided to make a point of using vendor products and run test on the vendor products as soon as you pay for and expect to be better secure right? ThatÕs what they claim. And in some case they are but we want to make sure that they're are what they say they are.  So I start looking at these vendor products running on equipment, network equipment.  I started with big cloud fabric controller. And I log in as admin as a little privilege user. Look through the command.  I like the word bash that word sounds good to me and got shell there and if you are paying attention on the account slide you saw what that is hidden and disabled? They wanted you to stay away.  Discourage you from touching the switch. They want to big cloud fabric to take care of everything. so with that in mind we have hidden and stable account.  They don't want you to touch the switch. they donÕt want you to  touch that.  There is no password on that.  So how likely are you to change that password? All the hidden stable does is to log you in. Once you have shell you can go ahead and switch over with your password then you are of course immediately right there.  So I'm thinking to myself. I wonder if I can do that on switch light.  So I log in as admin do a little extra word there.  Type enable first.  Give a bash and I came it as what  I was a bit surprise.  So check my new ID and it turns out that UID 0. So I like that.  And where he have where he so we have access there as well.  I think I start looking around for flash devices.  And there is one nice and named. ONEE. So we see with the privileges we need to write to it.  So that's good we got that.  Cumulous I want sudo it up. I can just go ahead FC shadow change the pass and switch do whatever you want.  Sudo everything. And then mill knocks, I will open the back door I will tell you what that is soon. At the end there. Net cad very useful, connect it to the back door. Admin obviously and I cad I see password.  Admin at UID 0.  And the then an extra account.  Good to know other ways to get in. Extra account there at UID 0. And I mention with cummulous there are ways to restrict informations. Since you are dealing directly with linux. You can add new users. there are some management problems though. And in the tool that [inaudible] gives you for low privilege or less privilege users. And source and limitations. So sorry QXlenux what do you know about this.  Now I necessary don't come out with a lot 0 days and sort of brand new vulnerability of a product. I donÕt celebrate.  Is show off.  Yes, across the top there.  QX Linux several command land tools. That they have set aside for less privilege users. I won't name them cause they are tongue ties.  They are meant to be used by low privilege admin account just taken care of the switch you go ahead and enter your arguments some commands and perimeters. Though some command and parameters get pass to server.  I can say that pretty well.  And that goes to the comparison with rosette. Rosette basically this is what's acceptable.  And its acceptable it lets it through.  Problem is there is command injection there is filter basically. There is a command injection that allows you by pass with filter.  So basically have to have sudo server to do whatever you want.  Rosetta or not.  And run is ready.  Sort of limitation to put on you.  And you can go ahead and side step those. I'm running out of switch again so here we go that is a switch with a license and [inaudible] IÕm hoping. so there it is running on a switch there CMB server I'm going to go and demonstrate this.  I'm not as resourceful as the guy that brought a whole safe here. I donÕt want to carry or ship a top of the line switch onto a plane.  Right.  So we are going to go head and make sure I get my address for this.  I already have it.  I'm a low per it user. I got a lots of VM running here. Take just a second here.  Okay.  Is off the screen again.  All right.  Here we go.  I have a touch pad up here.  I'm greedy.  And of course I have limitations IÕm a low privilege user. Let say I forgot the command injection.  They have patched this for already.  I don't feel so bad. But weÕre going through this with complete detail.  So I that one tool that I have sudo privileges for.  And there is my arguments. What I have done I have injected so that my second command my injected command looks like part of the label.  You have to do that.  You have to otherwise it looks at that label that first piece command tries to process it.  And of course there rosette and it fails.  I need to shove it all together so it sees all the label and it ignores it. There is no VM there is no guts right there. But it does stand on command. But inject nonetheless.  Now you notice there no spaces there and to get around that.  You go ahead and make a script. Put a script here. You have your own low privilege home directory. And put any command that you want out there. Just adds another user with no sudo limitations. You donÕt see anything. A little bit of a cliché.  Here we go. Then I have everything there. No claps? 
[ APPLAUSE ]
>> Thank you.  We worked all year for clap, come on.  All right. I donÕt feel satisfied. I feel fulfilled.  So this will by pass any sort of sudo limitation, for password change and use your account but obviously you don't want nobody to know that are you around.  So I'm going to go head exit and pause my VMs. So I can use resources.  All right am I back on this screen all the way? So once you of course have your privileges this live switch you can go ahead and you can look for the entity devices and find onee dump from the block device.  You have your privileges right there to modify and put it back.  A big part of this implication there is been talk about this of course.  That this is the kind of problem. We have grown up with other issues with different types of devices.  Is important to know that personal implications are greater.  If you have a former [inaudible] single server.  Yeah is important data but it is a single server.  But if you have a firmware compromise kernel not possible on your switch.  You have the network on.  One server, whole network.  Bigger implication.  And an important to talk about how this is done because they assume is done firewall is safe.  We want to talk about scenarios that are very possible.  That makes it not as safe as they think it is.  So I'm going to play the go tee. The admin administrator.  And just going to have a number of difference ways. You can browse the internet. You need to drive by and download. You can open a bad attachment that's what I'm going to do.  I'm going to be infected a piece of malware [inaudible] that i put together named big malware. Now big brother is going to do is a uno binary that will infect the window system and is going to key log off the ones that you are stuck with.  For simplicity sake. Very easy can get key log in with the connection of the switch. Once big brother sees one of these accounts in use, key logs a password.  As a network administrator you going to touch the switch at some point in time. You are going to log in and it will wait for you for days. When you do this your done usually you're done well he logs in.  And he writes a Linux compatible binary to the switches file system.  Nothing is downloaded from the internet. Carrying himself his payload it self [inaudible] a little brother. Writes it out.  Unpacks the little brother from the back door. He unpacks the firmware shoves a little brother in there.  And you know what are big brothers are for right? Reps his only back up put onee back. Before he does this he modifies onee so that any time onee installs in a conference system onee also puts little brother back. And thatÕs persistence over and over again. So overtime you install in the conference system little brother comes back. But it doesnÕt stop there. HeÕs a big brother. He helps out. He pivots. He connects little brother in the back door. And he is also connected to a C2 server. And the C2 server can come anywhere. This helps it pass Dlans, ACL, firewalls, what he does he browses. It is a reverse HTTP shell. Uses header and cable abusive using a proxy.  Likes to blend in.  Relay command between little brother and NC 2. Out in the great wild world.  So with are going to demonstrate this.  Got that attachment here. Im not a malware writer. Is a little [inaudible] not really?  It does hide a little bit but not hard to find.  Make sure he is running there.  Then as network administrator.  You know I've got [inaudible] I move on with the rest of my day.  At some point in time I'm going to touch that switch.  It works. This and a wrapper is like switch light. All command to get to the shell.  This is actually what big brother would be escaping him.  Now I'm going to double check just because the demo gods and all.  There.  All right.  What IÕve done to this thing. We're going to go to C2.  A little bit of a delay there. So is going to fire up.  So its works like a little bit web server. Of course right. Something browsing.  Just going to go head and listen and as soon as the connection is made we'll is a prompt a little bit slow in starting up. Hopefully it will come up soon. There we go.  Can we get that on the screen?  So little brother has been started the connection has been made. And the big brother has started C2. There is though relay on anything.  ItÕs gonna load x to Little brother behind whatever firewall that's there.  Then move around a bit and look at that switch real quick.  So thereÕs some timing. Wait. Just need to be patient. There you go. So we found a file systems error.  But things are not forever.  Entirely, some point in time to be notice what this mysterious connection.  As an administrative you may see this sort of thing. You might see a strange connection. You might freak out. You go, who installed that operating system. We really fix things so much. Especially with device like this.  The desktop. We re-image right.  So we go head and we re-image. ItÕs an infection. Right, thatÕs what we do. I'm going to go head and bring up onee and install another file system network operating system. This one happens to demo operating system.  That's going to do it. When it comes up and we'll go ahead. IÕm going to stop this because its obnoxious.  I'm going to get my command and get my operating system. I did this because  I did not have access to the operating system at the time.  I did this but I made sure it operated just like switch light. I wanted to be realistic.  So I'm going to do a thing.  WeÕre gonna move onto fixing this short of stuff and we'll come back and see little brother resurrected.  I have been characterizing as poor choices. And that is what we have seen with the vendors. This is about fixing it. The solution in addressing primarily vendor and what they're going to have to do.  Have hardware.  Install environment, inter conference systems agents inter-mediation stuff. The vendor stuff is in mediation. And for network administrators themselves architects right it changes enterprise architecture from mitigation. things that you have to do in the meantime.  Hardware obviously.  Trusted platform module. Im sure big network has put in for bay switches. You know there are not any new power PC designs I have not heard of any come up so that maybe difficult to do for them to add them in the power pc switches and if we can that will be great we want to start using TPM. We want platform security. To make sure any short of amount of modification onee. Onee will not fire when the boot fails. And we know something is wrong.  They also working on, we get them on the hardware the next time of course to get them in use and they also working in getting in standard as a partnership of course through the vendor and the people that making a standard is making a slow progress but they're working on it.  That is keyless networks. The ones that develop onee and got it adopted they're working on that.  Install environment. the root telnet.  Is secure we can use SSH it is okay. And with SSH we can increase key entropy. Enforce a password change.  If we a capable of remembering a password or using a password or we choose not to.  Factory reset. You here has not done a factory reset. We forget these things we screw up. Factory reset we can do that. Remove IPv6 and waterfall.  There is nothing wrong in having DHCP provider administration URL. We can protect the DHCP server. We can make sure we are only one running but the half ways of doing this.  No reason that we can't use a DHCP server.  And of course a good one will be [inaudible] installations best one but there is always problems with keys and signing and that sort of thing.  Understanding this problem, ultimately that would be the best solution.  Operating system before we hit that.  Let's check out little brother. Operating system has been installed again. LetÕs check out C2. C2 is able to issuing a reconnect command. If you enter a command and you don't get anything back you know there is been a disconnect. Likely the network conference systems needs to be reinstalled again. I did that myself so I know there is disruption there.  So I'm going to reconnect. Big brother will close his end of the socket. Reestablishes connection with little brother.  So there we go.  Survive.
[ APPLAUSE ]
>> Thank you. Thank you so much. You have no idea.  Thank you.  We still have our we're persistent. Ans even after the installation.  All right.  As operating system.  Now we obviously can't do everything.  These are ways to harden the environment.  Make this platform a lot more than other forms because thereÕs a lot of writing in this firmware. A lot more than the firmware that we have seen.  Bigger amply indication more is being compromise.  First one changeable names.  New ID accounts.  Change new ID 0 account names.  They give us to do that. These are privilege accounts. It will be also be great if we could add a user. But you cannot add users. Now if you can add user but you have two types.  Admin and monitor. Which one you think you will use to find, configure to take care of the switch. Admin right? Admin has the able to change the password. Change the password of the original admin. Get your UID 0 back.  Force password change.  Don't awe allow [inaudible] I donÕt I have mentioned this in Black hat but one thing IÕve seen in network ops is the password that they have used, the shared passwords. First of all they usually have a shared password.  And the password sticks around for a long time. They have al to of equipment they have a big team.  I have seen those passwords go for years and not be changed. So enforce the password change. And of course move UID to 0 for the admins. And tighten shell access. Switch like. I would like to see a one time password.  They have self service portals for support.  Why can't they self-service them shell the support to get a one time access.  Who's going to give it?  Make it more difficult. More resilient. How ability the wrapper? And also with a one time password. And now it does a pretty good job of protecting that wrapper, there is a way to get shell access from the wrapper but taking way too long for me to reverse engineer the code. So I took a shortcut. [inaudible] If you unpack the firmware you can get access there. you can plug that into bashing and you're in.  If you have thinking maybe modifying or modify puppet modify password or shadow add user.  No, I thought of that.  Agents.  This is common.  STM platform.  They're not using TLS.  So they need to use TLS. They need to not only use for encryption you can also use it for ops mutual ops. And of course there is a concern with certificates in key distribution. But you also have good ops there. And the DNS to do the heavy lifting. Have them also lift these certificates and the keys.  And of course architecture. I see the man in the plane. We need more than the dlamp. We need to get as close as you can. I understand it is difficult physical separation. It is already done. Thats why I say try to get there right. Get as close as you can. If you have been in network ops they have 20 monitor in that dame wall.  Network admins has two monitor and a desktop.  Why can you remote to a jump box. Maximize that in one of the monitor and use it just like you would in the workstation.  It will take a little longer to get used to but it will up your security significantly. And then we have audit switches.  I said the A word. I have to learn to love audit. ItÕs a painful process but you know. Audit has good uses. They are your friends. Make sure your password changes and then it wouldn't hurt to have only partition to make sure it has not change.  I have a couple minutes left and I wanted to get to these last slides i ran out of time for questions but we can talk afterwards.  You're seeing something here that's familiar, right? Seeing that just raise your hand? We want to talk about once again about impact and security. And keeping pressure on the developers. And that is why we are here right? We are here to freak them out.  And talk about the difference that we are trying to make.  Getting product speeches to market is important.  They're out to make money and they have to be out there first. But desktop operating system have been through this before. They have something they called best practices. I donÕt think you're reading them. I think you should.  I think you should develop your own best practices. Start using them. Cause you are not. We start this merry-go-around again. Every year at these conference we hack it, you fix it.  And turn or do clean up.  Hire some for security.  Your stash or your small business units or a large business with limited amount of people.  One guy for security. We have that one guy in the security that is there all the way. So mid way though  development life cycle maybe an assessment at the end before you release the key is the before you release.  Honestly settle this thing one in the end and when you go try to sell it.  Security can be a feature too.  All right so we want to make a difference here. We want to learn from desktop and serve the operating systems.  We want you to begin to harness these new platforms dev ops and SDN. For most part they are taking over taking responsible for is defining the switch and configuring the network. They need to take care of the entirety go the platform. They need to be responsible for the whole thing.  Whether is checking permissions?  Release coordination. Audit. Are they  checking the audit. Or check the audit of events or following making a part of the rest of the entire platform. Switch light can do that.  To follow up to big cyber maybe a separate platform  to consolidate all go that. How about logging.  Checking logs in the individual flat form. Or you are consolidating log in. So you have visible there.  So you are now taking responsibility for the entire platform and not just part of it. And logic terms. And that is my general term here. And then I have some electronics. And that what come up. You can check things. You can check states. You can make sure that the platform is in a state that you expect it to be. You can a good one.  Hash the only partition.  Is a nice thing to do.  Make sure everything is intact. So final thoughts here, we are good on our time, the security in network ops is critical. The security that you have seen has been neglected. And because they think the security is safe. They're assuming that you are falling all the best practices.  They're assuming is an idea situation the solutions you purchase and the different layers are operating perfectly.  That is a perfect world and they're catching everything and this is not the case.  A single piece of malware can make the call from windows to linux and I did it right here.  I'm not the first obviously.  And when that happens of course that pivots. The network administrators makes a great pivot. You then are able to compromise those switches and you are able to turn [inaudible] that immeasurable.  Because you have visible now and entirely of network.  And we are here today trying avoid that.  That's all.  Hopefully we can work together so we can avoid that.  Links.  If you are wondering about all these stuff. The different products.  They're pretty cool products. And that is the end. Thank you for coming.
[ APPLAUSE ]
>> I think I'm right on time, right?