>> Welcome everybody.  If -- here to talk to us about linux containers.

>> [Applause] -- Linux containers.

>>  I've been doing security awhile.  If you don't know who NCC group is, we were brought by entity group from UK but same people same place.  Given some talks before on some different things.  So this is myself first DEF CON talk my 12th DEF CON10 second motivational slide to not wait as long as I did and not be a slack consider if you got an idea we got lots of inspiration until talks keep that going and do research urge submit to urge -- and give talks don't wait as long as I did.  So, I want to start with a story.  So Bob here's Bob here, he's got some web app or he's got something on the internet it's on Linux somehow there's a bug there's some kind of -- [indiscernible] something like that.  And he wants to add security but he's not really sure.  So somebody says, well you should do CH route that's legitimate.  You know open SSH uses it it must be good, right.  As you know it's broke Ken if you have -- and then you can go up to the route and yeah, so that's well-known.  Somebody else oh you should use -- NSA made it.  Must be secure.  If you no anything about SELinux designed for multilevel security really designed for government things when you've got different things to class tie.  It's not support on -- but other than that its you know not so great.  Some of those reasons complexity really complicated.  Created [indiscernible] most of the other modules have where you know he doesn't really believe in any of them.  If you walk into a box that has S -- are high.  And you know at the end of the day the colonial got a colonial if there's vulnerability -- he loves to just and then dig at them as he does.  So somebody else comes ago to Bob and says you should do jar jar and yeah, it really is in a lot of ways but notes root really solving some the problem.  If he has word pres, command injection -- going to do nothing you are doing a lot of other things unless you do all that -- turn everything to 11 and everything, not going to solve that.  Somebody else says just use VM that's got to be very secure.  They are not perfect you've got a lot of other problems.  So second story is Glen.  He knows a lot about security.  He's -- and so he talks to potential source he's really really paranoid he wants tolan Linux he doesn't trust OSC Malware on windows all the cool kids run Linux its got to be super secure right.  Not really he's one exploit away from having his calculate popped -- last time I updated my [indiscernible] that didn't go so well.  Third story is Margaret, Margaret works at IOT company.  She is tired of getting these everything running as route and everything getting popped and not any kind of execute on the little platform.  Then I actually inserted this slide after I saw Charlie and -- Charlie maybe Margaret works own -- team and doesn't want to run, no ability to control things.  Really a lot of that is done because money obviously takes time to add security.  There's no, you know, there's no way of virtualize something on -- this I mean bedded thing running Linux there's got to be something we can do.  And really this story -- have in common the worse attack service a -- sand box and containers it would be great to see them being the norm not the exception if you look at chrome and a adobe, any kind of modern sand box environment that should be everywhere there's a battle we are fighting and you are going to lose, right.  And so, until we can win, written in some crazy -- or whatever it may be we have to cut our losses you can do that with sand box and you can do that with other isolation.  For Linux open source loves to reinvent the wheel.  How I would like to set this up being in VEGAS one of my favorite movies there's this movie [indiscernible] where they come out of this house tearing at six guys with guns drawn and you know, it's a fire fight to paraphrase.  And there's kind of -- investigate who says what if it wasn't six guys with six guns but one guy with six guns he's William DEFOY -- one guy with six guns the person what if it wasn't a whole bunch of colonial but one colonial -- that's really the core idea of containers, right.  There's also not a new idea this has been done a long time before everything came long.  So before we keep going I just want to say that this is -- if you are really scared about security or paranoid and -- you do not -- on any kind of -- dash you need guys with guns and fences actual air gaps not things connected by wires.  Really, I'm not up here to say that containers can do everything.  But, but I am going to say you don't want to depend on anyone single mental.  Containers are getting a lot easier, more powerful.  And if you can do anything, you might as well just add things in layers as we've seen.  So, to get to how this actually works.  Lt first major area is name spaces win the -- just like any other cool stuff on -- it started ten plan anyone where they had the idea of [indiscernible] but the Linux colonial is split up into five spaces, and the way you get into one of these names is you -- -- kind of the new fancy fork the kind of phrase you can -- concern until execution context, the way do you that is by when you call clone, you add special flags at the end depending on the name -- that you want to enter the.  The ad d first a long time ago.  Basically let's your process have a -- of what the file system is.  We can keep going through these.  Host name.  They can have their own view of the other processes.  And they can be nest Ted there's cool tricks you can do.  If you look at how this actually works.  If you create attain never and you run PS in it you pretty much see the processes that are in your container which leads me to the awesome snake oil solution you can't hack what you can't see.  So the new, there's also a network name space that you know isolates your IP fire wall, your routing table and user name space which is the newest one which was added really important for accident occurring containers and essentially let's you be inside of attain never that's still treated as a low rights user in the context outside of the container.  Which obviously also high risk area of the container that you are really controlling a lot of how the U ID system works win the colonel and very sensitive area there have been a few vulnerability using user name space to break things outside the context of containers.  The way t looks is you know if you attach to our food container, if you your route you can see inside run sleep and outside of it you are U ID a hundred thousand essentially -- so the other major area of containers is capability.  The idea is you take the user route, right who can do -- and a whole bunch of little pieces you know whether you need to bind to something lower than a thousand 24, whether you need to be able to run [indiscernible] whether you need to be able to you know change network settings.  And [indiscernible] are great let's us get rid of this God mowed and split it up into a whole bunch of little pieces real confidential do only what -- you want the process to be able to do with root.  They started lumping them together in weird ways.  You end up with capability model that somewhat works but is confusing hard and can be messed up.  You know, kind of -- ability that you might commonly encounter things like being able to receive -- or a thousand 24 or change resource controls or send -- everyone asks what capability should be dropped should I drop this or that really you want to drop all of them.  You want to set up everything for your name space ahead of time.  When you pivot into it you want to throw away all -- and just live in your little -- and you can't do anything at all.  And then you're going to get questions on forms or even just people as implement tongue this you are thinking what if I leave this one capability enabled what happens the answer is it depends.  So if we go and look at example we look at [indiscernible] from CD record set ID root pop shells on LINN U Q you can get instant route pink still has route and obviously attack service pink pounded on a bunch.  If you copy PING to somewhere else you will lose -- and if you try to run that PING you will get option not permitted raw socket for PING.  So, the way to fix that is you can stackability of cap net raw on that new [indiscernible].  If we, you know, look at that we can see we have cap net raw, run it even though -- that's at the only thing I can do, only capability that I have.  Obviously there's a lot of dangerous capability that to you -- that can be bad.  Override you know discretionary access controls or turn off the miniture access control system.  You know, things like that.  Definitely things you don't want to have.  Other bad thing really cap -- that sound pretty important.  Its basically route.  There's a whole lot of things that it can do.  There's also a great post you by  that goes into all the detail and all different cape pallet and how they can be used to get route or do bad things and that's the link to it and so the other major aspect of containers is control groups.  Basic idea of those inheritable system for controlling resource across a set of processes.  Process sees.  And that can be devices or CP U usage or physical CP U memory amounts, rates of a certain device.  Network.  And it's really you limit on -- if you want to think about it that way.  Urge its used to fill gaps as main spaces there's no -- there's no name space scooped of have to get around that a little bit by using C groups in a way that that some one intended to do that but not really.  They are typically controlled by a [indiscernible] file system, so, it's just a directory and you put, you know, create directories and corresponds -- and things will read that the good thing it's a file system based and everything on -- as file used to be at least.  Following in that model but at the end of the day you know you can do tricky over amount attacks and other things that is a side effect of having it be a file.  So, you know it can be controlled through CG manager but really most of the container platforms with -- when you put all that together name spaces ice sew late the elements of the colonial capability help enforce those name spaces and limit the, you know, capability win that -- and then the C groups limit access.  So really that those three elements along with some other you know magic sauce will create containers and it is better than -- there's a lot of background to those.  You get special amount options you can do things like overlay file system where you know part of the directory is shown part of it is shown to the host there's go between you go do [indiscernible] so where API container is being used now on servers a lot of platform is a service system you know EC two you can do [indiscernible] Google API engine uses containers, [indiscernible] has been doing it way before anybody did and [indiscernible] everybody probably more than I can list.  Also being used in compliance.  Chrome is a huge user of these technology they have done a lot of hard work on making awesome hand boxing and chrome, it's some what used in ANDROID cool sand box tools if you are into those that use these.  Skipped of major one that started on the kind of path of [indiscernible] was LQE van nil language interpreter I won't go into the details controlled through a template you set various options you know what devices you want to have, some security things like [indiscernible] API architectural more what to keep or drop.  Recent advancement win the last year or two, of containers.  So, unprivileged containers I mentioned user name space creates ability for non privileged user then be root in all treated as a non root user when it comes down to it.  Obviously there's some weird things that can happen with that situation, so there's more work that needs to be done that will be a things where you know that doesn't have rot will be able t create roots and do things.  There's a problem we talked about attack surface one of the main entry points to the colonials there was a lot back in two point two now there's a whole lot more.  What are all those does your API really need all those, you know, 300 some odd says calls probably not.  So then this thing came along, guys that Google chrome team doing a ton of awesome work to push boundaries of what you can do herein very good way.  This has been tried before.  But the way they are doing it is much better than some of the methods that have been tried in the past.  Important to point out [indiscernible] is not a sand box its purely made to just limit the attack surface to things your API doesn't really need to have.  So you can also not only can you fillet on certain calls like this process just crunch chinning numbers just needs to call return and open and -- you can actually control calls and say -- you can only deal with these flags and this file.  That's pretty power F you do encounter -- when you got a lot of filter there will be some impact I think figuring out what that is.  But definitely is something to consider in fur going to be rolling it out.  Right now you will need LQE or -- or use end box or something like that.  If you are going to use [indiscernible] you have to use a slash couldn't crib or wait until one point ate which is I think when we are going to launch [indiscernible] LXE has it right now and you can crew it.  Its typically done through PR control.  You basically can set one of two flags, -- which should be used for just like a you know once you get process into a spot going to be crunch and numbers or basic you can put that on there you can filter which everybody remembers from -- this T-shirt contest how you would use in an actual program directly.  Berkeley packet filter is pretty cool you can use  -- some of the language that -- I don't know how it works underneath but its cool K where is it being used right now lots of places.  You be turn it on in lots of place at least it's not necessarily on by default it is definitely in chrome they have been doing a lot of good work they also did tour.  So, containers you know whose doing these, right.  So [indiscernible] there's interesting other companies that are doing these in the back grouped or offering them as a service, offering it as a service.  Big shout out to -- they have a crazy like customer moved system they do [indiscernible] for everything.  And they are super serious about security and everything else.  And also the [indiscernible] is interesting everything in that kind of minimal system runs in attain never.  It's also funny I don't know if it's been fixed when you would Google that it would bring up [indiscernible] Google would fix it they might have some problems.  You know, we're really here to talk about the big two when I usually mention dock consider this is the response from sill couldn't valley this is usually the response I get it's really packaging kind of development focussed.  One thing that is really good about that I think it's also very API focussed the kind of philosophy of dock consider it's not just attain never with you know some things in it, it's just, its -- one process in that container and that's it.  And that you know as a security person I really like that because then not only is it that one process nine container one process just libraries that it needs and just the files it needs and nothing else and so you really are cutting off a lot of attack service there and the you know the big dilemma dock and why its gotten so pop her makes -- in kind of the harder mowed.  DOC KER you know you just do -- didn't magic happens.  But isn't just containers there's all these other things that they've done.  Used to be based off of?     A    -- written in go they also have chain and all these other things.  They have arrest API by default uses dark dash also budget kind of down sides of it that they are working other than fixing related to the fact there's this call call -- all your containers that runs as root you have to be root to interactive act with it that makes people run as root on the container host and you know its never a good thing.  So you know you  -- dash you can map things from the host into your container.  Also had this idea of [indiscernible] had you been for containers you know you can do commits and all this stuff S all the orchestrating communication management I don't think the word orchestrating was mentioned much at all in the last hundred years except maybe last year.  Any way so [indiscernible] S other big one.  Really started out as amine must OS for hosting containers.  They are luncheon rocket in API container spek they are trying to distance themselves main idea that they are trying to be more secure and lean and be what dock consider should be or used to be.  And but you know why would anybody use any of these systems and its mostly just to make it easy or to package a lot of the functional will the with it one way you can relate to it you can secure that but -- or you can run OS10 easy but you kind of have to take some things, you know, for grantor Ted about control.  And so, you know it's kind of van nil language interpreter let's you do what you want you get bleeding edge you have to know what you are doing to do everything correctly especially based on default do a lot of stuff for you drop most of the cape ability that you don't need kind of disable those if your API doesn't need them we'll get into that.  So you know hard mowed you got flexibility [indiscernible] easy you have costs or risk [indiscernible] its new.  It's very very new.  We'll get to that.  So but we are here to talk about attack that's all just stuff.  If you think about this there's a lot of different ways that we can go about thinking what are the attacks in attain never what should I be worried about.  Containers to other containers.  Things in the container against itself essentially trying to elevate privileges win attain fer if you are not just a single process container to the container host.  You know, things  -- whether that's orchestrating and all that other stuff or whether you know other things on the network.  And you can go from there.  So, kind of starting at the top, you know, the colonials obviously is going all the virtual sayings stuff so in the colonials all [indiscernible] file systems, pretty much nobody compiles acorn until from scratch anymore that has all this stuff you don't need.  Inherit tongue vulnerability, special -- thing in -- that you don't use on your server S the other way this goes bad is not dropping disability so [indiscernible] there's been I don't know how many different  -- tolt station where you can go from -- to something else that you shouldn't be doing that isn't related Ted to network country the way that's happened in a lot of cases being able to force to load colonials module that you then control.  But you know speaking to have dropping typability -- sung by this they kept around for a long time the cap [indiscernible] capability if you read and maybe this is what they did you know it says something like you know, you can read some files related to permission, you can also invoke open handle act which is -- which doesn't have documentation they probably thought oh, whatever.  But it turns out that if you actually use that says call and you do other tricks you can -- I need host and read it from inside the container obviously there's other attacks you can do related to that that was something that really gave the disability model a good example of why you need to drop the ones that you don't really need because they are complicated.  And you know if you don't drop those you have to rely on some other thing to to you know enforce the rules on your container and you know that's typically done through a access control unit but if you don't use that, the [indiscernible] you are still root on a host unless you are using the user name space which doesn't have a ton of support yet or adoption.  You also have the problem of limiting access we mentioned how the colonials -- that is not a name space for [indiscernible] there's not a name space for [indiscernible] you have things in the container that are exposed from the host or that bridge that the host and the container so things like  where you can expose the colonials memory into the -- you have things like [indiscernible] everyone forgets about butting LXE for a number of years we are leaving that open.  You also have things like being able to make new devices or CD message that's not a ton of attack service there you are leaking things like fire wall rules.  Unattended access to the -- bridge system buy are default.  A bridge is a switch.  So, most of the default system I think almost all the containers will create a bridge and let your containers talk to each other it isn't something most people realize when they die employ they don't think they will be able to communicate but they can unless you've written fire wall rules or switched to some other network and type.  Container with my favorite function there you are going to be fork bombing the host.  Relating to you know [indiscernible] sung by this a couple times this d ideas of taking your application packaging it up putting a lock on it and shipping it off everywhere you know that sun really happen what happens when there's a vulnerability and you need to update it what do you do you know you can't just you are not supposed to run -- on containers kind of a no no in the container world because there's supposed to be -- [indiscernible] so what do you do and so you kind of need to have a system to adopt have a patching system.  And then the lack of -- so most of the container systems also have some form of API armour support that works quite well obviously not perfect there's been bugs recently publish there.  But at the end of the day you know the colonials is still enforce sink things this is the perfect solution one facet of many different things.  And we mentioned networking dough faults.  For [indiscernible] the defaults are quite bad for dropping that relates to the way that the model of LXC you have been -- dash its supposed to be a feudist things maybe -- whereas idea of one process.  Speaking of [indiscernible] does a pretty good job of dropping capability but if you read between the mind they mention they drop all capability except for those required but if they don't know what you are going to be running how do they no the ones that are still retired.  You have to to be root to use it that end up being bad.  I've been on a number of engagement people have been added to the group but you know if you are in the group, you can essentially root the host .  There's also things like -- doesn't have authentication if you turn that on then you have some problem where some outside attacker can get to it or some -- or some other thing where you can get to that system and it could even be exposed to containers through that -- if to you don't have fire wall.  Also has -- which I think is not just -- has the problem where you know not something like it's in including this from these five veep dorse and they just accept they commit request from five random all that code being -- he doesn't that the code he's committing is going to data center and a crazy company and all it would take would be one commit.  There's been back door -- AMI published that happens with get had you been and all that.  It also I mentioned it doesn't drop all capability by default -- [indiscernible] you can still make devices in your container API armour you know most of the time your API isn't going to need those so you should drop them.  When you bind a port to  binds it to all the interfaces and so if you have a multiple interface going to be on all those.  Doctors some other things reHACK late Ted to base images a lot of people dock consider file main way of [indiscernible] they'll -- but that pulse in like 200 visit megabits of pack damages that you don't need and you know that's kind of a problem where you think you are just running a single process in that container really there's a local bunch of other stuff that could be used or try to attack to break out of that container.  I mention mention giving user access to -- essentially giving them root.  It doesn't have support right now for -- or user name spaces which are two really critical security benefits but it will very soon I think one point ate what they've mentioned about that.  And the other thing I mentioned that I've seen is people for trying to do -- with container something in the container be able to see what's going on with itself they'll expose the [indiscernible] inside the container don't do that.  That's giving your container root on your host.  Speaking about the user name space I have to poke at them a little bit more.  They mentioned on hooker news that you know, on the  you know, we will soon have support for user name space when we feel comfortable the [indiscernible] can run [indiscernible] we will say so clearly that was a year ago.  No problem l more than a year ago now.  And they still don't have it. To give them a little credit I think that one of the major reasons why they don't have the user name space that was a limitation of go and not necessarily their fault but they are on it now.  Hopefully it will be soon.  Also, every container talk has to have an image of some broken container, so I can check that box.  So, rocket is the other one that kind of came on the stage.  It's still you know their whole selling point there's no root Damon there's a [indiscernible] that you can run.  There's no thing after running on your host.  Still requires root to run it.  So, I don't know if they really solved the problem.  Also doesn't drop almost any of the dangerous typability doesn't support the user name space.  Let's see what else.  Oh yeah doesn't support [indiscernible] kind of support -- ins to documentation.  They had a bug where you couldn't run things inside attain never without running them as a root inside the container which seems crazy. They don't have in -- so you know if you read the documentation they have this very impressive design with this multi stage system and it sounds really good, but unfortunately I think this is -- and you know I hope it improves I like to see exesion but I think right now this is where they are.  So then very reply cents l there was I think at [indiscernible] they announced open container project.  I was really really bum bd that they didn't go with this logo because its awesome and a great movie actually changed their name to the open container initiative.  And they really have this specification for containers and kind of standardized the system and have all -- they launched something called run -- which is a very very money mal container essentially is a kind of exesion for proct it's very minimal you know you just define container with -- something like that.  And then its backed by -- which is supported by -- but they still are not work [indiscernible] all that stuff sound kind of bad and messed up and you know what can we do.  So, there's a lot of recommendations I have.  I actually a -- that I'm going to be publish shinning really soon I kind of designed my slides to be high level of each area but the white paper goes no atone of detail all this such really the big message is -- I know everybody says that you know to the a horn a bunch really if you are trying to defend you want to build defense in layers that's really the only way you are going to do it.  So, if -- start with the colonials, cause, its got acorn until, so you want to do GR security everyone should be running it it's ridiculous especially on the server where there's no special hardware you know sometimes on a laptop can be a little tricky with drivers or whatever it may be.  But, on a server when you are just running a few things or you are doing jousted it and so, you know obviously dropping all capability that you don't want trying to design for the smallest possible set you are giving these out but you want to be careful about the what they are and try to fully understand what capability really can do.  You also want to use API armour or GR security -- or some other thing if you can most of the container systems will have API armour by default enforced enabled which is awesome.  It also can be nest Ted it's kind of crazy to think about API armour profile for attain never in that container if you want to have something to -- limited API arm nor profile of that first one you can do that T that's kind of need.  For [indiscernible] specifically you don't want to allow user to run [indiscernible] if you don't want to give them root on that host.  You don't want to run containers that are [indiscernible].  As we've seen either using IP table trick or using the [indiscernible] ozonate need -- you do not need to be root -- if I don't know how many times you've been I'm sure you guys no you've been on a project or looked at security only reason why they are running it because they need to [indiscernible] and that's super and doesn't exist anymore and nobody should be doing it.  Small base images is really cool there's a -- crazy tiny [indiscernible] and this is going to be documented using [indiscernible] if you can so right now with LQC you can use [indiscernible] you want at a -- allow your API to be able to call.  In ways that is typically you know accessed.  You want to do all your normal -- that's a long list of things there's a lot of stuff to do there that's what should be done.  You want to avoid things that are -- or just keep that inned that in mind that cross container network king the other thing that's cool if you have these different interface it is really easy to tell what is going aphony process [indiscernible] so the core picture is you know you've got this harden application you've got [indiscernible] user name space dropped capability [indiscernible] protection read only except for maybe like the log files or something like that.  And then, you got a main mal container and then you do [indiscernible] and then you've got -- on some other kind of virtual machine to gain some like actual hardware separation and then you isolate that whole thing by some trust that you give it do that differently for other things so you know, your containers that are getting hit by API on the front end of your system those should be treated than different trust to you don't want to have those on the same system if you can avoid them -- that that's not ideal.  So where do we go from here more name spaces hopefully will be coming that will help clear that up edge cases where [indiscernible] is a -- so, there's the thing reply sent l in tell came out with clear containers essentially like a 20 megabit [indiscernible] that will give you some hardware virtual sales aspect that you can apply to containers that's cool that I think is going to be need to explore.  -- or more refined going forward.  And also, you know, these -- because this is Linux on Linux anywhere it can run you can do this kind of stuff on I mean bedded devices isolation mechanism or run containers on a phone or whatever where virtual sayings or other things is less keypal.  We talked about server threes no reason why desk top Linux can benefit from all these isolation -- or using containers hopefully there's a lot more effort there I personally run everything in attain never that speaks anything externally or parse anything from the outside.  So Mike services idea which is somewhat new it's kind of cool you end up with something looks like this.  I think budget ways to think about it that actually isn't a micro service picture that's the chrome sand box so if you think about it, the way that Mike service is the idea of them you know splitting up everything into a hundred -- that sounds like a pain in the ASS it let you do a lot of these privileges you can do a lot of crazy cool security wear.  Everyone drives all these boxes on a white board this is what our system look like by the end is of the day there's a whole bunch of other piece that get connected there and they don't have the security they should it's not my off service it can talk to some other thing nobody rights fire wall rules for all that stuff.  So you know the service model kind of splitting everything up into pieces running it all in little containers.  Having, you know, message C U ES and all kind of things in starts off something like that and ends up something like this.  So really it's not about perfect security containers you can't drop that and think you ever security its impoving the work the number of times I've been on a pen test or red team or something I've been stopped by popping something and having it be inside a sand box or container close to zero.  And that's just silly these days.  Not that hard to do containers. It can secure a lot of existing systems that are Linux micro service dark dark that's going to be a lot more popular I actually like it from the security aspect of it.  So coming soon my white paper is coming out covers everything I talked bane whole lot more depth covers all kind of past attacks and locating at new areas and everything hopefully it will be relieved soon.  If you want to make sure that you get it I'm going to be writing everybody that emails mein text file you can follow me on winter I'll talk about it at some point.  But, yeah, any questions or comments?  I'll be around this is the last talk.  Thanks a lot for hanging out.  It's kind of late, but i appreciate it.