>>Um right, so my talk is Are all BSDs Created Equally? A survey of BSD kernel vulnerabilities. Um so who am I? My name is Ilja van Sprundel um I uh work at IOActive I’m the director of penetration testing, uh I do pen test, code review and I break stuff for fun and profit, basically, and then it’s somebody else’s job to fix it. [laughter] Um so my outline is sort of really easy, I have a one slide introduction that sort of puts something out there and then I tried to collect some data uh which is essentially vulnerabilities over the years in uh BSD or BSD- BSD kernels and then sort of lacking more than what I want um I sort of do this sort of test by audit and then basically um I sort of rung through some common attack surface and some issues I found uh and then I sort of of uh lay out some results and draw a couple conclusions. Um so what’s this talk about? About BSD kernel vulnerabilities, a comparison between different BSD flavors uh the audience I think should be- pretty self explanatory, right? If you’re a low level security guy, or you’re UNIX or BSD geek, or if you’re a Linux guy you might enjoy it too. Um or people generally curious about you know low level OS internals, I think they might enjoy this um as far as knowledge goes um I mean I do expect that you have some basic understanding of the BSD kernel um it doesn’t have to be too deep but just a general concept would be nice. Um yeah before I wanna continue um this is not just- I didn’t just create this out of thin air, um there was work by others that I’ve uh sort of relied upon and sort of- that came before me um and I’m sure I’ve forgot some because you always do uh if you feel like you should be on the list and you’re not I’m- I’m the only guy at fault here, um but most all of these guys had basically had contributions at some level at- in sort of the uh- offensive BSD kernel stuff. So this is my entire intro it’s basically a small rant by Theo De Raadt um from about 12 years ago um where he gave some Forbes magazine review about the Linux guys and how they suck um and he follows up with some uh- some uh- um some posts online and he goes uh ‘well you know if the Linux people actually cared about code Quality as we do, they wouldn’t have had as many you know local kernel holes in the last year’ and he goes like ‘how many has it been, like 20 so far?’ right? That’s the entire slide um and now that thing is 12 years old but somehow that’s been stuck in my head I’ve never gotten it out and I’ve been like ‘well you know is that- is that really true? Like, can you back that up?’ um and so I went to look- looking for some data and sure enough most of these things have CVEs um and there’s a website that collects them all and you basically go there and say, ‘please show me all Linux vulnerabilities’ and it gives you this beautiful grid and says, ‘these are all the bugs per year, types of bugs’ all that type of stuff. Um and so if you look at this thing you- you sort of see from 99 til 2007 um and this- this- this- this grid’s actually slightly older this is from uh about three weeks ago the numbers from two thousand and s- 2017 aren’t quite right, it says 346 there, but it’s really today it’s 353, that’s the number of Linux kernel vulnerabilities this year, right? Um but if you start looking at the- at the- at sort of the bo- the- the top, 99 til about 2003 you know the numbers were pretty mild right it was like oh you know 5 anywhere between 5 and 20ish right? So that- and then- and then sort of as of uh 2004 it kind of sort of- you know it sort of hockey sticked that goes on right? Um so the numbers basically get worse and worse and worse, more and more security vulnerability is discovered and disclosed every year in Linux kernel, essentially, right? So now what do those numbers look like for the BSDs? Well, I went to the CVedetails dot come and said uh you know what were BSD kernel vulnerabilities and sort of [mouth noise] zilch and it says net BSD kernel vulnerabilities nothing, CVe kernel vulnerabilities, zero, well I know that’s not true um [laughs] I had some before, they should have at least shown up um but I do know they publish their own advisories um and I guess to get it so BSD it’s neurota um and it’s it’s essentially the same thing except it’s uh much s**ttier to pa- to get data out of um so I spent like an evening writing some uh web crawl and some s**ty protocol that parses HTML and then like f***k around in Excel to get the numbers lined up. Um but I did that and so you sort of you know build a similar ish table um and you’ll see that the- the numbers from 1999 to about 2003 more or less are in the same ballpark as Linux uh and then Linux after 2003 kind of you know goes like this but then all the BSDs kind of sort of stay the same, I mean it’s- it more or less line up, goes up, goes down a bit, it’s usually anywhere between zero and like 20 or something um and that- that hasn’t changed so if you look at numbers even for this year, they’re more or less in line. Um so looking at these numbers um Theo was right, you know, the Linux guys are doing a lot worse than the BSD guys right? A very astute observation um and when he said 20 in 2005 um that was you know a very conservative view if you look at the table here, it’s 133 not 20 [laughs] so you know he was uh- he was off but uh the- his- his sort of uh the impression he wanted to give seems to sound right um. But then I wonder well, you know if we see this kind of hockey stick um in the Linux numbers why don’t we see it in the BSD numbers, right? Is it really true? Is it- is it- is it code quality, is that what it is? And so I wanna sort of- in my presentation I want to I wanted to basically figure out okay are these numbers equal footing, are the Linux guys doing something different, or is everything else the same, and it’s all about code quality. Um and so when I was tal- thinking about this I said well you know there- there’s you know this uh- this many eyeballs thing right? And and and I know I know cause the many eyeballs thing is mostly like you know this sort of it’s not really true and of course not all bugs are shallow no matter how many people you have looking at code, some bugs are just ridiculously hard to find. Um but the general idea behind the many eyeballs thing I do like and so I’m- I’m thinking is there some truth to it? Um cause surely there are more eyes looking at the Linux kernel code than there are at OpenBSD just by sheer number right? Um and so I wanted to figure out if that’s- if that has- if that has remotely anything to do with this. Okay so um first of all I- I sort of looked around and sort of has anybody else done this before right? Cause the way you would do this is you go and look for vulnerabilities in these things and see what the numbers are. Um and it turns out somebody did that um 15 years ago, Um so- uh uh uh a very very nice Australian guy named um Silvio and he did uh um he did a Black Hat presentation about this um and I and I vaguely remembered this and I looked over the slides and I’m like this- you know you get so nostalgic looking at it again and you’re like all these slides are so f***king awesome um and then you start looking at them and then- and then it sort of get to the last three or four conclusion slides and and um Silvio basically goes well you know there’s really not much of a quality difference between the BSDs and the Linux in terms of security. Um now that was over a decade ago, it was 15 years ago, so have things changed, right? The- the history conclusions are 3 years before oh Theo’s rant so things may have changed in those 15 years, right? Um the time spent um in the slides compared between BSD and Linux is like he spent like a week on all the BSDs combined and then like 3 months on Linux right? So the- even- even- like there that’s apples and oranges right? And then the last one sort of the um uh Silvio basically put out this okay I’m going to go look for trivial integer overflows and some info leaks and and so that’s kind of it, right? And so what I wanted to do is I was like I don’t want to limit myself to just those two I wanna be a little bit wider and then see if I can find some other stuff as well and how easy those things are to find. Um so I sort of set- set out and said, okay I’ll do the same thing Silvio did, just 15 years later um so I spend a April, May, June, and a little bit of July um auditing BSD source code. Um and when I set out to do this I was like okay well um where where do I think the bugs are? Where would they be, right? Whats- what’s the interesting attack surface right? And so you come up with this sort of short list of you know where I think they are, right. Number one eh- I mean it’s stuff we’re- a user can talk to it essentially, right? So number one is system calls, right? Users are gonna have to talk to the kernel and they’re gonna have to do it through system calls, right? Uh number two is obviously TCP/IP Stack, right? If you’re network connected [cough] someone's going to be sending you packets at some point so that’s attack surface and so that’s sort of the super common that’s the one that you know my- my mom could come up with those and then sort of a little list of like slightly less common to like you know sort of rare-ish or exotic or something, right? Um and so at the top of the list is sort of the drivers and specifically you know ioctls um cause you know most devices need just someone to talk to it and configure it, and get data out of it Um [coughs] and that’s sort of uh- I started thinking a bit more and I was like oh well these bs- BSDs um most of em you know it comes with these compat layers um that’s probably interesting to go and find- look for bugs um and then um I was doing something else where I had this idea that I’ll- I’ll talk about in a second and based upon that I was like oh you know the uh the trap handlers are probably interesting too and this is where it gets sort of get a little bit more and more esoteric and then I sort of did okay well what about filesystems and it turns out there’s some attack surfaces in filesystems as well and then I’m like okay well, there’s also networking that isn’t um TCP/IP stack like you know wifi, I mean it kind of ties into it but it’s really I mean it’s not, you know you open up TCP to illustrate it it doesn’t talk about you know- uh a wifi frames right? Um and there’s other things like bluetooth and IrDA and I wanted to cover those too, but I ran out of time so I’ll just in that section I’ll only talk about the wifi stack. Um and then I was like, okay let’s just dive right in, and what I’m gonna do is I’ll sort of um have a little slide that sort of introduces the thing and then I’ll show you a little bit of code of a- of a bug and I’ll describe the bug and for 4 or 5 of them I’ll basically run through a little bit of a demo [coughs] So let’s uh [clears throat] start off with syscalls. Obvious attack surface, right? Um it has exactly how the user talks uh to the kernel um without system calls your OS would be essentially useless. Um it says the BSDs have you know evolved since the 70s um they no longer have 4 different system calls uh they- their- they number in the hundreds uh and they do all sorts of funky stuff and performance improvements. Um and these, I mean, th- uh- essentially these are things like you know, open a file, read a file, you know, create a network connection, send- read something from a network connection, close a file, you know, send a message, get my UID, that- that’s the kind of stuff it does, right? Um and so FreeBSD at the top of the list, they have like 550 ish and then um NetBSD is slightly below it with 480 and then at the bottom is like OpenBSD they only have 330 system calls um and functionality wise you might say okay well you wanna have more, wanna know less because that you know have more functionality and while that may be true, from a security perspective, the less system calls the better, right? Um cause no- less system calls means less attack surface, means more security, right? Um so just looking at these numbers you go like oh go OpenBSD, right? Um and then I- one of my assumptions basically that well given that their such obvious attack surface and they’re so well tested and they’re constantly used by apps this is probably you know less likely to contain security bugs, right? [laughs] and so let me show some code, so that’s a system call, this one basically you know drops a sys- a sys log message right? And it takes a buffer and it takes a length and it takes your buffer and your length and it passes on through this function called dosens uh- dosendsyslog and dosendsyslog basically takes the length field and then basically puts it in uh um in on uh uh IO structure and then from there sort of takes out a link- the length field again and then passes that along to uh the kernel malloc function. Um now the kernel malloc function is sort of um historically in BSDs it was very special um the free [inaudible] BSDs have sort of evolved away from that um but OpenBSD still has the original model where um malloc basically goes and says if your length is too big um you’re doing something really horrible and instead of just hanging your system, I’m going to c**p out um and basically panic and then it looks like this, your system essentially dead after you do this. Um and so essentially this is the uh it’s a- it’s an unbound malloc in OpenBSD um that triggers a kernel panic um and it’s- it’s easily triggerable through dosendsyslog system call. And let me see if I can- man I ca- I really can’t see that [mumbles under breath] is that it? What? One more? Uh this guy, there we go okay, so that’s a [laughs] I- I- I seriously can’t see it cause my screen’s different than what’s there uh anyway so that should tell you the U name that is NetBSD uh- OpenBSD sorry, right? Yes, yes! And I- I precompiled the syslog binary um and I will be throwing the physical thread out after my talk uh I just don’t know where yet um but if you- if you wanna figure out where it is um shoot me an email and I’ll tell you. So if we run this thing, boom there you go. RIght there that’s a kernel panic, the box is dead. >>Woo! >>Awesome, thank you! [applause] [laughs] there we go, oh yeah hold on. Yeah that looks about right. Okay so that definitely triggers a kernel panic, right? Okay let’s uh do an- let’s look at a non system call cause we don’t have just wanna look at OpenBSD right? So there’s this system code called kldstat which is um uh in uh uh FreeBSD basically it basically you call system call and you say hey give me a um a list of loaded drivers or for that particular driver, tell me everything you know about it and sort of this- syscall goes back and says, here’s what I got. And the way that works is um it calls this thing called copyin which takes the data from the user and then it calls kern kldstat and it basically collects a bunch of data and puts it in this structure called the stat structure um and I don’t have the function for um kldstat in here cause I didn’t want to spoil- I didn’t want to like pollute um the slide but essentially what this function does is it fills out two strings uh that can be about 1000 bytes long um and it sort of um copies an alternate string into it uh but then everything else after the no byte is never initialized so you basically end up with about a half a page for the info leak stuff just data that’s on the stack never initialized, right? Um [clears throat] this is in uh FreeBSD 11 is when I- when I looked at it went back and saw that this book has been around for ehhh more or less about a decade um so [clears throat] let me see if I can demo that too. [mumbles under breath] okay this works better. Uh I think that one’s right, yes So this basically um triggers the bug over and over and over and what you’re going to see is like a long stream of like random kernel bytes um [indiscernible] boom there you go! All that stuff is that’s all you’re leaking, you’re leaking, you’re leaking. Yeah, there’s some weird bug where sometimes it hangs but it basically just keeps leaking and leaking and leaking, and leaking, that’s endless amounts of [indiscernible] um I was thinking of doing a demo where I grabbed some stuff out of it um but I didn’t um ran out of time. Um. Oh yeah um d**n it. There we go, okay. Oh yeah. Um [clears throat] so now that I’ve seen two of these bugs um and I have several several more uh syscall bugs uh I didn't want to spend all my time just demoing those those things um but essentially you know the previous assumption where we go ‘oh well you know these things are well tested, they’ve been around forever, it’s highly unlikely to get bugs’ well that’s not really true, turns out bugs do occur in system calls with some regularity um and particularly newly added system calls, right? Um that do senses locking has only been around for one version um the KLD one happened to be happens to have been around for a decade um there’s a bunch of others like um I have- I have a bunch of other like FreeBSD ones um like you know that show up um so you know that the assumption is isn’t really true like if you just assume syscalls won’t have bugs well that just isn’t true. Okay so let’s move on to the TCP/IP Stack right, hopefully presumably everybody knows um uh what that is and what that means um it you know it handles networks you know [clears throat] the low level stuff IPV4 and 6, Udp and tcp and the sequence numbers and you know, Ipsec and how it all ties together and then how you route it back and forth to userland app and those kind of things. Um this is obviously very well known attack surface [clears throat] it’s been around forever, again the same assumption here is that this is very well tested code, it’s been around since the early 80s uh very unlikely to find security bugs in there. Right So here’s another piece of uh OpenBSD code this basically takes uh P- pppoe vert net packets and sort of passes them all to to this handler and it sort of loops over the the packet data and it looks for a tag in the length when it finds a tag in the length it goes into the switch and the switch goes ‘oh hey, this tag, this tag, this tag’ in this particular case, there’s a um an error [clears throat] some error tag in this case I guess SName error and it basically goes there and says ‘okay let’s me- set the error message to this’ and then send a flag which is error tag and then end of the uh switch case you kind of there’s this code that kind of goes ‘oh hey, if we got this particular error message go and and and go do this’ and then um they obviously pull out the thing out of the packet um before they can actually touch the bytes and this is- this is like a bizarre um uh uh they have these things called Mbufs which are basically very pa- specific buffer structures that are designed to hold network data, right? And there’s all sorts of APIs around them to work with them and so they’re, they’re made in a way so you can chain it together so that if you wanna like append or prepend or like ch- swatch- uh switch stuff out as you’re building a packet that- that they work, right. And as part of that um before you ever touch any of these things um you- you you’re not going to need to have a continuous buffer, right? Um and so there’s a function called Pulldown and basically what what Pulldown does is it says ‘okay what I’m gonna do is I’m gonna look at all of these things chained together, um, and when I successfully when I- when I- when I complete and it’s successful, I’m going to hand you back a you know a buffer that is completely continuous and you can start using that’, right, and so it does that um and if that works it goes on and does what it’s supposed to do. Now if that fails, it bails out um and it basically goes to this thing called Done and then Done basically goes, says, ‘Oh yeah this is your original uh Mbuf let’s go and free that thing’ right? Um and so that code looks fine except if you look at the main page for Pulldown um it says in uh the fine print it goes ‘well if Pulldown fails it frees the Mbuf and then it tells you it failed’ um so what’s happening here basically is if the Pulldown fails, the Mbuf is freed, and then it goes and frees the Mbuf again, right? So s- uh uh sorry. Um so uh uh a pretty trivial case of um you know double free um this isn’t o- I found it in OpenBSD 6 point 1 and it turns out it’s been there for a decade and change um and then um one of the things I did is I tried to cross reference, because they do have a fair amount of shared code, so I tried to cross reference um and I looked at the NetBSD code base and then I saw that in February of this year they discovered this bug independently, and they fixed it, but uh there was never an announcement made, it was silently fixed Um [laughs] [applause] and so again the- the previous assumption that this is well tested code, and it’s highly unlikely there’s a bug in there, well that isn’t really true, it’s- I mean it was well tested code, it’ll still have bugs uh and the other thing I discovered is that um so these mbufs are incredibly complicated and it’s very error prone, right? To- uh to do it right and never make a mistake, it’s kinda hard and one of the things is that um a lot of APIs have sort of handled these things um it depends on their failure cases, so about half of them basically go and say ‘okay if I fail I don’t free this buffer’ and the other half go, ‘if I fail I’m going to free these buffers’ right? And so that creates an inconsistency where if you’re a developer, right and you’re you’re- assume the API uh uh um frees on failure um and so uh you basically say ‘okay I’m not gonna free this thing because it frees on failure’ but if the API really doesn’t free on failure, then you’re leaking memory and the opposite oc- occurs too, right? Where if you go and say ‘okay I’m pretty sure it’s API on the return- on the failure doesn’t free- all the- will free it, you know it’s double free so this- it sort of you know inconsistency among API behavior um uh combined with the complexity of of the mbuf structure sort of creates this um situation where like it’s pretty d***n hard to get things right every time. Right. [clears throat] so let’s move on to drivers right. Ioctls you know everybody's favorite hack surface um you know lots and lots of drivers are out there um B- BSDs they’re no different, they have piles and piles of drivers for all sorts of stuff um and you know eh- obviously BSDs UNIX and in UNIX you know everything’s a file so the way you expose a device to userland is you put it in slash dev right? And you give it some name and obviously these things basically expose file operations to userland right? Like file and open and read and write and ioctl and things like that. And I guess you can mmap some of them too, things like that. Um [clears throat] so you know ioctl obviously is where you have an attack surface okay so uh let’s take a look at an ioctl, right? So um that BSD has this device called crypto um and basically what crypto does is um it takes uh uh uh an ioctl input and I actually- um anyway- it takes uh an ioctl input and it basically goes and says ‘okay well here’s uh here’s some structure that comes from userland and it contains this uh this account field’, and what we're gonna do is we’re going to create an array of number of account fields um and then we’re going to start running data into it, right? And then we’re going to start using it, and then we’re going to start like, manipulating stuff. Um and as as you can see there’s a count time size of the structure um and anybody who knows anything about writing code and security knows that that is a very class integer overflow. Um [clears throat] right that’s integer overflow right there, and that’s your memory corruption block, right there. The- I didn’t- again I didn’t want to put more code on the slide cause it would just pollute things but that crypto dev mkey stuff is basically you know starts uh uh uh writing stuff to to that buffer that was previously allocated. Um so yeah that’s uh uh uh uh classic integer overflow leaks memory corruption um it’s in NetBSD it has been there uh s- about 9 ish years um and then hold on let me see Yeah let’s uh There. So this is basically hopefully fingers crossed gonna trigger the um integer overflow and then cause memory corruption and it’ll basically trap into the debugger. There you go. Boom, it’s on- it’s gone, memory corrupted, rebooted [applause] [clears throat] so here’s another one this is uh uh this is uh um this is a device on FreeBSD called ksyms which basically gives you the ability to expose kernel symbols to userland, right? Um and it basically as I mentioned earlier, most of these drivers have an open callback, and this one’s no different. And when you uh when you call the open system call it goes to this callback and this callback basically sets up a few things so you can later do operations on it, and then it returns successfully. Uh one of the things it does in this case is basically um so it creates um a file descriptor storage specific private data which means there’s kernel private data associated with the final descriptor you’re going to use in userland. Um in this particular case what they do is um they uh uh look at the pointer to your uh um that’s basically contains all the data needed to describe your virtual address space for your- for your process and stores it in that pointer or or yeah inside that uh uh driver specific um uh uh uh storage um and then basically returns successfully. And then um later on there’s another system call that basically gets called if you do an mmap in that particular file descriptor um and as you can see it references um that particular pointer that was previously saved, right that uh sc pmap is saved here and it’s referenced here, right? Okay so the bug here is that uh um uh that pointer using mmap doesn’t necessarily still have to be valid then the one using open right and so the way that works is um- ‘cause usually you say okay well the file descriptor associated with the process and so as long as the process is around the file descriptor will be around and so that virtual address space map will still be around except you know OpenBSD er- er- at least the BSDs in UNIX in general is smarter than that and it gives you the option to basically not just hold on to a file descriptor but instead go to a different process and say, ‘hey you can now have this file descriptor too’ and then once that guy has the file descriptor, you can say to the other process, ‘okay well you go and die’ and so the other process is gone and so all data associated with it is entirely gone and expired and now you have that second process, and the second process is still holding onto the file descriptor and all of a sudden it says ‘oh well now sounds like a good time to call mmap on this thing’ and mmap goes around and says ‘oh look I have this- I have all these pointers to uh this particular process, this virtual memory uh space’ um and it goes and references that, does all sort of things with that, except the original process is gone, it’s dead, it’s entirely expired, um and so this guy um ends up- if you do that, you know it ends up doing really really horrible things. Um and I would love to demo this but um well I have a pock ‘cause it’s a s**tty race to hit uh this one took me a couple days to hit um I don’t have that- I don’t have that long um to show you um I will be releasing code um so if you wanna play a- play around with this uh you can just for now you you’ll have to do with a screenshot of a kernel panic [clears throat] okay so that’s it for this sort of super common attack surface. Uh so the compat code is another one where it’s like okay this is an interesting uh attack surface um and so uh the one there is basically so most BSDs have this thing where they tr- um offer additional binary support uh so for example uh uh FreeBSD NetBSD and up until a couple of years ago OpenBSD had support for Linux and what that means is you can have a Linux compiled binary, you would run it on FreeBSD, and it would just run. Not just ‘cause you have to set up like an environment but by and large the binary would work. Um and it could do other things like uh you know emulate [indiscernible] environment of the 64 bit version of of of of that BSD you have um a- a- and it could do um like old versions of the OS, things like that. Um and the way that works is basically what they do is they emulate a whole bunch of system calls, they fake a- the system call that you expect it to be and then they sort of funnel it to the real system call and then sort of you know parse the stuff back and forth um they uh now s- uh um compat layers are a great idea from uh a user functionality perspective, they are an absolutely terrible idea from a security perspective um and uh Theo De Raadt who’s the you know the leader of of OpenBSD and and you know a pretty go- d***n good developer um you know had some really interesting insight of the compat layers and if anybody would know, it would be him, he goes, ‘well you know, people who rely on this stuff don’t care enough to maintain it and the people who work on the actual kernel uh don’t give a d***n about compat layers ‘cause they don’t use ‘em’ um and he says ‘cultures aren’t aligned in the same direction’ and then he says, ‘compat layers, rot very quickly’ which means you write one today and tomorrow it’s gonna be just useless and just have nothing but security bugs in it. Um so here’s the uh uh um here’s the NetBSD um this is it’s this like svr4 which is like fairly ancient [indescribable] thing um and they basically emulate that on NetBSD so if you have original binary compiled for that and you move to binary NetBSD it’ll still work and it has this thing called streams which is sort of a predecessor to sockets ish um and so what they do is, NetBSD doesn’t support stream so what they do is they fake it and they sort of put a socket underneath. Um and so the- the original svr4 binaries will think they have a stream even though it’s a socket um and so this thing basically it’s it’s essentially uh um it’s essentially an an ioctl that’s supposed to do a socket operation um and it copies in the data structure and they put it in the BSD structure, and it passed on to this netaddr the sockaddr in and netaddr sockaddr in basically looks at this uh structure and it says ‘oh this thing contains uh uh you know a 32 bit offset inside it’s structure I’m gonna use this offset and I’m gonna add it to my pointer and I’m gonna give you back a pointer to where you can start reading from’ uh problem there being is there’s absolutely- uh no validation being performed um and so basically you can end up with a wild pointer that points anywhere um which you can either use to trigger uh kernel panic you know your box goes down or you can if- if you know what you’re doing you can very selectively pick pieces of memory and start leeking you know very selective pieces of of kernel memory. Um and then what’s really beautiful is that at the top of this file someone said ‘well we’re going to pretend to have streams, we don’t really, oh yeah by the way this code is really gross’ [laughs] Um so yeah this was found in uh uh NetBSD 7 point 1 but this things been around for uh uh 21 years uh it’s been there since NetBSD 1 point 2, it’s been around for a very very long time um and it- it gives you the ability to read arbitrary ish kernel memory or cause kernel panic. And so I submitted this bug to the uh NetBSD guys uh and they go ‘oh s**t yeah this thing- this thing is tot- total piece of s***t’ and they go uh ‘we fix- we fixed a gazillion bugs in this svr4 thing uh we should have never enabled this by default it is an absolute minefield’ um and so when I submitted the bugs and they fixed them they also changed uh their OS config file to be like- and and basically turn off svr4 support um if you want it now you’re gonna have to recompile your kernel. [laughs] [applause] Okay so another thing I was thinking, I was like okay, well, okay we have these things called trap handlers and um this is where things get a little bit exotic um so trap handlers were these sort of callback that you reg- that like very low level OS code registers sort of when it boots up and it’s goes eh- basically goes to the os- or the OS basically sets things up and so when some kind of a hardware thing happens these things get called, right? Some kind of exceptional fault happens and these are things like division by zero, a system call, a breakpoint, invalid memory access, um when it- when the hardware sees any of these things it sort of goes ‘oh I don’t know what this is but I’m gonna call this function pointer and we’ll have that deal with it’, right? Um some of them can be triggered by userland and some of them can be triggered by kernel um and the kernel has to trigger them correctly um the thing though with trap handlers is that they are unbelievably nasty pieces of code um and exceptionally architecture specific. Every architecture and like every little miniscule like little dif will have between architectures things are very different and then every little dif within the architecture might still have like tiny little changes between them um it’s an absolute pain to try and understand these things um and it is- it’s a ridiculous pain to look at this code so I didn’t look at it. Um so I said okay but I’d still like to play with this a little bit, where do I go from here? And I was like ‘Okay well what can I do to trigger um uh uh eh- exceptions an- and faults’ well what if I just start executing random instructions, what happens? Um surely they’ll trigger a bunch of uh uh uh uh uh uh uh e- exceptions and traps and then the OS has to handle it and then I have a slightly more um bigger fuzzer, this is essentially it this is uh sort of a one slide um sort of uh uh trap handler fuzzer where essentially what you do is you map a piece of code read and write in executable you sort of start a loop and every time you get into loop you basically you know read out op dev urandom- uh a page of random garbage, you fork off a process, you hand that phd process and the process basically goes, function pointer call’ and then surely crashes and it does this over and over and over and over and over again um [laughs] and essentially I did this on the previous Z and you know there’s a uh a Xen trap that kinda blows up my face and then [applause] [laughs] and there’s some other signal stuff that uh um sorta blows up and I have a demo of this, I’m not sure if it’s gonna work ‘cause this thing sometimes blows up fast and sometimes it takes awhile but I could at least um I could at least try and and show this thing. Uh … Oh man this is so horrible ehh. Tu- uh god d***n it! [laughter] I got em! Okay there we go, awesome! [applause] Wow look at that, this is I- this is a horrible set up I got here [chuckles] And plus my eyesight sucks um so this page functions for the [indiscernible] processes I don’t know if it’ll uh I may have to come back to it later ‘cause um give it a few minutes it might die um >>So these are all crashes of that second thread but like how many can- >>Yeah this has- doesn’t crash the kernel yet but eventually it’ll- it’ll crash the kernel um if we’re lucky, fingers crossed um I got a hit earlier this morning and- and usually it only takes a couple minutes but it’s like random garbage so it’s like >>And how are you recording which [inaudible] [speaker inhales] >>Uh so not in this version but I have a version that basically uh has utp socket and sends it to a server and so the server basically goes and says ‘okay this is- this is the garbage it generated- this garbage it generated’ and so when it crashes I look at the last packet and I go like ‘okay let’s decode what this- set of it- set of bites are like you can feed it to like Ida or something and it’ll tell you, this is what it means. Um anyway I’ll- I’ll get back to this later, I hope it dies [laughs] Uh um but essentially so yeah there’s uh uh uh uh uh uh a Xen trap I hit and then there was another sendsignal trap I hit which is this one um and so hopefully I ca- I can show this uh uh in a little while um [clears throat] so let’s mo- move on to file systems um so obviously when you think file systems you say okay well you have a file system image and you can mount it and obviously if you or or you know have USB stick and plug it in and have that mounted and if somebody doesn’t parse the file system correctly then you know bugs show up right? That is number one that is an easy attack surface. [sniffles] Um there’s another one that I sort of- I was playing with and I was like, okay that could be interesting, so in the last couple years all three of the BSDs have built in support for fuse, um and I don’t know if you guys know what fuse is but fuse essentially just a mechanism that allows you to have userland file systems, and the idea being there is that you know, writing file system sin kernel is really, really hard, um and so if you ofed userland it’s okay if your file system does something bad or crashes, the OS won’t go down. Um and so what that requires is, um, your normal BSD uh VFS layer now has to support a file system that’s in user as opposed to one in kernel um if you’re talking, if kernel- the kernel is talking everyone’s trusted, if kernel’s user’s talking, they’re not trusted. So you’re now taking this layer that is always talking to someone that it trusted and you’re basically now feeding it a bunch of data from userland that’s no longer trusted um so the VFS layer- layer- layers in theory should have been hardened um for fuse. Um so I looked at fuse for all three BSDs and basically sort of my impression was the NetBSD- fuse-they’re- they’re- this is funny right, so all the B- they’re family right? They all fork from each other except when fuse came around they all said ‘nope we’re all gonna sort of make our own little thing’ and so you have three distinct uh fuse implementations for the three different uh BSDs um and so NetBSD sort of um in my view made the most complete implementation which allows for- for the most options and flexibility. Um FreeBSD I think has the most controlled one in terms of uh the amount of stuff they’ll take from userland in terms of file system is highly controlled um and so what that means is there’s relatively little opportunity uh for consumers to make mistakes but there’s more opportunity for the parsing inside the fuse code itself to make mistakes. Um and then um OpenBSD sort of um uh has sort of just minimal implementation I mean they all uh um none of em on ioctl but they implement all of the other uh file operations. Um D***n I’m gonna have to skip through this ‘cause I’ve got only a few more minutes left I think and there’s conclusions I want to get to um but anyway this is the getcwd system call um for um uh for the BSDs [clears throat] uh for OpenBSD and that- you can hit that from- from fuse um anyway let’s- let- let’s skip over the file system stuff. Yeah the network is interesting you know there’s some bugs in the networks- in the network stack where if you send a packet the box goes down. Um [mouse clicking] blah blah blah oh and this is one of these things where there’s all- all these uh wireless drivers that have like these straight straight out heap smashes where they’re not expecting a malicious radio even though they’re taking data over over USB and so if you have malicio- k- k- a malicious radio all of a sudden the OS has all sorts of heap overflows. But let’s skip through that through right there’s some miscellaneous stuff I want to talk about but I don’t have time [laughs] um what oh yeah um so yeah results. Um I looked at this stuff for about uh 3 months um uh I found about 30 or so bugs in FreeBSD, uh 25 or so in OpenBSD, 60 ish in NetBSD, and it’s- thing is it’s hard to sa- ‘cause I sa-, I give approximate numbers because some of these things are like, it’s a single function and it contains 5 buffer overflows, okay is that 1 bug or 5, right? It’s- so I try to sort of be like ‘err it’s this many’ um depends how you make- or how you sort of look at these things but I think these numbers more or less make sense. Um in terms of bugs I’ve seen it’s like a very long list of, I mean there’s the straight heap and and stack smash like in those wireless drivers uh a couple of race conditions, expired pointers, double frees, recursion bugs [chuckles] integer issues, um very very interesting refcount issues um obviously overflows but also there was one uh NetBSD bug where they’re like ‘okay we’ll take a refcount on this v-note’ and then ‘okay we’re gonna take the mount point out and then we’re gonna drop refcount and now we’re gonna play with the mount point’ well there’s nothing tying them together so the mount point goes away, you're- your’re meaning well, pointers. Anyway, other things that I found were info leaks, out of bound reads, NULL derefs, logic bugs, um a typo where they literally called their own instructor, it was something something TX and it called something something RX um division by zero bugs, uh kernel panics and memory leaks. So um [clears throat] found bugs in all 3 um and basically you know, in the attack surfaces I discovered uh I mentioned earlier uh and sort of among the range of the bugs that I- that I just listed. Um winner loser is is in my impression fairly easy, when I started looking at this stuff OpenBSD came out as a clear winner um for a number of reasons, um they have massively, massively uh reduced their attack surface, um they got rid of loadable module support, they uh um have relatively few devices, um [chuckles] got rid of all of their compat code, including Linux, [chuckles] they deleted their bluetooth stack ‘cause it sucks [chuckles] um they have- more- they have more than 200 system- 200 less system calls than FreeBSD um and they cut support for a bunch of really s***tty architectures. Um on top of that um when Theo said they care about code quality, they do. Um they have virtually no integer overflows or signedness bugs they’re as good as gone in most places, except the ones where they didn’t know about the attack surface, like the wireless drivers. Um oh- yeah- um and then also very few info leaks, so the bugs they know about and the patterns that they know um they look for them you know like a madman and- and- they eradicate them. Um NetBSDs one of these where it’s like okay, comparatively they’re the clear loser, um tons of lazy code and compat code so that’s- you got an attack surface right there um [clears throat] they seem to be less consistent um when it comes to uh um code quality and too many signedness bugs. And I know how this sounds but- and I’m not trying to dis them because I know it’s incredibly hard to write an OS, right? If you think it’s easy to build, maintain, improve an operating system, you should try it, it’s really hard, like you will cry yourself to sleep. And then somewhere in the middle is FreeBSD um now when I started reporting these bugs um sort of- here’s the response that I got, right? When I emailed the OpenBSD guys uh it took about a week or so to get an email back from Theo and fir- he fir- he’s like ‘oh you know I’m so sorry I took a week, I was out on vacation’ and then within- you know after that week like bugs started rolling out, bug fixes started rolling out, you know uh the next couple of days, everything got fixed within a 2 week period. Uh FreeBSD guys responded as well um in uh uh about a week or so and then filed all the bugs, I don’t know the exact status of the bugs. NetBSD and this is where they really shine though, so I was kind of a lazy b*****d um and I was sort of uh uh [chuckles] uh procrastinating and I was like ‘oh I don't want to write up all these 60 bugs’ so I did only got them wrote up right before I got to Vegas and so then I sent them these bugs a couple days ago [chuckles] which is really unfair of me and I was thinking like ‘oh how am I going to deal with this, I don’t want to like make them look bad’ um and then something happened that was like ridiculous um they fixed all 60 bugs overnight. [laughter] [applause] Which is unbelievably impressive, and I told them that, and I told them like if you wanna call me out on being an a*****e you can, but I am in awe of you guys ‘cause that is insane. Um additionally they also turned off SVR4 which I think is a great deal and when they emailed me back they said ‘we have disabled this thing, something we should have done a very very long time ago’ Um so yeah pretty much and and my insights, more conclusions. Bugs are still easy to find. Uh very- various levels of quality. Um most consistent uh was OpenBSD um yeah one more point I wanna make is uh maintainers of these BSDs should talk to each other ‘cause I had several bugs where one guy had the bug, and then the other guy fixed it like 6 months ago, right? And it goes both ways, right, it’s OpenBSD and NetBSD and it’s FreeBSD um and uh I got another funny story where- funny story about that about a guy I met at a bar yesterday, [inaudible] he’s a developer, but I don’t have time to get into it. Um yeah code base is is OpenBSD is small, FreeBSD’s the biggest and so it’s interesting to look at these things uh it’s clear OpenBSD has the least amount of attack surfaces, FreeBSD likely has the biggest one. Um this obviously plays a part you know if you- if you don’t have code for something then you can’t have bugs in it, right? [laughs] and there’s a thing between accidental versus planned, right? Uh some things didn’t get ac- get- get implemented, just didn’t get to it and other things you made a conscious choice to sort of get rid of it. Um so yeah the many eyeballs thing, I think it’s a factor. Um based on my results I I think it matters, I think code quality alone doesn’t account for discrepancy, I think code quality has something to do- uh I think the the many eyeballs thing has something to do with it uh my my view is that you know say what you will about the guys looking at the Linux kernel, and you could say a lot about them, um there are just simply so many more of them and it shows in the numbers. Um yeah and then this thing leak- this thing sort of presentation got out before I did this thing, and so the internet saw this and they you know had some questions and and and I don’t I’m pretty much out of time but I wanna ‘cause somebody asked about uh um DragonFly BSD and I did not look at DragonFly BSD, I considered it uh maybe I should've, I just didn’t, I ran out of time. Um and then some links, and then that’s pretty much it, I think i’m out of time right? [applause] Awesome.