Uh, without further ado, please help me welcome James and Georgie.
Cool. Uh, thanks everyone. Um, so I'm James and this is Georgie. Um, alright, let's get
started. Uh, so, quick agenda. Um, so we're gonna run through our motivation for writing a
kernel fuzzer. Uh, we'll go over our, our architecture. Um, some caveats to be aware of if
you're doing your own kernel fuzzing. Uh, we'll run through Windows as a case study plus the
other operating systems we've looked at. Uh, we'll run through the results we've got, uh, and
what we're looking to do over the next few months. So, motivation. Um, typically now with
sandboxing, uh, kernel exploits are becoming more of a necessity. Um, it's quite hard to break
out of sandbox environments, particularly if you're looking at things like Chrome. So if you
look at the last Pwn2Own wins over the last few years, they've all used kernel exploits to
break out the sandbox process. Uh, so yeah, kernel exploits, pretty important nowadays, uh,
for, for things like Privest.
Um, also at MWR, we have a bit of a friendly internal competition going on. Uh, so Neil's
presented on his version of his Windows kernel fuzzer at T2. Um, obviously, we're trying to beat
the number of bugs we, uh, uh, that he got in his version. Um, but obviously, we're primarily
here to try and improve general operating system security. Uh, initially, and most of our
research has been focused on Windows, um, Windows 7, uh, but we've actually ported the fuzzer to
run on OSX and, uh, Windows 7. Uh, we've actually ported the fuzzer to run on OSX and Windows 7.
QNX. Uh, and we're looking to start running it properly over the next few months and trying to
find some bugs in those, uh, OS's as well. So, how hard can it be? Uh, if you look over at
Twitter, it would appear everyone has Win32 KO Day. Um, and so do we. Uh, so about a month
ago, uh, we did a run, um, purely a test run to make sure the fuzzer was pretty stable. So, as
you can see, 16 VMs, just 48 hours. We got 65 crashes, 13 of those you need. Which, yeah, I
didn't think was too bad. 13 O days in Win7. Okay, cool. Uh, as it turns out, though, we weren't
being particularly effective with our fuzzing. So, for those that don't know, there's a number of
syscalls in Win7 that are simply just sleeps, for all intents and purposes. So, we've got the
one there, NT Delay Execution. So, we managed to find that one, blacklisted it, and actually,
surprise, surprise, the fuzzer was a lot more effective. Um, so we, we ran it again last week.
We haven't triaged all the crashes, which is why we've not, uh, spoken in the slides, but we ended up
with around 150 crashes, uh, and 40 of those were unique. Um, in the end, actually, uh, we, we
crashed our database. We were pushing all these, uh, uh, crashes too, because it ran out of space
for it. So, that's, seems pretty cool. Uh, we are going to release the framework for the fuzzer.
Um, so, the core will be released, um, but we're not going to release any OS specific stuff. Uh,
but throughout the talk, we're going to go through how you would write the, uh, uh, the OS
specific stuff, and give you some examples.
You can go away and fuzz the OS's yourself. Um, it's worth bearing in mind when you look at the
code that we're hackers, not developers. It's pretty dirty in places. Um, so, yeah, bear that
in mind. Our sort of test is, does it compile? Great. It probably works then. So, the
architecture. Um, everything is nicely decoupled. So, you've got in there the center of the
framework core, um, which is basically what runs everything on the outside there, uh, which is the
OS specific stuff.
Uh, originally, we dev'd this up in Python, um, but we've completely rewritten it in C. Uh, it's
much more efficient, significantly faster, and seems to be finding a lot more bugs for us, so.
So, the first part is the OS API knowledge base. Uh, all this is, is the OS specific API that is
used by applications to interface with services provided by the OS. So, this is the things devs do
to, you know, do things with graphics, or interact with devices, or do networking, uh, et cetera,
et cetera.
Um, many of these will wrap system calls. Um, we'll provide two examples of how we've dev'd
these, uh, for Windows and OSX, um, later on in the talk. Um, but we won't be releasing all of
our ones. The next part, the next major part of the OS specific stuff is the system calls. Um, so
system calls are a low level method for user space to kernel space communications. Uh, these have
to be implemented per architecture and operating system. So, that's fine for open source systems,
you know, OSX, or any Nix based system. These tend to be open source. Uh, for Windows, it's a little
bit more difficult. You have to reverse engineer these ones out, or, or rely on other, uh, sources,
maybe React OS, or something like that. Uh, at the bottom there, you can see what a syscall looks
like internally to the fuzzer. So, we just have a syscall ID, uh, the arguments for that syscall,
and then just a return data type if it, if the syscall provides a, uh, a return. Also, we have a
number of, uh, fuzzers. Uh, we have a number of, uh, fuzzers that we can, uh, we can use to, uh,
fuzz values. Um, so, there, you can just see the, the fuzz values you can grab internally, uh,
from the fuzzer. Um, we use the words fuzz, but what we, we don't necessarily mean fuzz,
actually. Um, so, really, we actually want the library calls and syscalls to work, uh, and to
get these to work, we actually have to return proper values. So, whilst we occasionally return
a properly fuzzed value, most of the time, you'll return a normal value that is gonna ensure
that the library call and syscall call works. Um, yeah.
So, the next part is the object store. Um, the object store is used to maintain state across a
fuzzing run, uh, and it stores OS specific objects of interest. So, currently, this is handles in
Windows and file descriptors in Nix systems. Uh, it's implemented as just a global array of
structures. Um, it's deterministically populated by the fuzzer, and this is quite important for
when you want to reproduce crashes. Uh, if this wasn't deterministic, then obviously we'd just
never be able to reproduce any crashes we found.
Uh, it's, it's quite easy to retrieve, update, and insert new objects into the object store.
Okay. Uh, the next bit is the helper functions. Um, so, helper functions, we use these for things like
library calls that require a struct. It works in pretty much the same way as, uh, grabbing a fuzzer
value. So, if you want to call for a struct for a library call, you make a call to the helper
function, and it will return you a, a struct that should be, uh, or should work fine in the library
call.
So, logging. Um, this is actually quite difficult in kernel fuzzing when you keep hitting crashes.
Um, so, initially, we didn't have high hopes for the fuzzer, uh, so all we did was log, log a PRNG
seed. Um, as you might guess, as a fuzzer, we make heavy use of the round function. So, if we can
just seed this again, we'll always get a, a, a same run again, which makes reproducing quite
easy. Bit of an issue, though, is it doesn't generate a standalone test case, which is ultimately
where we want to get to. Um, so, what we've moved on to doing is logging seed statements. So, our
log files are actually complete source files. Uh, and all we do is copy and paste these into a
template. There are some issues with this. Um, so, occasionally, uh, if we get a bsod, we'll find
that actually the OS hasn't flushed the, uh, the write to the file before it's bsoded, uh, in the
Windows instance. Uh, so, we miss out on, let's say, the last call. Um, as a workaround, what we're
doing is we're usually able to, uh, we're able to, uh, we're usually able to, uh, we're able to
grab the stack trace and the arguments and everything else and figure out what it was doing.
Uh, what we're looking towards doing in the future, as well, is maybe using something like logging
over a socket or something like that, uh, which we're hoping will work a little bit better.
As you might guess, this is incredibly tedious. Um, so, this is an example of our logging from a
library. Uh, what we do is we grab a, uh, a variable ID to tag onto the end of a variable. Uh,
and then we assign that variable and call a function. Uh, as you can see at the bottom, um, we
log the function call before calling the function in the fuzzer, fuzzer to try and, uh, ensure that we
log the crash before it actually crashes. So, crash detection. Um, there's actually two methods
we tried with this. So, the first way we tried was just attaching a kernel debugger while we were
executing the fuzzer. Uh, obviously, requires one or more debugger processes, which slows
execution. So, you can either have, uh, on the host, uh, a debugger, or you can either have a
host debugging the VMs, or a VM debugging another VM. Uh, but the plus point is we instantly
analyze, uh, any crashes we find. The other option is unattended execution, which is what we're
working on at the moment. Uh, and this is just letting the OS handle the crash on its own. Um,
much faster execution, and we can run more VMs as well, and we have less, uh, wasted CPU cycles.
Um, we recover and analyze the crashes, uh, upon reboot. Another way we've been looking at,
actually, is, um, using hypervisors.
So, for example, VMWare will actually log when a host, uh, a host has crashed, and there's some
information in there that is actually quite useful. So, this is the way the logging, uh, the crash
detection, sorry, works. Um, when we reboot the box, the first thing we'll do is try and search for
memory dump files. Um, so, this is quite specific to Windows. Um, if we find a memory dump file,
we'll match that up with a log file. So, there should only be one log file on the disk, which is
the one that caused the crash. Um, if we get that memory dump file, what we do at the moment is we
under WinBag, bang analyze, and just grab those, uh, details out, and then the three of them, we
timestamp and just archive them all off. So, with a number of VMs, so our ultimate goal is to run
this across hundreds of VMs, is we need some form of way of the fuzzers, fuzzer to push this into a
central database. Um, so those three files I just described are actually pushed into a CouchDB
database that we just have sat in AWS. Uh, this is really useful for us because it means we can
actually do deduplication across all the VMs. So, what we do is hash the stack. Um, if we've seen
that, uh, hash before, we just drop the files and increment a counter. If we haven't, then we
push the files out into a CouchDB. It also means we can do some other categorization stuff. Um,
so, doing stuff like type of AV, 14 IP, bug check ID, et cetera. Closer to the mic. Okay.
Okay. So, that's the overall architecture for the fuzzer.
Uh, right. Yes. So, that was the overall architecture. So, this is what a fuzzing run looks like.
Um, the first thing we do is we have a little bootstrap, uh, script that we run to prepare a
VM. And, uh, I'm going to show you how we do it. We've got a bunch of script that we've got to
for fuzzing it's just a really simple python script it installs everything that's required
and just fine-tunes the system so we'll go through what the windows one of that looks
like later on in the talk we then launch a second python script which is a wrapper around the core
fuzzer but what that one does is go away and collects the memory dumps and submits it to
the database if it finds them etc etc as i just said we then launch the fuzzer binary
and the first thing we do is populate that object store and so we generate a bunch of
valid handles or file descriptors that we can then use for library and system calls straight away
okay so then for a predefined number of iterations we pick up either a library or system call
so we'll go away figure out what the arguments are and then grab either fuzz value fuzz trucks
or an object from the object store and then we invoke that call we check to make sure it
succeeded
and if it gives us a value back that's useful later on we'll pop that back into the object
store so you know library calls or system calls may return a hand or a file descriptor or something
similar we want to keep hold of that value and use it later on at the end if we didn't get crash
all we do is just clean up all our temporary files remove the logs and just try and revert
back to a clean state now currently all we do is reboot the box so this should mostly give us
a clean state to work from for the next fuzzing run
obviously we can't just start the fuzzer again because if we have done something in a previous
fuzzer run that we haven't logged it's impossible to reproduce that for test creation test crash
and it is useless to us we're actually also looking at how we can possibly revert back to a
vm snapshot as this will guarantee a clean slate to start from a few caveats to be aware of if
you're going to take this framework and use it yourselves we found a number of bug bugs in the
which has meant we haven't placed this up in the crowd i'm pretty sure people like amazon
will get really annoyed with us if we start crashing their hypervisors
so what we're looking at doing is running it under chemo which is a bit inceptiony but
should hopefully work you also need to look into things like we're also looking into things
on how to protect the fuzzle from itself so on a number of occasions somehow the fuzzle has
managed to get a handle to itself and kill itself which is kind of irritating
so what we're trying to move on to is a method of how we monitor the host that's fuzzing monitor
the guest story from the host so if the fuzz has killed itself or we've you know hit another sleep
or done something else stupid we can detect that and then start the fuzzle off again
cool so i'm actually going to hand over to georgie who's going to go through windows case study
yo um so
uh as james already mentioned we initially started with the idea of developing a windows
kernel fuzzer uh and this actually remained our focus despite of repurposing the fuzzer
for um os agnostic operations so the second half of the talk is dedicated
on um getting the fuzz work inefficiently on windows and basically writing all of these
custom modules for windows operating system so all of these operating system tweaks and
object store implementation details that you need to consider when you take the
framework and you want to basically use it for fuzzing windows i'll be going through several
examples for each one of these bullet points so basically examples on how we implemented
our object store some examples on what our collection of system calls
looks like in windows and basically some examples on the wrappers for the library course that we
have developed
um i'll be ending this with basically what's the process of bootstrapping the windows vm
before we actually start fuzzing so the attack surface we decided to focus on is
the win32k subsystem a pretty pretty well known attack surface in windows operating system
this subsystem implements mainly three components
the window manager responsible for our desktops windows menus toolbars the general user interface
and the graphics device interface also known as gdi and which is in charge of visualization on
the screen changing colors and basically just general graphics um win32k.sys also implements
some wrappers for directx calls and this is not something we have looked at until now and this is
probably not something we'll consider in the future there is counter component in user length
in terms of user mode libraries a bunch of other components in terms of user mode libraries a bunch of
DLLs provided for application developers to actually interface with the subsystem. So if
you're an application developer you wouldn't issue a syscall to the win32k.sys kernel
mod driver. You would be going on SDN and looking at how you interface with this kernel mod
driver using some library calls that will eventually wrap into a system call. In the Windows
operating system an object is basically a data structure representing some sort of system
resource and this can be anything from file, process, window, um, menu, etc. Um, there is
roughly three types, main types of objects in the Windows operating system. User objects, uh,
GDI objects and kernel objects. User objects are, uh, implemented to support the window
manager. Uh, obviously GDI objects implemented to, um, um, um, um, um, um, um, um, um, um, um,
um, um, um, basically managed graphic devices in space and kernel objects, uh, kernel objects
are basically there to implement basic operations in the kernel space such as process management,
memory management, um, IPC communication, etc. Um, one important thing to, uh, realize
when we talk about object is that when an application wants to manipulate an object,
uh, when an application wants to access an object, it needs to acquire a handle to that
object.
So if you want to do something with any of the system resources, you need a handle to each one of them in order to basically interface with them.
And this is how handles are defined, object handles are defined in the Windows SDK on a really, really low level.
They're just basically void pointers managed by the operating system.
Now, the implementation details are hidden from the application developer, and the application developer shouldn't really care about how these handles are implemented.
And this is just here for reference.
Furthermore, numerous aliases are defined for more or less any system resource available on the operating system.
So this is a snippet from windeth.h, header file from the Windows SDK, where a bunch of these declarations actually happen.
And this is a snippet from the Windows SDK, where a bunch of these declarations actually happen.
There's declaration for a handle to a window, declaration for a handle to a bitmap, et cetera.
So each one of these system resources will have a sort of specific handle associated, a handle type associated with it.
So knowing and realizing how fundamental object handles are in the Windows world, it comes as no surprise that we actually decided to preserve object handles in our object store.
The object store implementation itself is really straightforward at the moment.
It's just a globally accessible array of 120 elements, where we just preserve a bunch of handles to numerous system resources.
Handles are retrieved from this object store when we need a handle to be passed as an argument to a system or a library call.
And should a system or a library call return successfully another handle, we consider it.
this handle and we push it back to the object store sorry if we start running
out of let's say space in our object store and in this case it's a fixed
value of 128 let's say we start running out of slots in this array we basically
start overriding some of their handles from the object store we basically throw
out the oldest handles and we repopulate the object store with some new ones
one more thing to basically notice is the BH handle struct the BH handle
struct is our internal representation of a handle it has two fields which is
the handle itself as well as the index of the handle in the object store so
handles will be changing values per run and so basically if we grab a handle to
a particular window in one of the object stores it will be changing values per run so basically if we grab a handle to a particular window in one
run this handle will have one value if we grab the if you drop a handle to an
identical window in a second run this handle will have a different value so
how we drop the same handle twice why basically referring to the handle not by
its value but by its index in in the object store excuse me we have
implemented a number of functions to interface with with the object store I'm
talking about
the first one
one is basically there to bootstrap the object store uh
populate the object store with some handles before we even
start fuzzing so we can actually use some handles as
arguments for library and system calls. Um the second one is uh
the function get random handle this this function will
basically look up the store and randomly pick up a handle from
it. Um it will then wrap the handle in a BH handle struct
uh which uh helps us successfully log the location
of the handle in the object store when we uh issue logging
statement. Um we have another function for querying the uh
object store this one is get specific handle and this one
takes one argument which is basically the index of the
handle we want to uh extract uh free from the from the object
store. Um we also have a function for uh inserting
handles back to the object store. Again should the library
call return successfully a handle we just push it back to
the object store. Um for populating the knowledge base
of system calls. Well based on the knowledge base of system
calls basically for the majority of them we had to
reverse engineer them. Um we started with uh mainly system
calls implemented in uh in in the win 32k subsystem. Um but um
there's one thing when I say reverse engineering I don't
really mean reversing the logic of the system call itself or how
the system call is implemented under the hood. The only details
we're interested in are basically the system call ID uh
the number of arguments for the system call and and the data
type for each one of these arguments. So that's basically
the argument. Um optionally again uh the return type for for
the system call. Um so for the majority of the system calls you
have to uh do some reverse engineering. For some of them
you may also refer to react OS uh but um this is not
necessarily identical to the Windows revision you're working
on. Um to implement the Cisco invocation itself uh we had to
write some assembly snippets and these are um specific to uh
both uh the operating system and the uh and the uh the
architecture. A little bit more about the assembly snippets in
uh in just a second. Um this is what um uh an extract from uh our
collection of system call looks like. Um James already mentioned
uh what a system uh what our internal uh representation of a
system call looks like. The Cisco struct um the Cisco struct
has got uh three fields. The first one is the system call ID.
The second one is an array of uh data types for each one of the
system calls. And the third and optional field is uh basically
whether the system call returns any data type we may be
potentially interested in. Um the system call invocation again
is um implemented per architecture and per operating
system. Uh we have what we call system call invocation template.
Um the prototype for the template is the same across the
architectures. Um and this is the call uh with the name
buchan underscore Cisco. Um the system call invocation is a
system call uh this function this template function takes 33
arguments at the moment. Uh the first one being the Cisco ID and
the second uh basically the rest of these 32 arguments are uh
arguments for the system call itself. Now obviously not every
system call would be taking 32 arguments or I don't even know if
there is any system calls taking 32 arguments at all. The reason we
did uh a fixed number of arguments is because we didn't want to um
implement uh a function with a variable number of arguments.
Sort of print test like style. So what we do at the moment is we
uh set the system call ID as the first argument for this uh
function. We uh populate the first arguments um with the
actual data that should be passed to the system call and we
just pad the rest of the 32 uh arguments with some dummy data.
And then buchan Cisco is in charge of um setting the
registers for the invocation of the system call. Uh pushing the
uh pushing each one of the uh arguments to the system call.
Uh and then uh making transition in kernel lane. Um once we are in
kernel lane the system call is in charge of basically dispatching
the request. Uh taking care of each one and processing each one
of the arguments that are actually expected by the system
call. Uh the system call will then simply ignore uh the rest of
the 32 arguments on the stack. The system call will eventually
return back to the buchan Cisco wrapper. Buchan Cisco wrapper
will clean up the stack and return to the main fuzzing loop.
Um this is just a general reminder on how system call
invocation works on Windows. Uh for x86 platforms we have um
arguments pushed on the stack in reverse order. We have EAX set
to the system call ID and uh then a call is made to the KI fast
system call function. And now there is a caveat. This is
basically until this is the case until Windows 7 uh which is the
operating system we we have been mostly focused on. Um I think in
Windows 8 and Windows 10 uh the KI fast system call function is a
cast system call has been basically inlined in the code.
For x64, the first four arguments are set in registers.
These are RCX, RDX, R8, and R9.
If there's any additional arguments
that are pushed on the stack,
then RAX is set to the system call ID,
and the CPU instructions this call is made
to basically force transition into kernel space.
So much for system calls.
I'll show now several examples
on how to implement the OS API knowledge base.
The OS API knowledge base,
the information that we need for it
is publicly available on MSDN.
These are basically the library calls
that, again, application developers are expected to be using
when making requests to the operating system.
The OS API calls knowledge base is implemented
as one huge array of function pointers
to each one of the library call wrappers
that we have implemented.
I think until a few days ago,
we had about 500 library call wrappers.
So basically, 500 functions from GDI32.dll,
user32.dll, and some other user mod components.
As you can imagine, this is a really tedious process
of implementing wrappers for each one of these.
We looked at,
automating this by basically crawling MSDN,
collecting function prototypes,
and turning these prototypes into library call wrappers
in a more or less automated way.
But unfortunately, we felt miserably with that,
and we are looking forward
to basically improving our approach.
But at the moment, again,
each one of these are implemented strictly manually.
This is, I'm gonna give a few examples now.
This is the destroy carrot function from MSDN.
This function takes no arguments
and returns Boolean value.
And the wrapper for this call is really simple,
really straightforward.
Basically, we look at the call that we're about to make,
and we make the actual call.
That's literally it.
So in the log file, we basically end up
with the exactly same C statement
that we're making in our fuzzer.
Something a bit more interesting is,
the following one, destroy cursor.
On the top, you have the prototype from MSDN.
This one takes a single argument.
This argument is basically a handle to a cursor object,
and this function returns a Boolean again.
So what we do in our wrapper,
we declare the handle, but not as a handle.
We declare it as a BH handle struct
that wraps the handle itself, as well as the index.
We generate a unique, global unique variable ID,
that will be appended to the variable name
when we make a logging statement.
We then log the declaration for the handle,
while appending the variable ID
to the end of the name of the function.
We proceed by grabbing a random handle from the object store
and assigning it to the handle
that we are gonna pass to the library call function.
Then we log the handle,
we log the handle by its index,
rather than by its,
by its actual value,
and James mentioned this,
but the object store at any point of the fuzzle run
is populated in a deterministic way,
at least in theory.
So if we are up to a point
where we know the object store has certain handles,
we can refer to each one of these handles by its index,
and it should be the case of getting the same handles
that we got the last time we run with the same seed.
So we, the last time we run with the same seed, we run with the same seed.
So the last thing we do is basically log the final call
that we are about to make.
This is the actual call, the actual library call,
and yeah, we log this with the respective variables
and while, again, appending the variable ID
at the end of them.
And then again, the last thing is just simply make the call.
Something a bit more interesting,
so for example, this example demonstrating
how to use the helper functions.
The helper functions,
are functions that will generate a valid struct,
a Windows-specific struct that is consumed
by a library or a system call.
And I'll be talking about the helper functions
in just a bit.
But basically, this is a function taking two arguments,
a handle to a device context, and a CalRef Windows type.
So we start with the same thing again,
with the clarity variables that we are gonna use.
We're in the wrapper.
We generate the variable ID.
Again, this will be globally unique across the run.
We log the handle.
Again, notice that we log the handle as a handle,
not as a BH handle.
We basically hide the implementation details
for a BH handle when it comes to the reproducer test case.
We drop a random handle, assign it to our variable.
The next thing is log the handle byte index,
and the next thing is the interesting bit where we actually make
a call to a helper function.
This one is get CalRef, and each one of the helper functions
takes a single argument, which is the variable ID.
So get CalRef will basically hide the details for us,
but it will populate with mostly valid data.
Again, this struct that can later on be used
by our library call.
And again, the last thing is log the statement,
log the call that we're about to make,
and make the actual call.
And now, a bit more about the helper function.
There's just an endless number of custom structs
in the Windows operating system, and they are expected
by a number of library calls and system calls,
and this is where the helper functions come.
Basically, for each one of these, well, obviously,
we cannot implement for each one of them,
but for the most common ones, consumed by the Win32K
system and library calls, we have implemented
helper functions.
So basically, we have helper functions for returning
rectangles, points,
sizes, windows, et cetera.
And again, these are mostly valid,
and they shouldn't really be way too malformed.
At the bottom of the slide, you can basically see
a snippet from the list of helper functions
we have implemented.
Again, the helper function will return a valid struct.
The naming convention is get underscore name
for the struct, and the single argument
that will be consumed is the global unique file,
the variable ID.
This one is really obvious and really easy to implement,
because this is just typefd word in Windows.
So basically, we start by declaring the coloref variable.
We log the declaration by appending the variable ID
to the name of the variable.
We initialize the variable by calling get underscore
first underscore un32.
This will basically grab an unsigned integer from the code,
the first value uh store assign it to CR and then we lock this
value as it was assigned and return the uh the struct to the
caller which will be then passing this struct to the
library call. Um something a bit more interesting so for
example this one is uh helper function for returning uh
rectangle uh rectangle is a struct with um four fields uh
each one of them uh type long and so what we do is basically
pretty much the same apart from initializing each one of the
fields separately and for each one of these initializations we
um we log in in our log file. Um case study um uh fuzzing the
windows uh VM what do we need to do in order to basically get a
uh a decent uh VM uh basically the VM in a decent state for our
fuzzer to run uh in an in an optimal uh way. Um we start by
installing titan 3.5 and then we have a bootstrap script that
will set up the uh the uh the uh the uh the uh the uh the uh
set up the uh VM and do some general OS tweaks. Um this
includes installing WinDebugger uh installing the Python module
for CouchDB communication in order to submit our crashes to
the centralized database. Um we perform some minor uh tweaks in
the registry including disabling error reporting uh
disabling updates et cetera. Uh we enable uh kernel memory
dumps um basically this is where we get the information about
the crash. Um uh we enable spatial pool for the module uh
that we're uh fuzzing the for the kernel uh mode driver that
we're fuzzing which in this case is win32k.sys. Uh spatial pool is
basically the equivalent of page keep in in kernel length and uh
spatial pool will make it a lot easier to detect uh use after
free vons, double free uh and some general pool corruptions as
well. So it's very important to enable this. Um and the last
thing and obvious thing is to schedule the fuzzer control
script to start on logon um and um reboot the system. Upon
rebooting the system uh the fuzzer kicked off the system and
it kicks in. The first thing to do is look up the uh location
uh for where we store uh the uh memory dumps. Um so if there is a
memory dump we drop the memory dump. We run bang analyze using
KTXE. Um uh we store this in a log file that will later on be
submitted. We uh checked for a left or uh left over um fuzzer
log from from a previous run and we basically assume the fact
that um the this is the log file that caused the last B SOD.
We bundles everything uh we bundle everything together uh
which is basically the fuzzer log, the memory dump file, and
the output from bang analyze and we submit all of these all
together to uh CouchDB uh where CouchDB will um basically uh in
CouchDB we will be able to um do some triaging and um querying
based on the uh based on the crash. Um there was pretty much
about it when it comes to Windows. Again we have done some
stuff on uh Mac OS X and Q and X. Q and X just to prove the
uh the the concept that that it actually works and the fuzzer is
more or less platform agnostic. Um we haven't really spent too
much time fuzzing Mac OS X and Q and X but we've got some um
some uh again prototypes for it. Um when it comes to system
calls um in our operating systems they basically follow
the same calling convention which is uh system VAMD 64
application binary interface. Um and the object store for uh
POSIX and Unix operating systems. Uh we have uh a
we'll uh store uh file descriptors most of the time as
opposed to handles. And file descriptors are more or less um
similar to handles which is basically the most similar thing
we can get to handles. Um and for the crash detection we have
both uh the same choices either attach a kernel debugger or rely
on the operating system to get some useful information for us
in the syslog. Um this is just an example for a single
function um wrapping an IO kit library call. Um it's very much
similar to what we do for Windows. The only difference is
that we don't have handles, we don't have uh or Windows
specific helper functions. We have IO kit helper functions
that will return an IO kit object when it's needed. Um
again very much similar. Uh when it comes to syscalls on
Mac OS X we're kind of lucky because the XNU system calls
are all stored in a single file called uh syscall dot master.
Uh and this file can uh even uh be processed in an automated
way to extract all of this uh system calls with their
IDs and their arguments and basically turn this file into a
collection of system calls that can later on be consumed by the
framework. Um some of the results James already
mentioned uh we're getting some interesting crashes in
word2k.sys primarily in Windows 7. Um why Windows 7?
Well probably because most of our users are on Windows 7
unfortunately. Um of as it when it comes to other operating
systems um again we're looking forward to uh basically
spending more time on that. Uh the current implementation
um for a Mac OS X specific and QNX specific modules is just a
prototype to be fair. Um we have some crashes in hypervisors
which is something that we never intended to find. So this
is basically just an annoying thing happening at the moment
rather than uh something we can brag about. Um we also found
some interesting bugs in VMware tools and virtual box guest
additions. Um basically these are um bones that may
potentially be used for escalating privileges because
VMware, VMware tools is obviously running with
um with in with an escalated context. Um the setup that we
are uh using and this is just a setup for the stability runs. We
haven't even um thought about scaling it at the moment. Um
because mostly because we are satisfied with the results. Uh
is uh host operating system Win 10, hypervisor, VMware
workstation, guest operating system Win 7 64 bit. Um and the
VM specs is pretty modest to be fair. Uh just 2 gigs of RAM and
a single CPU per VM.
Um this is just to recap the uh results from uh what James
mentioned. Uh number of VM 16, stability run for 48 hours, 65
crashes, 13 of them unique. And this is before we started
blacklisting all of the uh syscores um causing a lot of
problems. Um breakdown on the type of crashes. We have 4 no
pointer derails, potentially exploitable Win Win 7. Uh
fortunately for Windows not exploitable on Win 8 and Win 10.
We have some uh use after freeze, 2 of them. We have
uh 4 uh pool buffer overflows. Uh basically pool uh pool
corruptions. And uh we have 3 under the uh category of
miscellaneous. Basically uh 3 um invalid Reduxes uh Reduxes
violations we haven't properly 3 out by now. Um this is a
breakdown on the crashes of the crashes uh based on uh the bug
check ID. Uh the interesting thing here is to notice that a
number of these crashes have been caught only thanks to uh
special pool being enabled. Uh the uh the uh the uh the
support for the uh Win32k dot sys kernel mode driver. So should
you run without special pool you end up with at least half a half
of these crashes. Um further work. What we can do to improve
the framework and to improve the um the Windows specific modules
for example. Uh obviously we want to increase the coverage.
Um how can we increase the coverage? The the easiest thing
and what I think is a very efficient way of increasing the
coverage considering the current architecture is object
tagging. So if you remember in the object store we preserve a
bunch of handles. But these are handles to any kind of object.
So object tagging would basically involve adding a s-
another field in the BH handles truck and this field will
specify the object type for the handle where this handle is
pointing to. Um so when we make a request to the object store uh
we will be uh specifying the type of object that we are
expecting to find on the other hand- on the other side of the
handle. Um we have a lot of problems with the uh the uh the
library calls. We have um well obviously we can implement more
uh more of these uh library calls. Um 500 is a good number
but there are so many of them we can uh we can implement. Uh we
have experimental implementation for user mod callbacks uh
based in uh based fuzzing. Um this basically works at the
moment but we had some problems with the logging. So none of
the none of these uh results that I was talking about are uh
with uh user mod callbacks enabled in our framework. Um so
once we get this working I expect even more crashes.
Another thing is multi-threading. Um if you um can
imagine the log file that we currently uh currently produce is
a huge log file of C statements that are plugged into a main
function and executed in a uh sequentially basically. Um so
this is a problem with multi-threading. All of these
runs have been from uh single threaded uh run of the fuzzer.
Um uh obviously we have to work on- work out how to implement
proper logging with multi-threading uh at the same
time.
Um and um we we can also look at some uh coverage feedback based
on uh CPU features. Um so if you know about NCC's Triforce
implementation well would it be possible to basically get
something similar for uh for Windows? Uh under miscellaneous
um we are always looking forward to improving the logging
because basically it's it's just a really tedious and and slow
process of implementing each one of these library calls with
the log statements for each one of the C statements that we have
in the file.
Um we haven't looked at handling uh hypervisor crashes at all. So
basically if a hypervisor uh crash occurs um these days um we're
just gonna have to um monitor the database and see that the number
of crashes is dropping and then we realize the hypervisor has been
down for a few days.
Um test case reducer we have considered uh looking at uh C
reduce. This is a program uh which is basically a test case
reducer language aware test case reducer that can uh successfully
reduce C files. Um we haven't really uh implemented anything on
top of that but the basic idea is that uh one VM will be reducing
the test case. We'll be feeding the this this test case to
another VM. This second VM will be compiling and executing and
we'll providing feedback back to the first VM which is uh
constantly reducing the test case. Uh and just in general we
definitely need to implement a decent logging for uh for the
VM and ideally this logging should be uh hypervisor
agnostic. So basically this monitoring should be working on
um VMWare, VirtualBox, KeyMuse and et cetera. Um these are the
people we want to thank. These are all people who provided
really valuable feedback and uh some useful ideas and
information when uh we implemented the fuzzer. Um and
the basically the core of the framework will be available
online um I think later next week um as soon as we get back
to the UK basically. Um and yeah keep an eye on Twitter uh um
we'll be bragging about releasing the framework I guess.
Um
uh if there's any questions I'm happy to take them at the
cafe area because I'm being told uh we're running out of
time. Uh but yeah thanks guys.