00:00:00.334-->00:00:05.539 >>So today we're going to be talking about machine learning, no one has not heard of machine 00:00:05.539-->00:00:10.844 learning, right? So uh, in specific what we are going to be talking about is how to bypass 00:00:10.844-->00:00:15.382 deep learning systems. Um, machine duping is what I'm choosing to call it it's a 00:00:15.382-->00:00:21.054 really corny name so if anyone has better ideas come to me after the talk and we can talk 00:00:21.054-->00:00:25.325 about it. So everyone has heard of machine learning my grandma knows what machine learning is 00:00:25.325-->00:00:29.830 at least if you've been reading the news you know that the best golf player in the world is now 00:00:29.830-->00:00:34.368 a deep learning system. So that's kind of weird, no one saw that coming in the last five 00:00:34.368-->00:00:39.373 years. Uh but let's not have a slight on what machine learning is and what machine learning can 00:00:42.442-->00:00:47.514 do for us that's something that's for a presentation two years ago, uh let's instead look 00:00:47.514-->00:00:53.453 at what we're trying to achieve in this talk. So everyone wants to be the flying carpet beagle 00:00:53.453-->00:00:58.058 in the middle of the venn diagram there. Uh most of us in this room have some sort of 00:00:58.058-->00:01:04.865 hacking skills, people hacking or computer hacking, we do stuff we're implementers so I'd say 00:01:04.865-->00:01:09.803 we're in that category over there, hackers. Um security researches have some kind of 00:01:09.803-->00:01:14.608 theoretical background if they're working on crypto and stuff then they have some kinds 00:01:14.608-->00:01:20.314 of math and stat skills, um I don't necessarily consider myself that, um, data scientists 00:01:20.314-->00:01:25.218 don't have any hacking skills but they do all the stuff in companies trying to maximize 00:01:25.218-->00:01:27.220 conversion they have math and stat skills but what we're trying to do today is to help 00:01:27.220-->00:01:29.222 all of you guys and convince all you guys that you really want to be in the center of the venn 00:01:29.222-->00:01:31.224 diagram and it's going to be important and increasingly important in the next few years 00:01:31.224-->00:01:34.861 to brush up on math and stat skills and to know about what's going on in the machine learning 00:01:34.861-->00:01:39.866 space especially for security folks. So, whether we know it or not, we are interacting with 00:01:52.646-->00:01:57.084 machine learning and deep learning systems on a day to day basis, if you don't like it, you 00:01:57.084-->00:02:03.957 have no choice. Um Google now, Apple, Siri, Amazon Alexa, they're all things that have 00:02:03.957-->00:02:10.931 been covered by the press, very high profile things, some common, some more common use 00:02:10.931-->00:02:15.902 cases of deep learning are in object recognition in self driving cars, if you know joe 00:02:15.902-->00:02:20.941 hotz in his cool comma.ai startup uh they are using deep learning tools to recognize 00:02:20.941-->00:02:27.414 objects on the road and build a really cheap self driving car system, uh, obviously on the top 00:02:27.414-->00:02:32.652 right hand corner you see AlphaGo beating the world champion at Go, and you have 00:02:32.652-->00:02:38.025 also pretty interesting use cases um like in medical research where they're using 00:02:38.025-->00:02:43.030 deep learning to predict the effects of newly developed drugs in patients and um I just have 00:02:45.365-->00:02:51.004 an example from a video there. Uh also in the security space there's also lots of stuff, if 00:02:51.004-->00:02:56.476 you've ever set foot at RSA or BlackHat Expo you know what I'm talking about, everyone is 00:02:56.476-->00:03:01.882 saying they're using machine learning, deep learning to do stuff. The extent to which that 00:03:01.882-->00:03:08.121 is true, I cannot vouch for. [laughter] So why would someone choose to use deep learning? 00:03:08.121-->00:03:11.792 Again, forgive me if you're an expert at deep learning, this is a DC 101 track so i'm going to 00:03:11.792-->00:03:15.462 be spending a little bit of time going through some basics of deep learning and machine 00:03:15.462-->00:03:20.600 learning and then I'll go into the interesting stuff, which is my research. Why would someone 00:03:20.600-->00:03:25.338 choose to use deep learning over more traditional machine learning methods like SVMs or 00:03:25.338-->00:03:31.745 linear classifiers, clustering algorithms. The first thing is that when you use a deep 00:03:31.745-->00:03:36.116 learning algorithm, you get some things for free that you would otherwise have to spend a lot of 00:03:36.116-->00:03:41.421 time doing. The most important thing that everyone would point out to you is that you get 00:03:41.421-->00:03:46.560 feature engineering for free. Again the extent of which that is true, you have to try it out 00:03:46.560-->00:03:53.433 and implement it and it depends on the use cases that you are using it for. Um deep learning 00:03:53.433-->00:03:57.170 helps to select the best features and you don't necessarily have to spend a lot 00:03:57.170-->00:04:00.207 of time doing feature engineering if you talk to any data scientist or machine 00:04:00.207-->00:04:06.179 learning engineer working in the large company uh then they'll tell you that most of their time 00:04:06.179-->00:04:11.051 is not actually spent on algorithms they're not trying to increase the efficacy of this 00:04:11.051-->00:04:15.589 'so and so' cutting edge algorithm that they're using to in their company's product 00:04:15.589-->00:04:19.726 recommender systems. What they're actually doing is feature engineering and data 00:04:19.726-->00:04:25.632 cleaning. So, they're like janitors you know, like I'm a janitor too, I spend most of my 00:04:25.632-->00:04:30.904 time cleaning data, I spend most of my time doing data feature engineering. So deep learning 00:04:30.904-->00:04:35.041 gives you that for free and that's why it's so appealing I think. The other thing which I 00:04:35.041-->00:04:38.111 think is the main difference between deep learning and other kinds of machine learning 00:04:38.111-->00:04:43.416 algorithms is that uh it tautes the promise of one infrastructure for multiple 00:04:43.416-->00:04:49.089 problems. So if you think about it deep learning really is just one, just one infrastructure. 00:04:49.089-->00:04:53.960 There's multiple layers of linear units and uh each one of these linear units interact in a 00:04:53.960-->00:04:58.398 different way with different linear functions to give you the result that you want to learn 00:04:58.398-->00:05:02.269 different things that you want it to learn. Compared with other kinds of machine learning 00:05:02.269-->00:05:08.108 algorithms like clustering you and SVMs and a lot of single regression decision trees, all 00:05:08.108-->00:05:13.680 of these require vastly different code bases whereas for deep learning the the 00:05:13.680-->00:05:18.151 differences in infrastructure are parametrized into the number of different layers the number 00:05:18.151-->00:05:23.890 of units in each layer the functions between each layer and other things like that. So it's 00:05:23.890-->00:05:27.327 sort of one infrastructure for multiple problems and I think that's what gives it its 00:05:27.327-->00:05:32.332 flexibility. The last two points are perhaps more relevant in today where there's so much data 00:05:34.601-->00:05:40.774 to deal with, um, deep learning allows you to do hierarchical learning. So you can split up 00:05:40.774-->00:05:45.879 the task of learning across different layers and we'll see an example of that later. So for 00:05:45.879-->00:05:51.084 example you have more shallow layers learning vastly different things from the deeper layers 00:05:51.084-->00:05:56.356 and you can extract how the outputs of each intermediate layer to exactly find the thing 00:05:56.356-->00:06:01.461 that you are looking for. The last thing is it's efficient and it's easy distributed and 00:06:01.461-->00:06:05.732 parallelized if you're looking at algorithms like clustering there's no straightforward way 00:06:05.732-->00:06:09.736 to really distribute it across systems and when you're dealing with terabytes petabytes of 00:06:09.736-->00:06:14.941 data, this is a problem. Of course there's message passing algorithms that uh that help 00:06:14.941-->00:06:20.347 cluster with that but they're a lot more complex. So deep learning allows you to 00:06:20.347-->00:06:26.686 distribute these uh problems up, distribute your infrastructure up and distribute the 00:06:26.686-->00:06:31.625 computation. Of course it's definitely not one size fits all uh deep learning is not 00:06:31.625-->00:06:34.527 something that you would want to use for any random problem. If you're trying to predict let's 00:06:34.527-->00:06:39.132 say the prices of oranges against time you wouldn't be using a deep learning 00:06:39.132-->00:06:45.605 infrastructure. That's like you know using deep learning to um you know predict a problem space 00:06:45.605-->00:06:51.344 of two dimensions. You wouldn't do that. What you would use deep learning for is in problem 00:06:51.344-->00:06:56.116 spaces that have multiple dimensions. So like hundreds or thousands of dimensions usually 00:06:56.116-->00:07:01.421 these things come from nature. So you can think of images audio video and prediction problems 00:07:01.421-->00:07:06.626 that are in a very complex problem space. So let's just spend a couple of minutes to go 00:07:06.626-->00:07:10.964 through the two steps involved in training a deep learning architecture, a deep learning 00:07:10.964-->00:07:17.737 infrastructure. So this is a diagram that a lof of you may have seen before. It's It's a 00:07:17.737-->00:07:23.543 453 neural net architecture. It's a very simplified version of it um, so ignore all of the 00:07:23.543-->00:07:29.215 all the nasty white lines between those units. Each circle represents uh a linear unit and 00:07:29.215-->00:07:33.453 there's activation function in it. Activation function just means that if you input a 00:07:33.453-->00:07:39.059 certain value into this circle than it either outputs it outputs a certain real numbered 00:07:39.059-->00:07:44.064 value according to the, the function. And each connection between two units between layers 00:07:46.599-->00:07:53.506 is uh, is weighted by the weight 'w' and also there's a bias unit. What's the purpose of the 00:07:53.506-->00:07:59.679 bias unit? It's simply to help to skew the results of the output by a certain amount If 00:07:59.679-->00:08:05.452 you can think of just linear algebra you have y=3x+b, this bias unit is equivalent to b 00:08:05.452-->00:08:09.723 which controls the intersection between the x axis and y axis. So all this is just theory, 00:08:09.723-->00:08:15.161 don't don't worry about it if you just take like one of the ten thousand books out there to 00:08:15.161-->00:08:22.068 learn deep learning. But let's just look at a simple example of how training works so we can go 00:08:22.068-->00:08:28.808 into how to bypass them. So the first step is the feet forward step um each unit receives the 00:08:28.808-->00:08:32.879 output of neurons in the previous layer uh in the case of the first layer it just receives 00:08:32.879-->00:08:37.884 the input um and then uh the bias is added to it the weight, the output, the output of the 00:08:41.321-->00:08:46.960 first unit is weighted by w1 and then it goes on and on and eventually when it comes to the 00:08:46.960-->00:08:52.766 output layer uh it outputs logics which are just numbers which is an array of numbers 00:08:52.766-->00:08:57.404 it's fed through most of the time a soft mix function for classifiers which just scales it 00:08:57.404-->00:09:01.808 into a probability distribution function and so in this particular case you see that um 00:09:01.808-->00:09:06.813 this dummy matrix, this dummy vector that's output is .34 .57 .09 and this just means that if 00:09:10.050-->00:09:15.889 you have three classes 0, 1, and 2, uh this classifier predicted that class one is the predictive 00:09:15.889-->00:09:22.228 class however that is wrong in our case so, according to the labels because 'cause is 00:09:22.228-->00:09:27.233 training, so you know the labels. So uh the error is .57 uh because the ideal case would 00:09:29.736-->00:09:35.975 be that you predict with probability one that uh the output is actually zero. So you 00:09:35.975-->00:09:41.748 feed it backwards and backpropagation, is really the crux of deep learning. So what 00:09:41.748-->00:09:45.952 backpropagation is, it's a pretty, well it's not the most straightforward algorithm that 00:09:45.952-->00:09:49.789 you'll, that you'll come across in machine learning and when you're taking some books like 00:09:49.789-->00:09:54.694 the stanford one on Machine Learning by Andrew Ng he'll also say that backpropagation is a 00:09:54.694-->00:09:59.532 pretty hard thing to grasp. I think the easiest way to explain it, the easiest way to think 00:09:59.532-->00:10:04.370 about it is that back propagation is just assigning blame. So let's say you uh you 00:10:04.370-->00:10:08.308 were the head of the board of directors and you had a bunch of people on the board that were 00:10:08.308-->00:10:13.913 giving you uh suggestions that were advising you on stuff um and some of them just talk 00:10:13.913-->00:10:19.819 bullshit all the time uh you wanna listen to them less, so that's what this is doing. 00:10:19.819-->00:10:24.958 [laughter] So basically you're feeding it input data through a deep learning network and you're 00:10:24.958-->00:10:30.663 trying to figure out what gives you wrong answers and you do that by sending it input data 00:10:30.663-->00:10:35.068 and then seeing when the answer is wrong, you know the answer, you know the right answer, and 00:10:35.068-->00:10:40.907 seeing what answer is wrong, and then you trace, you trace the path that this answer took 00:10:40.907-->00:10:45.645 through the network and you find out exactly which units are responsible and how much they're 00:10:45.645-->00:10:51.351 responsible because of their weights. So let's say you find a particular path of units is 00:10:51.351-->00:10:56.322 particularly bad at giving you recommendations or predictions for something that you're trying 00:10:56.322-->00:11:02.695 to learn then you just decrease their weights and you block them out. So all of this can be 00:11:02.695-->00:11:05.965 optimized with certain algorithms like stochastic rate and descend which is just 00:11:05.965-->00:11:10.970 basically trying to find a local minima in a in a particular problem space. So, there's going 00:11:14.274-->00:11:19.279 to be lots of demos in this talk, so if you've been ignoring me for the last ten minutes, you 00:11:24.951-->00:11:29.956 can uh, look up now. K, is this big enough? Let's make it a little bigger, ok so what we're 00:11:33.893-->00:11:38.431 going to look at here is an example of a deep learning system that is really accessible 00:11:38.431-->00:11:44.637 to everyone this is Tensorflow um it's um Google's uh open source deep learning machine 00:11:44.637-->00:11:50.944 intelligence framework. I think it's probably the easiest deep learning framework to use and 00:11:50.944-->00:11:55.949 what we're going to be walking through is a small example of how to do of how to use deep 00:11:55.949-->00:12:01.688 learning to solve the easiest task, the task that's used most commonly in most tutorials and 00:12:01.688-->00:12:08.494 examples and then we'll look at how to bypass that. So, Tensorflow, okay. So what we're 00:12:08.494-->00:12:14.467 trying to do is use the MNIST database which is um which is created you know like 20 years 00:12:14.467-->00:12:20.406 ago I think by Ian McCoog who is now the Facebook EI Director and um will be having some 00:12:20.406-->00:12:25.411 virtualizations done with this pretty cool uh virtualization tool. So, let's look at the code 00:12:27.814-->00:12:32.552 a little bit. What this is doing is basically taking in the training data and the testing 00:12:32.552-->00:12:39.459 data and labels and then you are creating a validation set, oh this is is is uh just um sugar, 00:12:39.459-->00:12:45.431 and then here is the actual definition of the model. This is the model and the layers are 00:12:45.431-->00:12:50.703 defined line by line, you have the first convolution layer, then you have a pooling layer, 00:12:50.703-->00:12:56.209 convolution two, pool two, and so it just goes down from top to bottom and you get from shallow 00:12:56.209-->00:13:01.447 layers to deeper layers as you go down and as you see the logits and the soft mix function 00:13:01.447-->00:13:06.452 actually creates the output of this uh neuronet. So this is a demo of MNIST Classification. 00:13:09.822-->00:13:14.827 What MNIST is, is just digits, handwritten digits. It looks something like this. So these 00:13:17.630-->00:13:22.201 may not look like digits that you and I write like I would never write a seven with a weird 00:13:22.201-->00:13:28.574 thing at the end uh but you know what can you do? You can always train a model on on different 00:13:28.574-->00:13:33.012 handwritten data sets, but this is a standard and it's used for comparison between between uh 00:13:33.012-->00:13:38.017 researchers and in academia. So, let's see here, this is uh a real time, um a real time time 00:13:41.788-->00:13:48.661 classifier predictor for digits. So if I were to write like a two here, let's see if it predicts 00:13:48.661-->00:13:54.734 it correctly. Woops, okay two. Even though there's that weird thing. So it's pretty good. 00:13:54.734-->00:13:59.739 Let's do like seven, oh seven, that's great. Only point seven accuracy. I mean confidence. 00:14:04.377-->00:14:11.084 Let's try something that's a little bit more challenge for the model, like a six. Okay so 00:14:11.084-->00:14:15.688 this is the wrong, this is the wrong classification because it thinks it's a five with point 00:14:15.688-->00:14:20.693 eight six confidence. See if we can make it better. Nope. [Laughter] nevermind. So, you 00:14:31.137-->00:14:37.043 can see, this is not the cutting edge in handwritten digit recognition [Laughter] Like if 00:14:37.043-->00:14:42.715 if uh I think uh this this particular this particular implementation represents, 00:14:42.715-->00:14:48.788 correctly classifies 90% of digits. So that's nowhere close to what a human can do. Right? I 00:14:48.788-->00:14:53.292 mean, if you or I can recognize nine out of ten digits then you should see an eye doctor 00:14:58.531-->00:15:05.438 [laughter] but anyway that's that. And uh let's continue. So what that was doing um was using 00:15:05.438-->00:15:10.309 convolutions and this is what you call a convolutional neural network. So these are just 00:15:10.309-->00:15:13.746 different flavors of neural networks. Just different algorithms that researchers 00:15:13.746-->00:15:19.752 publish papers on um to get tenure and stuff. But this is a really cool algorithm um it was 00:15:19.752-->00:15:24.757 developed 20 years ago I think um and uh what this does is it uses convolutions to gather 00:15:27.693-->00:15:32.698 insights on different details different levels of details in an image or in stuff that have 00:15:35.201-->00:15:40.106 adjacency in relationships. So what are convolutions? If you remember from your transform 00:15:40.106-->00:15:46.112 days in school or not, um convolutions are just filters so uh if you apply a filter a 00:15:46.112-->00:15:51.751 convolution on a matrix let's say in this case it's a 2D matrix then what you have is you 00:15:51.751-->00:15:57.123 do a matrix multiplication and then you end up with a single value for the convolution 00:15:57.123-->00:16:01.594 applied on the particular space that you are applying a convolution on. And so what this 00:16:01.594-->00:16:06.599 allows you to do is layered learning. This is an example for facial recognition where the 00:16:06.599-->00:16:10.403 layer the shallow layer is actually at the bottom here and then you're going up as you go 00:16:10.403-->00:16:16.542 deeper into the network. In layer one you are learning very very uh very very fine feature 00:16:16.542-->00:16:22.782 like the shape of your eyebrow, no, the the shape of your eyelashes and and maybe the 00:16:22.782-->00:16:28.221 wrinkles in your face. As you go further up, as you go further down the network then because of 00:16:28.221-->00:16:33.960 the convolutions and pooling then you actually get more zoomed out features, like the 00:16:33.960-->00:16:39.198 shape of your eyes the color of your eyes, what if you have like a mole somewhere or something. 00:16:39.198-->00:16:45.204 At higher levels you get the shape of your face, more general characteristics of you like if 00:16:45.204-->00:16:50.209 you have a mustache or not. So um this is interesting because you can extract the results out 00:16:52.511-->00:16:57.350 of intermediate layers to do certain things. Like I was at a talk by Facebook security team 00:16:57.350-->00:17:03.623 once and they say that in order to find uh images that are spammy, so when spammers try to 00:17:03.623-->00:17:08.261 spam their network with images, they often tweak the images a little bit or change the 00:17:08.261-->00:17:12.999 language or text in certain ways that you can't just do a pixel by pixel comparison. So there 00:17:12.999-->00:17:16.736 are certain ways that you can solve this by shingling or by doing some kind of fuzzy 00:17:16.736-->00:17:21.474 matching. Uh but the most efficient way they found and the most effective way was actually 00:17:21.474-->00:17:26.412 to pass these images through a neural network and then get the second layer out and compare the 00:17:26.412-->00:17:32.018 apple to the second layer. Then they able to reliably find images that were spammy and then 00:17:32.018-->00:17:36.355 group these images together and then have a human actor come in and see whether this actually is 00:17:36.355-->00:17:43.029 a spammy image or not. So this is this is just a diagram of the convolutional neural network 00:17:43.029-->00:17:48.034 that uh was used to to uh to classify the the digits uh basically if you look at the 00:17:51.470-->00:17:56.475 small squares that are zoomed in, one part of the digit is uh one part of the digit is uh, fed 00:18:00.179-->00:18:03.683 into feature maps and then you do sub sampling on that and they perform convolutions you perform 00:18:03.683-->00:18:08.688 more sub sampling and then you do a prediction on that. So besides convolutional neural 00:18:11.757-->00:18:16.028 networks there are also things that are called recurrent neural networks these are slightly more 00:18:16.028-->00:18:19.932 complicated but not actually that complicated. What you have is just recursion in the neural 00:18:19.932-->00:18:25.771 network. So instead of feeding in through uh, instead of feeding in input one way through 00:18:25.771-->00:18:31.344 a neural net you have recursion in it so the output of each time stamp and each intermediate 00:18:31.344-->00:18:35.615 output actually gets fed into the next time stamp. So this introduces the concept of memory 00:18:35.615-->00:18:39.085 and memory is important because when you learn things you don't necessarily learn things one 00:18:39.085-->00:18:45.791 frame at a time. That would be really weird um and uh this allows you to learn things like 00:18:45.791-->00:18:51.630 music, or or things that like audio or video which have some kind of a relationship between 00:18:51.630-->00:18:56.302 frames and relationship between inputs. And so this is an example of a generative 00:18:56.302-->00:19:02.575 recurrent neural network. You can teach a network to spell words. And so In this case if 00:19:02.575-->00:19:08.748 you see um that you already have the letters 'y', 'o', and 'l'. Then 'o' is likely to be the 00:19:08.748-->00:19:13.753 next word because of the memory of the network having 'y' 'o 'l' in its in its in its uh buffer. 00:19:17.723-->00:19:23.129 So there's also stuff that's more on the cutting edge. This was actually used uh to a 00:19:23.129-->00:19:27.566 certain extent in AlphaGo long short term memory. So when you're looking at things with 00:19:27.566-->00:19:31.771 slightly more context but you don't want to scale the depth of the network or the depth of 00:19:31.771-->00:19:37.443 recursion indefinitely then you want to use things like 'LSTM's long short term memory networks 00:19:37.443-->00:19:43.482 um because this allows you the concept to arbitrate um longer term concepts, arbitrate longer 00:19:43.482-->00:19:49.722 term data to store for a longer time and not just have a have a single uh five OQ that you that 00:19:49.722-->00:19:55.494 you store your memory in because that's not how we learn. So to make good predictions we need 00:19:55.494-->00:20:00.633 more context and you can think of things like a system that converses with you. When you're 00:20:00.633-->00:20:05.171 talking with somebody and someone mentions that he's from let's say France uh five minutes 00:20:05.171-->00:20:09.175 ago in a conversation you don't just forget that after five minutes because your memory 00:20:09.175-->00:20:13.679 buffers through you have to remember things like that and so the beauty of LSTM networks is 00:20:13.679-->00:20:20.419 that they have certain get it functions in in the recurrence of the network that allows you, 00:20:20.419-->00:20:25.724 that allows the network to learn what's important to remember for a longer period of time and uh 00:20:25.724-->00:20:30.730 what's not. So this is just a simple diagram of how deep learning has helped with speech 00:20:34.934-->00:20:40.740 recognition over time. The different lines in this diagram, this is uh the y axis is 00:20:40.740-->00:20:45.244 logarithmic right away, so the different colored lines represent different data sets uh 00:20:45.244-->00:20:52.151 the holy grail is definitely the red line it's conversational speech. So, a very very, a very 00:20:52.151-->00:20:57.156 very sterile kind of speech is red speech or broadcast speech where every word is annunciated 00:20:59.658-->00:21:04.597 to to you to to to the nail. And um all of these over the years have seen pretty good 00:21:07.366-->00:21:14.140 performance you see like up to 2% word error rate which is WER word error rate performance in 00:21:14.140-->00:21:19.145 air travel planning kiosk speech was a pretty weird data set um and then for conversational 00:21:21.280-->00:21:26.285 speech um you see in 2011 it went down all the way to about 20% so that's that's great. By 00:21:28.387-->00:21:33.526 the way the conversational speech data set if you ever um if you ever have a chance of 00:21:33.526-->00:21:38.497 finding it or looking at it, it's not out in the public but it's actually pretty weird. Um 00:21:38.497-->00:21:43.502 it's actually from blind dates [laughter] so uh when i was listening to it was it was very 00:21:45.738-->00:21:52.178 interesting how some of these dates turn out [laughter] but yeah there's some pretty 00:21:52.178-->00:21:57.183 disturbing stuff in there [laughter]. So okay now for the now for the fun stuff, how to 00:22:00.319-->00:22:05.324 comb. Okay so there's a short video, this is Gunter he's uh a a dog. He's a [chuckle] he's a 00:22:12.598-->00:22:17.603 mini schnauzer and he's my best friends dog. So let's just see how this analogy pans out. Cool, 00:22:28.714-->00:22:33.719 let's see if this video plays. Okay, cool. So this is Gunter, he loves ice cubes. I'm not sure 00:22:36.355-->00:22:41.260 if all dogs love ice cubes they think like he thinks it's alive. So this is a training phase 00:22:41.260-->00:22:45.464 where i'm basically teaching him that the clinking sound in the bowl represents that there's an 00:22:45.464-->00:22:51.036 ice cube in there. Sorry [chuckle] the clinking sound in the bowl represents that there's 00:22:51.036-->00:22:53.839 an ice cube in there and he knows he's going to get an ice cube because he has the taste of 00:22:53.839-->00:22:59.979 it. Then, I mislead the model. First I show him the ice cube I put it back in the bowl and I 00:22:59.979-->00:23:04.917 don't actually throw it, I throw something else that's not an ice cube and he's confused. The 00:23:07.286-->00:23:12.891 model is bypassed. So Gunter is the model now and this is kind of a like a lame analogy I draw 00:23:12.891-->00:23:17.896 from the stuff I'm doing. Forgive me. Uh but what we see here is that the dog doesn't 00:23:21.000-->00:23:24.703 know that I'm throwing the the ice cube. He just thinks that I am throwing the ice cube because 00:23:24.703-->00:23:29.908 he sees like the big hand motions. Um ice ice doesn't have much smell and he he he doesn't 00:23:29.908-->00:23:33.946 have great eyesight so he sees me throwing something and I've done it a few times before so he 00:23:33.946-->00:23:40.052 thinks that an ice cube is there at the other end of the yard waiting for him. So, that's very 00:23:40.052-->00:23:45.057 similar to what we are going to be doing um we're going to be feeding in images to classifiers 00:23:47.159-->00:23:52.665 um that will mislead these classifiers and these images are crafted in a certain way that 00:23:52.665-->00:23:59.171 will uh facilitate certain classifiers from being misled more than others. But first 00:23:59.171-->00:24:03.242 let's look at the attack taxonomy so we can look at the uh the attack factors and stuff 00:24:03.242-->00:24:07.980 like that and the security speak. Um there are two kinds of general attacks that you can do 00:24:07.980-->00:24:11.650 on these machine learning or deep learning systems you can have Causative attacks or 00:24:11.650-->00:24:17.389 Exploratory attacks. Causative attacks are relevant when you have access to the training step 00:24:17.389-->00:24:22.661 um so for example when you're looking at like an online translation engine like Google 00:24:22.661-->00:24:27.499 translate they actually do rely on some kind of online reinforcement learning when you 00:24:27.499-->00:24:31.603 see that something is a really really bad translation which happens pretty often I'm sorry 00:24:31.603-->00:24:37.209 but um when you see something is really bad you can report it as really bad and you can maybe 00:24:37.209-->00:24:42.381 give the right answer or something that's more relevant. And so this is how it how it 00:24:42.381-->00:24:47.720 does reinforcement learning. Um in Gmail when you have when you receive email that's not marked 00:24:47.720-->00:24:51.824 as spam but actually really is spam then you can mark it as spam and I urge all of you guys 00:24:51.824-->00:24:56.428 to do that because that actually helps with the with the reinforcement learning model. Um 00:24:56.428-->00:25:01.900 and uh this will help to train them all to recognize such examples as as spam in the 00:25:01.900-->00:25:07.406 future. So you can see how you can influence a model in such a way that's a little less 00:25:07.406-->00:25:11.810 interesting that's what I that's what I that's what I did last year in talks. Uh but what's 00:25:11.810-->00:25:17.116 more interesting I think are exploratory attacks. So in this attack model you have no access 00:25:17.116-->00:25:21.820 to the model you have no access to the training phase and what you're doing is just a black box 00:25:21.820-->00:25:27.426 attack in in in some scenarios where you're feeding it just weirdly crafted samples that 00:25:27.426-->00:25:32.631 look correct to a human but uh the machine just gives you a wrong result and so that's 00:25:32.631-->00:25:37.236 really that that really throws some people off even for machine learning researchers because um 00:25:37.236-->00:25:42.641 they think the deep learning model or the machine learning model is learning exactly in the 00:25:42.641-->00:25:49.314 way that the human learns. Like we don't learn to recognize alphabets or letters by uh 00:25:49.314-->00:25:53.719 looking at the pixel or the angle between uh the horizontal line and the slanted line in an 00:25:53.719-->00:25:59.892 in a A. We learn it in a more general way and I think it's still active research area into 00:25:59.892-->00:26:04.830 how to represent these things in machines better. Uh so this is still the dog keeps coming up. 00:26:06.965-->00:26:13.405 Uh this this is still active research area uh and uh that throws people off. So they'll 00:26:13.405-->00:26:19.178 target an indiscriminate attacks when you try to uh move the decision barring a certain way 00:26:19.178-->00:26:23.081 to cause an intentional misclassification or indiscriminate attack when you 00:26:23.081-->00:26:29.988 uh just try to decrease the integrity of of uh these classifiers. So this is a simple 00:26:29.988-->00:26:34.993 example of a misclassification. So in the early days, MNIST and digit class- digit recognition 00:26:40.466-->00:26:47.005 used to be used in recognizing digits uh written in in checks. And I know these these digits 00:26:47.005-->00:26:52.411 don't look very realistic but I'm using stuff from the MNIST data set. And what I did 00:26:52.411-->00:26:57.616 beforehand was to was to generate some adversarial images. And then fill in two 00:26:57.616-->00:27:03.555 copies of the checks. One with normal images and one with adversarial images. So if we 00:27:03.555-->00:27:09.061 just go back a bit and look at this this is the adversarial one and this is the normal one they 00:27:09.061-->00:27:15.667 look pretty identical just look at the digits portion. So it's nine three seven eight and this 00:27:15.667-->00:27:21.440 is some simple code to use the pretrain MNIST model with the standard tensor flow um MNIST 00:27:21.440-->00:27:26.411 example training with that it takes about four hours to train this model which is trust me 00:27:26.411-->00:27:32.384 really good speed using a CPU in the deep learning world. You see something that takes longer time 00:27:32.384-->00:27:37.389 to train later. And so what we're going to do is to just basically read the check um this 00:27:40.025-->00:27:45.030 just divides the image up into a pi- in into a pixel matrices and then it reads digits so it's 00:27:48.734-->00:27:53.739 loading the model it's predicting uh the digits 9378.00, 9378.00 so that's 00:27:55.941-->00:28:00.879 that's correct. And so if you look at that that's what it is. Okay? And now we're going to do 00:28:04.616-->00:28:09.621 the same for the adversarial image. Yeah. Okay so you expect this to be the same but no you 00:28:23.168-->00:28:27.205 actually expect this to be different because it's called adversarial. Um so the output of 00:28:27.205-->00:28:32.210 this is something totally different. 0524.84 So it looks the same but it gives you a 00:28:35.180-->00:28:41.587 different output using the same model, using the same code. And uh that's adversarial machine 00:28:41.587-->00:28:46.224 learning. This is something a little bit different it's the CIFAR ten data set so CIFAR is 00:28:46.224-->00:28:51.563 um is a data set of images So what you're trying to train them all to do is recognize images in 00:28:51.563-->00:28:56.034 10 classes there's c410 which is for 10 classes of images there's c4100 which is for 100 classes 00:28:56.034-->00:29:00.973 of images. The classes are here they are uh like you have things like airplanes, automobiles, 00:29:08.313-->00:29:13.452 birds, cats, deers, the interesting thing is that these images are not high resolution 00:29:13.452-->00:29:18.690 images you don't take them with your phone they're 32x32 pixels large so this is the actual size 00:29:18.690-->00:29:23.695 of it actually maybe bigger and so we're looking at two sets of images here, dog and automobile. 00:29:27.532-->00:29:32.537 I just chose them because dog is cute [laughter]. So what we see here is that, well if you see 00:29:37.409-->00:29:42.414 the preview window on on the right here uh that's actually preview on on Mac it's not 32x32 00:29:42.414-->00:29:47.419 I think Mac OS does some aliasing so your pictures don't look like shit and uh what we're 00:29:51.390-->00:29:55.894 doing here is to eval this and what this evaluator does very similar to the MNIST classifier 00:29:55.894-->00:30:02.000 is that it classifies this this image into a class so it tells you whether this image is of a 00:30:02.000-->00:30:07.139 dog or of a or of something else. So you see in the add ten image it classifies it as a 00:30:07.139-->00:30:10.275 ship. In F1 it classifies it as a ship as well. So you can see that the images are pretty 00:30:10.275-->00:30:15.280 similar um but there's actually differences in them. Let's classify the automobile one just 00:30:21.319-->00:30:26.324 for completion automobile okay this is what it looks like, let's push the image, automobile 00:30:34.533-->00:30:39.538 okay that's correct. And what the different numbers after add means is to what degree they are 00:30:41.973-->00:30:47.612 perturbed so to what degree we're injecting stuff in the image to make the classifier 00:30:47.612-->00:30:53.552 think it's not what it is. So you see it becomes a little bit more grainy but to a human it 00:30:53.552-->00:30:57.723 still looks like a car, I mean you wouldn't say that that's a that's a cat or you're weird. 00:30:59.925-->00:31:05.931 Yeah? So let's look at exactly what differences exist between these images. Let's open the 00:31:05.931-->00:31:10.936 Python shell, ah, SciPy, let's read these images in and then uh look at what the difference is 00:31:17.109-->00:31:22.114 between them. By the way a lot of Python libraries actually don't read pngs or don't write 00:31:24.850-->00:31:29.855 pngs exactly pixel for pixel so if you're trying this at home, use these libraries. So this is 00:31:33.391-->00:31:38.396 the this is the standard representation of the png it's just um a 32x32 pixel 00:31:41.166-->00:31:46.171 representation there's three channels, R, G, and B and the value of each pixel can be 00:31:49.107-->00:31:54.112 between 0 and 255 just pretty standard So we're reading the adversarial image now and we can 00:31:58.984-->00:32:03.922 see that it's printed out and we can see that okay the numbers are slightly different from 00:32:05.957-->00:32:11.830 before let's print out exactly what's different. Let's look the size first. We can see that the 00:32:11.830-->00:32:16.835 shape and the shape of these two images are the same 32, 32, 3, and let's let's calculate the 00:32:20.172-->00:32:26.111 differences between the adversarial image and the normal image convert it to N64 to 00:32:26.111-->00:32:31.116 prevent any overflows. If this is boring you out just zone out for a moment and I'll wake you 00:32:33.685-->00:32:39.424 up. So doctive one let's print this out, you can see okay, the differences between these images 00:32:39.424-->00:32:44.996 are of between one and two pixels or one and two pixels and you or I wouldn't be able to 00:32:44.996-->00:32:50.468 detect that but the classifier learns things differently so it can tell when there's a when 00:32:50.468-->00:32:54.105 there's a pretty significant difference or when there's a calculated difference between 00:32:54.105-->00:32:59.110 these two images. Let's save this. Kay, typo. I was lazy to redo the demos uh typos there. 00:33:02.581-->00:33:08.553 And save. Okay save the image and let's actually look at it. This is the difference between 00:33:08.553-->00:33:15.193 the two images. So if you add the normal image and this noise vector and this noise image you 00:33:15.193-->00:33:20.198 actually get this. Yeah. And let's look at a the difference between adversarial one and 00:33:24.870-->00:33:29.875 adversarial ten. You'd expect that F10 has larger has larger perturbations. Typos... I have 00:33:37.382-->00:33:42.387 to calculate doc at ten first. Okay so you see now that instead of having perturbations of 00:33:59.537-->00:34:04.476 length one or two now you have larger perturbations. Okay. and same for the automobile. So why 00:34:13.018-->00:34:19.224 can we do this? Basically it's an open research problem and not everyone in the research 00:34:19.224-->00:34:25.230 community agrees on why you can do this but mainly it's the concept of blind spots. Um 00:34:25.230-->00:34:31.703 machine learning models learn in ways that are vastly different from how humans learn and 00:34:31.703-->00:34:37.642 there's this concept of the data manifold which is the mapping between the input and output um 00:34:37.642-->00:34:41.446 and there are gaps there are pockets in the data manifold that allow you to do such 00:34:41.446-->00:34:47.752 things. So how do you generate images like that, how did I generate images like that? Um 00:34:47.752-->00:34:52.924 the intuitions are just three steps, the first thing is you have to run the input through a 00:34:52.924-->00:34:58.697 classifier model then based on the model prediction derive a tensor a vector or a matrix that 00:34:58.697-->00:35:02.334 maximizes the translator's misclassification. You can do this in three methods that we'll 00:35:02.334-->00:35:07.138 touch on a bit later. And then you scale the perturbation by some magnitude resulting in a 00:35:07.138-->00:35:12.410 perturbation which is the noisy image that you add to your original image and then use it 00:35:12.410-->00:35:17.082 as adversarial image. So that will result in an image that tricks you that tricks 00:35:17.082-->00:35:21.987 classifiers but not humans. And obviously if you scale the perturbation tensored by a 00:35:21.987-->00:35:26.424 larger magnitude you have a higher chance of tricking classifiers but then you also 00:35:26.424-->00:35:31.429 have a higher chance of a human detecting that this is that this looks weird. So a couple methods 00:35:33.765-->00:35:39.337 you basically traverse the manifold to find blind spots um in an input space there's 00:35:39.337-->00:35:46.044 there's also optimizations for that that help you to uh do this more efficiently and do this in 00:35:46.044-->00:35:51.216 a period of seconds instead of hours. And then there's also like better optimizations that 00:35:51.216-->00:35:56.654 allow you to look at how much each particular pixel actually affects the output this is 00:35:56.654-->00:36:02.227 called the salina selamat and i think it's really cool and you only change you you change those 00:36:02.227-->00:36:07.365 um uh pixel values by a smaller amount than you than the other pixels that don't affect the 00:36:07.365-->00:36:12.370 output as much so you can affect the output more without changing pixels as much. So we'll look at 00:36:14.639-->00:36:19.678 the threat model. The more you know about the model of course the better you can do against 00:36:19.678-->00:36:26.317 something like this. Um if you know a lot about the architecture uh If you know the 00:36:26.317-->00:36:31.756 training tools used, even the framework or library used then you can simply use that and 00:36:31.756-->00:36:36.661 train the model and and generate some adversarial images and you'd be good. So that's easy. 00:36:36.661-->00:36:41.833 The hard thing is when you have only label test samples. Let's say you were doing you were 00:36:41.833-->00:36:45.370 dealing with an online service like Amazon Machine Learning as a service or other machine 00:36:45.370-->00:36:50.275 learning as a service start ups and you wanted to induce some kind of misclassification on 00:36:50.275-->00:36:56.081 them then that's a bit harder but you can still do pretty well. So you can do a lot with 00:36:56.081-->00:37:00.151 limited knowledge you can make good guesses and infer the methodology from the task for 00:37:00.151-->00:37:03.755 example if it's image classification digit recognition you can use something like 00:37:03.755-->00:37:08.026 convolutional neural net, speech recognition then you use something like recurrent neural 00:37:08.026-->00:37:12.697 network and if its general services like machine learning as a service than you would use 00:37:12.697-->00:37:16.968 a shallow network because these networks can be easily generalizable. So what if you 00:37:16.968-->00:37:23.341 can't guess? Can you still do anything? So this is a small example of a Captcha Crusher a 00:37:23.341-->00:37:29.914 Captcha Crusher is this really cool project and what I'm going to be doing is reading captchas 00:37:29.914-->00:37:35.220 with Captcha Crusher and and uh testing them on Captchas that I'm generating with cool PHP 00:37:35.220-->00:37:40.225 Captcha. So let's just generate some captchas here. Okay this is the evaluation model um it just 00:37:47.132-->00:37:52.137 resends the samples and then tests them one by one and gives a prediction um it prints out 00:37:54.873-->00:37:59.410 the actual label and the predictive labels to you can compare them on the command line 00:37:59.410-->00:38:04.482 and then precision at 1 just means the top prediction so the top confidence for the for the 00:38:04.482-->00:38:10.388 prediction so usually when you run such classifiers they'll give you like a ranking of um we 00:38:10.388-->00:38:13.992 think this is the most likely and this is the second most likely so precision at 1 just 00:38:13.992-->00:38:20.665 means just compare the precisions for the top most likely. Let's generate some 00:38:20.665-->00:38:26.504 captchas with cool PHP captcha, generates it generates captchas that are pretty similar to what 00:38:26.504-->00:38:31.476 you see out there um you can train the model to to work better for different kinds of 00:38:31.476-->00:38:35.980 captchas different kinds of perturbations but uh I have problems reading some of these 00:38:35.980-->00:38:40.985 so I think that qualifies as a good captcha according to what I'm seeing on the web now. Okay 00:38:45.290-->00:38:50.295 so they're just random captchas let's generate some new ones to use for tests, PHP Captcha again 00:38:54.499-->00:38:59.504 okay. Okay then, let's run it. So this is the training of the model um training deep learning 00:39:06.211-->00:39:11.683 models take a pretty long time uh this is an output I don't want to bore you for thirty 00:39:11.683-->00:39:16.688 hours uh but you can see that I started training this model at on July 12th 5:53am and it 00:39:21.559-->00:39:26.564 completed, completed July 13th 9am so that's about 30 hours. Okay? So it does pretty well 00:39:32.837-->00:39:38.276 model accuracy now 8.2% so this is like no humans involved it's just reading just reading 00:39:38.276-->00:39:43.881 captchas reading the images and then it's learning how to read them so you know like that death 00:39:43.881-->00:39:49.621 by captcha services that use humans and interesting ideas to use this and make some money 00:39:49.621-->00:39:54.626 with server farms so you can make money solving captchas by using this tool. Okay so I see a 00:39:59.130-->00:40:04.135 mixed card predictions skip forward a bit. Okay so in this case you know there's only ten 00:40:06.638-->00:40:12.410 samples so it predicts all of them correctly ah you can see that for the last for the first 00:40:12.410-->00:40:17.415 sample it actually read IACTGB and let's look at the actual image oh let's look at the last 00:40:24.589-->00:40:29.594 image okay no. [chuckle] so let's look at tricking this model. The interesting thing is 00:40:33.197-->00:40:38.202 that um you can generate adversarial samples for these models with life model is just a 00:40:47.345-->00:40:52.350 edit of of the of the two and this is just a walk through of the code um it's online and then 00:41:02.360-->00:41:07.365 let's test it out and you can see that now it's printing something very different so 00:41:15.606-->00:41:20.611 let's say you were facing some kind of a captcha solving tools problems on your website um then 00:41:23.348-->00:41:28.219 you maybe want to use something like this if you suspect that someone is using deep learning 00:41:28.219-->00:41:32.724 to bypass captchas on your site then this would be an interesting thing to do where 00:41:32.724-->00:41:38.696 you would predictively break these tools and decrease their accuracy by a lot. So this is an 00:41:38.696-->00:41:44.302 interesting cat and mouse game because the deep learning models are used to bypass your captcha 00:41:44.302-->00:41:48.706 can just take these new images and train them and this is how you would make your models more 00:41:48.706-->00:41:54.045 robust you would take um these adversarial images and train them it on your model to to to 00:41:54.045-->00:41:58.449 make it perform better against adversarial images. So there's no end to this. So why can we do 00:41:58.449-->00:42:05.189 this? Two things transferability is the first one. So the adversarial samples that fool a 00:42:05.189-->00:42:11.262 particular model have a good chance of fooling some other model. So even if it's a vastly 00:42:11.262-->00:42:16.267 different architecture let's say you're training your model using um a decision tree um and you 00:42:18.436-->00:42:23.441 see that you can bypass a you can bypass this model using adversarial samples generated 00:42:25.676-->00:42:30.681 with a deep learning network with a pretty good with a 79.31% accuracy a 79.31% chance. So 00:42:34.185-->00:42:38.689 this is this is kind of weird, why why can you do that? Like, It's still an open research 00:42:38.689-->00:42:44.128 problem, The second thing is that substitute models you can always just use substitute 00:42:44.128-->00:42:49.834 models to train the target model and generate adversarial samples with this target model 00:42:49.834-->00:42:55.106 substitute model and then you can use them on these networks. Open research problem. So what 00:42:55.106-->00:42:59.510 this means for us is that deep learning algorithms are susceptible to manipulative 00:42:59.510-->00:43:04.482 attacks and you shouldn't make false assumptions about what the model learns, you should always 00:43:04.482-->00:43:10.121 evaluate the models resilience and not just its accuracy and these are some ways that you can 00:43:10.121-->00:43:14.158 use to make your models more robust and I'm introducing this framework today called Deep 00:43:14.158-->00:43:19.163 Pwning which is a metaset from machine learning and it's just this there's a get help page 00:43:24.902-->00:43:29.740 which you can find Deep Pwning it allows you to generate adversarial samples for 00:43:29.740-->00:43:34.745 arbitrary models and use this to test your network. So please play with it and contribute um 00:43:38.549-->00:43:43.888 this is important because more criminal systems rely on machine learning and thus there's more 00:43:43.888-->00:43:48.226 importance on ensuring a robustness and we need people with both statistical and 00:43:48.226-->00:43:54.131 security skill sets to evaluate these systems and to actually know when someone is trying to 00:43:54.131-->00:44:00.838 bypass and know how to protect it. So in other words learn it or become irrelevant. That's it. 00:44:00.838-->00:44:05.843 Thank you. [applause]