So today we're going to be talking about machine learning. No one has not heard of
machine learning, right? So in specific what we're going to be talking about is how to
bypass deep learning systems. Machine duping is what I'm choosing to call it. It's a
really corny name so if anyone has better ideas come to me after the talk and we can talk
about it. So everyone has heard of machine learning. My grandma knows what machine
learning is. At least if you've been reading the news you know that the best Go player in
the world is now a deep learning system. So that's kind of weird. No one saw that coming in
like the last five years. But let's not have a slide on what machine learning is and what
machine learning can do for us. That's something that's for a presentation two years ago.
Let's instead look at what we're trying to achieve in this talk. So everyone wants to be the
flying carpet beagle in the middle of the Venn diagram there. Most of us in this room have
some sort of hacking skills.
People hacking or computer hacking. We do stuff. We're implementers. So I'd say we're in that
category over there. Hackers. Security researchers have some kind of theoretical background.
They're working on crypto and stuff then they have some kind of math and stats skills. I don't
necessarily consider myself that. Data scientists don't have any hacking skills but they do a lot
of stuff in companies trying to maximize conversion. They have math and stats skills. But what we're
trying to do today is to help all of you guys and convince all of you guys that you really want to be
in the center of the Venn diagram and it's going to be important and increasingly important in the
next few years to brush up on math and stats skills and to know about what's going on in the
machine learning space, especially for security folks. So whether we know it or not, we are
interacting with machine learning and deep learning systems on a day to day basis. If you don't like
it, you have no choice.
Google now, Apple Siri, Amazon, Alexa, they're all things that have been covered by the press, very
high profile things. Some common, some more common use cases of deep learning are in object
recognition, in self-driving cars. If you know Joe Hart and his cool Comma.ai start up, they're
using deep learning to recognize objects on the road and build a really cheap self-driving car
system. Obviously on the top right hand corner you see AlphaGo. You can see the
Go beating the world champion at Go. And you have also other pretty interesting use cases, like in
medical research where they're using deep learning to predict the effects of newly developed
drugs in patients. And I just have an example from Nvidia there. Also in the security space,
there's also lots of stuff. If you've ever set foot at RSA or Black Hat Expo, you know what I'm
talking about. Everyone is saying they're using machine learning, deep learning to do some kind of
stuff. The extent to which that is true, I cannot vouch for. So why would someone choose to use
deep learning? Again, forgive me if you're an expert at deep learning. This is a DC101 track so
I'm going to be spending a little bit of time going through some basics of deep learning and
machine learning. And then I'll go into the interesting stuff, which is my research. Why would
someone choose to use deep learning over more traditional machine learning methods, like SVMs or
Linear Classifiers, clustering algorithms? Well, you know what I think this is a great good question.
first thing is that when you use a deep learning algorithm you get some things for free that
you otherwise would have to spend a lot of time doing. Um the most important thing that
everyone would uh point out to you is that you get feature engineering for free. Um again
the extent to which that is true you have to try it out and implement it and it depends on the
use cases that you're you're you're using it for. Um deep learning helps to select the best
features and you don't necessarily have to spend a lot of time doing feature engineering. If
you talk to any data scientist or machine learning engineer uh working in a large company uh
then they'll tell you that most of their time is not actually spent on algorithms. They're not
trying to increase the efficacy of this so and so cutting edge algorithm that uh they're using
to in in their company's product recommender systems. What they're actually doing is feature
engineering and data cleaning. So they're like janitors you know. Like I'm a janitor
too. I I spend most of my time cleaning data. I spend most of my time doing data feature
engineering. So deep learning gives you that for free and that's why it's so appealing I think.
The other thing which I think is the main difference between deep learning and other kinds of
machine learning algorithms is that uh it touts the promise of one infrastructure for
multiple problems. So if you think about it deep learning really is just one just one
infrastructure. There's multiple layers of linear units and uh each one of these linear units
interact in a different way. So if you think about it deep learning really is just one just one
way with different linear functions to give you the result that you want to learn different
things that you wanted to learn. Compared with other kinds of machine learning algorithms like
clustering you and uh SVMs and and logistic regression decision trees. All of these require
vastly different code bases. Whereas for deep learning the the differences in infrastructure
are parameterized into the number of different layers, the number of units in each layer, the
functions between each layer and other things like that. So it's sort of one
infrastructure for multiple problems and I think that's what gives it it's flexibility. The
last two points are perhaps more relevant in today where there's so much data to deal with. Um
deep learning allows you to do hierarchical hierarchical learning. So you can split up the
task of learning across different layers and we'll see an example of that later. So for
example you have more shallow layers learning vastly different things from the deeper layers and
you can extract out the outputs of each intermediate layer to uh to get the results that you want to
uh exactly find the thing that you're looking for. The last thing is it's efficient and it's
easily distributed and parallelized. If you're looking at algorithms like clustering there's no
straight forward way to really distribute it across systems and when you're dealing with
terabytes, petabytes of data this is a problem. Of course there's message passing algorithms
that uh that uh help clustering out with that but they're a lot more complex. So deep learning
allows you to distribute these uh problems up, distribute your infrastructure up and
distribute your data. So deep learning allows you to distribute your infrastructure up and
distribute the computation. Of course it's definitely not one size fit all. Uh deep learning is
not something that you want to use for any random problem. If you're trying to predict let's
say the prices of oranges uh against time you wouldn't be using a deep learning
infrastructure. That's like you know using deep learning to um you know predict a problem
space of two dimensions. You wouldn't do that. Uh what you would do, what you would use deep
learning for is in uh problem spaces that have multiple dimensions. So like
hundreds or thousands of dimensions usually these things come from nature. So you can think of
images, audio, video and prediction problems that are in a very complex problem space. So let's
just spend a couple of minutes to go through uh the two steps involved in training a deep
learning architecture, a deep learning infrastructure. So this is uh a diagram that a lot of
you may have seen before. It's it's it's a it's a four five three neural net architecture.
It's a very simplified version of it. Um so ignore all the all the nasty
white lines between those those units. Each circle represents uh a linear unit and there's a
activation function in it. Activation function just means that if you input a certain value
into this circle then uh it either outputs uh it outputs a certain real numbered value um
according to uh the the the function. And each connection between two units between layers is
uh is is uh weighted by the weight w and also there's a bias unit. Uh what the purpose of the
bias unit is simply uh to help to skew the results of the output by a certain amount. If you can
think of uh just linear algebra you have y equals to ax plus b. This bias unit is equivalent to b
which controls the intersection between the x axis and the y axis. So all this is just theory.
Don't don't don't worry about it if you just like take any one of the ten thousand MOOCs out
there to learn about deep learning. But let's just look at uh a simple example of how training
works. So we can uh go into how to bypass them.
So the first step is the feed forward step. Um each unit receives the output of neurons in
the previous layer. Uh in the case of the first layer it just receives the input. Um and
then uh the bias added to it the way it the output of the first you the output of the first unit
is weighted by w1 and then it goes on and on. And eventually when it comes to the output layer
uh it outputs logits which are just numbers which is an array of numbers. It's fed
through most of the time a softmax function for classifiers which just scales it into a
probability distribution function. And so in this particular case you see that uh this dummy
uh matrix, this dummy vector that's output is .34, .57, .09 and this just means that uh if
you have three classes 0, 1 and 2 uh this classifier predicted that class 1 is the predicted
class. Uh however that is uh wrong in our case so according to the labels cause this is
training so you know the labels. So uh the error is .57 uh because the ideal case would be
that you predict with probability 1 that uh the output is actually 0. So you feed it
backwards and back propagation is uh really the crux of deep learning. So what back
propagation is, it's a pretty, well it's not the most straightforward algorithm that you'll
buy, you'll come across in machine learning and when you're taking some MOOCs like the
Stanford one on machine learning by Andrew
Ng he'll also say that back propagation is a pretty hard thing to grasp. I think the easiest
way to explain it, the easiest way to think about it is that back propagation is just assigning
blame. So let's say you uh you were the head of the board of directors and you had a bunch of
people on the board that were giving you uh suggestions that were advising you and stuff. Um
and some of them just talk bullshit all the time. Uh you wanna listen to them less so that's
what this is doing. Uh so basically you're feeding input data,
through uh a deep learning network and you're trying to figure out what gives you wrong
answers. And you do that by sending in input data and then seeing when the answer is wrong, you
know the answer, you you you know the right answer and you're seeing what answer is wrong and
then you trace, you trace the path that this answer took through the network and you find out
exactly which units are responsible and how much they're responsible because of their weights.
So let's say you find that a particular path of units is
particularly bad at giving you recommendations or predictions for something that you're trying to
learn. Then you just decrease their weights and you block them out. So all of this uh can be
optimized with certain algorithms like stochastic gradient descent which is just basically uh
trying to find the local minima uh in in in a particular uh problem space. So there's gonna be
lots of demos in this talk so if you have been ignoring me for the last ten minutes uh you
can uh look up now. Okay is this big enough? Let's make it a little bigger. Okay so what we're
gonna look at here is an example of uh a deep learning system that is really accessible to
everyone. This is Tensorflow um it's uh Google's uh open source uh deep learning machine
intelligence framework. I think it's uh probably the easiest uh deep learning framework to use and
what we're gonna be walking through is a small example of how to do uh deep how to use deep
learning to solve the easiest tasks uh the tasks that's used most commonly in most tutorials and
examples. And then we'll look at how to bypass that. So Tensorflow. Okay. So what we're trying to
do is use the MNIST database which is um which is created you know like 20 years ago I think by uh
Ian LaCoon who is now the Facebook AI director. And uh we'll be having some visualizers. And uh we'll be having some visualizers. So if you're
looking at the code here it's all visualizations done with this pretty cool uh visualization tool. So let's look at the code a
little bit. What this is doing is basically taking the training data and the testing data and
labels and then you're creating a validation set all this is is is uh just um sugar. And then here is the
actual definition of the model. This is the model and the layers are defined uh line by line. You have
the first convolution layer, then you have a pooling layer, convolution layer, you have the second
convolution 2, pool 2 and so it just goes down from top to bottom and you get from shallow
layers to deeper layers as you go down and as you see the logits and uh the softmax function
actually creates the output of this uh neural net. So this is a demo of MNIST classification.
What MNIST is is just digits, handwritten digits. Uh it looks something like this. So these
may not look like digits that you or I would write like I would never write a 7 with a with a
weird thing at the end uh but you know what can you do. You can always train a model on on
different handwritten data sets but uh this is a standard and it's used for comparison
between between uh researchers and in in academia. So let's see here this is uh a real time
um a real time classifier predictor for for digits uh so if I were to write like a 2 here
let's see if it predicts it correctly. Oops. Okay 2 even though there's a weird thing. So
that's pretty good. Let's do like a 7. Oh 7 that's great. Only .7 accuracy. I mean
confidence. Let's try something that's a little bit more challenging for the model. Like a 6.
Okay so this this is a wrong this is a wrong classification because it thinks it's a 5 with
.8 it's 8 6 I I.
Confidence. Let's see if we can make it better. Nope. Nevermind. So you can see this
is this is not the cutting edge in uh handwritten digit uh recognition. Like if if uh I
think uh this this particular this particular implementation reper uh correctly classifies 90%
of digits. Um so that's nowhere clear cl uh close to 100% of ah uh the uh the the the uh the
to what a human can do right? I mean if you or I recognize 9 out of 10 digits then you
should see an eye doctor. But anyway that's that and uh let's continue. So what that was
doing um it was using convolutions and this is what you call a convolutional neural network.
So these are just different flavors of neural networks just different algorithms that
researchers publish papers on um to get tenure and stuff but this is a really cool algorithm
um that was developed 20 years ago I think um and uh what this does is that it uses
convolutions to uh gather insights on different details uh different uh levels of details in an
image or in stuff that have adjacency relationships. So what are convolutions? If you
remember your Fourier transform days from school or not um convolutions are
just filters so uh if you apply a filter or a convolution on a matrix let's say in this case
a 2D matrix then what you have is you do a matrix multiplication and then you end up with
single value for the convolution applied on the particular space that you're applying
convolution on. And so what this allows you to do is layered learning. This is an example
for facial recognition where the layer, the shallow layer is actually at the bottom here and
then you're going out as you go deeper into the network. Uh in layer one you're looking at a
learning very, very, very, very fine features like the shape of your eyebrows, no, the shape
of your eyelashes and maybe the wrinkles in your face. As you go further up, as you go further
down the network, then because of the convolutions and pooling, then you actually get more
zoomed out features like the shape of your eyes, the color of your eyes, whether you have
like a mole somewhere or something. And at higher layers, then you get the shape of your
face, more general characteristics of you, like if you have a mustache or not. So this is
interesting because you can extract the results out of intermediate layers to do certain things
like I was at a talk by Facebook security team once and they say that in order to find images
that are spammy, so when spammers try to spam their network with images, they often tweak the
images a little bit or change the language or text in certain ways that you can't just do a
pixel by pixel comparison. So you can't just do a pixel by pixel comparison, you can't just
do a pixel by pixel comparison. So there are certain ways that you can solve this by shingling or
by doing some kind of fuzzy matching, but the most efficient way they found and the most
effective way was actually to pass these images through a neural network and then get the
second layer out and compare the outputs of the second layer. Then they were able to reliably
find images that were spammy and then group these images together and then have a human actor
come in and see whether this is actually a spammy image or not. So this is just a diagram of the
convolutional neural network that was used to classify the digits. Basically if you look at
the small squares that are zoomed in, one part of the digit is fed into feature maps and then
you do sub-sampling on that and then you perform convolutions, you perform more sub-sampling
and then you do a prediction on that. So besides convolutional neural networks, there are
also things that are called recurrent neural networks. These are slightly more complicated but
not actually that complicated. What you have is just recursion in a neural network. So instead of
feeding in through, instead of feeding in input one way through a neural net you have recursion
in it so the output of each time step and each intermediate output actually gets fed into
the next time step. So this introduces the concept of memory and memory is important because
when you learn things, you don't necessarily learn things one frame at a time. That will be really weird. Um,
And this allows you to learn things like music or things like audio or video which have some
kind of relationship between frames and relationship between inputs. And so this is an
example of a generative recurrent neural network. You can teach a network to spell words and
so in this case if you see that you already have the letters Y, O, L, then O is likely to be
the next word because of the memory of the network having Y, O, L in its buffer. So there's
also stuff that's more on the cutting edge. This was actually used to a certain extent in AlphaGo
long short term memory. So when you're looking at things with slightly more context but you
don't want to scale the depth of the network or the depth of recursion indefinitely, then you
want to use things like LSTM, long short term memory networks, because this allows
you the concept to arbitrate longer term concepts, arbitrate longer term data to store for a
longer time and not just have a single FIFO queue that you store your memory in because that's
not how we learn. So to make good predictions we need more context and you can think of things
like a system that converses with you. When you're talking with somebody and someone mentions
that he's from let's say France five minutes ago in a conversation, you don't just forget that after five minutes.
Because your memory buffer is full. You have to remember things like that. And so the beauty of
LSTM networks is that they have certain gated functions in the recurrent, in the recurrence
of the network that allows you, that allows the network to learn what's important to remember
for a longer period of time and what's not. So this is uh just a simple diagram of how deep
learning has helped uh with speech recognition over time. The different lines in this diagram, uh,
this is uh the y axis is logarithmic by the way. So the different colored lines represent
different data sets. Uh the holy grail is definitely the red line. It's conversational
speech. So a very very uh a very very sterile kind of speech is um red speech or broadcast
speech where uh every word is enunciated to to you know to to to the nail. And um all of
these over the years have seen pretty good performance. You you see like up to two percent
word error rate which is w-e-r. Word error rate uh performance in air travel plan planning
kiosk speed was a pretty weird data set. Um and then for conversational speech um you see in
twenty eleven it got it went down all the way to about twenty percent. So that's that's great.
By the way the conver conversational speech data set if you ever um if you ever have a chance
of of finding it or looking at it I
it's not out in the public but it's actually pretty weird. Um it's actually from blind
dates. So so uh when when I was when I was uh listening to it it was it was very interesting
how uh some of these uh dates turned out. But yeah there's some disturbing stuff in
there. So okay now for the now for the fun stuff. How to pwn. So much time we have left.
Cool.
Okay so that's a short video. This is Gunter he's uh a dog. He's uh he's a he's a mini schnauzer
and uh he's he's my best friend's dog. So let's just uh see how this analogy pans out.
Cool. See this video plays. Okay. Cool. So this is Gunter. He's uh he's a dog. He's a dog.
Gunter he loves ice cubes. I'm not sure if all dogs loves ice cubes. They think like he
thinks it's alive. So this is the training phase where I'm basically teaching him that the
clink clink sound in the bowl represents that there's an ice cube in there. Sorry. The clink
clink sound in the bowl represents that there's an ice cube in there and he knows he's gonna get
an ice cube because he had a taste of it. Then I mislead the model. First I show him the ice
cube. I put it back in the bowl. And I don't actually throw it. I I I throw something else that's not an ice cube.
And he's confused. The model's bypassed. So Gunter is the model now. And this is kind of a like a
lame analogy. I I dropped the stuff I'm doing. Forgive me. Uh but what we see here is that the
dog doesn't know that I'm throwing the the ice cube. He just thinks that I'm throwing the ice
cube because he sees like the big hand motions. Um ice ice doesn't have much smell and he he he
doesn't have great eyesight. So he sees me throwing something and I've done it a few times
before. So he thinks that an ice cube doesn't have much smell and he he he doesn't have great
eyesight. So he thinks that an ice cube is there at the other end of the of of the yard waiting
for him. So that's very similar to what we're gonna be doing. Um we're gonna be feeding in images
to classifiers um that will mislead these classifiers. And these images are crafted in a
certain way that will uh facilitate certain classifiers from being misled more than others. But
first let's look at the attack taxonomy so we can look at the attack the attack vectors and stuff
like that in security speak.
Um there are two general kinds of attacks that you can do on these machine learning or deep
learning systems. You can have causative attacks or exploratory attacks. Causative attacks are
relevant when you have access to the training step. Um so for example when you're looking at
like an online translation engine like Google Translate uh they actually do rely on some kind of
online reinforcement learning when you see that something is a really really bad translation
which happens pretty often I'm sorry. But um when you see something that's really bad and it's
really bad you can report it as really bad and you can maybe give the right answer or something
that's more relevant. And so this is how it how it does reinforcement learning. Um in Gmail when
you have when you receive email that's not marked as spam but actually really is spam you can
mark it as spam and I urge all of you guys to do that because that actually helps a lot with the
with the reinforcement learning model. Um and uh this will help to train the model to
recognize such examples as as spam in the future. So you can see how you can use this to help you
can influence the model in such a way. That's a little less interesting that's what I that's what
I that's what I did last year in in in talks. Uh but what's more interesting I think are
exploratory attacks. So in this attack model you have no access to the model you have no
access to the training phase and what you're doing is just a black box attack in in in some
scenarios where you're feeding it just weirdly crafted samples that look correct to a human but
uh the machine just gives you a wrong result. And so that's really that that really throws some
people off even for machine learning researchers because um they think that the deep learning
model or the machine learning model is learning exactly in the way that the human learns. Like we
don't learn uh to recognize alphabets or letters by uh looking at the pixel or the angle between
uh the horizontal line and slanted line in an in an A. Um we learn it in a more general way and I
think it's still active research area into how to represent these things in machines better. Uh so
this is still the dog keeps coming up. Uh this this is still active research area uh and uh that
throws people off. There's also targeted and and indiscriminate attacks when you try to uh move
the decision boundary in a certain way to cause an intentional misclassification or indiscriminate
attack when you uh just try to decrease the integrity of of uh these classifiers. So this is a
simple example of um misclassification. Uh so this is a simple example of um misclassification of a
classification. So in the early days MNIST and uh digit class digit recognition used to be used
in recognizing digits uh in in checks. And uh I know these these digits don't look very
realistic but I'm using stuff from the MNIST data set. And what I did beforehand was to was to
generate some adversarial images and then fill in two copies of the checks. One with normal images
and one with adversarial images. So if we look at the MNIST data set. So if we look at the
MNIST data set. So if we just go back a bit and look at this. This is the adversarial one and
this is the normal one. They look pretty identical. Just look at the digits portion. So it's
nine three seven eight. And this is some simple code to use for pre-trained MNIST model with
the standard tensor flow uh MNIST example trained with that. It takes about four hours to train
this model which is, trust me, really good speed using a CPU in the deep learning world. You
see something that takes a longer time to train later. And so what we're gonna do is to just
basically read the check. Um this just divides the image up into a pixel matrices and then uh it
reads digits. So it's loading the model. It's predicting uh the digits. Nine three seven eight
zero zero. Nine three seven eight point zero zero. So that's that's correct. And so if you look
at that that's what it is.
Okay. And now we're gonna do the same for the adversarial image. Yeah. Okay. So you expect
this to be the same but no. You actually expect this to be different because it's called
adversarial. Um so the output of this is something totally different. Zero five two four point
point eight four. So it looks the same but it gives you a different output using the same
model, using the same code. And uh that's adversarial machine learning. This is something
a little bit different. It's the CIFAR 10 data set. So CIFAR is um is a data set of images.
So what you're trying to train the model to do is to recognize images uh in 10 classes.
There's CIFAR 10 which is for 10 classes of images. There's also CIFAR 100 which is for 100
classes of images. The classes are are here. They are uh like you have things like airplanes,
automobiles, birds, cats, deers. The interesting thing is that these images are not high
resolution images. You don't take them with your phone. They're 32 by 32 pixels large. So
this is the actual size of it. Actually maybe bigger. And so we're looking at two sets of
images here. Um dog and automobile. Uh so this is the actual size of it. Actually maybe bigger.
Just chose them cause dog is cute. So uh what we see here is that, well if you see the
preview window on the right here uh that's actually preview on on Mac uh it's not 32 by 32. I
think Mac OS does some aliasing so your your pictures don't look like shit. And uh what we're
doing here is to eval this and what this evaluator does very similar to the MNIST uh classifier is
that it classifies.
This this image into a class. So it tells you whether this image is of a dog or of a or of
something else. So you see in the F10 uh image it classifies it as a ship. Ship. In F1 it
classifies it as a ship as well. So you can see that the images are pretty similar. Um but
there's actually differences in them. Let's classify the automobile one just for completion.
Automobile. Okay this is what it looks like. It's a pretty shitty image. Automobile. Okay
that's correct. And what the different numbers after add means is just to ha to what degree
they are perturbed. So to what degree we're injecting stuff in the image to uh make the
classifier think that it's not what it is. So you see it becomes a little bit more grainy but to a
human it still looks like a car. I mean you wouldn't say that that's a that's a cat. Or you're
weird. Yeah? So let's look at exactly what differences uh exist between these images. But
let let's open a python shell uh scipy. Let's read these images in and then uh look at what the
difference is between them. By the way a lot of python libraries actually don't read PNGs or
don't write PNGs exa- exactly pixel for pixels. So it's not exactly pixel for pixels.
So if you're trying this at home use these libraries. So this is the this is the standard
representation of of the PNG. It's just um a thirty two by thirty two pixel uh representation.
There's three channels R, G and B and uh the the the value of each pixel can be between zero and
two five five which is pretty standard. So we're reading the adversarial image now. And uh
and we can see that. Let's print it out. We can see that okay the numbers are slightly
different from before. Let's print out exactly what's different. Uh let's look at the size
first. You can see that the the shape and the the the shape of these two images are are the same
thirty two, thirty two, three. And let's let's calculate the differences between the
adversarial image and the normal image converted to uh in sixty four to prevent any overflows. Uh uh
this is boring you out. Just like zone out for a moment and I'll wake you up. So docdiv one
let's print this out. You can see okay the differences between these images are of between one
and two pixels or one and two pixels. And URI wouldn't be able to detect that. But the
classifier learns things differently. So it can tell when there's a when there's a pretty
significant difference or when there's a calculated difference between these these two images. Let's
save this.
Okay typo. I was lazy to redo the demos uh typos there. And save. Okay save the image and
let's actually look at it. This is the difference between the two images. So if you add the
normal image and this noise vector and this noise image you actually get this. Yeah? And let's
look at uh the the differences between adversarial one and adversarial ten. Um you would expect uh
f ten has a larger has larger perturbations. Okay more typos. I have to calculate doc f ten
first. Okay so instead you see now that instead of f ten I have to calculate f ten and f ten
I could sem differentiate it so instead of having perturbations of a of a length one or two
then now you have larger perturbations. Okay. And same for the automobile. So why can we do
this? Basically it's an open research problem and not everyone in the research committee
agrees on why you can do this ah but main it's the concept of blind spots. Um machine learning models learn
in ways that are vastly different from how humans learn. And uh there's this concept of the
data manifold which is the mapping between the input and output um and there are gaps, there
are pockets in the data manifold that allow you to do such things. So how do you generate
images like that? How did I generate images like that? Um the intuitions are just three
steps. The first thing is you have to run the input through a classifier model then based on
model prediction derive a tensor, a vector, a matrix that maximizes the chances of
misclassification. You can do this in three methods that we'll touch on a bit later and then
you scale the perturbation tensor by some magnitude resulting in a perturbation which is
the noisy image that you add, you add to your original image and then use it as an adversarial
image. So that will result in an image that tricks, that tricks classifiers but not humans.
And obviously if you scale the perturbation tensor by a larger magnitude you have a higher
chance of tricking.
Um classifiers but then you also have a higher chance of the human detecting that this is,
this looks weird. So a couple methods, you basically traverse the manifold to find blind
spots um in, in input space. Uh there's, there's also optimizations for that that help
you to uh do this more efficiently and do this in a period of seconds instead of hours. And
then there's also like better, better optimizations that allow you to look at how much each
particular pixel.
actually affects the output. This is called the saliency map and I think it's really cool.
Um and you only change, you, you change those um uh pixel values by a smaller amount than
you, than the other pixels that don't affect the output as much. So it, you can affect the
output more without changing pixels as much. So we'll look at the thread model. Uh the more you
know about the model of course the better you can do against something like this. Um if you
know a lot about the architecture. Uh the, the, the, the, the, the, the, the, the, the, the, the,
the training tools use even the framework or library use then you can simply use that and train
the model and generate some adversarial images. And you'll be good. So that's easy. The
hard thing is when you have only label test samples. Let's say you were dealing you're dealing
with an online service like Amazon Machine Learning as a Service or other machine
learning as as service startups. And you wanted to induce some kind of misclassification on
them. Then that's a bit harder but you can still do pretty well.
So you can do a lot with limited knowledge. You can make good guesses and infer the
methodology from the task. For example, if it's image classification, digit recognition, you
can use something like convolutional neural net. Speech recognition, then you use something
like recurrent neural networks. And if it's general services like machine learning as a
service, then you would use a shallow network because these networks have to be easily
generalizable. So what if you can't guess? Can you still do anything? So this is a small
example of Capture Crusher. Capture Crusher is a really cool project and what I'm going to be
doing is reading captures with Capture Crusher and testing them on captures that I'm
generating of cool PHP capture. So let's just generate some captures here. Okay, this is the
evaluation model. It just reads in the samples and then tests them one by one and then
gives a prediction. Um, it prints out the actual label and the predicted label so you can
compare them on the command line and then precision at one just means the top prediction so the
top confidence for the prediction. So usually when you run such classifiers, they'll give
you a ranking of we think this is the most likely and this is the second most likely. So
precision at one just means just compare the precisions for the top most likely. Generate some
captures with cool PHP capture uh generate cap generates captures that are pretty similar to
what you see out there um you can train the model to to work better for different kinds of
captures different kinds of perturbations but uh I have problems reading some of these so I
think that that qualifies as a good capture according to what I'm seeing in the web now.
Okay so they're just random captures. Let's generate some new ones to use for a test. PHP
capture again. Okay. Okay then let's run it. So this is the training of the model. Training
deep learning models take a pretty long time. Uh this is an output I don't want to bore you
for 30 hours uh but you can see that I started training this model at uh on July 12th 5 53
a.m. and it completed completed July 13th 9 a.m. so that's about 30 hours. Okay. So it does
pretty well. Model accuracy is now 8.2 percent. So this is like no humans involved it's just
reading just reading captures reading the images and then it does a pretty good job of
it's learning how to read them. So you know like that death by capture these services use
humans and interesting idea is to use this and make some money with a server farm so you can
make money solving captures by uh using this tool. Okay so you see it makes correct
predictions. Skip forward a bit. Okay so in this case you know there's only 10 samples so it
predicts all of them correctly. Uh you can see that uh for the uh for the uh for the uh for the
last for for the first sample it actually read IACTGB and let's look at the actual image.
Now let's look at the last image. Okay no. Okay so let's look at tricking this model. The
interesting thing is that um you can generate adversarial samples for these models. You can
do this. With live model it's just uh edit of the of the tool. And it's just a walkthrough of
the code. Um it's online. And then let's test it out. Okay so let's test it out. Okay so
you can see that now it's predicting something very different. So if let's say you were facing
some kind of uh capture solving tools problems on your website um then you maybe want to use
something like this. Uh if you suspect that someone is using Deep Learning to bypass
captures on your site then this would be an interesting thing to do where you would
predictably break these tools and decrease their accuracy by a lot. So this is an interesting cat and mouse game.
in because the deep learning models uh the deep learning models that are used to bypass your
capture can just take these new images and train them and this is how you would make your
models more robust you would take um these adversarial images and train it with on your
model to to to make them perform better against adversarial images. So there's no end to this.
So why can we do this? Two things transferability is the first one. So the adversarial samples
that fool a particular model uh have a good chance of fooling some other model. So even if it's
a vastly different architecture uh let's say you're training your model using um a decision
tree um and you see that you can bypass uh you you can bypass this model using adversarial
samples generated with a deep learning network with a pretty good with a 79.31 percent
accuracy. You have 79.31 percent uh chance. So uh you can bypass this model using
this. This is this is kind of weird why why can you do that like it's still an open research
problem. The second thing is that substitute models you can always just use substitute
models uh to train the the target model and generate adversarial samples with this target
model uh substitute model and then you can use them on these networks. Open research problem.
So what this means for us is that deep learning algorithms are susceptible to manipulative
attacks and you shouldn't make false assumptions about what the model learns. Uh you should
always
evaluate the model's resilience and not just its accuracy and these are some ways that you can use
to make your models more robust and I'm introducing this framework today called deep honing
which is a meta spot for machine learning and uh it's just this. There's a github page which
you can find deep honing. It allows you to generate adversarial samples for arbitrary models
and use this to test your network.
So please play with it and contribute. Um this is important because more critical systems rely on
machine learning and thus there's more importance on ensuring the robustness and we need people
with both statistical and security skill sets to evaluate these systems and to actually know when
someone is trying to bypass and know how to protect it. So in other words learn it or become
irrelevant. That's it. Thank you.
Thank you very much.