00:00:00.200-->00:00:06.206
Give it up for Brad
Dixon! [member of audience]
whoooo! [audience applauds]

00:00:06.206-->00:00:11.211
Thank you. So, I'm Brad Dixon. I
work with Carve Systems, uh, we
have a lot of fun working with

00:00:13.313-->00:00:18.552
systems that involve, uh,
embedded or IOT devices, uh you
know, trying to do security

00:00:18.552-->00:00:23.557
assessments and penetration
testing, uh, on the whole
system. And, uh, this is going

00:00:23.557-->00:00:29.229
to be a presentation, a quick
one, about, uh, uh, an attack
technique that we found

00:00:29.229-->00:00:34.735
repeating itself uh as useful uh
more times than we thought it
would. And I'm gonna describe

00:00:34.735-->00:00:39.740
this as as being useful because
it works, it's easy, it's uh
it's pretty dramatic, and

00:00:39.740-->00:00:44.011
because it provides a teachable
moment about designing more
secure systems. But it is a

00:00:44.011-->00:00:49.449
novelty and it's a novelty
because it is uh it's risky,
it's crude and it's perhaps

00:00:49.449-->00:00:53.887
redundant to a lot of other
GREAT techniques that are out
there, but it sure is fun. And

00:00:53.887-->00:00:57.958
when you do this, uh, you do
this device, like you can, you
can demo this attack to your mom

00:00:57.958-->00:01:04.431
and she'll be like "Oh, I get
it!" So let's just, quick, get
to the demo! So to set the stage

00:01:04.431-->00:01:08.669
here, uh, we've used this on a
lot of devices, but this one in
particular is, is an embedded

00:01:08.669-->00:01:14.274
Linux device with, uh, u-boot as
a boot loader. There's no JTAG
that's actually being turned off

00:01:14.274-->00:01:20.113
with micro fuses inside the CPU.
So that's gone, can't find it.
JTAGulator was great, but

00:01:20.113-->00:01:25.085
couldn't find JTAG at all. And,
because this device, it had some
challenges, uh, before, let's

00:01:25.085-->00:01:30.090
say, um, it was, it had, uh,
had, uh, home-grown secure boot
loader added. [typing in the

00:01:33.026-->00:01:37.631
background] So, getting to the
demo here, this is the system
booting up and you'll see why we

00:01:37.631-->00:01:42.269
call this ping to pone in a
moment [indistinct)] This is my
trusty, like nineteen

00:01:42.269-->00:01:47.607
ninety-five multimeter just
jamming into the flash chips and
I'll show you where in in just a

00:01:47.607-->00:01:52.112
moment, but as the system is
booting it's doing a
cryptographic checksum across

00:01:52.112-->00:01:58.585
the flash image, and when that
checksum doesn't match what it's
expecting, it does a fallback to

00:01:58.585-->00:02:03.857
a secondary partition and it's
going to start that in just a
moment. And, as you can see,

00:02:03.857-->00:02:08.929
using a serial console on these
devices, you can check what's
going on. Um, I didn't put the

00:02:08.929-->00:02:14.968
clip there but I am again poking
the flash device to get the
second partition, uh, to fail as

00:02:14.968-->00:02:20.807
well. And they designed this
device to, to not respond under
any serial input. But what

00:02:20.807-->00:02:25.746
happens is that they
misconfigured it, so that if it
fails twice in a row, the

00:02:25.746-->00:02:30.384
primary and the secondary
partition, I get a u-boot
prompt. Excellent! That gets me

00:02:30.384-->00:02:35.389
in! [audience murmurs, applauds]
So the primary misconfiguration
was not checking their failure

00:02:41.595-->00:02:47.467
paths. That happens, as I found
out, more often than you would
think, and there's also gonna be

00:02:47.467-->00:02:52.339
two kinds of flash devices
you'll see in the demos. Uh,
this is a parallel flash device.

00:02:52.339-->00:02:56.777
It comes in a standardised
package, typically a TSOC
forty-eight package. You can

00:02:56.777-->00:03:01.415
actually attack it from the left
or right side, uh, I'm attacking
it in the right side between two

00:03:01.415-->00:03:06.753
of the data lines that feeds
data out of the flash device uh
when it's called, uh, called by

00:03:06.753-->00:03:13.193
the CPU. Um, and, and, as you
saw, it's pretty low-tech. You
just poke it. Um but there's

00:03:13.193-->00:03:18.365
been a lot of prior work on
this, uh glitching as a form of
attack has been done numerous

00:03:18.365-->00:03:22.836
times, uh and, just some really
amazing stuff that, like,
extracting cryptographic keys

00:03:22.836-->00:03:28.408
with transient glitches induced
by a number of manners. Um,
there's also been other blog

00:03:28.408-->00:03:33.980
posts about this precise thing;
using you know a transient
electrical fault to get a

00:03:33.980-->00:03:38.852
failure mode that's advantageous
to doing a penetration test. Um
and so there are some links you

00:03:38.852-->00:03:43.824
can check out but what I want to
do today is just provide more
details so that when you sit

00:03:43.824-->00:03:48.762
down, pull out some devices from
the closet, try this, you might
be surprised, as I was, how

00:03:48.762-->00:03:53.767
often this works. Um, I have to
warn you, uh well first of all
this is Grog, Grog's our mascot,

00:03:56.303-->00:04:00.440
uh sometimes you know, you try
elegant techniques and they
work, sometimes you just beat

00:04:00.440-->00:04:06.379
stuff with rocks, um, and and
you get what you want that way.
Um but you can definitely break

00:04:06.379-->00:04:13.086
your hardware doing this; I
haven't yet destroyed anything
but you can and and the way you

00:04:13.086-->00:04:18.158
do that is by exceeding, um,
there's an absolute maximum
current that can go out of each

00:04:18.158-->00:04:23.163
IC device and if you exceed
that, for too long um then that
device might just be you know,

00:04:25.232-->00:04:30.337
kaput; the black smoke comes
out. You can also temporarily uh
cause a device to fail; they

00:04:30.337-->00:04:35.375
have sometimes protection
circuitry so that if you exceed
the operating conditions that

00:04:35.375-->00:04:39.579
pin just shuts off. Usually
power cycling will fix that. Now
I've had that happen a few

00:04:39.579-->00:04:45.385
times, um and and of course,
depending on what access you
have to the device, like if you

00:04:45.385-->00:04:50.824
have JTAG you don't need to do
this. Um, if you have other
means to get what you need, use

00:04:50.824-->00:04:55.862
the safer means certainly. It it
will you know prevent you from
breaking something. Uh if there

00:04:55.862-->00:04:59.733
are any time-travellers in the
audience, go back and listen to
Joe Graham's and Joe

00:04:59.733-->00:05:05.005
Fitzpatrick's umm "101 ways to
brick your hardware". I I think
I'm adding a new manner to do

00:05:05.005-->00:05:10.010
so. So let's get to the details
about how can you mount this
attack yourself. This is a

00:05:13.113-->00:05:19.386
general architecture diagram for
the kinds of devices that we
work with- CPU devices, 32 or 64

00:05:19.386-->00:05:24.524
bit uh processors, running
Linux, using typically a boot
loader like UBOOT, um and you're

00:05:24.524-->00:05:30.997
trying to interrupt the
communications between the
external flash device and that

00:05:30.997-->00:05:36.002
CPU. Uh, these can be either
serial or parallel flash
devices. The reason that this

00:05:38.205-->00:05:44.044
works is that systems boot in
stages and what's being shown
right here is the activity on

00:05:44.044-->00:05:48.548
the flash bus, you don't need
to, you know looking at trying
to decrypt the details of this

00:05:48.548-->00:05:52.953
by zooming in is not helpful for
this but you want to get an idea
of like the wall clock timing on

00:05:52.953-->00:05:57.657
this and figure out well when is
the boot loader being loaded,
like it is shown on the left,

00:05:57.657-->00:06:03.029
when is the kernel being loaded,
what's the duration where the
device is booting and then where

00:06:03.029-->00:06:08.568
does the user space and init
process kick off. And you can
actually attack in two different

00:06:08.568-->00:06:14.874
places, the most successful for
me has been interrupting um the
loading of the kernel, so that

00:06:14.874-->00:06:18.778
you fail to UBOOT prompt. But
I'll show you an example in a
second of something that was, uh

00:06:18.778-->00:06:24.684
was actually much more
surprising to me. So in this
example, this device had

00:06:24.684-->00:06:29.289
actually been pretty well
secured. Um, this this was a
device where where JTAG wasn't

00:06:29.289-->00:06:36.162
going to be an option, uh you
know, based on how they design
the hardware. Um but I/we found

00:06:36.162-->00:06:42.669
a point later during the knit
process where poking the serial
flash device caused the uh init

00:06:42.669-->00:06:47.907
process to fail and give me
exactly what I wanted out of
this – a root shell on the

00:06:47.907-->00:06:52.912
device. YEAH!!! [audience
applauding] So this kinda
misconfiguration is much more

00:07:00.287-->00:07:06.293
rare, the the, forgetting to
have a a useful failure mode for
load uh UBOOT, that's pretty

00:07:06.293-->00:07:12.165
common, I've seen that a lot.
Um, but this one was more rare
and the reason for it, is uh is

00:07:12.165-->00:07:16.836
uh this this embedded Linux
system had been really cleanly
set up, it was it was great

00:07:16.836-->00:07:21.941
work. Um and they had, but they
had left something in there to
help developers out, that when

00:07:21.941-->00:07:26.012
the primary application started
up, it would grab all the serial
ports, throw up an

00:07:26.012-->00:07:30.116
authentication prompt, it wasn't
getty, it was something that was
built into their application. Um

00:07:30.116-->00:07:35.121
and then um but then if their
application for whatever reason
failed, the next step in the

00:07:37.924-->00:07:42.896
init process would be just
*snaps fingers* but a shell. Now
another thing that happened, er

00:07:42.896-->00:07:48.001
another characteristic of the
system was important to this. Uh
this system was using BusyBox.

00:07:48.001-->00:07:52.372
BusyBox is like what's called a
multiple binary, it is like a
swiss army knife, that has all

00:07:52.372-->00:07:57.043
the things that a typical Linux
system needs but it does it with
just one binary. And so since

00:07:57.043-->00:08:02.082
that most of the pages for that
had already loaded before the
application we caused to fail,

00:08:02.082-->00:08:07.153
has started, when that
application failed, even though
I was screwing up flash for it,

00:08:07.153-->00:08:11.825
BusyBox was already resident
memory. I think it might have
been different if it had to go

00:08:11.825-->00:08:18.732
load other pages because I
wouldn't have been able to time
that attack very elegantly. This

00:08:18.732-->00:08:23.636
is also uh an example of a
serial flash device. These
typically have an eight pin...

00:08:23.636-->00:08:29.342
uh pin out varies standardise
much bigger device, you now so
uh, you can use a multimeter

00:08:29.342-->00:08:34.547
probed it, you know to poke at
this uh and I was uh poking
between the chips select which

00:08:34.547-->00:08:39.552
says "hey flash device, read me
out some of that data" uh and
the actual data output on that.

00:08:43.823-->00:08:50.096
So here's another example. This
was an LT router. Um on this
device, uh as you can see up in

00:08:50.096-->00:08:55.869
your top left uh there's a
little DIMM looking uh CPU
module. The flash device was on

00:08:55.869-->00:09:01.775
the underside of that DIMM
device so that's outlined in in
the red box below. So to get set

00:09:01.775-->00:09:06.379
up for this I had to pop that
device out, flip it over, put my
pin where I wanted, throw a

00:09:06.379-->00:09:11.284
little blue ta-uh tape on it and
then pull that pin out just a
fraction, you know, a couple of

00:09:11.284-->00:09:16.389
milimeters so that it would be
ready for the attack. I'm
working on the left side of the

00:09:16.389-->00:09:21.594
parallel flash device, another
TSOC 48 pin out. It's a
standardized pin out so you can

00:09:21.594-->00:09:27.400
find this pretty frequently and
I'm uh probing uh the chip
select, uh sorry the uh command

00:09:27.400-->00:09:32.539
latch and address latch lines.
Shorting those two out and what
those do is when the when the

00:09:32.539-->00:09:36.142
command latch line is toggled
[indistinct] it says "Here's the
command, make sure you read this

00:09:36.142-->00:09:40.447
in and do what I'm asking" and
then here comes an address. And
just by screwing with the logic

00:09:40.447-->00:09:47.053
on that you've got this flash
chip confused enough to do what
I wanted. So this is a quicker

00:09:47.053-->00:09:53.793
demo on this device, just a
little ‘boop' on the pin and uh
we're gonna end up back at our

00:09:53.793-->00:09:58.798
UBOOT prompt which was where we
want to get on this. So doing
your own pin to pone attacks are

00:10:00.867-->00:10:05.138
pretty straight forward. You
need to survey the hardware, pop
that thing open, you don't need

00:10:05.138-->00:10:09.509
to take anything apart at the
PCB level, but just the key
things you want to do is find

00:10:09.509-->00:10:14.180
all the flash storage devices,
figure out how you can access
them. You need some way to

00:10:14.180-->00:10:19.285
monitor the boot process, a
serial console's great because
that's probably the access that

00:10:19.285-->00:10:24.557
you're gonna get. The developers
will turn the serial port on
when something goes wrong um but

00:10:24.557-->00:10:28.795
you know, there's other ways you
can do that perhaps, you know
monitoring with a logic analyser

00:10:28.795-->00:10:33.233
and just getting a wall clock
time on it. Um you need some
data sheets to figure out on the

00:10:33.233-->00:10:38.304
flash storage devices that you
have, um where can you probe
successfully. There is some

00:10:38.304-->00:10:42.575
trial and error on this, not
everywhere I tried worked on
each device so you're gonna have

00:10:42.575-->00:10:47.747
to take a few cracks at it. And
you need a way to be able to
understand when the device

00:10:47.747-->00:10:52.886
fails, maybe it's LEDS, maybe
it's something on the serial
console; there'll be something

00:10:52.886-->00:10:57.156
because the last thing a
developer wants, is for
something to go wrong in the lab

00:10:57.156-->00:11:02.095
and for their device to be
bricked. They left a door, find
it. Um after you've selected

00:11:04.664-->00:11:09.502
your pins you're gonna you're
gonna start poking at that. This
took me a try, usually about 6

00:11:09.502-->00:11:14.173
or 8 attempts per device, just
to get the timing right on it.
Um, but you know it's something

00:11:14.173-->00:11:19.679
you can work through pretty
quick. Ah, make sure you power
off between each test. If some

00:11:19.679-->00:11:24.684
of that uh protection circuitry
in the IC gets engaged, power
cycling it's gonna help you out.

00:11:26.920-->00:11:32.926
And then monitor for a different
operational state, one that's
not the normal one. Uh getting

00:11:32.926-->00:11:38.064
UBOOT prompts uh pretty common
but some devices have different
failure modes like, uh, enabling

00:11:38.064-->00:11:43.369
networking ports or, uh, failing
to like a USB device firmware
upgrade. You'll need to find

00:11:43.369-->00:11:48.942
those. And you need a little bit
of luck on this too. So we
pulled out all the devices in

00:11:48.942-->00:11:53.146
our closet and we said "Hey
let's go for this, let's see
what other devices can do this."

00:11:53.146-->00:11:59.085
And you know about, a little bit
over 50% of the time we were
able to get some failure mode

00:11:59.085-->00:12:03.122
that was helpful to what we were
doing. You know, getting root of
one of these devices is really

00:12:03.122-->00:12:07.060
just the start, that's kinda
like day one of the project. It
just helps us do the rest of the

00:12:07.060-->00:12:12.298
rest of the work um but it's
really cool to be able to demo
to people and say "watch this!

00:12:12.298-->00:12:18.004
BOOP!" and you end up with what
you want, um it's it's a great
demo. I highly recommend it. But

00:12:18.004-->00:12:23.943
let's talk about some of the
places where I was unable to get
uh to get root. The devices that

00:12:23.943-->00:12:30.483
have thoughtful consideration of
how to fail, those are pretty
resistant. The best ones uh were

00:12:30.483-->00:12:35.588
the ones that would reset if
they if they were unable to read
flash on it. From a consumer

00:12:35.588-->00:12:40.460
perspective that device is
bricked, that's probably maybe a
bad thing for business um but

00:12:40.460-->00:12:45.465
from a security perspective
that's a proper reaction to it.
Ways that you can improve your

00:12:47.900-->00:12:52.572
design or uh you know give
recommendations to others, you
know, if you can't all turn on

00:12:52.572-->00:12:57.243
uh watch dog timers early in the
init process at the boot loader
level, and it starts servicing

00:12:57.243-->00:13:01.881
in the user space, so this
something interrupts in between,
you're gonna confound and tag to

00:13:01.881-->00:13:08.488
a degree. Um, you know, who is
trying to get at this device. Um
and just be very cautious, you

00:13:08.488-->00:13:14.761
know, about shipping failed
debug mode systems. Those are
exploitable, you just gotta find

00:13:14.761-->00:13:19.766
the way to get in. The other one
is really a design, uh decision.
Hydra pins, hydra traces, um BGA

00:13:24.671-->00:13:29.575
or ball grid array devices; they
have an IC with little balls
underneath that mounts directly

00:13:29.575-->00:13:33.846
under the PCB and the PCB
designer can actually take
signals and route them

00:13:33.846-->00:13:38.951
immediately through via to an
inner layer of the printed
circuit board. When you add an

00:13:38.951-->00:13:45.258
inner layer, it's it's hard, to
get at them. Um uh you know
traces that are on the uh outer

00:13:45.258-->00:13:50.730
layers of the PCB you can
actually scrape off this outta
mask and get at those, um but

00:13:50.730-->00:13:56.602
you're gonna have less options
and so it just makes it harder
for an attacker. So BGA devices

00:13:56.602-->00:14:01.541
uh some security conscious PCB
design, and PCB routing can make
a big difference. And and the

00:14:05.011-->00:14:08.981
last, which uh I think you know,
everybody should be doing when
they lay out these devices to

00:14:08.981-->00:14:13.920
help improve security is just be
very terse about what you're
doing. If you have a serial

00:14:13.920-->00:14:18.858
console certainly don't accept
input on it unless you really
want that for some reason, um

00:14:18.858-->00:14:23.730
and boot that thing fast! Uh you
can get like embedded Linux
systems to boot you know, well

00:14:23.730-->00:14:27.900
under a second and it just makes
it hard for a hand-timed attack
like this. You could do

00:14:27.900-->00:14:32.638
something more elegant and
crafty of course uh but you
know, that's that's that's more

00:14:32.638-->00:14:37.643
work in an attacker, they'll
move elsewhere. So that's the
last bit of this um Max is here.

00:14:41.080-->00:14:46.919
Somewhere. Max is right there,
we're gonna uh be outside to
take some questions and if you

00:14:46.919-->00:14:50.790
ask some good questions we're
gonna have something for you,
but I'm Brad Dixon, uh thanks so

00:14:50.790-->00:14:55.194
much for your attention.
Appreciate it. [audience
applauds]