Thank you guys for coming. My name is Joe Grand. I am a Portland based electrical engineer,
hardware hacker, product designer. I love electronics. My name is Joe Fitzpatrick. I am a
Portland based electrical engineer, hardware hacker. I love electronics. Yeah, so sometimes
it's hard to tell us apart. We ran into each other a little while back and realized that we
had a little bit in common. Yeah. Like we both have brown hair. That's right. He has a much
better looking beard though than I do. So we've been designing electronics for a long time and
we've run into all sorts of problems and failures with usually unintentionally but sometimes
intentionally of breaking electronics. So we thought it would be good as like a DC 101 talk
for people who want to get involved in electronics and get involved in hardware sort of to learn from
our mistakes and we can sort of share our pain with you guys hopefully so you don't have to
go through the same thing. So yeah, who has ever bricked something before? Yeah.
You're the right audience. So you guys can all come up here too. You should be teaching us.
We'll take turns. Who wants to go first? Share your stories. So yeah, we sort of broken this,
we broke this down into 101 different sections. 101. Yeah, oh sorry 101. 101 whatever. Binary.
Yeah. That would be a long talk. We have fooled you. There's only five or whatever. What is this?
Yeah. Five. Starting at one. So we're sort of cheating. We're not starting at zero. We cheated.
So yeah, so we have a few different, you know, kind of broke it down into different sections.
So first we got to define what a brick is. So the authoritative source of all this is what? Of
course Urban Dictionary. A pound or kilogram of any drug item requires clarification
from speaker as to the amount intended. Yeah, so that's what a brick is. So we're going to talk
about a hundred, no. Well we got to do the, I get my dope straight off a brick. Oh, there you
go. Yeah. We got to give the example of it. I got a lot of bricks at home to get dope off of.
So brick. To brick something. All right. So yeah, this is the action of rendering any small,
medium-sized electronic device useless. This can happen while changing firmware, soldering, or
other practices involving hardware software. This was actually in Urban Dictionary? Yeah.
Someone added that in. Yeah. It was Jules Verne. Awesome. So it's an actual real verb now.
So now it's like beyond DEF CON, it's a real thing. I bricked my mobile phone when I tried to
install Linux on it. That guy's a noob. Yeah. Okay, so we have three questions. One of
two different types of bricks. There's the soft brick. So the soft brick's kind of easy, you
know, like you did something, it doesn't work, it doesn't turn on. But it does have signs of life,
right? This is when you get like this, the little message on those, those, those Jesus phones
that like, oh, you got to plug it into iTunes, phone home. Or on an Android you get the, the
little Android with the, the little, little belly virus thing sticking out. Operate on me. So
yeah, but this, you know, soft brick, that's a software problem. Let's talk about hard bricks.
This is what's really hard. Who, who, who loves the hard bricking? Yeah. Oh, you do. Yeah.
Yeah. Yeah. Hard bricks are awesome. Yeah. So yeah, these are the things that actually
require some sort of hardware hacking, um, modification or fix or something usually if you
can get it unbricked again. So there is this sort of variation. We are focused pretty much
exclusively on hard bricks. Yeah, and the great thing about hard bricks is it's, it's a hard
brick. Like, you're not going to brick it harder, right? Right. Once you, yeah, once you're
done, you're done. Yeah. So. You can only undo it. So yeah, we'll go through a bunch of
different sections. Starting with probably the most common and the most obvious is, is
bricking through messing up firmware. Yeah. Um, so we have lots of, you know, crazy
examples. I should mention, so these are examples of our actual bricks. And we have, you know, a
lot, a lot of them here that we're sharing. Um, the same things could happen on your stuff,
right? So you, you might accidentally go through these same problems, just not with the same
products. Um, starting with the DEF CON 18 badge, how many of you guys have a DEF CON 18
badge? Like five people. Are you serious? All you guys, all you guys are newer since then?
They ran out. Oh. No, no, no, no. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh. Uh.
Uh. Apologies. Cool, that's awesome. Alright, so no one has it, which is better because
you probably didn't run into this issue. But welcome to DEF CON. Um. So the DEF CON 18 badge
was one that I designed. It was the last one I designed. And, um, this particular one had an
MC, uh, 56F8006. It was a free scale based digital signal controller, microcontroller, that
had lots of good hardware functionality for sort of DSP types of functions, but in a
microcontroller. And, um, I had a bootloader in there that you could load through USB, uh,
new code in to try to make it easier for people to hack on their badges without needing JTAG
hardware and debug hardware and all that stuff. So just through USB. But if you mess up during
the linking process, like with your compiler, and if you point your code in the wrong spot,
and if you don't include the bootloader back in, and you reprogram it, and you screw something up,
then the badge isn't going to work. So it's sort of a lesson in proper compiler configuration.
And the only failure, so at that point, it would not work at all. And the only way to fix it is to
use the JTAG interface, the standard development tools to reload everything. And JTAG being an
industry standard debug interface that is useful, but more of a pain in the ass because now you need
the tools to connect up to it. You've got to solder on a connector and do all of that. So that's
sort of a standard thing, is you mess up a bootloader, save it with JTAG. And that's something
we see a lot with people hacking routers and phones and things like that, is they can always
recover it, usually through JTAG, though not necessarily.
And like most devices start their life as a non-functional block of something that's programmed
for them.
So, you know, there's got to be a way to get something that doesn't have any code on it to get
code on it. And JTAG's usually that way.
And usually you would put JTAG to load the bootloader, and then the bootloader to load your
code. But if you brick the bootloader, then you've got to start all over again.
Or you just buy a new one.
Yeah. Or get someone else to buy a new one.
So, wiping critical sections. So this is a Chromebook firmware. Who's ever used a Chromebook?
They're kind of cool. So they do some fancy bio stuff. They're all, the ones that are based on
Linux platforms, they have what's called a manageability engine. And you see the difference
between the two?
Yeah.
Between these two histograms right here. So there's this big block that's up here. And, you
know, it's got a lot of stuff right there. So this is a histogram. It's a tool called binwalk.
It analyzes binary files, firmware images. And it's missing in this other one. So what happened
is, is if you go and you take your Chromebook, and you're like, oh, I'm going to tamper with
this thing. So let me get a backup first. You take a backup from software, you get this. You
get this big bunch of zeros right here. Right? If you go then and you get the, oh, I bricked it.
It doesn't work.
I need to open it up. I need to use my bus pirate or something else. Or I need to plug pins on
there and reflash the chip. I'll just flash this on there. You'll totally brick it. Because this
block of code is a block of code used for the manageability engine on the Intel PCs. If it's
not there, the thing doesn't boot. So it kind of sucks. So you get a software dump. It's
different from your hardware dump. And you flash it back via hardware. And you bricked it.
So the lesson being, if you can get hardware access to get code out, that's probably a safer
thing to do. Yeah.
If you happen to have a backup of it before you go. Yeah.
Before you mess up. So another one on the Chromebooks. I do a lot of poking at these
Chromebooks because I have a lot of them. So you can mount the read only file system as read
write. Okay. That makes sense. Then you can make changes and you can reboot. Okay. You do
that. That's cool. Now the kernel verifies the root of s before it mounts it. And it doesn't
match. Okay. So it checks the signature. It checks all this crypto stuff that the math people
can figure out. But all that matters is that mismatch causes the error. You don't want to do
that. You made a simple like change that shouldn't have bothered anything. But you, you
tampered with the whole chain of trust. And now you have a brick. Chrome OS is missing or
damaged. Please insert a recovery USB stick or SD card. Note, the blue USB port will not work
for recovery. So backup, backup, backup before you tinker. Yeah, yeah, yeah.
Hardware backups. It's the only way. So then. Oh, this is a good one. Oh, yeah.
Who's done this? Who's done this? Okay. So DD, like, copy blocks of stuff. And, you know, you
gotta copy to this USB flash drive. And you gotta copy a hundred of them. And you're like, okay,
sudo dd interface, install that ISO, OF equals dev SDA. So SDA is generally the first serial disk
in your system. So that's probably the drive you're booting off of. But you have to do it as
root because otherwise you can't access the, the low level block devices. So you just erased, uh,
everything on your system. Who's done that before? Back everything up. So many people are
admitting it, too. Back everything up. Yeah. Acceptance is the first step. Are we taking
pictures of all of them? So here's the other thing. Like, now I've got a laptop and it's got
an X, uh, uh, EM, uh, what's it called? NVMe. Uh, non-viable memory express. So it, uh,
storage drops in and connects to a piece of express. So in Linux it's dev NVMe one, uh, NVMe.
Uh, and that's great, but except I plug a USB drive in and now I need to put something on it,
it shows up as dev SDA. So I do this all the time now, and if I ever get a new laptop, I'm gonna
wipe it on a daily basis. That's, that's just evil. So make backups. So, yeah, so unbricking
these types of firmware issues, um, if you have a backup that's good, you know, if you're gonna
hack on something, try to get a good known image before you start messing with stuff. Um, yeah,
what, who? I don't know, directly rewrite the storage media? He wrote that one. Uh, I did.
Yeah. Oh, yeah. If you, uh, if you really want a backup, don't trust your operating system. Don't
trust your CPU. Just go straight to the device and read it, right? If you have a chip of some
sort, read it with a programmer. Don't read it with, uh, software. Yep. And other hardware
things, too. It's funny, because you can unbrick your, you know, firmware using hardware. Um,
swap out the flash device, the memory device, whatever you've bricked. If you have another
backup, or maybe you take one from a product that is good, that has the same content, take one off
one board, put it on another, or copy one, the raw, dump, put it in another one, and then use
debug interface. If it exists, it usually does. Either it's JTAG or a vendor specific interface of
some sort that will let you reload new code back in, assuming you took a good backup in the first
place. So, yeah, if you've got those backup, it's great, you know. Um, if you don't, you might
have to buy a new device, and that's sometimes expensive. So swapping out the physical flash
device, whatever the device is that you actually broke, sometimes is a lot cheaper than
replacing the whole system, right? So if you, if you borked a flash chip,
you've worked something else like that, just replace that chip, you're good. And then just
return the, the one you, uh, just bought. Yeah. I don't condone that. People are
shaking their heads, no. Cash only. Yeah, just stop working. That reminds, actually,
I'll, we'll tell a story about that with this, uh, I've never, I've never done that. I've never
done that. No, me neither. Um, alright, so the, the next, actually, we gotta, uh, can anybody in
the audience identify what's wrong with this PCB? Oh, yeah. It's a little quiz. It's a tough
one. If you're new to hardware, it might be difficult.
Yeah. Okay, so, now we're getting into some actual physical destruction. Um,
bricking PCBs. A main concern when people get into hardware hacking is, am I gonna damage the
board? Am I gonna damage the chip? I don't know how to solder. Um, normally, it takes a lot.
Like, circuit boards are pretty robust to heat. Chips are designed to withstand a decent amount
of heat when they go through a reflow oven when they're being soldered. And typically, the
failure modes are thermal cycling if you're removing a part, putting it back on the board,
removing it, putting it back on the board.
Um, but sometimes, you get a little overzealous, and other problems happen, like we'll talk
about here. So, yeah, um, who's ever, who's ever tinkered with, like, a wireless router or
something like that? Okay, they're fun. They're fun to program. And you open them up, and you can
find all sorts of neat headers on them. So, if you're poking around, can I get over there? I
can't get over there. You have to describe it with words. I have to describe it with, I don't
know words. Um, so, what we've got is a bunch of, uh, pins on here. And this is just a, a bare
header that's sitting there on the motherboard. And we need to get that solder out of the board.
It's all there to get a header in there, so we can use a JTAG adapter. So, it's unpopulated.
And, you know, sometimes, you just want to get something done, and you're just sitting there,
and it's like, okay, you know, crank the iron all the way up. Come on, come on, melt solder,
melt solder. You really don't have patience, and, uh, you need patience, because too much heat,
sloppy work. Um, I basically completely peeled out the through-hole lining of each of these. Um,
I usually do a better job soldering, but that's kind of like, that happens sometimes.
This is a great job soldering, what are you talking about?
So, yeah.
Actually, I've seen, I've seen, I've seen worse jobs.
I actually have, too.
I've done worse jobs. This week.
And the way, the reason this happened is the way that circuit boards are manufactured is you have
a conductive layer, and then you, it, it's basically glued down to a non, non-conductive
layer. And that glue will get softened with a lot of heat, and get pulled right off the board.
Yeah, so you can, you can pull the copper off of the board, you can pull the layers of the
board apart, and, yeah, it gets really messy.
So, yeah, patience really is the answer. Um, and also, you know, this is a really common
problem, because most devices that have through-hole parts in them are assembled with
what's called a wave soldering station, or a wave soldering machine, where, like, a big wave of
solder, basically, solders all of the through-hole parts that are underneath, you know, coming
through the board. And that's why, when you get products, consumer products, all of the holes
are already filled with solder, because of that wave flow, the, the reflow, and the wave
soldering, and the surface tension pulls up the solder into the holes. So, the first thing you
usually have to do when you're hacking stuff is, like, suck that stuff out.
So, another one, um, I play with this, these things a lot. Pogo plug is this, like, ten
dollar network, network atta, network attached storage device. Um, and, and, and, and, and, and, and,
um, I think I dropped, uh, pictures of it, uh, yesterday, uh, where I plugged PCI cards into
it, but that's beside the point. I was really trying to figure out where the JTAG pins were on
this guy. I knew the CPU, I knew where they were, I knew where the, the pins were on the
CPU, but I, I thought there had to be test points. And so I just decided, okay, you know
what, I'm just gonna, I'm just gonna take the chip off, I'll figure it out, I'll look at the
traces underneath. In the process of doing that, like, you can see the exploded view, in the
bottom right, uh, it's kinda tiny, but there's a couple traces that, in the process of taking the
chip off, you know, you, you sit there with a hot, uh, hot iron, uh, hot iron, uh, hot air, and
you blow it on the chip, and the chip gets warmer and warmer and warmer, very patiently. And
what's really annoying is the last thing to melt is the solder, right? Because the solder conducts
the heat away, so you just sit there, you gotta be patient, gotta be patient, and when you
wanna get something to work, you are not patient. So then I'm like, okay, well, it's almost off, so
I stick something in there, I try to lever it up, and in the process of doing that, the thing I
stuck under there, I scratched a whole bunch of traces off, so they were disconnected. And, you
know, pull it off, uh, and after all that effort, I find out that there was, uh, there was a
bunch of, there wasn't even test points for JTAG anywhere. I thought they were routed
underneath the chip, they weren't. Um, so, you know, in this case, though, I actually kind of
said, screw it, I don't care if I break this one, I mean, it's 10 bucks, and I was fine with
losing the 10 bucks, but I learned something from it, I guess. Well, and this is a good
example, too, of if you are hacking on stuff, if you can get multiple units to have a
sacrificial lamb to do something like that, if you do need to look what's underneath the part,
it's like, alright, if I break this one, I don't care, now I'll get the information I need to do
an attack on another one, like, that's okay. But if you break this one, I don't care, now I'll get
it. If you only have one, then you're screwed. Yeah, and then you return it, and you're like,
hey, it's bricked, I don't know what happened. I just opened it up. Um, alright, so,
shorting traces. This is, this is totally, uh, you know, something that happens a lot. Um, and
this is, this is a Hurst scramble pad, so this is a, uh, um, an access control device that is
used at, like, the White House and other federal buildings and airports and stuff, and it's, it
designed in the 80s, I think they've updated it since then, but you've probably seen them before,
you push a button on the pad, and the key, the, the number ordering changes every time, it goes,
like, do-do-do, and, like, changes every time. So, someone can't, you know, look at the wear
marks or your fingerprints on the thing and try to narrow down the key space. It also has
really narrow viewing angles. Um, so I put one of these in my office, which is funny, because
you could just kick the door down if you wanted to. Don't get any ideas, by the way. Um, so, I
got one of these things on eBay, I was messing around with it, uh, I had some batteries, I was
kind of testing out the system, taking some measurements on the, uh, the, uh, the, uh, the, uh,
the linear regulator, just a standard, you know, run-of-the-mill LM7805, a 5-volt linear
regulator that was taking in, I can't remember what it was, 12, 12 volts in, um, so I wanted to
measure the input, make sure I wasn't going to fry the rest of the circuitry, because this
particular board was from 1992, and I didn't have a backup of the, uh, of the code on the
microcontroller in case I broke something. Um, I don't know what happened, but one of my probes
slipped and shorted the input, which are these very high-capacity batteries with very high
current output, um, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and, and,
uh, directly to ground, which causes a short circuit and a spark and damage to the, uh, to the
board. You can sort of see in the exploded view, like, all of the solder mask and part of the
board is actually missing, um, and I was really scared that I'd just completely ruin this device.
Luckily, the, uh, the regulator is pretty robust. I didn't do any damage, still had a stable
5-volt output, um, but I sort of, uh, sat in the corner and whimpered for a while. I had a bruised
ego, uh, but I immediately sent a picture to Joe, and I was like, we can use this in our
presentation. Yeah it was great. I mentioned this idea to Joe and he's like oh I can brick
this, I can brick that, I can brick that and he went and broke everything he had. Which is
great because that's what I've been doing for the past two weeks before that. So here's
another one. Burning traces. This was a fun example. I was reverse engineering a vacuum
sealing food, you know like a food thing to vacuum seal food. I was working on a project
designing basically something very similar to that. So I was reverse engineering this board
to figure out how it was designed and made a really beginner error. Using my oscilloscope I
wanted to visualize some of the traces on the board but what I didn't realize is that I was
creating a ground loop and I was accidentally measuring a signal, an AC signal that I
shouldn't have been measuring the way that I had things set up. Probably should have maybe
been using a multimeter that was isolated. And there was one trace on the board that is
designed to be a fuse.
So that circle there you can sort of see the square and then it kind of goes out in a right
angle of like a really thin trace before it gets to the rest. That was a fuse designed into the
circuit board which saved not only me but it saved my oscilloscope from actually you know
getting destroyed. And normally you know this is kind of what happened.
And what happened? Bang!
Yeah so that was Dave Jones. If you haven't seen his EEV blog videos you should check him
out. He's a very interesting engineer with
lots of good technical detail and lots of opinions.
Actually you want to go back to the picture of that PCB? There's a moral to be gained from
this right? This is a food saver V850 okay? Joe's hacking on a food saver V850. This is not a
smart food saver. This is not an internet enabled food saver. This is like the vacuum thing
that you put like steaks in so you can freeze them.
Or vegetables.
If you're bored and you need something to hack don't just look at computer stuff.
Everything is hackable.
That's right.
Open your mind.
That's right yeah. Good point. This was all digital logic, no microcontroller or anything. So it was a good experience
actually in learning how to reverse engineer analog electronics. But then I eventually just gave up and
designed a digital system to do the same thing.
But you know what's great is there's no firmware to Brick right?
That's right no firmware. And so the key thing here is learn how to use your oscilloscope properly
which after this I went and studied up on ground loops and hooking up you know AC things to oscilloscopes and needing
an isolation transformer.
Bang.
Yeah. But if I had broken my scope that would have been really bad.
You just return it right?
Yeah. I don't know if that one would work. So ways to fix unbrick your PC boards. Be patient in the first place.
And don't just go straight at it with a you know don't turn the heat up to 11 on your soldering iron.
Blue wires will actually work. Like you know the little wire wrap wires that you see on boards sometimes to fix
prototypes if you get some 30 gauge wire wrap wire or some
magnet wire or some angel wire I think they call it to fix broken
traces and to fix you know things on the circuit board that
you're not going to be able to fix a blown area but you can
just patch it with wire is a good way. Oh yeah and PCBs are
actually kind of really resilient I mean they're just
like fiberglass and metal and they work right so if you even
if you have that board from the front that's like torn you know
you line those things up you put some glue down there and you
like solder it up well enough that board would probably still
work. Assuming it's not a multi-layer board with stuff
inside. Well yeah you know. But for the ground planes you'd be
fine. Level of detail. Hey Scott. Yeah. So the question is
do we ever use ChipQuik? ChipQuik is a special alloy used
to help you remove surface mount parts from boards and it
basically reduces the overall melting point of the solder so
if you have multiple pins coming off the part you use
ChipQuik and it melts everything at once and you can
slide it off the board. So the answer is yes.
And the good advantage with ChipQuik is that it doesn't heat
your part too much. The disadvantage is that the stuff
stays molten for so long that it will dribble and get stuck on
other parts and if that happens you're gonna have solder this
solder alloy everywhere. I'm completely capable of bricking
hardware without ChipQuik. Yeah so you have to be really
careful to use it but yeah so sometimes you use that or you
just use hot air rework but it sort of depends. All right
anybody recognize this beast up here? Does anybody remember
why it's blinking red? No? Bad connection. Yeah you don't
remember blowing in your in your cartridge and trying to get
better connections? So yeah bricking connectors this is you
know messing up more mechanical physical things of systems. So
I mentioned before something about the Chromebooks the CF720s
I kind of like them because I got a lot of them and the reason
I got a lot of them is you know because I can get them cheap
right you go look around you can get it for like a hundred bucks
each but I'm really cheap so like I'm always looking for a
little bit cheaper.
So I got a lot of ten broken ones on eBay and I'm like hey
what's the worst that can happen right forty bucks each that
sounds like a good deal. So I open them all up I I I got ten
of them and of the ten one had a cracked screen so not much I
could do with that. But then I went through the other nine and
actually the the tenth one as well they all had broke like
loose cables in the display so if you look in the back panel of
a display it has a sorry the motherboard has a cable it goes
up through the hinge up to the back panel of the display and it
has a panel display on this one this one model just has a thing
you keep opening and closing and opening and closing it it just
got tugs a little bit and so the little edge of that connector
slips out the tiniest bit a fraction of a millimeter and
that's enough for the display to not work. So all I did is kind
of pop open the displays tighten these connectors and I
suddenly went from four hundred bucks worth of Chromebooks to
nine hundred bucks worth of Chromebooks. So that was kind of
fun and there was something else I was going to say about this.
This was happening with normal use too right?
Yeah so it was normal use you keep open and close and open
and close and you eventually kind of wore it out a little bit
too much. Sort of bad design.
Yeah. Oh yeah this one. Yeah so this is a little mini PC that I
was using and I actually ended up using this to build an AR
sandbox. You ever seen those? It'll be a picture later. And
the problem is it was very poorly designed it was a little
micro USB connector that was used for power input and it
wasn't just a regular like USB cable it's this is like Intel
x86 bay trail four core thing you do.
And what would happen is uh it had a three amp power supply five
volts three amps it's quite a bit but it used to use tiny
traces inside and it kept like burning out those traces and uh
heating up and melting this little connector so um the
traces really weren't well sized for the amount of current
regulate thermal recognition wasn't well controlled um if you
start using the CPU too much like it wouldn't be able to
supply enough power over the connect connector and it would
just disconnect it would just shut down. So um I kind of got
sick and tired of that I tried replacing the cable I thought
like maybe bad micro-users were going to be able to do that. So
I just started using the USB cables because that happens a lot
to me probably because I use cheap cables. But uh I just
basically said screw that I opened up the case I soldered the
power lines directly to a ground point and a power point very
messily too. Not as bad as that other one that through hole one
but I soldered it up it worked and uh yeah it still works.
Sometimes it doesn't have to be beautiful to work right?
Yeah.
That's what my wife tells me all the time.
Take that as you want.
I didn't mean it that way.
You guys are disgusting.
You should hear what he says to her. Um so uh another one you
know again I play with all these systems I've got these tablets
these are the cheapest tablets you can possibly buy that run
Windows and actually you can buy them even cheaper because
everybody goes to the store and they buy them because they're
cheap and they can take them home and they can't run anything
on them so they return them and spend more money. So I go in and
buy all the open box ones because they're even cheaper.
Anyway uh this TW 700 tablet it's a little 7 inch Windows
tablet and it's got this like micro-users that you can just
plug in and it's got a micro USB connector um for power
charging input and you know I use these a lot I charge them all
and I just charge them all and charge them all um but every
time you plug in that USB cable it wobbles this connector a
little bit. Um the housing the case of the cable uh connector
the case of the tablet does not have uh a flush case around this
connector there's a little bit of wiggle room and that wiggle
room keeps wiggling every time you plug it in and take it out.
Solder is never ever ever designed to hold any load or any
shock. It's a little bit of a pain in the neck. It's a little bit of a
strain or any physical strain. Right it's solely designed as an
electrical connectivity. So what happened is after a while those
four little tiny or five sorry five little tiny connectors on the
bottom of that USB connector all got broken. Um so yeah it was a
pain in the butt I had to replace a bunch of them but hey it works
now. And this is a good example of if you're designing electronics
try to use a connector that has through hole uh uh ports on the
side or whatever they are for mechanical stability and that's
something that companies don't like to do because it's an
additional step to solder them in but it's gonna prevent that from
happening. So here's another uh another thing that I've done many
many times and this is just the most recent example. Um this is a
low cost consumer device. This is another cheap Chromebook cause
you know I got a thing for Chromebooks I guess. Um it has
USB audio running over a flexible printed circuit to the
side the other side of the laptop. So they make the
motherboard small it goes on one side they put all one connectors
on one side and they have a little ribbon cable that floats
through the case to their side. If you open the case without
that that cable is there um you're very likely to like tear it
or pull it. If you're lucky then it disconnects it just pulls it
out of that black socket. If you're not lucky then it pulls at
an angle and it tears the a bunch of the traces. And you
know it's just a a piece of plastic and metal but the thing
is that for some reason these are really expensive to buy as
replacements at one off. Uh. Relatively expensive. Relatively
oh yeah. Cause that's a low cost. Well we're talking about
Joe Fitz expensive which means it might cost like 10 bucks for
this cable. But sometimes it's a little bit more expensive than
what you're looking for. But sometimes someone else plays with
much much much much more expensive toys. So notice how
there's no detail on this slide. Let's just say that it's a very
expensive uh consumer device. And this mistake was very costly.
And if you look on that circle this is a flat flex cable so it's
a flexible circuit board where normally you might have one or
two layers uh on a flexible board. This is a multi-layer
flex board connecting very expensive pieces of equipment
together that I accidentally tore. So it's a very expensive
product. And um tried to fix it. I was like oh some wires right
like if you can solder the top and bottom. But they were multi
layer and it was horrendously embarrassing and never to be
spoken of again. Right. Now that it's on film. You just
returned it right? Actually we just returned it. No lie. Uh I
think that the moral of the story I think behind that is uh
do not hack on what you cannot afford to lose. Yes. Yes. Or
return. Or that you don't have enough credit for. Or that you
can't return. Save your receipts. Save your receipts. Um
okay so solutions to unbreak your connectors. Mechanical
reinforcement is actually a really common one. Like just use
some tape. Use some epoxy. Yeah on those Chromebooks that had
that wiggly port. If I had just gone in and run a drop of epoxy
on each one when I got them new. Which is what I do now. Um they
would have all been fine. They would have never broken in the
first place. And vendors are just too cheap to do that. Yeah
too too cheap. Epoxy's expensive. It costs cents. Yeah. Uh
fractions of cents. Uh so electrical reinforcement like
Joe did patching over weak connectors and putting in
better better connections there. Um learning how to you
know locate replacements if you do mess something up. See if you
can source a part. You know looking at common distributors
for various places. Um reading mechanical drawings so you
know which part to use. And DigiKey's your friend. You
know you can get parts shipped same day. Delivered next day if
you need to to you know continue on with your project. Yeah if
it takes a while to get the skill to like actually find
anything on DigiKey. But usually you just you know keep
searching. You find something close. Find stuff in that
category. And then they ship it really quickly. Which is what I
like. You do it. You do it. You do it. You do it. You do it.
Okay. So you do the parametric search and then you just
narrowed down until there's like a few items on one page you
just choose one of those. Or you buy all of them. And return
the return the rest. Yeah. Um okay so now we're getting into
breaking chips. So actually integrated circuits on the
physical circuit boards themselves. Um. You're talking
about absolute maximums. Yeah sure okay. So absolute
maximums I think we might actually have an example of
this. Oh we do. But. Okay. Integrated circuits are are
sensitive to their voltage levels. Um whether they're on
on signal pins or power pins and data sheets of these parts will usually tell you the maximum
allowable values and things like that and usually if you go above them the manufacturer
is not going to let you return it and then you're sort of let out the magic smoke and
you're done. And it's kind of an RTFM case because if you look over here it says pretty
clearly operating range use this voltage range. If you're not going to read the data sheet
then like oh well whatever. Who reads data sheets anyway? Until you brick something.
Yeah. You go that's why I should have read that thing. Yolo wiring. So speaking of
Yolo wiring I found another tablet because I tend to acquire a lot of these cheap tablets.
This is a cheap Chinese tablet and it's got a 1.8 volt spy flash chip and this is like I
think one of the first ones that I poked at that was actually 1.8 volts so I didn't really
expect to worry about it. So I just you know whatever opened it up popped it open grabbed
whatever tool I was using probably an FTDI chip wired it up and tried to dump the spy flash
contents and then the system didn't boot. I also didn't get any spy flash content
so I was trying to figure all that out and it turns out it's actually I needed to level shift
right. All these tools we've got um there tend to be 5 volt and 3.3 volt tools some of them are
5 volt tolerant some of them can work at lower levels but if you do something at a higher
voltage than the device is made to uh withstand you're going to do something bad. You might not
totally brick it you know in this case what happened is I actually just erased the flash
contents so the flash chip still worked the CPU still worked but uh the process of trying to read
it at 5 uh 3.3 volts made that 1.8 volt flash chip die. And a lot of chips do have internal
protection diodes on IO pins um so to protect you from accidentally doing that but you're not
supposed to rely on those like those are almost like it's like getting catastrophic health
insurance or something right like you don't want to rely on that um in case there's an accident.
So yeah uh another thing I was playing with oh is that good yeah there we go onward. So pulling
up too much current. So this is uh what someone might call an FTDI chip but it's actually a
FTDI cable it's not an FTDI cable it's just a USB to serial cable it's got a chip on it that
says that it's a prolific PL2303 you plug it in on one end to a USB port it's got TXRX power and
ground on the other end. Um I bought a bag of like a hundred of them and cause they're cheap
that way they're like a dollar each. And this one I was using I forget what I was doing with it
but it kept like stopped working and I would go and what would happen is I would look in the
dmessage log of the system I was the USB side system and it kept saying like oh this device is
connected device disconnected.
So I'd have to go and unplug it and plug it in and I left it walked away I come back a little
while later to do to work on it again and it's not working so I go to pull it out and um my finger
sticks into the plastic and it mushes it around and I'm like huh I don't think that's how it's
supposed to work. So I pulled it out and uh ran water over my finger and uh loped it up and the
board is definitely a little bit singed right there. Um I don't know whether this was just shoddy
manufacturing and there was like a ball there or whether I was hooking it up wrong and I was
actually drawing too much current. But it got really hot and hot enough to melt the plastic and
blacken the board. So did we did we wipe your fingerprint from that image before we submitted
these to DEF CON? I don't know. It doesn't it doesn't actually look like my fingerprint.
Joe's thumb is there. Uh oh. Use biometrics. Yeah good thing the hotel doesn't use biometrics
right? Um okay so another example of pulling too much current. This is at an actual tiny little chip
level. Um this was for a product I was working on uh for a consumer device. So I had some
prototypes. Sent those to a manufacturer to start getting ready to ramp up for full
production. And they had made some changes to some parts which is not uncommon. They might say
oh we have a we have a supplier that can provide a similar second source part. We're gonna put
that in place of this chip that that you the engineer slaved over to specify. So they sort
sometimes just put in what they think is the right replacement and not tell you or tell you
later. Um so these came back and um we noticed that once in a while we would have failures of
this particular linear low dropout linear regulator. So taking power in bringing it down to a lower
voltage. And um we just couldn't figure out what it was. And this was a company I wasn't very
familiar with. I think they were Chinese based. I'm not sure. And um the only thing we could think
of is let's decap the chip. Let's take the plastic covering off of the chip itself and look at
the die. Look at the actual integrated circuit to see if we could uh locate any failure. So we
sent a bunch of chips to Chris Tarnofsky who is I would say the best chip hacker in the world. Who's given talks at DEFCON and Black Hat.
And all sorts of crazy satellite TV hacking and smart card hacking. Um and had him decap the chips and
look. And um he went and and very quickly realized that there was damage on the physical die. So like
that you know the previous one who just showed had damage on the board. An integrated circuit is
really like a like a circuit board at a microscopic level. So there was damage on that die. Because
the the way this system was designed is um basically there was current flow to that giant tab.
But that tab wasn't actually designed to handle current flow. It was just mostly
designed for thermal heat dissipation. Um so maybe that was a designer error or they swapped
that in or it was just a mis-misdesigned sort of under design of the part. Uh relying on the
engineer to read the data sheet before they actually designed the board. Yeah but that sounds
like work. A lot of work yeah. YOLO silicon. Uh so yeah we gotta we gotta figure out ways to
unbrick these systems. And it says unbricking your ICs but really we gotta talk about unbricking
the whole system. Cause most of the time if you've done like electrical damage to the chip you're
just gonna have to replace that chip. So replace it. But.
Figure out the problem first cause if you go and you fix your connection issues your board issues or
you fix your you know uh you don't fix all those other issues first and you replace the chip you're gonna
end up with two dead chips. And then you're like wait wait what happened and you replace it and you
get three dead chips. It's like a chip killer. Yeah. Right. You fix the what do they say you
treat the treat the cause not the symptom. Yeah yeah. So again DigiKey is your friend. Yeah. We're
not sponsored by them we just like them. Yeah. There's other distributors too if you like
Mouser you could Mouser is still your friend. So personally I like DigiKey
cause they have USPS shipping that's generally very quick and very inexpensive. Yeah. So shipping
is not as obscene compared to your ten cents worth of parts. So like two dollars of shipping for a
dollar worth of resistors instead of like eighteen dollar shipping for a dollar worth of resistors.
It makes a big difference. Um alright so here we are at our one hundred and first uh section um
one oh one. One oh one. First. First section. One oh onest. Bricking scenarios that we couldn't
think that fit anywhere else but sort of like WTF what is going on.
Um we have uh. What? Yeah. Yeah. So uh. Anti tamper. So anti tamper mechanism. So this is a photo
from inside of an AT&T micro cell. There were a couple of people who talked about these a few years
ago um and uh the what happens is you open this guy up and inside there's this little gray thing
that holds a bunch of jumpers right and those jumpers may either connect or not connect um the
thing to the case. When you pull the case open the case is designed so that it pulls those jumpers
out and you don't know what arrangement they go when you put them back in. Right? As if you do
this and you don't notice what you did and you power it up it sets a tamper flag right and it
phones home and it tells AT&T that you've been doing bad things. And that you probably shouldn't
expect a return from this customer. Yeah. Well there's actually so so some discussion about this
if you search online. People have you know tried to open these up and um you know pull out the
jumpers and try to put it all back together. They're like oh no I saw something fly across the
room. And then they call up AT&T and they're like oh my micro cell is not working. They're
like well it looks like it's been tampered with. And they're like oh it must have fallen on the
floor or something. And they're like okay. Yeah sure. Um yeah so anti-tamper mechanisms are you
know things that are physical security to protect you from tampering with an electronic device.
Here's another one. This is from a Verifone um pin pad 1000SE. This is a uh you know point of
sale terminal thing you'd enter in your pin. Uh there they have a lot of mechanisms on this
particular device. And I had I purchased a whole bunch of different pin pads at a surplus store
and uh this just happened to be one of them. Um that had multiple multiple mechanisms. So you
open up the device. There's a there's a button that gets depressed. Um but the coolest thing
about this one is that there's a active um circuit board there. It's like a multi-layer
circuit board. Four layers. The top and the bottom layers are copper planes. And then the
inner layers are like a mesh of wire like a maze. So if this thing is powered on and you try to
like drill through it or remove the cover. The system's going to know and give you some sort of
you know tamper.
Tamper detected and not working. You'd have to rekey it and everything. So this was just sort of a
fun one and another great reason to have sacrificial lambs um if you can. Because if you tamper
something like this on your first one and you only have one you're going to be in a lot of
trouble.
So another comment on the whole like having sacrificial lambs. If you're ever like just
looking for devices to hack on and like you decide to go like AliExpress which is a place to
get like really cheap junk from China. Um buy more than one because you buy one and you go and
you take it apart. You're like hey I hacked this thing.
It's really cool. Let me go buy ten more. You buy ten more. They're going to be different.
Right? They're going to have the same color on the outside. It's going to be the same picture with
like eighteen logos that have been you know photoshopped out and written over watermarks. But
you're going to get a different device inside. So buy them all at once. Buy a bunch at the
begin and then just hack them. Yeah they just grab it from a different factory or just like
iterative their design process is just too crazy. Um so some weird environmental conditions
which are the worst things to debug. If any of you guys have worked with RF systems before um you
know the sort of black magic around RF design and it's really really sort of a nightmare. So when
the environment conspires against you to mess up your circuitry like that's really it's a hard
thing to deal with. Um this particular design is a uh an RFID read write module. Um I've
designed a series of RFID readers and writers uh for Parallax which is like a hobbyist
electronics company. This was the fourth in a series that I thought would be a really easy fix
um to add on USB functionality to our standard read write serial version that would take a
TTL level serial. So this one um with USB I was like well I could just add a a uh serial USB
device, take the TTL level serial, add a USB port on there and we'd be good. Not so true. Um
about three years later after debugging this on and off for three years just I just realized this
and we were able to put it in the slides. Is that I I was receiving all this noise on the RFID
receive line like I was seeing demodulated data when there was no tag over the reader. So it was
like a mystery of what was what was being used. Um and then I was like oh my god this is a
device that's being demodulated. And it turns out that my reader was demodulating noise from
the environment because the antenna was too sensitive. And I didn't know that. It was one
change in capacitor value to decrease the sensitivity and it really reared its head when
you're powering directly from a uh a USB device. And I was sort of treating the USB you know
power five volt power line as being clean but that's not necessarily true. So it was just
generating lots of noise and I was having all these problems and thought I was a
horrible engineer. Um and then once I realized I could change one capacitor now I thought I was an
awesome engineer. But I'm never touching RF stuff again.
Uh so uh this is another one and the picture doesn't do justice to what what happened. This is
a uh an AR sandbox which is where you have a sandbox that's full of sand and a connect that
looks down at it and uh and looks at what you've got and uh when you when you move the sand away
and make a pile it turns it into a mountain and colors it green. When you dig a hole it makes it
it turns blue. Um so I had this set up I brought it to TorCamp which is a great awesome like
outdoor hacking camp event. Um but it is in the Pacific Northwest so it is a bit moist and the
next the second morning I turned on my connect and my thing and it just was not working
properly. And this is a this is a partial failure. What actually I had at TorCamp is I had the
very corners of the frame would show up as mountains they'd be red and the middle was all
black. Like what the hell is going on? And this is an environment conspiring to work against you. I
had this thing out in the middle of the room and I was like what the hell is going on. And this is an environment conspiring to work against you. I had this thing out
side I had it underneath a black sheet which had worked through all my testing because the connect
uses infrared to find out what's going on. I thought I had like left it out and the moisture
had like uh you know melted or you know fried something inside and it was no longer working. I
could replace the connect. Turns out the sheet that I used had a mesh you know pattern on it and
the infrared light from the sun would cast down and make a uh an infrared pattern that the connect
was recognizing and not throwing any color on properly. So this is like okay I thought I had bricked
hardware. I thought everything was broken. Turns out it was just a sunny day.
Yeah. Darn. Damn those sunny days. They're the worst. Yeah so you know test your systems in you
know the right environments. Um so optical glitching. This is something that we just sort of
threw in here because when uh when Joe was over we were working on the slides one day and I
needed to take a picture of the scramble pad to show in the earlier slides. I had my camera with a
big flash on it and when I took a picture of the scramble pad I went locked the door. We went
back in for something later and and my access code wasn't recognized. I'm like huh that's weird.
And then I remembered about optical glitching that you could actually cause failures inside of
chips due to photons hitting things the wrong way. And since my that scramble pad had a uh e
prom to store the program code which is UV erasable I actually caused the system to fail and
erase all of the access control pin pin numbers in there. So I I sort of bricked it. I was afraid
that I actually like changed some of the code.
Uh but really like you know chips do not like light. Uh but you can sometimes bend that to your
will if you're intentionally trying to use optical glitching to like you know skip over something
and do some glitching on the die itself. That's sort of a crazy advanced attack but it sort of
surprised me and it was like wow light can damage things. Even making a slide deck about
bricking can brick things. Yeah that's right. Subtext.
Yes. Yeah so the the Bosch BMP 012345 0805 pressure sensor is also sensitive to that. Um there's
also the example of the raspberry pi zero which maybe that was. Or two. Or raspberry pi two that
uh oh that was the power regulator. So yeah there's lots of things you assume it has a package
over it right a a a plastic package but light and photons can still get through and mess with you
and that's like it's pretty interesting. So these these WTF scenarios like it's kind of tough. Like
what the heck did you do? What did you break? You didn't change anything. It worked yesterday.
It worked today. And it just stops working. Um so what can you do? You can get another piece of
hardware. Be more careful. Um you get another piece of hardware and do like a a manual diff.
Compare every single component. Test step by step. Swap them out one by one. Or uh the best one I
like. Uh you just grab a bite to eat. Take a nap. Maybe it'll work tomorrow. Maybe someone else
will fix it. Maybe it'll work tonight. Yeah. Yeah. These types of things like are are actually
the worst that make you hate engineering. Um but then you solve them and it you know everything's
okay again. So as a little recap. Like we got the best ways to brick and the best ways to to
avoid it. So bricking your firmware. Right? Just wipe your flash. It's wiped. Um yeah. Cut
traces. You know. Yup. Smash connectors. Smash connectors. Uh applying the wrong voltage.
Work on anything the last minute and that's when these WTF scenarios uh really win. So avoiding
it. Back up your firmware. Um yeah. So you know have a good work space. Don't rush things.
Take your time. Have protective measures so you don't damage components. The P1. The P2. The P3. The
word. Patience. Yeah. Uh double check your pinouts and voltages. Read the read the manual. Read
the data sheet. Um have a critical set up. Yup. So. And and uh unbrick. Who who cares about that?
That's no fun. Um restore your backup cause you've got one right? Yup. Yup. Um enhance your
soldering skills so you know you don't make mistakes with disgusting soldering. Right. DigiKey
is your friend. Order parts and DigiKey is your stower friend no matter what it is you're
bricking. And like you said don't hack what you can't afford to lose. I never listened to that
one. Yeah whatever. Uh huh. Um so yeah. So best way to do this is to go to the internet. Uh
maybe you brick one but then you learn from it anyway because hacking is all about learning
right? And maybe you learn something of like okay now I know how to defeat that next time. Now I
know how to not make that mistake next time. Um and share your mistakes right? Like it's sort of
embarrassing to stand up here and say like I fucked that up. I fucked that up. But. Actually
it's kind of fun. Yeah I guess it is. Yeah. But there's you know there's lots of failures in
like sharing those. People can learn from those. We learn from them. Blog posts are great. I see
lots of blog posts of people bricking things that I would have bricked if I hadn't read the
blog post. Yeah. So. And that you know that's that's what I'm trying to do. Yeah. And then you
know again that's the way that's the way to learn. Everyone's gonna make mistakes and don't be
afraid. So step one. Brick hardware. Step three. Profit. Yeah. And uh yep. So thank you for
coming. Yeah apparently you can make a whole presentation about this and thanks for sitting
through it.