So I will get them started but I have to go on record just being so honored that the stage is staring Dustin Hoffman presenting for us today. Give Dustin a round of applause. >> Thanks that never gets old. I'll tell you. >> [Off mic]See, if you were not a speaker, I’d say fuck you… >> You can still tell me, dude. You can tell your friends later. The only people that seem to get that are really old bank tellers or people not from this country. Like oh, that's a famous actor. Hey, thanks for coming everybody. I appreciate you being here. You can be at any other talk right now but you chose mine so I am thankful for that. This is What the Watchers see, Weaknesses in Municipal Mesh Deployments. I’m Dustin Hoffman. >> And I'm TK. >> Wait. I have to talk about me. I am my own and also a senior engineer at an IT services firm and principal of various other concerns and I also like to cross fit so we can compare times and this is my colleague TK. >> Yeah, I guess I'm a senior engineer. We don't do the title thing too much but that one seems ‑‑ we make them up pretty much every time. I don't have a whole lot to say about that apart from being the principal of research. >> How this got started for us was ‑‑ excuse me for a second. Oh shit. >>Excuse me for a second we have a tradition here at DEF CON. These are first time speakers. >> We have one first time speaker so they're both taking it. But the tradition is we have a ‑‑ we welcome new speakers by allowing them to enjoy a shot with us. >> I'm going to go on record. You do a much better job of pouring than Proctor. >> Cheers gentlemen. >> Cheers. [Applause]. >> Oh, thank you. >> Thank you guys. >> Carry on. >> Thanks. So how this all got started for us obviously in the post 911 world a lot of municipalities are taking Homeland Security money or any moneys available when they have shrinking budgets and using technology and provide security or oversight or surveillance throughout their city. We started to see these camera pods and there will be pictures later. Being the curious souls that we are, we’re curious about that. In certain places, my phone, it would be like hey, there’s an open WiFi network available. And they had interesting names like Police Department [laughter] And this is before the era of you know, FBI Surveillance Van 004. This is 2008, 2009. >>So while drunkenly hanging around downtown one night a few friends and I were wondering around and we find this fountain. I decide to climb on top of it which may not have been the wisest thing in the world but I did that and here I am acting like a clown and a voice came from above and said, this is the police. Please get off the fountain. And it sounded like it was about the 150th time the guy said that. I see, oh, of course the camera pod Dustin was talking about. And I don't know, I guess I wasn't really all right with being observed so it put me on wanting to look at these things and see what it's all about. You can see the network available even just walking by and a number of things we'll cover. A little creepy. So one of the first things I thought of after thinking about the fact that even in a relatively small town around 70,000 people by no means a large city, I thought of this article that I encourage you to read on Wired, called Transparent Society. Has anyone read it,Transparent Society? This short story described, two societies. Both were surveyed. In one basically the authorities have the camera. In the other every citizen has equal access to the video and it's an interesting experiment. One of the things I think civil libertarians and people like myself are concerned about with this kind of thing is the information is asymmetric and certain privileged people have access to it but people like us don't. There's no audit trail. We don't know how people are using it. So those are some issues that we'll cover along with the vulnerabilities. You might ask why are municipalities deploying such networks. Once you deploy a network it's like building a road. All kinds of other fantastic things can spring up alongside it. It's like a utility. Cities are using them to obviously monitor traffic in intersections and on main and other roads. Monitor public spaces. They have two way audio. I know in the city, the specific city that we'll be talking about in our talk today they used infrared sensors to monitor parks after hours. So if you go to a city park after dusk you will get the same voice, city parks are closed at dusk. >> They have a real nice video of some kid running away. You see him looking then scatter. >> Presumably that's what they're looking for. Frequently in the RFP's for these things you will see cities talk about we have a shrinking police budget, but it let's us be everywhere. Whether or not everyone is okay with them being everywhere is another question. I think of the audience mated speeding ticket robots if you will that they deployed in Arizona that the voters, you know, flipped out about. Maybe rightly so and complained about. If you were speeding these devices were on the side of the road. Your current speed is this an if you're going too fast it blinks at you. This would read your license plate number if you were going too fast and mail you a ticket. So that's one of the reasons the cities are deploying things like this. Implementation is almost universally a mesh network. Why not a wire network? Nobody wants to trench. Like I said the budgets are shrinking. To wireless technology a city council or other committee has a dramatically lower cost. I don't know if you've been involved in stringing fiber. The licensing process and the costs are substantial. It will easily exceed the cost of cameras and what not. You can deploy much faster. Most of them operate on 2.4 gigahertz, 5 gHz. These are not special. Who actually does the install? I think this will be a key part of our talk. You have specialized companies that really focus just on municipalities and other kinds of government organizations. One in particular Leverage IS they have special qualifications and basically, basically they're able to fill out a long RFP. So these tend to become the prime vendors in installs like this. They're not technical qualifications as you will see. >> Not at all. >> First picture on their website is a smartly dressed police woman. Presumably providing real‑time response to real‑time crime. Including climbing on fountains. If you look at their Twitter site which hasn't tweeted in a year or two they only follow other police agencies. Implementation hardware itself is not terribly interesting. Things you already encountered. Maybe it's just packaged differently. Specifically we see a lot of fire tight vendor. Bosch cameras, normal DVR's. Here is some of the equipment we picked up off E bay. 2.4 giga hertz antennas here. They usually stick these in outdoor enclosures. There's plenty available on E‑bay. We don't have a big budget for DEF CON. You can even on E‑bay you can buy some units I don't know if you can use the license but they have radios to operate on restricted public services networks. I think it's 4.9. So information specific to Firetide. You should talk. >> Well basically they're just little boxes and as far as the people using them are concerned in one end you plug in the Ethernet cable and the other end and magic happens and the package makes it to the other side. We did a small amount and I have to thank you guys from another company I worked out for helping me out. Jordan Hays did indeed attend the conference. All these guys I don't know if I could have done it without them. >> My favorite story is oh you want help for this? Oh for DEF CON, sure. Like this one in a couple weeks. So thanks for that. >> Yeah they really pulled it out. Well really honestly, you know, I'm losing my track here. We were able to find out a little bit of information. It kind of looks like these things or using something like they're using ‑‑ what am I looking for? Oh man. Automated source routing or reverse sort‑‑ I don't remember the name of it. In any case where in every packet there's a header that contains every single hop that the packet must take in order to go forward and they added on bits as well. Signal strength to help with their routing so they can make sure it's a good system that automatically heals itself. In this case I think they sort of relied on the fact that Automesh is sort of a trade secret and I don't know if it's really trade secret but they deliberately made it hard to understand. Rather than actually using security. Um, I don't know if there's anything we didn't already say. >> If you use any embedded box, any small low power system it's going to be similar to this. Really the only unique thing about Firetide as a vendor is they have a patented proprietary protocol they call Automesh. It's derived from other actual documented protocols. A large amount of our time was spent as far as we know with the first decoding of the protocol. A full decoding would permit full inter-operability. Typically you will see with these deployments 2.4 gigahertz, open standards in this case with Firetide equipment and uses Automesh which is not documented. In all the city's documents which you know get on the website 2.4 is used but it's not mentioned. They only mention 900 megahertz and 5 gigahertz. So it makes me wonder if you see the sloppy implementation they did if they're aware the 2.4 gigahertz radios are turned on. When we talk about mission general, if you don't have a clear understanding on this with them ‑‑ with any other network let's say especially a WiFi or wired network you assume a client node let's say streaming video my phone has a path to the switch or some other more centralized distribution point. With mesh you don't. All the nodes are peers, this peer may be able to talk to the backhaul, whereas this one can’t. You don't have to string fiber to the city. So here in the first example with the wire you see all 5 outer nodes presumably are reliable, semireliable and linked to the center node in a mesh nodes can route the traffic through other peer nodes. This is actually if you are ever going to do anything malicious scroll the city's website because the information they provide is incredibly useful if you had ill intent. This image is right off their website. It documents the specific antenna replacement and buildings they route off of. Is it terribly small? Oh it is. I'm sorry. Things you can't know before you get here. My apologies for the small images. You see the camera pods described and all the hop locations. From this you can derive which antennas go where. Like I said Automesh, not documented. They claim they have 19 patents to date. There's things you can assume that it will do; a lot of the things that all the other mesh protocols will do. Link from node to node. From reading many, many technical documents we discovered it embeds things like each node for a particular altitude, GPS coordinates preset during install, things like this which can contribute to the idea that with Automesh you don't have to do as much pre configuration. >> So we set up and when we're looking at the traffic that's actually going out over the air it starts out with kind of standard ‑‑ I wish I could point at it but I can't. You can see it beginning there. There's a standard issue, headers and all that stuff and additionally there's an IP header which is standard and that points to ‑‑ oh I can point. How about that? That points to the mesh IP which looks a little weird. As far as I know he everything is normal about that. The protocol is set to 134 which is allegedly for RSVP. I know what follows is not in fact RSVP. Soon then we move on to the header right here. This is what actually follow that will last IP header and the Automesh header. And this is what I think is the mesh ID. There are only very few. >> Automesh uses this to specifically identify which mesh, say you have multiple meshes deployed in the same geographic region, which mesh this node is a part of. This is part of the Automesh secret sauce. This is one of the things you define prior to deployment. We'll talk about it later in more detail we believe this is part of what most of these deployments believe provide some of the security they think they have. Specifically that it's not documented. You can't load up for example Wireshark and get very far because it chokes on Automesh. One of our goals was to help that along. How many people use Wireshark? It's super helpful. Right? It's great. It chokes on some of this. Specifically because there's no public definition. >> We haven't entirely decoded it here but you can see there's a couple fields that are probably flagged. Actually the next one is ‑‑ this right here is a unicast frame and there's just not very much information. This is a broadcast frame and I think what they've done here is they've attached a little bit of information but maybe some flooding for getting routes and stuff like that. And you can see this right after the mesh ID the last bit is 1. I was not able to find any packets that had that set at 1 that was not broadcast and you have units here. There are a couple values here. This is probably some routing info. Then there's this chunk that comes here. This looks like it might be IP addresses and this here are definitely the IP addresses of external ‑‑ they match the external IP of that first IP header we looked at. So I assume it represents the idea of the node that we're entering. Then it goes to a multicast address. Conveniently they've left for us ‑‑ oops, conveniently they left a marker at the end that you can use to sort of strip this out entirely which when they have it unencrypted which they sometimes do. There's an easy way to skip all the auto you mesh stuff so you can look like we were able to send a package across the stuff and pull it out of the air which takes away their safety there then this is just an example of something. This is a CDP packet. Not much to talk about here other than you can see that it’s in the clear. >> Our specific case ‑‑ thanks for sticking through the background. I hate to skip ahead and not have you appreciate what you see in these things. In this specific city you see we mapped out most ‑‑ a lot of ‑‑ there's apparently ‑‑ how many cameras are deployed? >> 122 I think. >> And that's not including what they have to put overall the school sites. The city and school district apparently joined hands and they kind of aggregated all their cameras together. The primarily collection here is downtown. This is kind of the downtown. Oddly enough from the time the initial RFP was released in 2006 until 72 hours ago the entire system was unencrypted. So no WEP, nothing. So friends there's a lot of things we can't know in life. Like one of the things I try to figure out is is it actually supposed to look like a Wu Tang clan? The world may never know. In this case I'm going to try to put myself as best I can from servicing other IT clients and put myself in the shoes of the City Manager and maybe the people that did the original install. My best guess is the Automesh alone would provide the security because it was obscured. Because you couldn't just load it up and read it right off without additional efforts. It was secure enough. I think this is part of the problem with when we use words like security. I don't know about you but I think security is over qualified. I think it was said best in the books. There's two kinds of encryption in it is world. The kind that keeps your kid from reading your diary and one that keeps companies and multinationals from reading your diary. At the end of the install the QA manager, I'm being generous that there is one, they didn't look at the form and say is it secure? Yeah it's secure. We can check that box. Or it's a matter of well I turned the knob and the door didn't open. If you lean on it a little it will pop right out. It's kind of one of those things and I think as taxpayers, especially with the next slides you want to cringe because this was all tax money. This is something you and I came out of our pockets and here we are frankly ill served by ‑‑ you know I was thinking of this the morning. I thought of the Obama care website. Shoddy government contractors took a lot of money and provided not a great product. I feel like you will have shades of that repeatedly as we go through the slides. But 72 hours ago WEP was turned on. So yea for government efficiency. Right? Was it because of our talk that was ‑‑ I don't know. I'm going to say it was though. I'm going to take credit for it. Moving the bar of security forward. One peg at a time. >> You should read these later. So here's what you'll commonly see. These are the camera pods watching you. Here's a loud speaker. Right? Here's the little all weather enclosures. When you see things like this I get nervous. I have to wonder if, you know, those directional antennas will provide security. Right? They only send the signal to one other place. I wonder if ‑‑ think about when you didn't know as much as you know. If you did things looking back you would be like oh yeah that was a bad idea. That didn't work out as well as I thought. I have moments like that. When I started the company I run now I thought that was a really bad idea. Not at all safe or secure. Maybe you got by by the skin of your teeth. That's what happened here. We have Automesh. Then we can read the video screens. We have directional antennas. No one else is going to see the Redland's police pop up. And there's one of the Firetides right there. Having vendor names here, maybe not the best idea. They were installing another pod right by our office and they had some vendor out. Like a bucket truck. And I stopped to take pictures because he had the door open to this thing. And that guy got upset. I was trying to take pictures with my camera. Here's the highest point in the city. This 8 story building. It has cameras ‑‑ this is the high point they route it back to. You can see these things are popping up everywhere. This picture makes the city look really bad. This is more cameras. More PA devices. I'm not sure what I find more obnoxious is the cameras or PA things. This makes it okay though because of a notice it's under surveillance by the police department. I think they have run afoul. >> This was not sufficient. You have to have notifications a lot more places than they have them. >> It's not exactly downtown either. It's kind of like on one of the ways that you can get into downtown. But no where else. >> I've never seen this sign which is weird. So as it was until 27 hours ago which is not fair that I got here and they changed it right before we came. So I'm so smarting on that. Here are the security problems and potential issues with having it as is. And when you have one vendor doing the install, I'm confident you will look around, when they get a government contract they're going to tell everybody. It furthers their prestige the small community they're in. Santa Monica has a deployment by them. Other cities larger than us. So assuming like smart people like us do that directional antennas and network protocol is not enough. At the minimum ‑‑ and I'm going to date myself but I was saying this back in 98 at my last corporate gig people will deploy wireless network not knowing they reach beyond their office building or home. All of a sudden you have access to the transmission media. With the wire you have to put a tap on it. Wireless gets in places you didn't plan on. That's exactly what we have here. Another one, how many of you ‑‑ this is my favorite. I'm not sure what to do about them. I play along but how many of you got my call where it's like this is your bank center calling. Can you read me your 16 digit number? I have. I'm worried there's fraud. Right? So I probably give it to them. Was anyone here at the talk the gentlemen gave ‑‑ it deserves more recognition then I will give it but they impersonated a cell tower and there are warnings outside the door. None of you were here for that? Okay. The lights are bright. Okay. They don't call me eagle eyes. So yeah. Great talk. And what was the key thing I took away was you don't have both parties authenticating each other. You have the same thing here. With this set up E‑bay hardware you can do a simpler method and you can become a node on their mesh. You just have to know the mesh ID which can be read right off the air. That mesh ID. There's only 256 unique combinations here. You read it off and you can join their network and participate. What can you do with that? If you're a member of the mesh flower a mesh presumably because it's wireless the health of the network is changing based on fog or all kinds of weather conditions or magnetic transmissions. One node can report hey I have a great view to the back hall. You all should route your traffic through me. A node on the network unlike a wired network ruled to be in the right position. Not so with a mesh network. You can have the traffic routed through you by issuing a favorable packet. Saying I have a fantastic link to that last hop you need. I don't have to sit and sniff just with the cameras I'm in range of. I can actually request all cameras. Or whatever other city services is present on that network. All the equipment presumably also supports encryption of their own transmission. Cameras, the audio transmitters, all those do. Never turned on. Like I said using a directional antenna to secure the medium simply not sufficient. If I'm picking it up while driving around a round something is wrong. Some of you may have worked with optical transmitters where you shoot a laser to another ‑‑ to make a wireless link. You actually have to get literally in between the two, the transmitter and receiver to pick it up. Not so here. Antennas leak in directions you're not prepared for. I mentioned a fantastic feature of Firetide. I really feel bad for the city and I'm putting the blame on the vendor. Firetide actually supports mesh node authentication. PKI via certs, there's one node on the mesh to all the other nodes. It's fantastic. Not turned on. That box didn't get checked apparently. Again Automesh or any other protocol ‑‑ don't look at my typo. Simply not sufficient. So really what we've seen a botched thing. Security is hard is it not? What do we see? Is it actually broken? Certainly not. What was broken? Someone screwed up. Someone thought managing it that way was a great idea and it wasn't. Someone thought doing something a novel way instead of a proven way was a good idea and it wasn't. Someone here thought they could ‑‑ because of other things in place that it was sufficiently secure or they thought security wasn't a concern at all. We'll never know. So really this is really the integrator. In some ways it's fun to knock your local cities or governments or what not but really we put so much in the hands of private companies who take government money and are supposed to provide a service. I presume the mayor pro tem of a small city like the one we're doing up here that guy ‑‑ you have to feel bad for him. He's in no position to evaluate the finished product even more than my grandmother is; should we knock her for saying oh honey you have the whole internet on your phone. What do you expect from her? What do you expect something from a city mayor or manager? They depend on the integrator. That's who they paid substantial sums of money for. A near by city was paid half a million dollars for something like a handful of cameras. It wasn't really related but it was a recent award or proposal, half a million dollars. Will they get vendor security? It's 2014. I hope so. Is anyone in the city capable of evaluating that install? I doubt it. The city of LA, city of Chicago, city of New York. Sure. I would imagine so. Is there even an IT department. Is there anyone that can do a post install test? One of the things I talk about in terms of providing IT service toss our clients is do we test it in ways ‑‑ you know, here's something from me. Back in the days of going desk to desk my last corporate gig when I set up a new computer for an employee I forgot to install the printers probably because I don't print anything. I don't have a need to print out paper. I never really have. But these people do. I don't understand. Our clients print more than anybody. If that printer isn't there I will get a call immediately. In the same way I feel like we often assume in the security world, you know, we ‑‑ here's another one. Filtering services based on IP. Right? Certainly IP’s on the network. Did anyone actually test it from an IP? Did anyone actually test it? Thank you. That sign was clear as day. Thank you. I was nervous about missing that. Did anyone actually test it? I presume ‑‑ I'm guessing not. I think that's what I call the IT people, guilty here. If we do self reflection we will see. We know what we're doing. We've done it a few times. A lot of times maybe. It's worth checking. I know residents are upset that they're being monitored all the time. I saw a comic recently that said how do you want to sell this to the American people. Protecting children or fighting terrorists? They sold this in terms of public safety and from there the public was calm enough, I'm guessing people wouldn't be happy if they found out their sister's brother's kid could watch the cameras too or perhaps much worse going through these potential threats beyond just security weakness. So here are some of the things we theorized and tested. These things work. Big warning here. Federal and states are having lawsuits that may apply. I'm not a lawyer or accountant. So seek the advice of someone that would know about these things like the EFF. Don't go running afoul of wiretapping laws. So we've talked about implementation like this one prior to 72 hours ago observing video streams is trivial. Or because they're multicast you can subscribe yourself. Why not? Like with any wireless system but especially one you can participate in the network in a meaningful way. There's denial service opportunities. Simply by flooding, always subscribing. Like I said these are normal off the shelf. Nothing special. They're not prepared to have a thousand bogus subscriptions. ARP spoofing… You can become a node on the network or the network video recorder. Dirty. Dirtyer though and totally worthy of a Doge is joining the mesh legitimately. There's only 256ID's. All you have to do ‑‑ you can just read it and go oh this is mesh number 3. Now as far as anyone can tell you're a legitimate node on their network. I'm guessing that there aren't a lot of separation. There isn't a lot of separation between these ‑‑ the cameras, the network DVR's and the rest of the city's networks. Often they left themselves wide open to anyone who can listen or participate in the network with accessing services that they're not prepared to have public access to. Honestly guys for the police department you've got ‑‑ you have to start thinking about things like reliability. I mean I'm not a lawyer but it would concern me potentially evidence could be tampered with from 50 miles away. >> You're basically plugged in. You might as well walk in. >> Find a hot port in your office and you can participate in the network. Video manipulation. Think Ocean's 11. You have the security guard with all the monitors and it's in the middle of the night and he's watching and the guys you are supposed to root for takeover one of the cameras and play back a loop. I call it all is well loop. An empty corridor. Whereas in reality when the camera goes all the good guys are making their way down the corridor. Entirely impossible. Videos usually is transmitted through packets. This is how it looks. This is my illustration from this morning. You have a municipal camera connected to the municipal DVR. These are packet numbers. Which are incremented as the next packet is sent. With a malicious node, thank you, with a malicious node your own stream all we have to do is jump ahead. So if the highest packet they're on is 106 I will jump to 200. My video packet is 200, 201, 202, 203. Think about what happens next. The DVR goes whoa, I’m getting ing packets out of order. They've expired. I can ignore them. I've literally now just by incrementing the count have taken over and the DVR has no reason not to, right? It's functioning as intended because we're using EDP we can throw away all the packets ‑‑ now these packets will continue to arrive and they're always ignored because somewhere along the line you watch the traffic and see this happen. Packets arrive out of order. What do you do? You discard them. So just with this, just by incrementing the UDP count you can completely takeover the network. Here is the other danger. The cameras are trusted. I mean you can say hey let's have Godzilla walk down the street. Or project all is well at the city bank but in reality it's being robbed. I'm getting calls but I don't know Frank it looks fine from here. There's also ‑‑ you can also do the opposite. Send resources else where through injecting false data like that. Not to mention look I have a friend ‑‑ I don't want to identify him. I know someone that works at verizon wireless. He tells me people look at all your dirty texts. Someone has access to them all. Everyone can look at everyone's dirty pictures. Who do we trust to look at all this data that's collected that's archived? These are privacy implications along with the weaknesses that go along with it. Things that maybe not everyone thought deeply enough about. Security is complicated and it interacts with other government things. I think some times we think of ourselves as tech people and not citizens that have to participate in the 2014 world. So our demo is sad because of WEP. >> Because we can't break it. >> The person who writes the check to us is now actually thinking because we ought not try because of potential liability. I know we promised a demo and I'm very sorry. But thank you for coming and filling this whole room. I think it's always a speaker's concern that, you know, when I spoke two years ago there was a really, really, really great talk at the same time as mine and I walked in and half the people bailed before I started. So thanks so much for your attention. If anyone has questions we're going to be out front. [Applause]