>> Here's Mr. Tal. I'll let him talk to ya and start the talk. So, enjoy. [Applause] >> This is working, great. All right, good morning. Good morning everyone. Good. I'm getting an interactive audience, good. Um, so this talk is titled "I Hunt TR-069 Admins." Let's get it started off. So just a quick disclaimer here. Corporate legal wouldn't let us you know, actually do most of this stuff but you'll see. So first of all I would like to credit my personal role model which, and first thank Taylor Swift for providing inspiration you know, and words of wisdom which I've tried to incorporate into some of the slides. [Applause]. So yeah. But seriously if you're not following this account, you should because it's hilarious. Okay. So and obligatory who am I? My name is Shahar Tal. Yeah, such profile pic, much coolness, very professional. I have an amazing wife and daughter waiting for me back home. I've just spent almost 10 years as an officer in the Israeli defense forces. Last year I joined check point where I lead a team in the malware vulnerability research group. We occasionally do some interesting things and hopefully you will get to see us more this year. So this is my right leg. Is that, can you see that? I mean, ok. This is my right leg. My left one has a hexadecimal dump. And if you want to see it just come after the talk. So starting off with an agenda, I'm gonna give you an introduction to what TR-069 is. Okay. [Laughter] And I'm going to tell you why you should care and I'm gonna try to give you a sense of what the landscape feels like. We'll move on to show a few examples of footage and conclude with a short summary. So just beginning with you know, residential gateways. Sometimes referred to as home routers or Soho routers. And we all know that this is an incredibly unprotected domain. In recent years it just gave us more and more good examples of that. So we have the Internet census which showed us that there are literally millions of devices out there waiting to be controlled with default passwords. We've seen rapid severance, great coverage, VPN exploitation, and last year showing us it's also quite easy to run code on so many devices. We've also seen campaigns targeting devices for criminal activity and you know, there are really too many to mention but some of these guys, maybe some of them are actually here, have been doing great work demonstrating really, what, that the state of affairs of this domain. And I'm here to kind of extend this notion. So what is TR-069? TR stands for Technical Report, just kind of like a, kind of a RFC and you know, contrary to what you'd expect, the title and numbering were not given by a giggling teenager but a more or less respectable body called The Broadband Forum, which is previously known as The DSL Forum and The Frame Relay Forum. And they're basically known as-- they are a consortium of many players in the broadband market and they're attempting to find standards and common goals. So in 2004 they released this CPE WAN management protocol and which is CWMP or more commonly referred to as simply TR-069. And the specification went through five amendments actually which were released late last year, and to tell you the truth it's a pretty boring 228 page document filled with, one sec, filled with fancy words like mechanism specifications or schema definitions and you know, they actually describe pretty simple things. So let me try and summarize 200 pages for you. This is how TR-069 sessions or provisioning sessions work. Okay, on the right side we have the CPE. Which is the TR-069 client. That's your home router. On the left side you have the TR-069 server which is called an ACS or an auto configuration server. And they talk in basic SOAP RPC and XML or HTTP and it's that simple. It's important to mention that the client always initiates the session which is a single TCP connection over which RPC's are made back and forth. The client begins with an inform which is studying the reason or why is the session is initiated and the ACS follows with provisioning functions such as get parameter values, or set parameters values. It really is that simple. And there is a dual authentication mechanism. The CPE should make sure it has a verified ACS and the verified ACS should only accept sessions from authenticated CPE's. And here's a quick example just to get, just to be-. This is an ACS set parameter value call for a certain CPE. And this is in fact -- let me see if I can -- yeah. Okay. So this is set parameters value call and this is assigning a new value for the management server URL key which basically means this is updating the configuration ACS URL for this device for ACS super-securized.com, whatever. So these are only some of the names -- oh, sorry. You see some really big names up there have active TR-069 deployments. And as you can understand this is not some obscure protocol. This is a growing trend to adopt it everywhere. And this is a widely used defacto device management standard. Yeah so research in 2010 estimated 115 million TR-069 devices online, 70 percent of which are Gateways and you know, today it's probably more. How about this? Some very interesting results from the ZMap guys, Zachary and his friends at University of Michigan. And they show that 7547 which is a default WMP port is actually the second most popular open port in the world. This, I mean, this is no joke. This is a serious, serious protocol. So let's talk about some of the functional use cases for TR-069. So what can you do with ACS's? You can provision devices which is what you call zero configuration. Just remote management by the tech report guys, you know, when you call the call center and they fix your router, whatever they do there, can monitor for faults, for malicious activity even, diagnostics, performance measurements, you can replace, fix faulty configurations. And also deploy upgraded firmware. So I'm betting at least some of you are surprised with that last bullet you know, and this is so you can understanding the power that's entrusted in these sessions. Right? So this, and this is a power that service providers don't necessarily want you to know about. I mean, think about this for a second. You know, without getting too deep into the trusting your compiler issues you know, who do you really trust to run code on your devices on demand? You know? How about when it's done silently? Remotely? With elevated permissions? You know I might trust some of these you know, very big vendors like Microsoft, they have heavily protected update servers, whatever, I might trust them, but my ISP, I'm not sure. So yeah. ACS's have that trust and they can remotely manage all their things. So here's a short story. This is my home router or at least it used to be. You know I'm starting to learn about TR-069 and I'm poking around looking for the TR-069 configuration and there's no evident pages with these settings. I know you can't really read it from there but nothing says TR-069. And you know oh, well I should go into expert mode. Right? Oh why not? I'm an expert right? So I go into expert mode but still nothing saying TR-069. So maybe it's under remote management. I clicked it and then no, that's only for web management, which is, you know, properly turned off by the way. And this doesn't makes sense to me because I know, I know these devices have TR-069 enabled and on the specs I've heard that my provider uses this. So as it turns out my provider customized the firmware to hide the fact that they were in control this was done using the highly sophisticated HTML commenting technique. [Applause]. So you know, I entered the URL, and then there I get the details and I have the set up screen and you know suddenly I see that there is no legitimate way to turn this off. So the disabling, the informs is just like the periodic hello messages, it just, I mean, it still has no real effect. What you actually could do is change the ACS URL but never mind about that. So you know in the later firmware version they actually changed this page to be you know, just a status page, now there really is no legitimate way without you know, getting a root on your device, to turn off TR-069. And so the point is someone out there knows that they have a lot of power in their hands and they clearly don't want me to know about it. So with that context in mind let's take a broader look on TR-069 architecture. And, sorry. Again on the right side we have a CPE, the client which can also act as a proxy from any other home devices, such as VOIP phones, Nest devices, set up boxes, whatever wants to be controlled. And on the other side we have the ACS. That's the, you know, the one server that's in control. On its north bound side it is attached to ISP internal assets like the OSS, BSS, Billing, Policy, the call center of course, and this south bound interface is where the CPE's go. So, wait. What? There's a single machine in this design you know, that's in control of an entire fleet of devices for that ISP which would make it in fact the single point of ponage. So if ever there was a high value target in an ISP that would be it. Well luckily this south bound interface is using the most trusted and secure line that you can think of which is the public Internet. [Laughing]. So you know, looking at this from the APT or you know, nation state attacker's perspective, so you can take control of this one target, one accessible target and you're in complete control by design over millions of devices. This is like, that's the NSA's dream right there. So now that we all understand that TR-069 is very powerful what could you actually do if you had control of an ACS? I see my shot coming up. So I'm just gonna pause for that one sec. It's very, very dramatic, very dramatic. [Applause]. >> Keep talking. >> I'm gonna keep talking. All right. So what could you actually do? >> I see you've already taken your shot. >> I apologize for that. [Laughing]. It was a long night. You know, I copied all those Swift On security quotes like late last night. So what could you actually do if you had control of an ACS? You know, and that's a big if right now because we don't know how protected this is. But you can actually get private data, you can get the SSIDs, you can get the host names, Mac addresses, usernames, you can get phone numbers. With some vendors you can even get complete configurations including passwords. You can set every parameter you know, starting with the classic DNS change. Right? We all know that one. You know, but you can go for more sophisticated stuff like adding your hidden WiFi. >> Shut up! >> I'm shutting up. [Laughter] >> Come on you guys know the drill. How about a little love for the brand new speaker? [Applause]. Scumbag ACS? >>Yes sir. >> What would an attacker do if... [Laughing]. Maybe I should attend this talk. >> That's right. Okay so you can add the new hidden SSID, you can add like a new hidden WiFi to that router and just connect to it whenever you're around or you can even you know, replace the WAN service with a tightly controlled tunnel. You can download logs, configurations, firmware from the device, and you can actually you know, go all the way and custom make your own firmware for all the clients of that. You can do this selectively, you can do this for everyone. This has crazy abilities. So now, you know, we understand this is a critical component in modern Internet infrastructure and with this understanding let's go out and research TR-069. So a couple things just with cropping up Google when you look it up. And you know, there's Luka, which is the main developer for CWMP, that's an open source TR-069 implementation. But his talk is mostly about interoperability issues, and much less about security. And just a nice blog post by these guys, 3S Labs and they actually make some few good points but they never follow it up with anything afterwards. And that's about it. And I mean when we were seeing this, we were really surprised. Because you know, this is why we think that this talk is so important because no one is talking about this yet and this is important. We think more people should and hopefully you guys will. And so why is that? And the problem is this is software in a niche market. So this is service provider world. This is out of most consumer's and researcher's reach. You know, where is the TR-069 community? Right, I mean, are there like TR-069 parties out there? I don't know if I'm not invited. Maybe I should. I don't know. I mean you know how there's a subreddit for everything, I mean, nothing for TR-069, this is-. You know, we were looking for this, you know, a TR-069 media outlet, something that covers you know, TR-069 happenings. And you know what? Apparently there is one. So this actually is a website called TR-069 Central. It's a WordPress blog. And you know, don't laugh about it because you know, they have 16 followers on Twitter. [Laughter] Yeah, let's see if I can -- yeah they're following 23 people including me. [Laughter] And, you know, of course I'm kidding because they actually do a decent job of covering TR-069 happenings and they have independent reviews and such. But, and when I say they I mean a pretty nice guy called Max and they compiled a list of ACS vendors, some smaller names, some bigger players and, you know, we started browsing the vendor sites. As you go along you get this certain feeling. Let me try to show you what I mean. Let's hope this works. >> This is a presentation on SS/SI for TR-069 auto-configuration service. >> This is great music, by the way. >> Let's begin by logging into ACSI and enter your username and password. >> Okay so maybe that's enough... So you know the feeling is much ACS vendors, much TR-069, many features have such a 1999 look and feel. Right? So you know, the more we read, the more we understand that ACS software, are basically web applications that are filled with CPE controlled values that go through lots of processing, and you know, and we all know that clients can be like totally trusted. Right? So we can look at TR-069 problems in two planes. So we have insecure configuration and we have insecure implementation. Of course we have the combination of both. More words of wisdom there. [Laughing]. So how do you find ACS's in the wild? So first off you can hack a single router. Right? We all know that's kind of easy. And then you just get the ACS URL, that one. But you can do more interesting stuff like you know, scanning, you know, ZMap, Masscan, for the win, you know. Scanning 7547 and friends. There's also UPnP which are a lot of fun actually. And there's public data that you can search like the Internet census and the DNS census. Of course there's some fun Google dorks if you're looking for a specific product and of course you can also go for, let me show that for you, and look for interesting stuff there. But anyway there are vulnerable ACS's online. I guarantee it. So how is ACS authentication performed? So SSL is recommended. [Laughter] Which means that a whole bunch of lazy ISPs avoid the trouble and just go with the shared secret which is HTTP authentication. Right, so either basic whatever. And we've actually done a little survey, we've collected a few hundred ACS URLS belonging to a few hundred ISPs and would you like to guess how many of those were using HTTPS or SSL? So yeah so here we go so, 81 percent were actually not encrypting any of this, you know, super critical provisioning sessions. And this is like -- I mean, what -- [Laughing]. You know that means, that all it takes is one freaking you know, NSA quantum packet, you know, single quantum TCP packet to take control of a provisioning session and give you a new firmware or change any of the other parameters I've just described. Here's a nice trick. So our interfaces, right, we take the ACS passwords, which is, you how the ACS is supposed to know your, the real CPE and, but as I mentioned before they allow you to change the ACS URL, which basically means you can change it to a web server that you control and you know, even downgrade to basic authentication and just make it even easier to parse. So yeah, nice Jerry. So let's talk about certificate validation. And the technical report states that if TLS1.2 is used then the CPE must authenticate the ACS using the ACS provider certificate. You know, that sounds reasonable. That actually sounds right. But this is how I imagine this is perceived in most ISP management forums, and I did a little field test. This is obviously a fake certificate right, it's a self-signed certificate that we just created for this, for testing purposes. And so you claim to validate SSL certificates but the fact you accepted my self-signed one determined that that was a lie. Yeah, so first big hard face palm for the day. [Laughing]. So let's run through a short recap until this point. So I hope you're convinced that TR-069 is you know, dangerously powerful and TR-069 servers are high value targets and, you know, we've also seen actual proof that a lot of implementations just aren't serious enough. And now we're gonna take it to the next level. So at this point I have to apologize in advance. I wish I could give some more details on some of the following slides but however we do realize that there are many unpatched instances out there and we don't want to make it too easy for the bad guys. So our first target was a Java based open source project, it's called OpenACS, you know, and we started auditing the code and three days later we get remote code execution. So we have reflection and file upload and this is like you know, at this point we're like -- and this is exactly what we expected to find. Right? We expect to find poor code, this is awesome. I mean, let's keep it up. So we move forward. We pick Genius.acs, which is you know, another open source project. This one, this is fairly modern. It's under active development. No JS, MongoDB, all the new hip stuff. And we started editing, auditing and two days later we got a functional remote code execution exploit. So this one actually is kind of nice. Let me try to show you that. So this is a regular expression that's filtering out bad characters. And an input that follows to JavaScript eval but they forgot the G at the end of the regular expression meaning that you put in one bad character and it just removes that and lets you do whatever. And the real bad thing is this is running as root. No JS servers, yeah. This is bad. Lets you dump like AC shadow and stuff. [Laughing]. Yeah, that would be our faces when we saw it. Moving on, some more ponage. Here's some green text for you. Scanning for IvP4 ACGS and we detect an instance, a vulnerable instance in Middle Eastern ISP, and when I say Middle Eastern I'm talking about a major, a very Muslim state and you know, the northbound interface is exposed which means the internal API, I mean you can actually make calls to internal API, which means that we own that. And then we are like, you know, face palm. But OP actually delivers the vulnerability report. The ISP support center was not thrilled with an Israeli calling about the vulnerable infrastructure. [Laughing]. [Applause]. But we showed them this, and you know, practically all blanked out. It's a few thousands devices. Eight out of 10 would report again. So yeah, I mean what if you're leaving a critical Internet internal asset listening to the entire world with no protection whatsoever? The next case is kind of a big one. We can't disclose the vendor at this point in time but he has a massive install base, locally, globally, and our findings include the internal API authentication bypass code injections there. And denial service attack. The combination of which allows writing arbitrary files to any location including C slash I net to ASB enabled Internet server. So you know, remote execution, right? And we went ahead and contacted one large provider to test this with permission. And eventually we ended up owning just over half a million devices and this is a major ISP. And this is just an example provider. We just picked one. And you know this has been unpatched for at least two years now so it is very possible that we're not the first to discover that flaw but we don't really care. Appreciate it. Appreciate it. [Applause]. So you must be asking yourselves what can I do? What can I actually do? If you don't trust your gateway there really aren't many things that you can do. But, you know, adding another security -- well first of all you can hide under the couch and never connect to the Internet, but that's obviously not a good solution. So the first thing you can do is you know, audit your own TR-069 settings. You know, actually ensure that you know, SSL is turned on, that there is proper certificate validation. You know, if you're unsatisfied you can disable it, if you can disable it. And, you know, adding another network security layer is not bad. You know you can add another router with NAT or firewall capabilities. I mean we sell these devices but actually, quite frankly you can use open source firmware alternatives and they'll do just great work. And you know, ask your provider about the TR-069 configuration. That's, I mean, it's their responsibility. So you can't fix the problem, so there is no easy fix because the bad implementations are out there. So TR-069 has to mature more. Awareness is key. You know, this is what we're here to do. TR-069 can be secure and this is up to the security community which is you guys. We're hoping to get as many you know, new research blogs out there as we can find. And ACS vendors they have to start you know, writing better software. Put your money in secure coding. I mean, show your security stance. I've never seen a bug bounty for TR-069 software and I think we should. Service providers, I mean you're the owners of the traffic and you have to protect your customers. That's your responsibility. So for some future directions. We're actually currently looking to extend this research. So you remember how 7547 was the second most popular port in the world? And his is by design. So we're looking into some client ponage, stay tuned for CCC so this opens up an interesting exploitation vector. Right. Thank you very much. [Applause].