00:00:00.167,00:00:04.004 >> Welcome, thank you all for uh skipping lunch if you're here. 00:00:04.004,00:00:06.707 Um welcome to my talk on Attacking Networking 00:00:06.707,00:00:10.978 Infrastructure. My name is Luke. Uh here's some boring info about 00:00:10.978,00:00:13.513 me. Uh I'm a security… security engineer originally from 00:00:13.513,00:00:16.850 Minnesota. I'm working in the Bay Area currently. Um I'm a 00:00:16.850,00:00:19.786 junior undergraduate student. Uh it's my second year at DefCon, I 00:00:19.786,00:00:24.258 spoke last year uh and I also participate in a lot of bug 00:00:24.258,00:00:27.127 bounties so. So if any of your companies run a bug bounty 00:00:27.127,00:00:30.264 you've probably heard my name as the jerk that submits bugs at 00:00:30.264,00:00:33.300 anything other than working hours. Um if you have any 00:00:33.300,00:00:35.669 questions about this presentation uh you'd lie to 00:00:35.669,00:00:38.639 send me legal threats, there's my contact information I'll put 00:00:38.639,00:00:42.743 it back up at the end along with uh the code and the slides for 00:00:42.743,00:00:45.479 this presentation will be linked at the end. >> [Louder!] >> 00:00:45.479,00:00:49.850 Louder, okay. Better? >> Get closer to the mic. Just try 00:00:49.850,00:00:54.855 getting closer. >> Trying to avoid tipping it over. Alright 00:00:54.855,00:00:57.591 uh let's get the boring stuff out-of-the-way first. Um here's 00:00:57.591,00:01:00.193 my lovely disclaimer. The views and opinions expressed in this 00:01:00.193,00:01:02.429 presentation are those of the author and don't necessarily 00:01:02.429,00:01:05.198 reflect the official policy or position of any current or 00:01:05.198,00:01:10.604 previous or future employer. Uh just don't sue me. Alright. 00:01:10.604,00:01:12.506 [Laughter] >> Um, as usual we'll start with a quick rundown of 00:01:12.506,00:01:15.208 what we're gonna be talking about today. Um, I'm gonna start 00:01:15.208,00:01:18.278 with what Internet Two is. Uh then move into some of their 00:01:18.278,00:01:21.081 products. Mapping their network and then explain some of those 00:01:21.081,00:01:24.551 products um in order to gain control of some of their devices 00:01:24.551,00:01:27.921 that are running on very large network uplinks. Now to make 00:01:27.921,00:01:31.091 this presentation work, there are two computers and four VM's 00:01:31.091,00:01:34.294 appear that all have to work together perfectly um and 00:01:34.294,00:01:37.731 there's only so many sacrifices you can make to the demo Gods so 00:01:37.731,00:01:40.968 please bear with me if things don't work well. Um, I've tested 00:01:40.968,00:01:45.372 them enough but yeah without, further ado let's get started. 00:01:45.372,00:01:47.975 So I want to give a bit of backstory about how I got 00:01:47.975,00:01:51.912 started looking at the software um and this personal product 00:01:51.912,00:01:54.047 which we'll get into to in a minute. Um the university I 00:01:54.047,00:01:56.917 attend has a nice website full of information about what 00:01:56.917,00:01:59.987 applications and services are available to me as a student. 00:01:59.987,00:02:02.990 When I'm bored I like to browse around and see, [cough] excuse 00:02:02.990,00:02:05.792 me. What I'm able to access. It's kind of amazing what and 00:02:05.792,00:02:09.229 EDU email address grants you these days. Um one of the pages 00:02:09.229,00:02:12.633 is called Internet two. SO the description is, the Internet is 00:02:12.633,00:02:14.468 a global system of interconnected networks. The 00:02:14.468,00:02:17.204 University connects to both the global Internet and a number 00:02:17.204,00:02:19.706 special research and education networks commonly referred to as 00:02:19.706,00:02:22.209 Internet two. These research provide high-bandwidth 00:02:22.209,00:02:24.111 connectivity enabling and supporting research 00:02:24.111,00:02:26.613 collaborations educational opportunities regionally, 00:02:26.613,00:02:30.717 nationally and around the world. Basically it's a private fiber 00:02:30.717,00:02:33.887 network run between universities it's used for sharing all sorts 00:02:33.887,00:02:36.757 of research data that you know would take a very long time to 00:02:36.757,00:02:41.962 transfer over the standard uh Internet. If you go to their 00:02:41.962,00:02:45.165 website and find it even more boring description about how 00:02:45.165,00:02:49.002 Internet two is a community of readers uh researching and 00:02:49.002,00:02:53.807 leaders in academia and um basically it's just a consortium 00:02:53.807,00:02:57.077 of universities. Uh there are some corporations government 00:02:57.077,00:02:59.413 agencies but it's mainly universities connected to this 00:02:59.413,00:03:02.482 and it's mainly for sharing research. But one of the other 00:03:02.482,00:03:06.386 things they do is they generated a great software for all the 00:03:06.386,00:03:10.991 people in this consortium so um they share the software between 00:03:10.991,00:03:13.927 all of the companies and universities that participate 00:03:13.927,00:03:17.130 and in doing so they also share vulnerabilities between each 00:03:17.130,00:03:20.333 other since they're all running the same software. Um they also 00:03:20.333,00:03:24.471 do collective-bargaining on everything from AWS to Splunk to 00:03:24.471,00:03:28.442 VMware. Um basically it's there to benefit al the companies that 00:03:28.442,00:03:32.145 participate. The other thing that it is is a private network 00:03:32.145,00:03:34.514 so this is what I'm talking about this is a map of their 00:03:34.514,00:03:37.551 actual dark fiber. Um it totals about 8 point 8 terabits a 00:03:37.551,00:03:40.120 second of optical capacity and about hundred gigabits a second 00:03:40.120,00:03:44.124 of ethernet capacity. Again it was mainly developed for sharing 00:03:44.124,00:03:47.027 research and technologies between universities. Uh and I 00:03:47.027,00:03:48.929 get really excited when I see something like this, because 00:03:48.929,00:03:52.399 it's not just a whole bunch of blinking lights but you know 00:03:52.399,00:03:55.035 these are additional routing paths between each of the nodes 00:03:55.035,00:04:00.340 on this network and Internet twos been around since 1997 and 00:04:00.340,00:04:03.543 a lot of people didn't really care about security back then. 00:04:03.543,00:04:06.279 And so there's a whole lot of risk here where you know these 00:04:06.279,00:04:10.317 routing paths might be trusted uh or maybe um not even 00:04:10.317,00:04:12.519 considered by some security teams cuz they've been around 00:04:12.519,00:04:17.657 for so long. So in addition to the actual network like I said 00:04:17.657,00:04:20.193 they produce a variety of products. Um actually most of 00:04:20.193,00:04:22.829 these products are actually open source which is really nice. 00:04:22.829,00:04:26.500 There uh um the most popular one they have is called Shibboleth. 00:04:26.500,00:04:30.604 It's essentially a federated identity management uh system. 00:04:30.604,00:04:32.739 Essentially it's a really nice sandal providers really 00:04:32.739,00:04:36.143 extensible if you ever done any penetration testing on pretty 00:04:36.143,00:04:39.513 much anything running at a US-based University. It likely 00:04:39.513,00:04:44.918 interacted with shibboleth for authentication. Um but 00:04:44.918,00:04:47.721 shibboleth is their most popular product. It's been poked at by 00:04:47.721,00:04:49.790 some other people before so I want to look at some of their 00:04:49.790,00:04:53.160 other stuff. Uh they have a lot of uh tools in the performance 00:04:53.160,00:04:55.795 and analytics category so because they run these fiber 00:04:55.795,00:04:58.832 networks they need to maintain the health of these networks and 00:04:58.832,00:05:01.401 so they do that through a tool called Bandwidth Control, which 00:05:01.401,00:05:05.038 is essentially a wrapper around iperf. It does a lot of hard 00:05:05.038,00:05:08.775 work in setting up a receiver and a sender on either end. Um 00:05:08.775,00:05:14.214 NDT is a diagnostics tool uh a Wamp is one way paying and then 00:05:14.214,00:05:18.385 perfsonar is a wrapper around all of those tools and it's 00:05:18.385,00:05:21.321 essentially an ISO that you download um and you can install 00:05:21.321,00:05:24.791 it on your servers and it makes scheduling bandwidth control 00:05:24.791,00:05:29.095 tests and WAMP tests really easy. Um we'll look at what it 00:05:29.095,00:05:34.067 actually looks like a minute. Um, first off, I just explained 00:05:34.067,00:05:39.105 that perfsonar is. Um for uh giving a closer example if we 00:05:39.105,00:05:42.008 are here in Las Vegas. We look at the Las Vegas node and the 00:05:42.008,00:05:45.178 network operator in Las Vegas wanted to make sure that their 00:05:45.178,00:05:49.049 fiber connection to Salt Lake City is remaining solid um they 00:05:49.049,00:05:52.953 would set up a perfsonar since the Las Vegas perfsonar since in 00:05:52.953,00:05:56.556 Salt Lake City and um because they're all part of the same 00:05:56.556,00:05:59.459 network uh they collaborate and basically you'll set up tests to 00:05:59.459,00:06:03.830 run say every 24 hours and they'll alert you if the network 00:06:03.830,00:06:08.935 goes down or if performance starts degregrading. Alright so 00:06:08.935,00:06:13.540 I'm gonna actually look I have two instances set up here. Two 00:06:13.540,00:06:16.509 perfsonar references they're called impact and torpedo for 00:06:16.509,00:06:19.179 easier things here so you can see they're on the same network 00:06:19.179,00:06:25.752 um and so a quick bandwidth control test here uh so this is 00:06:25.752,00:06:29.022 just showing how some of their tooling work. And so what it's 00:06:29.022,00:06:33.193 done here, is to use iperf, um it can you can customize it. So 00:06:33.193,00:06:35.562 you could say I want to use three lay or I want to use iperf 00:06:35.562,00:06:38.999 two or three um and then it schedules a test between the two 00:06:38.999,00:06:42.535 because the way iperf works you need uh both ends to agree on 00:06:42.535,00:06:46.506 when to set up a receiver and what ports to use. And the once 00:06:46.506,00:06:51.044 that time goes. We have our info here so you can see it got about 00:06:51.044,00:06:53.780 a gigabit a second. Makes sense since both of these hosts are 00:06:53.780,00:06:58.318 running on gigabit connections. Alright, now let's look at the 00:06:58.318,00:07:03.256 actual toolkit websites. So this is the if it loads. This is the 00:07:05.825,00:07:09.763 actual perfsonar interface so this is what their uh web… 00:07:09.763,00:07:13.466 interface looks like so its essentially just a queue for 00:07:13.466,00:07:17.637 that tool I just used. So you can see here and I set up a test 00:07:17.637,00:07:20.874 run every half an hour between impact and torpedo and we can 00:07:20.874,00:07:25.578 see there's the last time the throughput was 600 Mb a second, 00:07:25.578,00:07:28.415 um I can pull up and look at a graph and see how that's changed 00:07:28.415,00:07:32.419 over time. Now since this is a virtual machine it has gaps here 00:07:32.419,00:07:35.188 but you know it's easy for network administrator to look at 00:07:35.188,00:07:40.393 this and kind of see what's happening on the network. 00:07:40.393,00:07:45.498 Alright, back to the actual presentation. So one of things I 00:07:45.498,00:07:48.168 like to do when I'm first approaching a product is uh when 00:07:48.168,00:07:50.737 I'm looking for issues is look at what mistakes have already 00:07:50.737,00:07:53.573 been made in the past. So developers to make the same 00:07:53.573,00:07:57.544 mistakes over and over again you know it's just how it is right 00:07:57.544,00:08:01.448 now in the industry and so uh I went through and Perfsonar used 00:08:01.448,00:08:05.151 to be hosted on google code uh their website is back up. It was 00:08:05.151,00:08:08.988 down because the whole Google Code being deprecated. Uh so 00:08:08.988,00:08:14.127 issue 783 was found and basically it's a venerability in 00:08:14.127,00:08:17.063 the web interface that I just showed. Uh it was patched in 00:08:17.063,00:08:22.435 2013 and this is the patch for the issue. Um so if you look at 00:08:22.435,00:08:29.209 it's pulling in peals live XML library and is adding an extra 00:08:29.209,00:08:32.545 entity handler. Um that just always returns an empty string. 00:08:32.545,00:08:36.683 Um so we're going to look at what a uh what an external 00:08:36.683,00:08:41.020 entity is and then how to exploit them in the real world. 00:08:41.020,00:08:45.692 So if we start with a simple XML file. Hopefully everyone can see 00:08:45.692,00:08:49.462 this. We have a list of all the presentations I've given so we 00:08:49.462,00:08:52.298 have the name of the location and then the author. And the 00:08:52.298,00:08:54.601 authors always going to be the same every time. It's always 00:08:54.601,00:08:58.171 going to be Luke Young. XML has this feature where you can set 00:08:58.171,00:09:01.274 and define an entity. SO I've defined an entity called LY with 00:09:01.274,00:09:04.878 the value Luke Young. And then I can just reference this entity 00:09:04.878,00:09:08.448 with an ampersand, the name of the entity and a semicolon that 00:09:08.448,00:09:12.819 way if I ever change my name if I got married I can just edit it 00:09:12.819,00:09:16.489 here and it will edit and it will update you on the rest of 00:09:16.489,00:09:21.761 the XML document and most XML parsers for this by default so 00:09:21.761,00:09:24.831 when you go to get the value within Python or whatever you're 00:09:24.831,00:09:29.836 using it will just return Luke Young. It happens transparently. 00:09:29.836,00:09:32.906 The most or one of the most popular tax with this was 00:09:32.906,00:09:38.244 something called the billion balls attack so basically it's a 00:09:38.244,00:09:41.347 denial of service issue. So you start with a single entity you 00:09:41.347,00:09:44.417 define an entity that includes that one 10 times to find 00:09:44.417,00:09:48.688 another one that includes that entity ten times and you get um 00:09:48.688,00:09:52.091 an exponential growth year in memory when an XML parser tries 00:09:52.091,00:09:58.465 to uh unserialize this uh XML. And for the most part this 00:09:58.465,00:10:04.237 actually here expands to something like 16 GB of memory 00:10:04.237,00:10:08.141 and will actually crash most applications that have XML 00:10:08.141,00:10:11.277 entities enabled. Um but denial of service phones in this 00:10:11.277,00:10:14.948 context are kind of lame just crashing the software is boring. 00:10:14.948,00:10:17.517 It's not something we're looking for. Um so the more interesting 00:10:17.517,00:10:20.720 feature of XML is something called external entities. So you 00:10:20.720,00:10:24.624 can define a system entity with a file URL and basically what 00:10:24.624,00:10:27.760 this will do is it will actually load the contents of the file 00:10:27.760,00:10:32.232 and inject it into the XML. So in this case I'm gonna load in 00:10:32.232,00:10:36.002 etc password and fill it in right here. Uh this… it was 00:10:36.002,00:10:39.539 originally intended for people that have multiple XML files and 00:10:39.539,00:10:43.543 they can actually include other XML files within that that way 00:10:43.543,00:10:46.312 you just load in one and it magically pulls in all the rest 00:10:46.312,00:10:49.983 of the files but you can have a nice folder structure we don't 00:10:49.983,00:10:53.386 have to have everything in giant XML file. However there's 00:10:53.386,00:10:57.357 obviously a lot of potential for abuse here um because you could 00:10:57.357,00:11:02.695 actually include a uh file from the system. So… back to the 00:11:02.695,00:11:07.667 actual issue. Um the patch for this issue was to make whenever 00:11:07.667,00:11:12.138 loading an external entity it will return an empty string uh 00:11:12.138,00:11:14.641 that will not prevent the denial of service issue we just 00:11:14.641,00:11:20.513 reference but anytime something file URL, that will fail. So 00:11:20.513,00:11:23.316 first thing I'm just gonna do I'm going to take, I've actually 00:11:23.316,00:11:28.187 r synched one the filesystem off one of these devices and I'm 00:11:28.187,00:11:32.292 going to just search for live XML new without an external 00:11:32.292,00:11:36.029 entity handler defined. So we're looking to see if they missed 00:11:36.029,00:11:39.065 anything or someone added new endpoints where they forgot to 00:11:39.065,00:11:43.970 add this patch and if we run it right here we've immediately got 00:11:43.970,00:11:48.575 off the bat 13 matches of potential ways to get into this 00:11:48.575,00:11:52.445 application using external entity. Now this is actually a 00:11:52.445,00:11:55.748 bit of a false-positive because some of these are libraries that 00:11:55.748,00:11:58.585 are all sim linked and so lime thinks they're different files 00:11:58.585,00:12:01.955 but there's actually only about six different ones here. That 00:12:01.955,00:12:07.126 particular one that's vulnerable is in an MWG message. So as you 00:12:07.126,00:12:11.598 could see here it defines a live XML handler it doesn't set up a 00:12:11.598,00:12:15.201 way to block entities and then it parses in the file and if we 00:12:15.201,00:12:19.172 traced this all the way back up the stack um this is accessible 00:12:19.172,00:12:23.576 as an external user. So that requests looks little bit like 00:12:23.576,00:12:29.415 this. So we're gonna send a SOAP request um and if you've done 00:12:29.415,00:12:32.819 stuff with XML you probably know what SOAP is. Um and then we'll 00:12:32.819,00:12:37.023 define this NMWG message handler and then within here would 00:12:37.023,00:12:40.960 include the etc password. So this file right here. We're 00:12:40.960,00:12:45.865 actually gonna try and do this live now. Um so we're gonna send 00:12:45.865,00:12:51.371 a post request to the OPPD Daemon on the server which 00:12:51.371,00:12:56.509 traces all the back to that profile. If we run it, there's 00:12:56.509,00:13:02.048 etc password for the systems. So the next thing I want to do. The 00:13:02.048,00:13:05.718 authentication application is handled by etc shadow so I'm 00:13:05.718,00:13:08.688 just gonna read etc shadow instead. Um this is a file I 00:13:08.688,00:13:11.157 didn't show, it's just the exact same thing except for it's a 00:13:11.157,00:13:15.595 shadow. And it doesn't work. So the reason this happens here uh 00:13:15.595,00:13:18.398 it actually sends us an extremely verbose error message 00:13:18.398,00:13:20.800 that it can't read that file. So the reason that's happening is 00:13:20.800,00:13:24.604 because the OPPD Daemon isn't running as route on this device. 00:13:24.604,00:13:27.874 So we don't have permission to read this file and we actually 00:13:27.874,00:13:32.145 we can read other stuff off the system for example we can read a 00:13:32.145,00:13:37.750 sequel um somewhere in here. Sequel passwords off the system. 00:13:37.750,00:13:42.922 We can read configuration files um however none of it was really 00:13:42.922,00:13:47.060 exploitable actually so while we can read arbitrary files because 00:13:47.060,00:13:51.497 authentication is handled uh by ETC Shadow we couldn't get admin 00:13:51.497,00:13:55.201 users we couldn't get anything interesting. The sequel database 00:13:55.201,00:13:59.238 is blocked off so it's only accessible by localhost and so I 00:13:59.238,00:14:04.277 hit a like complete dead-end. So if we go back to the 00:14:04.277,00:14:08.281 presentation, the uh the point i was at here. I was able to find 00:14:08.281,00:14:11.517 cross eyed scripting kind of everywhere um but cross eyed 00:14:11.517,00:14:13.953 scripting vulns are kind of lame you have to get an admin to 00:14:13.953,00:14:16.856 click a link while they're logged in. No one logs into 00:14:16.856,00:14:20.727 these devices often. Um and it seemed there's a lot of ecstasy 00:14:20.727,00:14:23.963 as he saw there were other issues there however getting RC 00:14:23.963,00:14:26.799 just seem like an impossible task I should put it down for 00:14:26.799,00:14:30.670 like a month and finally came back to it later um and I found 00:14:30.670,00:14:33.940 something called bandwidth graph dot CGI. So when I pulled up 00:14:33.940,00:14:37.410 that graph a second ago that was bandwidth graph dot chi so this 00:14:37.410,00:14:41.914 is endpoint handles graphing historical bandwidth data of 00:14:41.914,00:14:45.985 tests and if you actually look at the data and the source code 00:14:45.985,00:14:49.122 for this we can see something interesting so there's a veal 00:14:49.122,00:14:55.628 call on an attribute from the XML data that is sent in here. 00:14:55.628,00:14:58.364 And if we trace this all the way back. Uh we're gonna get into 00:14:58.364,00:15:02.702 exploiting it now. Let's take a look at here let me show some 00:15:02.702,00:15:07.840 example performance data you can see it's basically iperf results 00:15:07.840,00:15:10.777 with the timestamp and the throughput. And if you look at 00:15:10.777,00:15:14.914 the throughput value it's a scientific notation number and 00:15:14.914,00:15:18.818 perfs in scientific notation in pearl is like five lines and 00:15:18.818,00:15:22.855 parsing it with eval is one and so developer was being lazy and 00:15:22.855,00:15:25.625 they decided to use e-val thinking that it was perfectly 00:15:25.625,00:15:27.794 safe. Um and so we can see why they made this mistake though. 00:15:27.794,00:15:32.498 Uh you know, you're in a rush, it happens. So let's actually 00:15:32.498,00:15:35.468 look at how to reach this code path because it's quite 00:15:35.468,00:15:40.273 complicated. So starting at the top of bandwidth graph dot CGI 00:15:40.273,00:15:43.276 uh we need a couple parameters we need a URL parameter which is 00:15:43.276,00:15:47.246 the measurement archive so this measurement archive contains all 00:15:47.246,00:15:52.685 of the data of tests that have been running overtime and so we 00:15:52.685,00:15:56.022 did URL to access that from because you can run this in the 00:15:56.022,00:15:59.959 cluster environment and we need to key to look it up by so if a 00:15:59.959,00:16:02.528 test has a name, it has a key. So assuming we have both of 00:16:02.528,00:16:06.232 those we get all the way down into this get data function 00:16:06.232,00:16:10.236 which goes and looks set up a data request makes a request to 00:16:10.236,00:16:15.608 the measurement archive and then pulls out this data XML 00:16:15.608,00:16:18.177 attribute. Long story short, it gets all the way down to 00:16:18.177,00:16:21.747 throughput. There's actually a second step in here to though. 00:16:21.747,00:16:24.317 So the way to measurement archive works is when it makes a 00:16:24.317,00:16:27.220 request it first sends an echo request. And we have to echo 00:16:27.220,00:16:31.090 that back with a success message before it'll request the data. 00:16:31.090,00:16:34.460 And so the reason that handshake is there is to actually avoid 00:16:34.460,00:16:37.897 actually a kind of an attack scenario here where you pointing 00:16:37.897,00:16:43.903 it at uh an attacker control system however since we have 00:16:43.903,00:16:46.505 complete access to the source code and this is open-source 00:16:46.505,00:16:50.443 were able to actually generate that correct echo request back. 00:16:50.443,00:16:56.215 So this is what a example echo request looks like this get sent 00:16:56.215,00:17:00.720 in and the important part here is the event type here as long 00:17:00.720,00:17:05.958 as this value has that string uh it'll be accepted by the server. 00:17:05.958,00:17:09.762 And then following that we will so this is will send back our 00:17:09.762,00:17:13.666 actual export straight. So if we look in the throughput parameter 00:17:13.666,00:17:17.970 here we put back tick ho am I back tick. Because it's uh it's 00:17:17.970,00:17:20.506 executing pearl and it pearl you can put back ticks and it'll 00:17:20.506,00:17:25.811 drop to a shell. So here's our example exploit, now I actually 00:17:25.811,00:17:31.017 have a script to do that. And all os these scripts will be 00:17:31.017,00:17:35.688 available actually are available right now um if you have the 00:17:35.688,00:17:38.124 link. Alright so we have a simple server here it's gonna 00:17:38.124,00:17:41.861 handle all of the magic of sending an echo request and then 00:17:41.861,00:17:45.264 sending the exploit string. And if we actually pull up the graph 00:17:45.264,00:17:48.668 dot CGI you'd see we provided the key parameter in this case, 00:17:48.668,00:17:51.404 it doesn't matter what it is because we control the server 00:17:51.404,00:17:55.841 and the URL to access our uh server admin. And we don't see 00:17:55.841,00:17:58.177 anything interesting here on the page but if we actually look at 00:17:58.177,00:18:02.782 the source code we can see right here in the source code it's 00:18:02.782,00:18:07.653 printed out the results of who am I. So taking that a step 00:18:07.653,00:18:09.188 further um we can put a full this is a just a python PTY call 00:18:09.188,00:18:10.523 back so we can actually get an actual shell on the device 00:18:10.523,00:18:11.857 instead of having to run commands one at a time. We 00:18:11.857,00:18:13.225 refresh the page and we have a full shell now. Alright so you 00:18:13.225,00:18:18.230 see we're running as Apache. So same thing here. We want a cat 00:18:30.443,00:18:35.615 etc shadow. And it doesn't work again. So were kind of stuck and 00:18:35.615,00:18:40.619 having regular etc fun. But we want root etc. So. Back to the 00:18:43.289,00:18:45.658 presentation. We're gonna talk about now about how we obtain 00:18:45.658,00:18:49.795 root on this device. So if we actually pull up these perfsonar 00:18:49.795,00:18:53.633 tool case, it has the ability to change configuration settings. 00:18:53.633,00:18:57.136 You can turn on and off services uh such as bandwidth control and 00:18:57.136,00:18:59.772 O amp. You can change configuration for those. You can 00:18:59.772,00:19:02.308 change the default port or you could change what restrictions 00:19:02.308,00:19:05.077 there are for example you can change your bandwidth controlled 00:19:05.077,00:19:10.850 to only accept TCP or only set UDP uh performance tests. And in 00:19:10.850,00:19:13.786 order to start and stop services on Linux you need route. Um 00:19:13.786,00:19:18.557 unless you made special changes so somehow the application is 00:19:18.557,00:19:21.494 obtaining route in order to do this but if we go back to our 00:19:21.494,00:19:24.797 shell, we don't have pseudo-privileges and there's no 00:19:24.797,00:19:27.900 really easy way to find root there off of any file 00:19:27.900,00:19:32.238 permissions or anything else. So first we look at the source code 00:19:32.238,00:19:35.841 again all the way down. They have a Daemon running his route 00:19:35.841,00:19:40.980 called toolkit config. It's a simple XML RPC server it's only 00:19:40.980,00:19:45.584 running on looped back and it exposes five methods. It exposes 00:19:45.584,00:19:49.655 a config firewall method which accepts no parameters so not 00:19:49.655,00:19:54.627 anything exploitable there. It exposes a right file start stop 00:19:54.627,00:19:59.031 and restart service so right file looks really interesting 00:19:59.031,00:20:01.434 ideally we just write a new file. A new [indiscernible] 00:20:01.434,00:20:07.506 root. Now we have escalation. And so here's the example code 00:20:07.506,00:20:11.744 to do that. We say load in the config clients set it up to 00:20:11.744,00:20:15.815 point to the loopback interface and then call the save file 00:20:15.815,00:20:18.284 method which is an alias of write file. I don't know why 00:20:18.284,00:20:21.220 they changed the method name in different parts of the 00:20:21.220,00:20:24.590 application. Uh and if we try to actual run this, it doesn't 00:20:24.590,00:20:28.194 work. Uh we have another issue here too. And so we look at the 00:20:28.194,00:20:30.996 source code there's actually a white list of what files you're 00:20:30.996,00:20:33.933 actual allowed to edit. Uh so they put a little thought into 00:20:33.933,00:20:36.836 this and you know decided we shouldn't let someone write 00:20:36.836,00:20:39.772 arbitrary files as root. That's a bad idea. yeah. So they built 00:20:39.772,00:20:44.043 this white list, here are all the files in the white list. Uh 00:20:44.043,00:20:47.246 it's a rather extensive list because this is an extremely 00:20:47.246,00:20:51.951 customizable application you can install other packages so 00:20:51.951,00:20:54.753 basically any config file that never want to be edited as part 00:20:54.753,00:20:57.990 of this application is in this list and there's a couple 00:20:57.990,00:21:01.627 interesting ones in here there's etc host so we have the ability 00:21:01.627,00:21:05.998 to redirect network traffic. Um there is etc MTP uh so if we 00:21:05.998,00:21:10.469 have any sort issues we can change the time on the host 00:21:10.469,00:21:15.207 along with a bunch of perfsonar software so bandwidth control 00:21:15.207,00:21:19.145 around NDT we can edit all of those. We can also write HTML 00:21:19.145,00:21:22.781 files um since were apache so we could drop a cross eyed 00:21:22.781,00:21:26.318 scripting payload but again not very interesting, we want root 00:21:26.318,00:21:32.391 on this device. So if we look at the bandwidth control uh 00:21:32.391,00:21:36.328 configuration. This is an excerpt from it it's got a user 00:21:36.328,00:21:39.031 in a group so it drops privileges immediately after 00:21:39.031,00:21:43.435 running and then a post hook parameter at the bottom. And so 00:21:43.435,00:21:48.140 what this is what happens after a successful bandwidth control 00:21:48.140,00:21:52.244 test and execute the post hoc and so since we can edit this 00:21:52.244,00:21:55.247 config manager we can change the user and group so that the 00:21:55.247,00:21:58.184 application never drops privileges its running as root. 00:21:58.184,00:22:02.221 And then we pointed to a host hook to control our apache user 00:22:02.221,00:22:05.391 that way when we trigger a successful test it's going to 00:22:05.391,00:22:09.728 trigger our host hook parameter as root. So in order to actually 00:22:09.728,00:22:13.599 do that it's a little more complicated we don't want to let 00:22:13.599,00:22:16.802 the network administrator notice that something is broken so we 00:22:16.802,00:22:19.171 have to do this as quickly as possible and then restore it 00:22:19.171,00:22:22.107 back to its original configuration uh as quickly as 00:22:22.107,00:22:25.845 possible so we're gonna back up the original config stop 00:22:25.845,00:22:29.448 bandwidth control write out post hook. Write the new bandwidth 00:22:29.448,00:22:33.018 control config, start bandwidth control, trigger a session which 00:22:33.018,00:22:36.722 has to be successful which will trigger our post hook, stop 00:22:36.722,00:22:39.325 bandwidth control remove our post hooks so we delete our 00:22:39.325,00:22:41.827 evidence and restore the original bandwidth control 00:22:41.827,00:22:46.532 config and then start it back up again. So I'm going to actually 00:22:46.532,00:22:51.237 try to do that now. So we have our shell, we're currently 00:22:51.237,00:22:55.808 logged in as Apache. Uh I'm gonna pull down shell dot PM uh 00:22:55.808,00:22:59.545 which is uh a script I've written. And we're gonna run it. 00:22:59.545,00:23:02.348 So that's actually gonna take about 60 seconds to run. So 00:23:02.348,00:23:07.186 we're gonna look at what this is doing here. It's pulling in 00:23:07.186,00:23:10.856 again the config client we're loading in I don't know why this 00:23:10.856,00:23:13.592 has to be here. i don't write pearl scripts but it crashes if 00:23:13.592,00:23:17.229 it's not. Uh and here is the post hook that we're actually 00:23:17.229,00:23:21.400 writing. So we're gonna copy and binbash to a different value and 00:23:21.400,00:23:25.004 then we're gonna set UID on that binary so that whenever we run 00:23:25.004,00:23:28.507 it we can become root and then the rest of this is doing all of 00:23:28.507,00:23:31.410 that work of restoring the original config. Here's our 00:23:31.410,00:23:35.314 exploit config with the post hook parameter inside of it. So 00:23:35.314,00:23:40.319 if we actually go back to the shell… hopefully. Oh. [Laughter] 00:23:43.255,00:23:48.260 This is why you don't do live demos. Let's try that again. Oh. 00:23:51.230,00:23:55.467 We are root. Okay. It did work. [Clapping] Woo! Awesome. Now uh, 00:23:55.467,00:24:00.406 that's fun, we have root on these devices. Who cares? Is 00:24:08.614,00:24:11.250 anyone actually even running these things? So you know I 00:24:11.250,00:24:15.087 happened to stumble across this. It's an obscure application. Are 00:24:15.087,00:24:18.223 these running anywhere in the wild? Was my next step so. Next 00:24:18.223,00:24:20.993 goal is trying to find out where these are running. I don't have 00:24:20.993,00:24:24.496 an ISP that plays nice with mass scanning the entire IPV4 00:24:24.496,00:24:28.367 internet space so I had to find a nicer way to locate these 00:24:28.367,00:24:34.373 devices. So if you actually look at a example um here's a live 00:24:34.373,00:24:37.209 instance of one of these running. We can see there's all 00:24:37.209,00:24:40.212 sorts of of information here. There is.. this is 00:24:40.212,00:24:43.515 unauthenticated you can view all this. You don't need any creds, 00:24:43.515,00:24:45.818 you can see what services are running. WHat ports they are 00:24:45.818,00:24:49.188 running on and more importantly you can see the interfaces on 00:24:49.188,00:24:52.491 the right there so you can see information about uh if they're 00:24:52.491,00:24:56.195 connected or dual homed and they have or are connected to an 00:24:56.195,00:24:59.298 internal private network using the Mac address of the devices 00:24:59.298,00:25:02.334 and you can actually see the speed of the card according to 00:25:02.334,00:25:06.572 eat tool. So we can tell if there's a 10 Gb card inside of 00:25:06.572,00:25:10.075 each device without even authenticating it. The other 00:25:10.075,00:25:12.678 thing we have here at the bottom is test results. So you can 00:25:12.678,00:25:16.081 actually see what applications or what other hosts each of 00:25:16.081,00:25:22.087 these uh instances is testing against. So the idea is we start 00:25:22.087,00:25:25.257 with one of these nodes, we ask it who they're testing against 00:25:25.257,00:25:27.559 and then we ask each of those nodes who they're testing 00:25:27.559,00:25:30.396 against and we map the entire network that way. But we still 00:25:30.396,00:25:35.200 need some starting nodes so if only there was a nice public 00:25:35.200,00:25:39.972 database of all these devices oh wait if you look in the corner 00:25:39.972,00:25:44.410 up here, there's globally registered which is pretty much 00:25:44.410,00:25:49.581 exactly what you think it is. Uh, they provide a actual 00:25:49.581,00:25:53.652 database on their site all of the globally registered perfect 00:25:53.652,00:25:58.123 perfsonar servers. Also unauthenticated even has a 00:25:58.123,00:26:01.560 pretty web interface. So here's the idea we start with the 00:26:01.560,00:26:05.197 public list because they're still unlisted instances and we 00:26:05.197,00:26:08.901 map the network from there on so you can see the great 00:26:08.901,00:26:11.036 [indiscernible] ones represent ones that are publicly 00:26:11.036,00:26:13.372 registered but we can locate them through the other ones. 00:26:13.372,00:26:19.011 Alright so doing that, actually wrote a it's about a three 00:26:19.011,00:26:23.182 hundred line go line script um that does this exactly what I 00:26:23.182,00:26:26.885 just described. It starts to pull down a list of all publicly 00:26:26.885,00:26:31.223 um publicly registered instances maps them all asks them who 00:26:31.223,00:26:34.226 they're testing with. Maps all of them and it pulls down the 00:26:34.226,00:26:37.196 interface data for each of those. It takes about four 00:26:37.196,00:26:39.765 minutes to map the entire network from my gigabit 00:26:39.765,00:26:42.668 connection uh that could probably be improved. It's not 00:26:42.668,00:26:47.739 actually saturating that. Uh my code kind of sucks. But it's 00:26:47.739,00:26:53.545 open source, someone else can fix it. Um… [Laughter] So what I 00:26:53.545,00:26:56.548 actually do is kind of take all this data and put it into 00:26:56.548,00:26:59.918 Splunk. So Splunk didn't sponsor me or anything I just like 00:26:59.918,00:27:04.490 Splunk. Using all that as of April 29th, when I mapped the 00:27:04.490,00:27:09.728 network, there were 970 publicly routable nodes um combined to 12 00:27:09.728,00:27:16.635 point 51 TB of RAM across all of them and 29 point 8 5 THz of CPU 00:27:16.635,00:27:20.372 cycles across all these devices. Uh it's easier to understand 00:27:20.372,00:27:25.377 terms. The average node has 13 GB RAM and 12 cores at 2 point6 00:27:27.746,00:27:31.884 GHz. Alright so next we want to look at what the theoretical 00:27:31.884,00:27:35.220 network speed of this device is. So each of the included in that 00:27:35.220,00:27:39.925 data is the information on the network card on the box so I can 00:27:39.925,00:27:44.363 tell if the 10 gig or a 20 gig or 40 gig network card. So we do 00:27:44.363,00:27:46.999 all that and sum of those together we get the theoretical 00:27:46.999,00:27:51.270 bandwidth of the perfsonar network which is 5 point 719 Tb 00:27:51.270,00:27:56.575 a second. Now theoretical speeds are kind of lame and I really 00:27:56.575,00:27:59.344 want to know what this was actually capable of because you 00:27:59.344,00:28:03.248 may have a 10 gig card and only a five gig uplink. And I can't 00:28:03.248,00:28:06.118 find any way to tell that without exploiting your server 00:28:06.118,00:28:11.990 which I like not going to jail. Um however I have an idea here 00:28:11.990,00:28:15.527 so. So I have a gigabit connection at home that can run 00:28:15.527,00:28:19.197 bandwidth tests from my server to one of the perfsonar 00:28:19.197,00:28:21.934 instances and find out information about their uh 00:28:21.934,00:28:25.270 bandwidth. BUt that has an upper bound, I can only find out up to 00:28:25.270,00:28:28.607 a gigabit a second since I only have a gigabit uplink. I'm not 00:28:28.607,00:28:31.910 about to go pay for a 40 gig uplink in order to test these 00:28:31.910,00:28:36.381 vulnerabilities. So I had to find some other way. Turns out I 00:28:36.381,00:28:39.985 have another friendly unauthencated API where you can 00:28:39.985,00:28:44.489 say run a bandwidth test against a different perfsonar node and 00:28:44.489,00:28:50.562 send me the results. So the goal here is actually uh enumerate 00:28:50.562,00:28:54.766 all the perfsonars and the Maximum interface speed 00:28:54.766,00:28:59.237 calculate their location based on GeoIP and then find the five 00:28:59.237,00:29:03.842 closest instances uh that have the same or faster network cards 00:29:03.842,00:29:08.747 within them. And then after all that's done we want to run tests 00:29:08.747,00:29:12.951 between them. And this sounds like some horrible messed up CS 00:29:12.951,00:29:17.189 interview question uh I can guarantee you I did not 00:29:17.189,00:29:20.092 implement this very efficiently.Uh there's the 00:29:20.092,00:29:26.064 Splunk query uh that does all of that. It works. [Laughter] It 00:29:26.064,00:29:31.069 takes like an hour to run but it does return results. So, once we 00:29:34.039,00:29:36.274 have all the data we actually want to run these tests. We have 00:29:36.274,00:29:39.544 to be careful here when running these tests because we actually 00:29:39.544,00:29:44.016 have the risk here of generating a denial of service when running 00:29:44.016,00:29:47.552 these tests. Um so we have to be careful we only want to run two 00:29:47.552,00:29:51.957 tests at the same time ever. Um we never want to run more than 00:29:51.957,00:29:55.560 10 at the same time ever and we never want to run two tests on 00:29:55.560,00:29:59.398 the same instance so if you have a ten-gig uplink but I run two 00:29:59.398,00:30:02.868 10 gig tests against you they both gonna get like five games 00:30:02.868,00:30:06.605 which is inaccurate I want only run one at a time and then some 00:30:06.605,00:30:09.408 hosts don't have bandwidth control enabled so well I know 00:30:09.408,00:30:13.445 they're exploitable I can't find out what their bandwidth is. So 00:30:13.445,00:30:16.782 we're actually losing out on some data here about hosts that 00:30:16.782,00:30:19.618 if we're exploiting this for real we would have been able to 00:30:19.618,00:30:22.054 attack but we can't because they don't have bandwidth control 00:30:22.054,00:30:28.126 enabled. So doing all of that which takes a very long time to 00:30:28.126,00:30:32.497 run um I was able to calculate and the actual demonstrated 00:30:32.497,00:30:38.970 total bandwidth the perfsonar network which is 3 point 7 Tb a 00:30:38.970,00:30:42.040 second. Um now in the title of the talk, I mentioned four 00:30:42.040,00:30:45.977 terabits, I didn't just round. Uh I did account for all of 00:30:45.977,00:30:49.548 those instances that don't have bandwidth control enabled but we 00:30:49.548,00:30:54.753 know that they're sitting on at least a 100 Mb uplink. And that 00:30:54.753,00:30:59.291 combine that altogether you get the four terabits a second. Now 00:30:59.291,00:31:04.196 the fact that I'm calling it uh um so they're… excuse me. Uh so 00:31:04.196,00:31:11.002 if you've if you are around last two years ago now um Cloud Flair 00:31:11.002,00:31:14.172 blocked an attack in Europe against Spam House. Uh it was a 00:31:14.172,00:31:18.710 three hundred gigabit a second attack and they had an 00:31:18.710,00:31:21.646 interesting effect they were seeing where where some of their 00:31:21.646,00:31:25.751 network can handle the uh the traffic with their upstream ISP 00:31:25.751,00:31:29.721 [indiscernible] falling over and that's one of the risks here 00:31:29.721,00:31:33.558 when you have that much bandwidth and that was 300 Gb a 00:31:33.558,00:31:38.096 second given that was two years ago this is four terabits a 00:31:38.096,00:31:42.033 second and we have complete control of the packets being 00:31:42.033,00:31:45.003 sent because we are root on this device this isn't something like 00:31:45.003,00:31:48.940 DNS amplification where you know if you have the right firewall 00:31:48.940,00:31:53.345 rule you can block traffic this is you know I could send you 00:31:53.345,00:31:56.715 four terabits a second of legitimate HTTP requests 00:31:56.715,00:31:59.751 assuming the network cards can push that out you know it's 00:31:59.751,00:32:01.953 really hard to filter something like that because it could be a 00:32:01.953,00:32:06.391 legitimate traffic. Uh given there are actually some 00:32:06.391,00:32:09.795 interesting ways to defend against stuff like this. Alright 00:32:09.795,00:32:14.766 so onto the live demo. Hopefully we're gonna take down the site 00:32:14.766,00:32:19.771 here. Not someone else's site. Again. [Laughter] So the initial 00:32:21.873,00:32:26.845 version of this uh talk I uh I have a couple perfsonar 00:32:26.845,00:32:30.115 instances running at home and I was planning on attacking a 00:32:30.115,00:32:34.753 server collocated in the data center and I launched the attack 00:32:34.753,00:32:39.291 while doing rehearsals and my phone blew up because I crashed 00:32:39.291,00:32:43.195 the network at the house and there were about 18 dudes pissed 00:32:43.195,00:32:46.364 off at me that their Internet didn't work in about 10 minutes 00:32:46.364,00:32:49.501 later I got a letter from the ISP saying please stop doing 00:32:49.501,00:32:55.207 that. [Lighter] So we're gonna cheat a little and were gonna 00:32:55.207,00:33:01.713 attack from Vm's here. Uh so we have a simple server HTTP server 00:33:01.713,00:33:06.718 on poncho here um I'm gonna download a simple ddos script 00:33:10.121,00:33:13.992 and hopefully it's really hard to see but that is just sitting 00:33:13.992,00:33:18.997 there spinning right now so ta-dah [Laughter] [Applause] 00:33:30.742,00:33:35.413 Alright. So on to the uh last part. Um so I reported all these 00:33:35.413,00:33:39.351 issues to perfsonar uh sorry to disappoint you, you can't 00:33:39.351,00:33:41.920 actually go exploit these right now though I would highly 00:33:41.920,00:33:45.156 encourage people to continue looking at this software it is a 00:33:45.156,00:33:48.827 legacy Pearl application I don't think I found everything by any 00:33:48.827,00:33:51.897 means um I kind of stopped once I had a full chain you know all 00:33:51.897,00:33:56.201 the way to root and it it is interesting and they are very 00:33:56.201,00:33:59.437 responsive so this is one of the pull requests since it's all 00:33:59.437,00:34:03.275 open sourced and I just fixed the issues myself um and the 00:34:03.275,00:34:05.911 team was extremely friendly. They fixed the issues, merged my 00:34:05.911,00:34:09.748 request within 24 hours and pushed out a new build pretty 00:34:09.748,00:34:14.552 much immediately and the great part is all these app, all these 00:34:14.552,00:34:17.589 instances have auto updates enabled so pretty much everyone 00:34:17.589,00:34:19.858 on the network is upgraded at this point. That was about a 00:34:19.858,00:34:24.296 month ago that build got pushed out so when you do find security 00:34:24.296,00:34:26.998 issues, they typically are patched very quickly by them. Um 00:34:26.998,00:34:29.968 so that's great. Um, I was very happy with the responses made by 00:34:29.968,00:34:35.106 them. And then finishing this up, the exploit code has been 00:34:35.106,00:34:39.811 released on my GitHub, uh along with the slides. Um you can also 00:34:39.811,00:34:45.116 go to that URL, bored dot engineer. Has my uh has links to 00:34:45.116,00:34:48.620 it right there if you don't remember that as promised here's 00:34:48.620,00:34:53.091 my contact info again we got out a little early this time so you 00:34:53.091,00:34:55.627 have some time to make it to your next talk. Uh if people 00:34:55.627,00:35:00.565 have questions, feel free to uh… [Applause] Ahh, that was a 00:35:07.205,00:35:10.275 really good question. He asked what I spent five dollars on. I 00:35:10.275,00:35:14.512 put it in the talk title . [Inaudible comment from 00:35:14.512,00:35:16.514 audience.] >> I could repeat the question. The question was what 00:35:16.514,00:35:19.284 did you spend five dollars on? That's a very good question, 00:35:19.284,00:35:22.420 it's in the talk title again in the initial version of this 00:35:22.420,00:35:26.658 application I was going to spawn up VPS instance for five dollars 00:35:26.658,00:35:30.829 and then launch an attack live across the Internet and then 00:35:30.829,00:35:35.033 course my ISP got very angry about that so I did not update 00:35:35.033,00:35:40.038 the title unfortunately that's what the five dollars is from. 00:35:43.408,00:35:44.776 >> [Inaudible comment from audience.] >> Total time spent 00:35:44.776,00:35:47.045 uh actually finding the exploits is probably like 10 hours and 00:35:47.045,00:35:48.380 then writing reliable exploits for them is probably like six 00:35:48.380,00:35:52.717 and then mapping the network was a colossal pain since I'm not a 00:35:52.717,00:35:58.490 stats person, and figuring out how to write those queries 00:35:58.490,00:36:02.961 correctly sucks so that, that was probably another like 10. 00:36:02.961,00:36:07.966 Forty hours roughly total. Thank you all, have a great DefCon. 00:36:15.140,00:36:18.343 [Applause] Thanks.