>> Okay.  Welcome everybody.  Before starting first of all, yes, you are not in the wrong track.  This talk about is looking glass mostly.  Not (inaudible) but how to access them by exposing something else and in this case it is why the title.  We are looking about looking glasses.  I'll tell you what looking glasses are if you don't know them and what we found in our research in order to exploit them.  Actually I'm not Emdel I am Luca Bruno and this is my colleague Mariano Graziano.  And we're going to talk you through this presentation.  I'll talk about the first part.  And it will give more details about (inaudible).  
Before starting we are both coming from Europe.  So we are researcher in Eurecom in France.  We are researching several topics in general which is in security.  I'm mostly into imbedded systems, networking devices in general and physical infrastructure.  While Mariano is a PhD student his topics are mostly malware and forensic stuff.  Here is currently in the U.S. here at Cisco doing internship.  I give you a small presentation about us so let's get into the case. 
First, I know this is going to be a bit of a bizarre talk because this is not something that you see every day.  I mean, looking glasses are not so commonly used.  And the reason of this research is also a bit bizarre.  So let me guide you through why we did this.  Imagine for a couple of minutes that actually you are an attacker.  Some kind of very lame attacker.  You are starting your first test in the offensive part of security and you are looking for your next target.  You are a bit lame in the sense that you want to go big so you are looking for something that is, let's say, quite a big impact.  But your skills are not so high.  So let's say you are looking for something that is some kind of critical infrastructure or global infrastructure where the impact of what you can do will affect a lot of people in different geographic areas but you don't have a lot of skills.  And you also don't have a clear goal right now.  Actually because we're not getting into a lot of expectation but we have just the general idea you want to cause a bit of damage. 
So let's think about this.  Let's say that we are    we want to target the internet because it is the most used infrastructure around the world.  Everybody is relying on that for economic, personal and whatever activities.  If you want to target something of the internet that is catching a lot of region encounters around the world maybe you want to target actually how the traffic is exchanged between different and several autonomous systems around the world.  So we'll talk a bit about internal system.  Routing.  I told you already that we imagine our skills are not so high.  So let's say that you are some kind of webs (inaudible) or something like that.  We know the basic of scripting, general injection.  We know a bit of Google (inaudible) and some more kind of injection in general and common injection.  And we still don't know what we want to do once we get to our target.  But in general we know that some kind of interesting targets will be the backbone of an operator.  So in general we want to go with our limited skill in the (inaudible) to something that is like a backbone or another autonomous system.  So what you are going to do is not attacking directly the route or the routers.  We are not coming up with a zero day for Cisco or Juniper.  Instead when we went through the mental process we found one really good candidate which is a looking glass.  Looking glass actually something that I will explain to you later but fits well in this model.  Where it just requires a bit of web knowledge in order to get big.  So I hope that everybody here know more or less what is a looking glass and how to use it.  If you don't know I will give you a really brief presentation of it.  So let's go to an introduction about looking glasses.  
First, let's talk about the internet.  Everybody is using the internet.  But actually the internet is not a single net or not a single entity.  The internet's composed we can describe it as a network of networks.  It is in fact composed by a lot of entities    commercial, noncommercial, private, university, government.  The internet in general we have a lot of operators.  We call them alternate systems.  AS.  And each of these autonomous systems have its own network.  There are different kind of networks.  Access networks, depending on what your target and what your economical business is.  And all these networks are in somehow interconnected by one protocol.  This is the BGP.  It is what is governing how the traffic is flowing across the internet.  There's something like the diagram that you see here where a lot of ranges, a lot of IPs are interconnecting and there is connecting system A to B passing through system B one, B two or whatever.  Let's keep in mind we are BGP that is governing all of the internet and let's see how in practice this routing works.  We normally talk about a BGP routing table that is something that we tell you okay in order to reach that IP address you have to go through so many alternate systems.  That's fine.  But in the case actually the reason a single BGP routing table.  What we have really is that every autonomous system has its own policy.  Has its own decision making process.  Also, there are different algorithms and different way to calculate which is the best route in order to reach your target.  So what we actually are seeing is something more complex where every single autonomous system has a local view, a local routing table that is used, that traffic is following in order to reach the destination.  But as you may imagine, different autonomous system may have a different looking view what is the routing table.  So there could have been some interesting situations.  I am not an expert in BGP so I am not going too much into details, but what can usually happen is some kind of anomaly where for example the same graphics could be announced by different system by error or order to attack.  I have one example here.  This is courtesy of (inaudible) analysis.  For example this is the incident where (inaudible) was announcing the graphics.  As you may see, the same graphics was being announced by different systems.  The current one of people on your right, the blue one.  And another one that was by mistake announcing the same graphics.  In this case some of the autonomous systems the one marked in yellow, red and orange were seeing the announce and following the announcement.  So whenever you were trying to connect to PayPal you were reaching India.  And in the other side there were some system that because of proximity following the correct amounts of people.  So you have some kind of semantic or non-uniform routing.  Imagine that you are the center or engineer of one of these autonomous systems in the middle.  One customer is calling you and telling you, hey, look I have some problems reaching Paypal.  You are a engineer at PayPal and somebody is telling you I cannot reach PayPal.  What can you do about this situation?  If you are PayPal and you know that your announce is correct, what you can do is trouble shooting.  Everybody does trouble shooting and de bugging of connectivity issues.  You need tools to do that.  Because you are now in a situation where your (inaudible) is fine but something else on the other side of the world is doing something bad.  And another, a third party, autonomous system is having problem and you want to de bug the situation.  You need tools in order to do that.  In particular you need something in order to get view on the remote status.  We can come up with several tools and several solutions.  One, is something like setting up a community, a distributed network of probes.  So let's say that we are four autonomous system.  Each one we are receiving different analysis and different BGP information.  We all collect this information in a central location and there is some kind of service that's is exposing all this data and keeping chronological track.  For example, we're coming from Europe and the RIPE labs  this kind of services in the right where you can look at the history of analysis from multiple points usually at public exchange.  Another solution that you can come up is some kind of private agreement.  For example, summaries like the some, ring where you participate in these ring and once you are part of this ring you will provide one system usually a server inside your autonomous system and giving us access to this system to other network engineer in other anonymous system.  In exchange you get the same access somewhere else so that every time you have a problem you can log in to a remote system, multiple remote systems actually, around distributed for example, trace roots and compare them to see what is going on and where the traffic is being lost or whatever.  
The third idea actually something more general.  So let's give the whole public not only network engineers and not through central point of failure.  Let's give the large public access to the net through our routers.  As you may imagine this is something that sounds a bit scary in the sense that you need some kind of web servers that is directly exposing some limited feature of your routers in order to do the debugging and get information on BGP.  So actually we're going to focus on this, this third tool, it's called a looking glass and focus on this software and see how does it work, what are the vulnerabilities that we found?  We also found some incidents.  Let's go through it because it's scary stuff. 
So what's in a looking glass first of all?  A looking glass became famous let's say in the 90s and in the first years of the 2000's They are web script in the sense they're usually composed of one file, couple of files, a bunch of files.  Nothing very complex.  Not a big architecture.  This is something that you just download, unpack and put in your web server root and forget about it.  They're usually written in Perl a lot are written in (inaudible) because it was the 90s and some are written in PHP because everybody loves PHP.  How do they actually work?  They establish a direct connection from the host that is hosting a looking glass to the router.  SSH, or whatever methods you have to administer your routers.  And obviously you need some kind of credential in order to log into a router.  So how does it work?  You have a config file with credential and whatever on the same host.  And the web will use this information to connect.  As you may imagine this is something that is becoming scary and scary as we go on.  Because if something goes wrong from all this stuff, okay you have a hole.  How does it work in practice?  Let's say that we are an operator the autonomous system on your left and you want to provide this kind of looking glass access to the rest of the world.  What you are going to do, like already have our routers, the three routes on the left which are handling our internet and the connectivity and everything else.  We then add another system.  Traditionally a Linux system that is the bottom one on the left.  This system actually has two legs.  Legs for links.  One links is going on the right to the internet.  The public internet.  This is where you expose your web access.  This is actually where the other operator, the other engineers that for example the two on the right are going to connect and use your web interface, your web service.  On the other side in the same host is another leg which is going to the left connecting this service to the private part of an operator.  Called the private network.  The internal system where you usually have all the console and administration services that you don't want to expose to the whole world because they are a bit critical and you don't need to do administration of them through the internet.  So actually this is something let's say that the looking glass is a bit critical because it sits at the border between the public part of your network which is fine something that everybody can reach and the private part where actually you have critical services.  How does it look like?  I hope you love the 90s because it is, like, clear, simple 90s interface.  Just have the minimal that you need in order to select which kind of program to run.  For example you have connectivity issues, you want to run ping and trace through prong system to your own IPs.  Or you want to see how the analyses are going so you can see some kind of advance or reduce in summary of the analysis.  Then you usually have one form where you can input some other additional parameters like which host, which IP address do you want to ping or trace it to.  And what I was showing before in the slides is that one looking glass can be connected to multiple routers.  Actually all the routers in a single autonomous system.  So you can select which router you want to run.  Imagine that you are geographically distributed so you are multiple regions and multiple exchange points.  You have routers at all the exchange points.  So you want to run comments on specific routers and specific geographical locations.  What we did in this research is okay let's focus on what we can actually review and check.  There are a lot of custom looking glasses.  So imagine that your internal system a looking glass is quite simple.  Just the web service that is connecting to SSH and (Inaudible) so you can write your own.  This is a bit difficult for us to review because we don't have the code and we cannot set it up in your own lab.  So in order to try to assess we have to attack an autonomous system and it's not legal.  What we did instead is take at least    there are several of public looking glass around the world, crawl through them in order to see if anybody or if a large majority of them are using open source software that we can download.  And actually the answer is yes.  There are several.  In particular there are these four that are mentioned here that are quite common.  Probably because they were the first to be written.  And they were highly needed and a lot of operators were just downloading them and deploying them.  As I told you already, a lot are written in Perl and (inaudible) is one we love written in PHP.  What we did in the rest research and Mariano will show you later is reviewing the code of these looking glasses and also looking for all we found vulnerabilities then looking how widespread deployment was and which kind of incidents we found. 
Before proceeding farther just to frame a bit how we're going proceed Mariano will show you several incidents.  We started from the base like assuming this is bug proof.  The administration can makeovers.  Let's say you are to deploy a web (inaudible)plus CGI but forget to enable the CGI or forget to protect your configurations and SSH keys.  This case we may see some exposed credentials.  There are routers credentials.  Then go one step farther.  Let's say that we try to find bugs in web application.  We look for the normal, usual code breaks.  Let's imagine that we can find some kind of (inaudible) shifting.  It could happen.  In this case we can use it for normal stuff like attacking other web services on the same site.  Some of the looking glasses may come with additional model.  For example they may come with additional malware for example, binaries or forked/modified modules.  If you find bug in these modules you can attack that through the host.  Not the software itself but the host. It's the one that is actually storing the credentials.  We already have a step stone into the private side of an operator network and we can also steal the credential.  Then stuff gets really scary because okay let's focus on the looking glass.  Looking glass is connecting to a router by allowing just a limited set of comments.  Let's see if it is really a limited set of comments or if we can inject whatever comment we want directly to the CLI of a router.  If we can imagine to do that we are escalating from a basic web attack to where we are attacking critical infrastructure.  And as a side case of this, let's say that, okay, we can do that.  We can log in and control multiple routers.  We can attack multiple autonomous systems.  We can actually change different vulnerabilities.  So the one that we found is the one already known.  To actually attack the network.  Once we are in the router, we can try to escalate and change the modification of the router in order to break the local a hook of the local traffic so the routing changing the local topography or announce bogus preferences to have more impact.  Some of you now may think this is for sure not possible because you have best practices.  We all know that BGP are (inaudible) at the border and whatever.  In theory yes that's correct.  But what we're going to show you through this presentation is best practice exists they are well written but often are disregarded.  We also have another paper that is describing more details on that and we actually believe that this kind of attack is feasible with what we have found.  But for clear legal motivation we cannot perform that.  Sorry for bugging you with this big introduction.  Let's get into the meat of it.  Mariano will go through a incident.  That's a scary part.
>> Hello everybody.  I'm Mariano.  
(Applause)
>> Okay.  Today I will talk about our finding.  That Luca told you are more on the defensive side.  Okay.  More on the (inaudible) side.  And in particular what during our research we have found several incidents affecting the most common deployed of looking glasses.  So okay if remind the definition of a looking glass.  Looking glass is something related to the web.  So it's normal the first kind of problems come from the web.  When we talk about web issues, we have two kinds of problems.  The first one is related to web misconfiguration and the second is the more related about the common web vulnerabilities.  Now I will give you two quick examples.  One about the web reconfiguration and one about common web vulnerabilities.  For example one very looking glass it's possible to retrieve the router credentials.  Give the IP address, user name and password.  Then it is possible actually because the configuration file is stored at the known address.  With the wrong permission. For example for common web attack we found a couple of (inaudible).  I know this sounds like a joke.  I mean, what can you do with a (inaudible) looking glass.  Actually sometimes with the device people buy them and then set up under the same (inaudible) other application.  Let's talk more in the case about web misconfiguration.  During our research we were able to find a couple of configuration files with Google Doc.  They are really simple.  You can see two very basic examples.  Once that you know the configuration file (inaudible).  Now let's have a look at the real Google Doc here.  As you can see here in the output, you have you have developed credentials.  This means the IP address belonging to the IP address belonging of course across the web then you have the user name and password.  Now, of course, you have the output is already in the Google query because of the looking glass is very confused and has been indexed by Google.  Let's look at the configuration directive of these Google results.  As you can see on the left side, on the right side, sorry, you have the configuration directory basically is we can lift it because it is very config ratable.  The configuration files then another interesting file is the log file for the looking glass.  On the left side we have the real configuration file of the system.  In this case we have a case for the cougar looking glass, is one of the most deployed.  We have one entry for every router exposed in the web interface.  Every router entry needs a router name.  And then, of course, the credentials.  The connection meter that can tell it the stage, NHS, SSH, RHS.  Then the user name and the password login.  Another problem related to web misconfiguration is the full configuration file.  For example if we don't load one of the most common deployed looking glass and we have a look at the root directory, we have several files.  Here it's really easy to spot the configuration file.  The LG.com.  Once that we know the name, basically we can just comb the web.  It's really easy to have a list of all the available looking glasses.  And basically you can create easily your own (inaudible) to check for the big address of the big looking glass address then the name of the configuration file.  If you are lucky, you will find several entries of data (inaudible) because they are readable and you can download them.  
Of course, there are best practices.  So if you ever look the reading file of all the read me files of these looking glass results, (inaudible) to warn the net operator about the security location of the configuration files.  For example, it's written clearly make sure you (inaudible) on the configuration file since it contains your password.  Hope you get it.  We prove some guys didn't get it and they were found about 35 exposed in which you easily are able to retrieve configuration files and it means the credentials.  Then let's have another look at the findings on our research.  We have a few well known websites.  One is the firm.  One is the governor.  They are some looking glasses.  Everybody here in the audience knows the (inaudible).  It is also quite famous in Italy.  The network institutions and universities.  We were able to retrieve the (inaudible) of the looking glass.  Basically common for everyone.  While for the gov, this is not the real looking glass.  An old one and it's not currently deployed.  But still we can access this custom PHP code.  If you look at the code, you can find several security issues and give you idea about the problems that a looking glass can have. 
Let's continue.  I just always related to the web configuration is related to the expose virus of the SSH keys.  If you ever look at the configuration file or one of the common looking glass software, the connection meter is for example SSH you can have a configuration file of the configuration file.  If you create your own code on that you know of course by private keys, and you start calling the web for the looking glass for the website, you can find  several.  For example one of the keys is here in this slide.  On the top part of the slide you can actually have the private keys of these SSH router on the back one.  Then if we for example try to read the dot SSH directory again we have a configuration problem we can list the directory and it's readable.  We have several files, of course private keys.  Then the client SSH configuration file and also the public key because, you know, I mean, we never know.  Then let's have a look now    let's focus more of our attention on the web.  I remind you now the definition of the looking glass is the web application that is directly connected to the backbone router.  
And they may (inaudible) on the web.  When doing our research, we have found there is a total lack of prevention of possible abuses.  In particular for example there are no capture at all on these softwares.  This means an attacker can easily automate several tasks.  Such as example, mapping, automate the command injection even for example attack from several devices a victim from a different system.  And I know this sounds really lame, but for example, create your own let's say fake dos botnik using all these devices because you don't have any capture and you can send comments and for example (inaudible).  
Let's have a look at the first kind of vulnerability we have found.  Simple cross site scripting.  Here the problem is in the HTML tag cycle and the address meet parameter.  It is directly taken from the web phone.  I know some people can complain about the cross site scripting.  But, again, some network operations services can run under the same origin domain and for example you can steal cookies and execute around Java script code for whatever purpose you have.  Here is an example.  You can see here we have the looking glass interface.  In the command we can see the address parameter.  That we have of course for example we are trying to trace.    
(Cheering) 
>> Okay.  Continue.  Sorry.  So here we have the parameter.  As you can see problem with the cycle.  Then you can inject your Java script code then execute it and do whatever you want basically.  Now this is the big part of our talk.  We have here a router common injection.  The looking glass definition we can easy understand that (inaudible).    
>> Now stop.
>> Oh now I have to stop.
>> Does anybody know what's going on?  
(Cheering) 
(Applause)
>> You do?
>> What?
>> Raise your hand if you are a first time speaker.
>> Raise your hand.
>> Welcome the new speakers!  Cheers!  
(Applause)
Good luck.
>> Thank you.  We can continue?  After the shot.  Okay.  It is able to execute code from directly from the web interface on the backbone of the console.  This is the case on one common deployed looking glass software in PHP.  Here the problem is again the algorithm parameters.  The algorithm is the parameter taken from the (inaudible).  Here they also have this looking glass as kind of a confusion between the HTML case meaning and (inaudible).  But here we have a look at the code.  Open it readable.  You have these algorithm parameters taken from the web phone that goes directly at PHP function.  They can remove basically removes all the spaces.  From the variable.  Then this parameter (inaudible) the function.  This is a simple wrapper around HTML, HPH function.  Try to translate all the possible traffic in HTML entities.  Then basically the argument it is directly centered with the backbone router.  So it is a simple, straighter HTML escape and your common comments from the web go directly on the backbone router.  Here you have the cool proof of concept.  As you can see here you just have to keep in mind that for example to calculate comments for example you use a new line.  But it's important to notice that the destination depends on the route of software so you can use whatever you want.  Depends on the router.  
Here the comment is for example data on our lab environment. Okay now another kind of bug we found on our research is the remote memory bug.  This kind is the problem that sometimes the looking glass has some other third party software embedded in  their config rating, installation package.  For one of them deployed looking glass, as the utility called fastping.  It's been written by a student.  Not me and Luca but two other students of course.  And here the fastping is a third party binary, the equal reply is not properly sanitized.  And we have memory corruption bug.  Again let's try to have a look at the code.  Okay.  First introduction I'm sorry but the code is written Italian.  And the comments in Italian. I hope you are familiar with Italian. The idea here is that you have these functions that is called (inaudible) but means fill tables.  This function has index 2 parameters, an index and a delay.  Okay.  Then we have a general of global array that is called delay that has the length.  Then this function basically applies directly to these global array using any index that comes from a field of the ICMP equal reply.  Of course it not provide any index (Inaudible) of fix it length of the memory corruption.  
The expectation here is the following.  We have a third party binary.  That sometimes okay is in the defaulter installation package.  But, of course, then the network administrator can decide not deploy it.  And the (inaudible) is no longer maintain and these are reasons why the current order of the looking glass software won't fix the issue.  Let's talk about the real exploitational part.  We keep in mind we are dealing with delays.  So the exploitation is time dependent.  The problem here is that okay we can basically have a work round about this problem.  Because from the request we have a field and we can (inaudible).  But then the other is that every time that we run a (inaudible) we are sending 100 basically ISMP equal request packets.  So the attacker can send up to 100 ICMP equal replies.  So, basically, we can write one long parameter where 100 the longest parameter in memory on the target.  The point is it's not trivial to subvert the flow of this application.  I still have a bet with Luca.  We are at Defcon so I will give you the challenge guys.  We are not providing any code.  Now I talk more about the web misconfiguration and that this kind of issue related to web Mobility.  But we also have other kind of problems.  For example basic configuration network design.  These are because for example network operators basically make the router interfaces public.  We can easily from the looking glass get IP address of these exposed routers.  We can for example with nmap to have a look at what services are running on these routers.  And we can see that we have the exposed the SSH and the (inaudible).  Now Luca will talk more about some fixes and possible mitigation of all these problems.  
(Applause)
>> So that was actually the cool fancy part.  We find out how to inject commands basically, vulnerabilities and incidentally in general we found a lot of disregarded best practices.  There are a lot of counter measures for these kind of issues.  So I see we can split it in three basically.  The first one is looking at the code.  The open source stuff in generally web code.  We have to dilute a lot about the code.  Then we have to improve our deployment methods.  And if you are a network designer, a network architect or network engineer, please apply some best practices.  First, let's start, everybody, understanding and acknowledging that if you are a web service that is exposing some kind of connection to a router, to some critical service to the web, it is, like, a dangerous case.  Some kind of security target and we must know it and have a proper look at it.  In particular, looking glasses are just one example.  There are a lot of services like that which have been written in the 90s usually.  And has been forgotten since then.  Okay it's 2014, 2015, let's have a look from the security point of view to the code and review it.  From the deployment side, we can do some steps in order to improve the situation.  For example, there are a lot of cases where you don't need to connect this kind of services between your physical infrastructure.  We found a lot of looking glasses that are connected directly to backbone routers and operational routers.  You can set up looking glasses in other ways.  For example, provide just one system that is mirroring all of the roots that you know and is providing just ping and trace or some kind of sound box or playing environment that even if a hacker can get into it is not a big problem.  Then, once you have deployed this kind of service actually check that everything is fine, that one public attacker cannot reach it over the web.  Or your SSH key.  Even if you did all of your configurations perfectly fine, check for other stuff that is on the same host.  For example we found some cases for the looking glass was perfectly configured.  But if you were accessing the same host through the IP or normal default dock root you were able to get to the same files.  The last point, if you are a network engineer set up proper ACL on your routers.  A looking glass doesn't need to have a root connection to your router.  We found root possible in a configuration file, a looking glass user just needs to run a couple of comments.  So set up a proper ACL just to do that.  Then a router consults have not been tough as possible public access point.  So there are some bugs.  We know there are a lot of bugs.  Sometimes very weak.  It's easy to crack them.  Use strong password.  Don't reuse passwords for them.  I don't want to see anymore SSH internet services exposed through the web.  I will show you some screen shots from our own connection.  We were able to connect to a router SSH console.  Nobody in the world needs to do this kind of backbone router administration from home.  Eternal system have their own private network and VLANS.  So let's use them and put administration interfaces there. 
	The talk is basically over.  In order to conclude everything, what I want you to take away from this talk is that, okay, there are a lot of best practices around.  We all know this.  But best practices are often disregarded.  We found a lot of cases where everything was basically exposed and this kind of critical software were not reviewed, sometimes were forgotten and there were clear bugs. We didn't show any advanced degree bugs.  So beware.  They are sitting everywhere in critical places.  Let's review.  Let's check them.  And something that we want is also this talk was probably a bit lame.  We didn't show any real advance.  But an attacker doesn't need something really advanced.  Attackers go through the weak link.  Looking glasses a really weak link.  Very simple, mostly forgotten and in critical places.  Once an attacker gets them it's easy to escalate.  We showed you can get from the web to a basic attack.  Something that find really interesting as part of this research is that I really understood the internet core, there are so many pieces and so many critical security flaws around that I am still amazed the internet is real running up to date.  That was all.  We are available for questions later outside of this room.  Just want to thank a couple of other guys from Europe that helped us and discuss add bit it and some very cool students bug finding and overall in the whole research.  So thanks to you guys from the NOPS team.  And that's it.  
(Applause)
Thank you.
>> Okay.  There are a couple of back up slides if you want to see them.  We found actually the partition is not that difficult because known bugs exist in a router.  For example Cisco there are weak issues all around.  Type five has basically just MD five.  And type four where there is actually critical bugs.  There were actually critical bugs.  Then once you have access to our router you can escalate quickly.  From Cisco and Jupiter existing known bugs to get from a normal user to a root user.