Alright. So this is Hacker Machine Interface, state of the union for SCADA HMI vulnerabilities. Um it it it's not uh you know I don't want it to the title slide here to indicate that this is about stuffy shirts and racks. I mean this is about hardcore exploitation in this talk. Um we're gonna cover an in depth analysis of a corpus of 200 plus uh confirmed HMI's vulnerabilities that have come through the zero day initiative program. We're gonna detail out the popular vulnerability types that have been discovered in these HMI solutions uh and we're gonna talk about how they're developed and and how the weaknesses actually manifest in the underlying code. We're gonna talk about some of the biggest SCADA vendors that exist on the planet including Schneider Electric, General Electric and Advantech. But the the all of the vulnerabilities that we're gonna be talking about can be applied to pretty much every SCADA vendor that's out there today. This talk will also cover uh and compare a pa uh a time type of vulnerability that we're gonna be talking about. We're gonna talk about how to patch performance for the various SCADA vendors in the industry and we'll also compare the SCADA industry itself against the other software other entities in the software industry. And finally we're gonna use the data that we presented uh to provide you additional guidance on what SCADA researchers should be looking for in HMI solutions and what we can expect in future attacks against SCADA HMI solutions. But first let me let's uh introduce ourselves so you know who we are. I'll let Fred Fritz introduce himself. Good day, my name's Fritz Sands. Uh my Twitter handle is Fritz Sands because I'm occasionally boring. I was a long time developer of Microsoft, 25 years in uh the Windows operating system and then I joined the trustworthy uh computing and secure Windows initiative in 2001 when the big security push happened to Microsoft. I uh left Microsoft in 2014 and joined the zero day initiative where I have been investigating software in the real world which has given me a deep appreciation for the code quality at Microsoft which I did not have when I was there. Uh so I'm the senior manager of vulnerability research in Trend Micro's Tipping Point organization. Uh my primary responsibility in this job is to actually run and manage the zero day initiative program which represents the world's largest vendor agnostic bug bounty program. Uh I've been working with Microsoft for 10 years. We've been in operation for 10 years. We spent over 13 million dollars on vulnerabilities over those years. Um we do a lot of root cause analysis, working with researchers around the world to buy bugs, uh define how they actually fire and help the vendors get them fixed in a proper way. I'm also the organizer of the ever popular Pwn2Own hacking competition where I spend probably half a million dollars this year just on exploits uh against the hardest attack surfaces in the world. So before we get started we want to give you kind of an overview and level set everybody in the audience about what we're talking about here, what the SCADA industry is, what HMI is and who are the heavy hitters in this industry. A lot of the marketplace if you look at it is really focused on developing hardware and selling control systems uh and not so much focused on the selling of the HMI solution itself. In fact many of them are freely downloadable which makes them good uh good targets for auditing. So in this case they kind of focus on hardware, hardware uh software um or software that runs on hardware and less on Windows applications and that really shows in the type of vulnerabilities that we're seeing come through the zero day initiative program. It is a highly regionalized market. So there are vendors in China that specifically develop SCADA software and hardware for Chinese uh for Chinese implementation. There's also ones in Germany uh and we've even seen code developed by China uh by Italian developers. Um so it's it's a very active market. Um so it's a very active market and if you if you are focusing on SCADA uh re products in one region it'll be completely different in another region. As you can see on the slide we've got a bunch of big names on there. That Wecon brand is the one that is that is actually for China and we found I personally found a dozen bugs in their in their HMI solutions uh and submitted them and and and had them fixed. Um Siemens is also a major brand uh and and GE electric or General Electric, Advantech which we'll talk heavily about in in this presentation. Now there is also a lot of mergers and acquisitions in this space. Uh it's very much like the rest of the software market. Lots of buying, lots of selling. Um but and the one interesting thing that we find is when we buy a vulnerability in some of the mom and pop uh SCADA develop uh HMI shops which there are a lot of them. We'll see that by the time the patch comes out they've actually been acquired by one of the bigger companies like Schneider Electric or Siemens. So there is a lot of merger and acquisition that goes on which makes the disclosure pos- uh process a little bit more complicated. If we look at the human machine interface what exactly is it? Well it's primary job is to provide status of the critical infrastructure. Things like alarms, notifications, they also provide highly advanced and customizable visualizations that give operators insight into what is going on in their critical infrastructure. Um and a lot of these you know there you can kind of develop these and customize these uh these visualizations for different components in your actual uh infrastructure. Now they're supposed to be air gapped and tr- and run on isolated and trusted networks but this is really not always the case and we'll talk about attacks here in a couple minutes where they took advantage of HMIs that were not on uh isolated networks. Now even isolation uh is is not guaranteeing security. If you ask the Iranians back when Stuxnet came out the air gap network didn't really provide them much value uh when they were being exploited using USB uh link vulnerabilities that existed. Um so if the developers are actually spending the time and and and and thinking that the fact that there HMI solutions are going to be used in air gap networks and not putting security in, that's what we're seeing in the code. That's what it feels like. They're not actually spending a lot of time implementing best practices of the industry. So why would you target an HMI solution as an attacker? Well it's because it controls the infrastructure. You can actually see and and get configuration, configuration information about the devices on the network and it can actually be used by itself without a vulnerability to shut down a network, to shut down a critical infrastructure. This is this is the case in the Ukrainian attack that happened at the end of last year. The Ukrainian uh the attackers who were going after the Ukranian power companies just used the HMI solution by itself to trip breakers and shut down the power. And they were not actually exploiting HMI vulnerabilities but they were using the HMI system to actually take uh take the systems down. Now it can also be used, you can actually attack these to deceive uh and disable alarm systems in the in the in the control system itself. And this is the case in Stuxnet where they actually deceive the operators um and and about the state of the centrifuges that they that they were controlling and actually sended send uh commands to trigger uh self con uh self uh uh destruction um conditions in the actual control systems themselves. So there are active attacks in in um HMI solutions. If we look at uh you know Stuxnet is obviously the most popular uh one that we we talk about. Everybody knows about this one. But it did leverage uh you know the vulnerabilities in in HMI solutions including uh Siemens semantic step seven DLL hijacking vulnerability uh along with a sequel server authentication bug. Alright so these are really simple bugs, very common bugs in HMI solutions and it leveraged those to deceive the operators about the state of the centrifuge. Now Black Energy is an ongoing sophisticated malware campaign against ICS environments um and it actually does target HMI vulnerabilities. The GE's simplicity path traversal vulnerability it's used it's it's uh we believe it to have used the some vulnerabilities in uh Siemens WinCC and Avantek uh remote web access. So quite famously in the ZDI program uh the GE simplicity vulnerability is actually one that we purchased from an anonymous researcher and disclosed the ICS cert and it turned out that that was actively being used by Black Energy. So it's kind of interesting to see that happening uh in the wild. Now another big player in the in the industry is ICS CERT and as a researcher you need to understand who this organization is and where they sit in the government so I'll rattle off the title and their location in the government. They're the industrial control system cyber emergency response team which operates within the national cyber security and integration center, a division of department of homeland securities, office of cyber security and communication. I mean that's a long name. I'm almost getting paid by letter there. So but in reality they are a very important organization and people who are researching in HMI need to know who they are and learn how to work with them. We work with them every day in our jobs because we're purchasing a lot of vulnerabilities in HMI and they do a lot of things. They actually release a report every year about all the stuff that they're doing and according to the 2015 report they actually responded to 295 incidences and handled 486 vulnerability disclosures and that's significant. That's a lot of vulnerabilities passing through that organization and that's a lot of things that we're seeing in this organization every year. Because it's so regional uh it's it's often hard to get a hold of these mom and pop organizations when you find a vulnerability in their solution and at this point you come to programs like the zero day initiative or go to ICS cert to help you disclose those vulnerabilities and get them fixed. So let's talk about attacks that leverage HMI features or vulnerabilities uh in uh in their active attacks. Um if you read the Verizon data breach report they talk about a uh uh an incident where their team went in and actually it was called in to analyze um the security of a water utility. Now they don't give the name of the water utility but they do talk about their findings and in this case they found that there was an internet facing AS 400 system responsible for HMI like capabilities uh like manipulating uh PLCs. But this system also did network routing and managed customer data. I mean how ridiculous is that that all of that information is sitting on one system connected to the internet. Both critical infrastructure and billing systems. Um this is kind of a an example prime example that there's no focus on the separation of responsibilities when they're architecting these critical networks. Now what they learned is what they discovered was there are four separate connections to this to this AS 400 over a 60 day period where the IPs were tied to hacktivist uh activities and they actually altered the the water flow and the chemical activity. So they actually altered the water flow and the chemicals in in that system. Now according to the report they they say that the attackers really didn't understand what they were what they were working against and they didn't couldn't really didn't do a lot of damage but they could have done a lot of damage by that by accessing that system. The most recent example of a really high profile SCADA attack was the it was in the Ukraine where there's several uh Ukrainian companies that experienced unscheduled power outage which is which is which affect almost a quarter million people. These were caused by a malicious actors and there's actually a really great report on the ICS website uh ICS cert website that describes all the details about that attack that are unclassified. Um so and they talk about how the attack was coordinated and after and they all attacked within 30 minutes and in this case they didn't use HMI vulnerabilities but they leveraged that HMI solution because they were because there's no isolation they were able to VPN into this into the network and get access to the HMI solutions and use remote administration. So they were able to VPN into the into the network and get access to the remote administration tools which dosed the operators from making any changes and they actually just tweaked the the the knobs in the HMI solution to turn off and trip breakers and as a result the power went out. They also uh put kill disk malware on the Windows based HMI systems which which basically brought them to their knees and really hurt the restoration efforts. Now this is obviously used to destabilize the Ukraine a little bit. I don't think they've actually attributed the attack but there is a lot of political stuff going on in that region so you can imagine. Now there's also some interesting report uh some interesting research that came out of a sister organization here inside of our company uh where they actually looked at the malware that was used in the Ukrainian attack and actually found links to malwares in other companies in the Ukraine including a rail company and a mining company around the same time. Now black energy was supposedly not used in the attacks uh against the Ukrainian power company but it it did exist in that network. So you can imagine that the the attackers who were going after those are probably the same attackers who had access to some rail and mining companies in the Ukraine as well. How did they how did how did our the our sister organization know this? They looked at the infrastructure of the malware and the naming conventions that were used and they released a white paper on it and it's actually really really interesting and it's worth a read on the it's on the trend micro blog. So let's let's talk about the prevalent vulnerability types that exist in HMI solutions and what the current state really is. So the state of the situation is the HMI solutions have not seen any benefits of the evolution of secure software development life cycles over the last 10 years. We have looked at a lot of the code you know dozens and dozens of code bases uh we've analyzed and looked at vulnerabilities and confirmed zero days and that's what we've learned. There really is no security built into that software. They haven't seen any any benefits of the secure development life cycle that Microsoft, Apple, all these other companies have and this is actually a really scary thing and in fact most of the solutions that we're vetting bugs in do not have ASLR uh safe safe safe SEH or stack cookies enabled which is really really scary and we actually urge SCADA vendors to turn on all of these mitigations including things like building 64 bit apps and and to to make ASLR better uh and actually reducing the reliability of heap sprays and also turning on just the the basic mitigations that are available by flipping a a toggle in the compiler. It's actually really embarrassing. It's also there's also a lack of understanding of how these are really uh these uh solutions are actually run. They they they seem to think that they're gonna be running that isolated environment and they're using that as a way to not implement security mitigations but they are continually being integrated and you've seen attacks in the wild that leverage interconnected HMI solutions to take down critical infrastructure. So this is probably the only pie chart that you're gonna see at DEF CON and I'm actually really proud of that because I did a lot of work to generate this pie chart. But in a what we ended up doing was we pulled all the 2016 and 2015 ICS cert advisories uh and identified all of the HMI solutions that had bugs fixed over the last two years. We cross referenced that with our 250 plus zero day vulnerabilities that we've purchased in HMI solutions to come up with what the most popular vulnerability most common vulnerability types are in HMI solutions. So we're going to do a little bit of cataloging here. Um we also catalog the CWEs to kind of get an idea of of what vulnerabilities existed and and here they are listed on the slide. So um the number one is memory corruption uh followed by credential management, usually hard coded passwords, insecure defaults, authentication and authorization and code injection issues. Now what about cross site scripting? What about cross site request forgery? Well most of these are windows based applications. There are some web based applications but most of them are windows and as a result you're not going to see a lot of that cross site scripting. So we're going to do a little bit of scripting stuff but there are some in that gray area on the slide. So what we're going to do is let no let's let's get down and dirty with this. Let's let's look at every single one of those categories and we're going to give you case studies of what these look like so you can understand how terrible this code base really is and what you need to understand to actually go find these bugs and to protect yourself against these bugs. And that's the most important part. So first we're going to talk about code injection vulnerabilities. This makes up about 9% of the common vulnerability types that exist in these products. And that you know it's the classic stuff. SQL injection, code injection, OS command injection. But there's other other domain specific languages that exist in this software and that's what we're going to talk about today. We didn't want to cover like stuff you guys already know. This is we're going to talk about gamma code injection right? So this is a domain specific language that uh that's used in in this industry. Specifically we're going to talk about Cogent DataHub. And and we're going to talk about CBE 2015 uh 3789. Now this allows this vulnerability actually allows an attacker to turn on an insecure processing mode in the web server which allows the attacker to send arbitrary scripts to the server and execute arbitrary code. This was discovered by an anonymous researcher and disclosed and purchased by us disclosed uh to ICS cert and fixed. And we do offer the ability for people to uh submit bugs to us in an anon an anonymous fashion uh and we do offer the ability for people to uh submit bugs to us in an uh anonymous fashion and we get a lot of that actually uh through our program. So what is Cogent DataHub? Well that's what you see on the screen here. That's one of those visualizations that I was talking about. Cogent DataHub is a real time middleware solution that is deployed across several sectors including chemical, commercial, critical manufacturing, energy, financial, et cetera. And it's used around the world. It offers the end user the ability to create those really uh intense uh uh advanced visualizations which you see on the slide here. Customize those that so they can monitor their their underlying code. So we're going to talk a little bit more about Cogent DataHub. So what is GammaScript? Well GammaScript is a domain specific language specifically designed for the use within within DataHub. It's a dynamically typed interpreted programming language specifically designed for rapid application development. Uh it looks like C and I'll show you some here in a second. And it has a range of built in features that's about libraries and everything. It actually has a fully documented a uh API that you can read uh on the internet. And it's actually pretty full featured for those application developers. Now the attack itself uh is a flaw in a valid expression method. And it allows an attacker to execute arbitrary code on the system. It actually sits uh and is uh accessible through an Ajax facility on on on port 80. And you simply supply a well formatted GammaScript which allows on the underlying code execution. Now the interesting thing about this is it's domain specific. So there's a lot of functionality in Gamma that's specifically used for developing that stuff. But unfortunately it did have the in that script the ability to execute system commands. And so what is the vulnerable code? It's right here on the screen. It's very very simple. A valid expression basically takes an expression and checks one flag. Am I allowed to have to execute this uh expression? And if it does then it executes the expression. Right? And this is whatever you want to send to the system. Now the the question is how do you actually get that to load up and how do you change that value? Well it also allows you to do that as well. And the exploitation steps are you send uh a request an HTTP request to the uh port 80 which will load the GammaScript libraries. Then you go you call Ajax support dot allow expressions and which will set allow any expression to true. And then you call a valid expression with whatever script you want and you execute code. So let's demo that exploit. So what you see here is the installation of DataHub. You can see you can kind of zoom in for the audience here. Uh oops. Hmm. Yeah well forget that. Um so what we're going to do is right here uh on the screen what what's highlighted is Cogent DataHub version 7 and it's running and sitting on on port 80. And what the first thing we're going to do is we're going to run a proof of concept here at the bottom that is a uh just basically a Python script that sends the uh 3 commands that we need and will actually disclose information on the server. It actually is disclosing autoexec dot that uh on that box. So then we're going to send another script uh which will actually execute calcu the evil evil calculator. And you'll see here it's actually a very very reliable bug and a very reliable exploit. You can just kind of send it over and over and over again. And there's those evil calculators you know. So that's a that's a really good a pretty fun bug. Um really simple bug and they did actually do a really good job fixing it. You can see here all the calculators being spawned by uh the the process. Um so. Now how do they patch the bug? Right this is kind of one of the interesting things for the ZDI program because when bugs get patched researchers will also submit bugs uh uh POCs that actually break the patches which is kind of interesting. But here it's going to be kind of difficult. So on your left is the old code and on your right is the new code and you can see up here that they actually um removed allow expressions so you cannot access that at all. So you can no longer turn toggle that flag in the system. Um they also removed uh eval expression entirely and they actually la- gave it a really great comment which is actually a best practice. Uh this method is dangerous. It could allow somebody to execute arbitrary code via an HTTP call. If you absolutely need it create a script and define it and then make sure that your web server is on a trusted network. So that code is buried in in the application itself so it's highly unlikely that a developer is going to go look at that but uh they're just going to call the APIs but it is good that the fact that they actually documented that so they won't regress that bug at some point. So that's how that bug actually works. So I'm going to turn it over to Fritz and he's going to cover uh the rest of the uh the prevalent vulnerability types and then we'll talk about some some disclosure statistics. Hello again. So the next uh section we're going to look at is authentication and authorization problems. And authentication bypass, improper access control, uh improper privilege management, bad authentication and what we're going to focus on is an Advantech case and you're going to hear Advantech a lot. And this is actually a pretty fun one. It's information disclosure. And uh this is uh CVE 20165810 and ICS cert says a properly authenticated minister can view passwords for other administrators but the terminology is a little unfortunate here because this is not a system administrator. This is an administrator of a given SCADA solution, a given project. And so that is sort of akin to unprivileged users of the system. So this is in essence saying one user can extract the password of another user. And this was discovered by Zuyu and disclosed by the Zero Day Initiative. And it's sort of fun. And basically they have a script, an ASP uh script that allows you to change your username, your password, your description. And this is great. But it can be abused. And the way you do it is you log in to the account you have. So this is not anonymous. You have to have an account on the system. But then you can change the URL to give any other name. And then pass that in. And it will bring you back the password of the second account. Now you can't see the password because it's got asterisks in front of it. Yeah. So here's the demo showing it. So first log in as the admin. And by the way you can also get the full system administration. So here's the demo showing it. So first log in as the administrator account this way. And you can see that there's a test one and a test two user. So now you log in as test one. And put in your password for test one. And that's all great. Now if you try to change, change to test two using the UI, it will quite properly give you an error saying you can't do that. But if you try to change to test two using the UI, it will quite properly give you an error saying you can't do that. But if you change to a user name of test two in the URL, it will pop it back. But it's got those asterisks. So we got to fix that. So you view the source. And there's the password. And then you can use that password of course and log in as anyone else. And including of course the complete system administrator of all of the solutions. And there you got it. And you're logged in. And what? Okay, here it is. So that one was sort of fun. And there's also a lot of insecure defaults in this space. Bad transmission of information, missing encryption, unsafe ActiveX controls. Yes, we're back to ActiveX controls. So the one we're going to focus on here is the Schneider Electric DSNVS. And this is a bad ActiveX control with memory corruption. Now even though it's memory corruption, we put it in here because this ActiveX control, well first it was set as safe for scripting from untrusted source. And, but what's also interesting is it was never meant as a control to be used in Internet Explorer. In a web page. So it should have been configured as automatically kill-bitted. So it's really bad configuration. So it was wide open to Internet Explorer when it should not have been. And this is CVE2015-0982. And the Schneider Electric PELCO here is an HVAC control. And this is a CVE2015-0982. And the Schneider Electric PELCO here is an HMI for digital sentry video surveillance systems. So it's really great. You can use this to, you know, get information on video surveillance systems, which is always fun. What I wanted to do was show this for people who are going and auditing looking for ActiveX controls and might be vulnerable. This shows an interesting second step you often need to take. There are two ways to tell the system that an ActiveX control is not safe for scripting. The standard way, the fast way, is statically in the registry to mark it as not safe for scripting. But if you note, to turn it on, to make it safe for scripting is to flag it as safe for scripting. But if it's not marked in the registry as safe for scripting, it can use the interface iObjectSafety and in dynamic runtime assert that it is safe for scripting. So even though it's in the registry, it's not safe for scripting. So even though it's in the registry as not marked as safe for scripting, it still is potentially vulnerable. So you've got to look at the dynamic situation as well as the static situation. So you can't just do one. And here's just the demo of how the memory corruption works. Which is you use Internet Explorer and you go to an attacking web page, which invokes the control in IE. And if you go to the control in IE, you can see that it does a stack buffer overflow and fills everything with your classic 41s that we all know and love. Now let's talk about some credential management problems. This actually, I was really shocked when I ran into this because it's like, you're kidding, right? This happens a lot, that they hard code credentials in IE. And this happens a lot, that they hard code passwords in the code. Hard coded passwords. You know, I thought we'd gotten rid of that 15 years ago with IIS. But, well, it's, we're in SCADA space, you're hacking like it's 1999. It's, it's awesome. It's awesome. We're back then. So this is, the one we're going to look at is GEMDS PulseNet. And it's got a hidden support account. And this is really fun. So this is used to monitor devices in industrial communication networks. And it's deployed in energy, water, and waste water sectors. And it's used worldwide. This is CVE 2015-64-56. So if you take a look at the user management panel using the UI, you see that there are exactly two accounts in the system. There's an admin and an operator. Well, that lies. If you act, go in and use, uh, I used Heidi SQL. But if you use anything that extracts information from the database, you see that there are not two accounts, there are three. There's a hidden account called GE support. Now, now it's really super subtle because it only stashes the MD5 hash of the password, not the password itself. You certainly, you, no one here could crack an MD5 hash. But it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's a MD5 hash, right? It, it turns out that the password is actually PulseNet. But they made it leet by changing the L to a 1. And here's the demo. Uh, you can see on the right the two users that are officially there. And on the left we will log in as the two servers that have been created and that we provide user that isn't there. And what I think is, is really cool is that even after you log in as the user that isn't there, as you're logged in as the user that isn't there, it tells you that, that user is still not there. Which, which is sort of slick, I think. Uh, there's also a lot of other misconfigurations. One of the other ones that we see a lot is where companies decide to roll their own, um, ACLs and they decide they don't want to put things under program files as Microsoft intended. And so they create their own top level directory with their company name under the C drive and they often put, you know, world has full access. Uh, and then they put their service binaries in there. So any local user can drop new binaries in and they will run as, uh, as a system service. So this is very standard. Now we get to, to the joy of memory corruption. Uh, stack-based buffer overflows, heat-based buffer overflows, out of bound, read-write, the, just the classic ones. And the, Advantech is our, our, uh, whipping boy here because they did a, an awesome job here. Um, we got a hundred bugs in one day from an anonymous researcher. Uh, this was like this, this data dump from heaven. And we analyzed them and passed them on and they were all buffer overflows. And it was quite impressive. And I'll go drill into one. In particular, uh, this is CVE 2016-0856. And it was an anonymous, uh, uh, researcher and disclosed by us. And this is their webpage. And what's really interesting about web access is it's a SCADA solution, but they also advertise, as you can see in here, that this is for internet of things. So this is widely deployable and it's awesomely vulnerable. It launches a service, web, uh, VRPC, in the context of local administrator. And it listens on 4952. And the web, the service calls are configured to look like Microsoft IO access control calls. So they've got an iOctal value and they do jump tables off of that to perform hundreds and hundreds of types of services. For this particular one, the, the, the, the, the, the, the, the, the, the tag bar is embeddable. Uh, as you can see, the, the, uh, the sidebar is an I-8 disk that's clean and ready and graphic in Uh, our acaba. It made it very, very complicated. Uh, running a test version to, uh, make you every, every time that you cannot uh, not only access to web access, which you can use upto, upto our channel that! Uh, uh, just have to, install, at least we have a software app that lets you or play any of this game. Uh, uh, let me, let me, I'm making a mistake I missed the, the skill, this would have been significa Then, let me Uh, let me, let me, uh, here. and so inside you've got this and the flaw is the stack based buffer overflow. Here's the classic S printf call. You know nothing of a surprise there. Here is the stack layout and this is sort of fun because you can tell the Windows name is at minus 80 and then zero is your return address. No stack cookie. Why no stack cookie? They didn't flip the bit in the compiler and linker. Probably because they first built this 20 years ago and they never changed their configuration to handle to add ASLR to handle safe SEH to handle stack cookies. So all you have to do is overwrite the return address, point it to the first of your ROP. You can handle the ROP chain well because there's no ASLR. Life is a game. Life is good. Life is really good. So you can see us jumping to an address and here I will pop the glorious calc. And this was fun to do. Bingo. And that's running at high privilege. Life is good. Well let's talk about the patch analysis. Look, S printf, Microsoft, published the banned API list a decade ago. And there's a reason why Microsoft published the banned API list. And so what they did when we reported this is they changed S printf into SN printf. Now SN printf is also in the benchmark. band API list. Now it's a better band API because it won't buffer overflow but if you give it too many characters it also will not null terminate. So if the stack is not pre cleaned out and it isn't it is possible for you to then use stirring manipulations on this window name where you think it's hex 80 characters long where it may be longer because it didn't null terminate with the copy. So there still may be problems. As I said a hundred bugs came in. A hundred bugs. Advantech fixed 75 of them. We have disclosed the other 25 as not fixed. You guys can enjoy. There are also when they did fix they were not fixed. There are also when they did fix they did not do any kind of global replace. They did specific point fixes of the ones that they fixed. There are thousands of string copies. S print apps etcetera in the code base. And I would not bet 10 cents that none of those can be reached by attacker supplied data. So have fun guys. Have lots of fun. Ah yes researcher guidance. So I'm going to give you a little bit of a quick look at a couple of it. What do you people want to do? Well the first thing to do is fuzz. Right? These things are easy to fuzz. They don't have CRC's most of these file formats are wide open. Just do bit flipping. Remember to turn page heap on on the process that's being attacked. That's a great way to find memory corruption because then it breaks at the corruption point not later on when its been found. So we have so much work to do. So I'm going to just turn this being used. Use your tools that you've got for fuzzing. Use your tools for analysis. SQL map is great for finding SQL injection possibilities. One of my favorite tools is attack surface analyzer by Microsoft. One of the reasons it's one of my favorite tools is I helped write it. Microsoft released a public version in 2012. It creates a snapshot before and after the installation of your target software. And then it will highlight security problems in the configuration and it will highlight increases in attack surface. So it will tell you your new COM objects, your new ActiveX controls, your new RPC endpoints. Here's an example. It shows you, for example, on the uh, on the uh, Advantech software of the new RPC information. And here is attack surface analyzer telling you that the web root directory, which is, you know, where files are going to go that are being executed in the high privileged web server context, that this entire directory has full, has, can be, has right access by the world. What could possibly go wrong? So, uh, ASA is a great, great tool to use. Now, if anybody in here has pull at Microsoft, uh, we need a new version drop of ASA because it doesn't work on Windows 10. And if Microsoft wants to be really cool, they could release the source of ASA because I know what needs to do to fix it. So it works on Windows 10 and it would take me about an hour if Microsoft would please release the source. Also, audit for banned APIs. Go look for the S print apps. Go look for the source of S print apps for stir copies, use IDA to trace the tainted data back and see if you can get to the source of these uh unsafe copy APIs if you can get to those from attackers by data. IDA's great just you know it's wide open people. And now back to Brian for more of the corporate things. Yes. So we wanted to give you an an understanding of how when you find a vulnerability how long it will actually take to fix. Kind of talk about the vulnerability exposure window. So what we did is we actually took all of the HMI vulnerabilities that we've received in the zero day initiative program again over 250 uh now um and looked at how long they actually took and if you look at the last 4 years it's not exactly trending down. You know it's pretty consistently about 140 days from the time that we disclose a bug to when the patch comes out. And the thing about the I the SCADA industry is that when they're applying those patches if the patch is bad or there's an issue it will actually denial of service the critical infrastructure as well which is not good. But that means that patching actually takes a really long time. It's almost you can imagine almost twice as long. So that leaves you know almost through probably around 300 days when there's when the when the patches are not being applied. So you know that's how long the bug it it are existing in the software even after you find them. So what we wanted to do is actually call out a couple vendors who uh who disclosed who we've disclosed vulnerabilities to cause that's what we like to do. Um and so what you see here is all of the vendors over those years and Cogent DataHub I want to call out as being one of the better SCADA vendors for doing patching. I mean in fact one of the first bugs we disclosed to Cogent DataHub their CEO actually emailed us and uh and worked with us on the fix and they fixed it in like 6 days it was amazing. Um and and they've continued that trend you know we've still purchasing bugs in Cogent DataHub and they're fixing them relatively fast. But if you look at the big vendors out there. You see you know ABB uh GE uh you know um Indusoft those over 200 days to release a fix for a zero day vulnerability that we've purchased that is known. So it's kind of interesting um you know it it averages out around 150 days um for for bug fixes a lot of these are going through ICS cert and so just to kind of call that out. Um if you look at the SCADA industry and how it compares to other industries you know micro we we in the highly deployed software we consider that Microsoft Apple Oracle the big name vendors they they do a decent job it's about 120 days for them to fix a bug when it's disclosed. Um and and SCADA and security products are kind of battling out for second and third with SCADA coming in the in third and and kind of worse of all of them is business software, things like HP you know and and and other big name businesses like IBM it takes them a long time to fix vulnerabilities. So you know we're almost 200 days for those types of vulnerabilities so just you know as as you find bugs and you uh just you know work our with ZDI to get them fixed or disclose them directly to the vendor. It does take a significant amount of time but in certain cases it does take more than just 180 days. So kind of wrap things up. Um we give we we present at these conferences and provide this level of detail because we want you to go find bugs. We want you to work with uh with the vendors to get them fixed. We want you to work with bug bounty programs like the zero day initiative uh to get compensated for your research. Um and so we're we're definitely interested in buying vulnerabilities and that's why we provide this detail. Um there is ICS in you know focused malware that is actively exploiting HMI vulnerabilities. These vulnerabilities these code bases are plagued with vulnerabilities. Uh and you can use those simple techniques to actually find them. It does take a long time for them to fix but they do end up fixing them. Um and so we're going to be continuing this research and we're actually going to be releasing a white paper in a couple months. We're going to release some proof of concepts and all of our disclosure data is publicly available on our website zero day initiative dot com. Uh for you to analyze yourself and draw your own conclusions. Um again we are the zero day initiative. We buy bugs. If you find zero days we are a white hat bug bounty program. We've been doing this for 10 years. Uh we like to watch researchers grow and uh provide feedback to make sure that they are finding better bugs and getting higher payouts. And so if you are interested you know come up and talk to us. We've got basically the whole team here uh in the front row. Uh and they you know we do a lot of research and look forward to working with you. Thanks for coming and spending the time with us. Thank you.