>>Hi all, uh I’m Tim, or Vulc@n more often uh during this weekend of the year, anyway. And uh, if you are here to see the, the CFT, CTF [laughter] uh talk, with uh, the last 20 years of, of organizers, which incidentally, is all the years. So uh, um we have from the beginning all the way to the ones that are currently organizing it on, here on, on the panel. And, um I believe the current organizers are, this is the last one. Right? So, so stepping down this year. And every, uh few years, the, the current organizers step down and there’s um some, process for passing the baton. And uh one of the reasons that, that this talk is on the on the docket on this year is to um, answer questions of people who think they might want to step up and take the baton and also to have sort of an archival reference for when this happens in the future. Um, so uh again I’m Tim, uh I’m going to um, do my best to um, moderate, uh these folks up here, when we get to the panel time. But I’m also going to uh, sort of lay a foundation of, uh um, sort of the basics of what uh CTF is. I’m not going to try and make too many assertions. Now, uh I am sort of inherently biased being one of the organizers for uh few years. And also being a participant for a few years. Um but I’ll make uh every effort to um, sort of remain neutral up here and pass the questions off to our panel of, of experts here. Now the talk is geared towards um organizing CTF. So, I’ll do a little bit of explaining about what CTF is, but it’s from a organizer perspective, right? And some of us, even do CTFs from time to time, uh professionally, uh but in most cases these slides are, are geared specifically to what, uh what CTF means in the context of Def Con. So, uh what is CTF? Right? So, we’re not going to cover uh, what is CTF, in depth. But at, at the basics it’s a cyber security based, uh capture the flag contest, right? So, it’s just like flags when you play with hills and kids and stuff. Except the, the flags are digital, uh also called an exercise or an event or a game. Um, typically it’s all geared towards the industry and some minimum bar of proficiency or ideally some bar of excellence in fields of, of cybersecurity. So, there’s different models for, for organizing this. And we’ll go over a couple then, uh in a bit. But to sort of stress different things and as an organizer you get to decide how you’re going to stress the, the different uh sort of areas of excellence. Um, CTF exercises are becoming um much more common. So, these days you can almost play about every other weekend. Uh, if you’re okay playing in these remote competitions that happen all over the world. So, there’s uh, uh circuit so to speak. And um, there’s also uh, uh multitude of contests that sort of just tack on CTF at the end where it’s very domain specific. Like the, um social engineering CTF or uh, um ICS CTF or something like that. And some of those bear more resemblance to the game we’re going to talk about today than others. Um but uh, in some cases this really just means contest, in other cases there really are flags that uh can be captured. So, going all the way back to the basics for capturing the flag we need to talk about like what the flag is, right? What’s the thing that needs to be captured? But from an organizer's perspective that actually has uh, uh there’s, there’s a lot to sort of suss out there. Um, is it just random text? Well if you just use random text, then from a participant perspective it’s difficult to understand whether, uh you’ve actually found the flag or whether it’s just uh, random text right? That’s uh, encrypted or compressed or something like that. It’s somewhat difficult to know when you’re done and when you’ve found the flag. Um, from the, uh uh, also from an organizer perspective it might result in a lot of guessing. Because the participants end up with this sort of um denial of service sort of situation where people are just guessing repeatedly hoping that they’ve found the flag but they’re not really sure. Um, so then the, the next thing you do is you try and add some structure. Ya know, it’s a certain size, it’s a certain format, it has a particular prefix. Um, all of these things sort of limit that space of brute forcing right so now you can increase the likelihood of guessing correctly for people who do want to guess. Um, you also open the door to all kinds of, of difference defenses. Like, consider writing a, a IDS signature for uh, for the prefix that you know is going to be there because the flag has a specification. Uh, so there’s these, uh trade-offs that happen when you, even define something as basic as, what is a flag. Um, so, uh even beyond defining sort of the specification for what a flag is, there’s uh, um, uh, a lot of other mechanics that happen around the flag. So, flags are stolen and you sort of have to prove that you’ve, you’ve actually stolen it right? So you have to go back and assert to the organizers that you’ve successfully stolen it and over the years that has really matured uh over time from um, uh verbal or writing on paper, to email, to uh now you sort of have rich-ish web APIs. Right? Solid services where there’s structure and, and protocols around how to, uh sort of automatically or in automated fashion submit flags to uh web APIs or scoreboard servers. Um, and then there’s all of these sort of game decisions that have to be made. Some of them made by the participants where, flags might be shared. Um, uh you have collusion between teams for pot- uh potential collusion. Um, do flags have a preset, predefined constant value or do, does it change over the time? Maybe being diluted as more people have stolen flags from easier services. Um, do flags expire? If you steal one at the beginning of a 48-hour period is it worth the same amount at the end of the 48 period. Is it worth anything? Um all of these things sort of go into the mechanics of how that particular game is going to operate. And uh adjust the game strategies that the participants are going to adopt. So then what is Def Con CTF specifically? Well there’s no shortage of quotes. These are just the top hits from when you do a Google search but um, the assertion that I think most of the people up on this panel would say is that Def Con is one of the, the highest regarded CTFs out there. Right? And uh, it’s kind of grown over the years from being one of the, the oldest and uh sort of one of the only to be the, the, the world series or the best of the best. Right? Um so typically the phrase Def Con CTF, if you hear somebody saying it they’re referring to the on-premise version of the game which starts tomorrow morning, right? At 9am and goes til Sunday, goes throughout the weekend. Um, in the beginning there were the Goons. Or more specifically there was Myles. Ok. And um, so Def Con uh CTF was one of the oldest and longest running. Uh there are some others that have been around for uh, for a while. Like the UCSB’s ICTF comes to mind. Um, but even ICTF is, is fairly restricted until just this year was sort of academic only. Right? And they opened up for you guys, right? So as a, as a pre qualifier they, they opened it up so it’s not academic affiliated. But, but Def Cons got a long history of being very approachable and very open and um, it’s one of the longest running CTFs. It’s also one of the oldest contests that’s still running uh at Def Con. Um, so back at Def Con 4, uh was the first one. It was just called capture the flag or network capture the flag I think a little bit later. And um, certainly grown uh, over time. So while this Dec Con 25, it’s CTF uh 22. Uh, uh oh. There we go. Um so if you look at uh, uh quick a timeline, again in the beginning there were Goons. Um, sort of later on, uh teams more formally organized and sort of had a multi-year approach to organizing. We may switch to different computer yet. Around the year 2000 it was clear that organizing needed to be a team sport. Let’s go back one and see, less, less epilepsy. [laughter] Um, now, now CTF, now with less, less epilepsy. Um, sort of around uh 2000 it was clear that organizing needed to be a team sport, right. Uh there were other hacking competitions, um HOPE for example had something. But there was nothing that really had, uh sort of the network component, the defensive component. Yeah, you got one more, see if this will last, ya know. Alright so in the way of a road map, which might stay there for a few seconds, nope. Um, let’s go back and forth here. Uh, quick, look at it, uh so in the way of a road map there’s sort of different milestones that happened over the years. Uh in the beginning. >>Hey Tim, can I interrupt you for a second? >>I’m sorry >>Just me, just me, Uh I just wanted to say, he’s had 2 problems here that are totally classic what happens every year when you come with new technology. And that happens every year with organizing these contests. We don’t’ know necessarily who’s going to be in the audience. What words to use to say to talk to make things work right and we definitely don’t know what cables to bring. [laughter] >>Um but they do provide testing harnesses in the speaker room with do work fine. It’s fixed. You had to go kick the cable apparently. It’s unfixed [laughter] [off mic discussion]. There’s so much in between here and there [off mic discussion]. Yeah, we’ll just, we’ll just skip over it. The uh, there’s sort of different milestones that happen throughout the years right. There’s, there’s the realization that teams are going to be competing, right even though it wasn’t necessarily organized as, as a team competition. There’s the realization that teams are required for organizing. Right. It’s too much for 1 individual or a couple of individuals to handle. Um, fast forward to there’s so many people that want to play that there needs to be some sort of qualifying round in order to gate the people that can actually make it to what’s now known as the finals of the Def Con CTF that’s on premise. And then, uh the sort of different things that happen sort of along the way. Like the inclusion of the game outside the game, the the meta game and the moving to IVP6 and badges that have uh game code on the badges. Uh, and then last year was actually the first year and perhaps the only year that the, the architecture and the format of the game were announced like long ahead of time so you could actually spend time developing tools as a team and come in with tools that you knew were going to work. Right. >> We actually, we actually announced last year that we were having uh, a custom architecture. It’s like a 9-bit middle ending thing, monstrosity. >>Middle ending is not actually a thing by the way. [laughter] >> No, No. [laughter] >>Oh, it totally is now. It is now. >>Yeah it is now. >> Um, so yeah, now we actually have a completely custom architecture that the teams just got this morning, they just got the manuals this morning. >>So, so preparation before 9am is perhaps of limited futility, right? Um, so getting back to, to what is capture the flag if we kind of grossly break them down into 2 categories. This is again been covered in other, other presentations so we won’t spend too much time on it but the 2 broad categories are, are attack defend and uh game board or Jeopardy style. And the big thing in attack defend is that the participants are directly connected to each other. So defense actually becomes a big component of the game. Uh also service level availability or SOA uh becomes part of the game, the, the part that need to be vulnerable and able to be attacked have to be there and running. Uh so everybody is typically level-set. Uh think of a VM that has custom software and is distributed to all the teams. There’s different ways to do that but that’s a good analogy and ultimately it's composed of a set of challenges and that’s another thing that’s matured over time. The concept of challenges or services and these are sets of custom software that uh typically run with some amount of concurrency. Um there’s debate on how many challenges, uh concurrently make a good CTF or not. So, uh conversely the Jeopardy style is much like the show, right you’ve seen the show. It’s a grid of questions, um participants are not connected to each other so defense is typically um not part of these games and you solve a series of challenges where the order might be determined, there might be say control by the leader by the organizer, and the categories might be completely arbitrary or they might be designed with uh certain ya know learning objectives or goals, such as gating for, for CTF, demonstrating proficiency in, in subjects that, that would uh work well when you get here for the weekend. There’s also hybrids and things that don’t quite fit but broadly these are the 2. Uh, so Jeopardy style world-wide or just proportionally with the number of CTFs that happen is by far the most common. Um it's uh possibly more diverse within a single event because defining what a question is, uh is very broad. Uh whereas you can find what a software service is in an attack defend game is perhaps relatively uh confining. Um, it’s arguably easier to, to organize for several reasons. So, so today Def Con, when you say Def Con CTF it means a thing hear, uh there’s typically a Jeopardy style one that gates everybody as a qualifying round. Um so this was introduced in uh 2004, Caezar, you did qualifying, right? >> Uh, turns out yeah, we were just debating that, discussing that. >>No, Caezar was like we didn’t do quals. I was like, I remember playing in your quals so I’m pretty sure you did. >>Yeah so the last, so on the, on the chart it should show but you can’t see it because it causes epilepsy. Um, but I think the last year that data hackers- >>Yeah that was all DD back in the day. Like that was literally… >>Although Myles brought us a long, DD implemented so much stuff it, you really would have to understand the guy that, how amazing he could pump out a user mobile and X router system with double back. >>Right >>Firewalling and stuff, like in a night, in a night, in an evening. >>They, they also had some pretty crazy constraints in their quals too like, I remember specifically playing and, and not being able to actually make any kind of connection into that box or out of that box, other than the literally just terminal IM. And so we had to write like GU encoder and decoder routines into like telnet so that we could move binaries to the box. >>There, there first qual was just a race to finish 8, 8 levels of the challenge, that was it, 8 questions. >>Yeah it was totally linear. Yep, yeah exactly. >>So, we’re having a sad realization up here that anything with pictures appears to anger the projection system. Uh which is unfortunate because there’s a lot of pictures in most of this, so. [laughter] Um, so you’ve seen the board. This is what the board looked like when DDTEK [inaudible] had quals in 2006 anyway. And uh there’s a starter question, there’s a hand that kind of points off somewhere, gotta download a file or go to some web service. And uh, maybe I should try and switch computers here. Uh but unlike Jeopardy uh the leader controls the next question but unlike Jeopardy, uh other uh teams can sort of catch up and answer the, uh the previous questions. I might just try and switch briefly here. >>We could probably fill the space. >>Yeah go for it. >>We’re gonna talk about stuff while he works. That’s usually how organizing CTF is. >>I was going to say yeah it’s pretty, it’s pretty AD-HOC to begin with. Basically something, something goes awry every time. This, this is on- this is its own sort of like micro example of how CTF actually works in real life. Because it turns out nothing goes to plan. It’s the old like, you know planning is useful or plans are useful of some sort of thing. >>And pulling out that Tim was a part of DDTEK. And DDTEK CTF’s were famously broken. [laughter] >>Oh yeah. There was one year I think you completely erase the scores from one day. And we were, we were upset with this. >>Yeah. >>Yeah, and I actually, uh, playing CTF uh most of my CTF experience was under DDTEK. My 2 black badges was from when in their first and final year. Um but a funny story we actually were, playing quals or organizing quals, uh I once wrote a, a mock web server with a CGI script underneath it. And, uh there’s, the bug was in the CGI script but all the, the web script kitties were like, yay a web challenge. [laughter]. And then uh, so they started uploading like PHP shells and python shells and one team in particular, uploaded a python shell connect back with a hard-coded IP address and port. So like, kind of, let me play with this. So, I killed it since it wasn’t actually part of the game. Or necessarily part of the game and they did it again. Alright, this is what’s going to happen. Killed it, just immediately started kicking back to them and then uh, finally it kicked back and they started typing commands. And me trying to L S uh slash. And then like, you know, cat something, uh let me type some other stuff. Eventually realized they were talking to somebody buy uh, [laughter] but uh yeah [inaudible] so much fun >>Yeah it took an amazing amount of self-control to not mess with everybody else’s exploits essentially like, people would land stuff on boxes and like watching it happen you could like connect to it faster than they could. And so like not going in and grabbing everybody else’s shells when you’re the organizer just for fun was actually like a massive, massive self-control. >>The reason I did though was because it wasn’t necessary to, to land the challenge. >>Right >>So, that was actually the year, so I don’t know if you guys know Geohab he popped the servers by figuring out that, dot bashRC was writable by general users. So, he wrote cat, uh flag in the- slash temp slash something. So he just waited until somebody else exploited it cause they were just going to run bash because it wouldn’t run bash so, whatever and so he just waited and just saw a that file pop up in temp and got the flag. It’s the way to do it. >>We actually, we actually had a whole, we had a whole um, we had a whole like sub like focus area that we, that we called the dirty tricks department. >>Yeah. >>Which was basically like trying to figure out all the, all the herent conditions in everybody else’s exploitation methodologies. So, like I remember one year there was a team that was doing really really well but they were always like, their payload would actually cat the flag in a file in temp and so we could just like walk around behind them and mop up all the flags that they were getting too. >>Nice >>One thing that I found, though is uh, nevermind are you ready? >>I’m ready >>Alright >>Want to finish your thought? >>No, no go ahead. >>Alright, too much of a thought. >>Yeah, you’re in the middle of your thing, go ahead. >>Um, I will say that I walked into the, the speaker room earlier today with my uh, laptop that’s running Linux and plugged it in and everything worked and then I didn’t tell anybody I was like, oh linux and it finally works, it understands how to do it. >>It’s the year of Linux on the desktop. >>So I went in there later and I tried it again [laughter] and it worked again and I got up here. I’m like, oh no, and now I have a mac plugged into the projector and it’s working fine. Um, so I guess it’s still not. >>Go back and show your timeline, I thought that was uh, that was a good, you should flip back to your timeline. >>Yeah agreed. >>Oh yeah. >>Nobody got to see that. >>Yeah, I’m a few slides down now. Um, great so now we have timelines with no epilepsy. Um, so uh I- I, kind of went over a few of the um the parts and this is by no means exhaustive it's just sort of like the things that stood out in, in, in my memory and sort of uh worked well for the slide desk right. Um, but uh, it definitely has a lot of the, the pieces that matured over time with the, the qualifier introduced in 2004 and then the tradition that continues now with returning the, the return in champion. So uh, so Kenshoto would be inviting the returning champion back for the first time in 2005 and there, there’s sort of these uh, I don’t know, um dirty, dirty tricks or, or twists or things that sort of make it hard to um, uh come in prepared right. So, like one of the things, this year makes it hard to come in prepared is that it's en- entirely custom architecture that wasn’t announces until 9am this morning right so that’s like a way that preparation is, is difficult. Um, some other examples we’ve got going back in 2006- to submit keys over DTMF. >>We, we were just talking too. I remember, I remember basically everybody, so we had, we had deployed like a, an actual like I think it was an asterisk >>Asterisks >>Yeah, and so basically all the tokens, all the keys were just ya know, really big entirely numeric digits and so we were making everyone like, they would pick up a phone and it was like press 1 to submit a key, press 2 to talk to a member of Kenshoto. And so all the teams had to like run out into Vegas trying to like buy modems and like had to like write their own DTMF like scripts to submit tokens. Because otherwise the first few tokens that were being submitted by people, they were first of all, they were like 32 digit numbers, like they were big. And everybody’s like [beep beep beep] [laughter] [beep beep] s**t I f****d it up [beep beep beep] [laughter] >>Specifically Hangout >>Yeah exactly. And then yeah, and you couldn’t submit more than one token per call so you had to actually, hang up the phone and do it again. [laughter] >>We, we had so many tokens we exceeded the available bandwidth to submit our tokens we couldn’t submit them fast enough. >>Yeah so everybody ran out, everybody ran out to Fry’s to try and buy, to try and find modems and like. >>Which also has ranging ramific- >>And people are like how, how do I connect something over serial to my like modern laptop. >>Which also has like interesting ramifications because then next year like teams would buy all the modems like just to >>Yeah and we didn’t use modems next year. [laughter] >>Good, they have good return policies. [laughter} >>They do. >>Uh, but there’s, there’s other things that sort of, uh go through the times. Uh, multiple hosts, uh IPv6, a lot of people were um, sad face when it was all IPv6 and they had a lot of IPv4 tools and shell code and so forth. Um so, cust-, so MSP430 architecture on the badges, sort of midway through, uh legitbs and, and, and again this year with the custom architecture. So, sort of the, some of the, um milestones by no means exhaustive, um but um as you already said, the one that I want to focus on now is the, the qualifying round. Because that gets into the, they style of game that’s game board style. >>Actually, before you jump off that slide can I, can I like mention one other thing? >>Yeah >>I think there’s an interesting pattern that’s shown uh as a result of this slide. Which is the like burnout cycle of hosting CTF. If you notice, almost all of those are the same-ish like maybe off by 1 year here or there or whatever. Because the, just herculean level of effort that goes into creating CTF in general literally causes like internal drama and burns out teams over time. >>It’s, it’s a second job. >>It’s, it’s literally, so like our as an example during, during Kenshoto CTF we had full time jobs and actual really life stuff to do besides you know CTF and then had an 8-month development cycle every year for, for CTF. And by the end of it we were just like nope, not doing this again. >>How many hours a week did you work? >>Oh my God it was literally, literally a second full time job for almost the entire team. >>Wow, we had DD so we went [laughter] 4 hours a week. >>I know, right? >>And he worked. >>And you guys were like, DD go do a bunch of stuff. [laughter] >>What’d you do this week? >>So um. >>What’s our contest like? >>Uh Lightening who wrote the uh architecture for this started in 2015. >>Yeah >>So >>Um, to be fair it took Intel a lot longer. [laughter] Um, so if you kind of uh, uh fast forward, the the previous slide showed a Kenshoto board form ’06. Here’s another one from ’08. I think maybe even the same code or the same style was used in 2007, it looks very similar anyway. Um, but you can see that the, the board is, is sort of uh, um staying the same, static, as Kenshoto evolved. Um you, you do get to the fine, sort of the areas of excellence in the categories, um there, there’s uh, uh an aspect of the organizer side of how approachable do you want the, the, the open qualifier. Anybody can sign up right. So how approachable do you want it to be for novice players? But typically, that’s solved with uh, sort of a trivia category and uh, and also the, the lower point value ones, uh where they’re, they’re very approachable to uh. >>Approachable enough that, that for all the years we hosted it the trivia 100 had the same answer. [laughter]. It was, well it was >>Hack the planet! >>There ya go! [laughter] It was literally hack the blank. >>So uh >>Blank the planet. >>The last… >>Hack blank planet >>And, and our last version was blank blank blank blank [laughter] >>Yeah. Yeah. They submitted a version blank blank blank and people just submitted hack the planet. >>We actually have a, we have a baby’s first category now. >>Yep >>So it’s all like baby’s first heap exploit. >>Yep, yep. >>So through the years, there’s uh, the next uh set or organizers. DDTEK, um obviously different uh look and feel but, but very similar. >>Really? >>Very [laughter] >>Uh similar semantics. It’s still Jeopardy, it’s still categories. Um, you’re still choosing the area of excellence so to speak. So it, its, when you go around the other clocks um, it’s it’s sort of all the same things right. Crypto, cross site scripting, buffer overflow, heap overflows, right. The, the, the boxes open into to any of the security related fields. Um, and then the, the current organizer right, um have a, a slightly more Japanese game show right. This is one of their, their earlier ones from uh 2013. Um, but the concept of qualifying the teams through this gate has, has persisted since it started. And it really uh filters down to the, the handful of teams that are gonna be present here >>Do you guys know >>Uh at Def Con >>Do you guys know anybody, anybody who’s been an organizer and stuff, do you guys know like off-hand the total number of like, I actually clicked, I am beginning to play. >>Yeah >>Quals? Cause those numbers >>If you guys will >>Are impressive >>Stop interrupting we’ll get them. >>Oh, okay. >>Um I, I should have said at the beginning that, that I made these slides and they haven’t actually see them, so there might some >>True story >>Disagreement and, but I’m, I’m trying to be, uh to be honest and, and faithful, I’ve uh, I’ve kept everybody in the dark. [laughter] So the basic organization of the game is the game, um has remained the same and um. You know other CTFs around the world sort of take a similar approach so they modify like how you get through the board and so forth. So, one thing you might’ve noticed, the uh, the astute observer, through the boards that we’ve looked at some of the same teams keep showing up. Like everybody up here and anybody out there that’s a team know that. But you see the, the sort of regulars the familiar faces that come back. And um, teams have really started persisting over time. Uh, um back even when uh the Ghetto Hackers, were, were playing and then, and then uh organizing. And uh these are teams that, that sort of prepare and, and practice and test tools and, and things year round. So it’s, >>Except for us >>It’s on a cycle >>Not the ghetto. >>What’s that? >>The ghetto did not. [laughter] Sort of I mean a few hours a week we, we, we drank a lot. >>You had DD. >>I said that teams existed Um, so they develop and maintain tools and teach processes and so forth and you can sort of track them. These, these are just points in time so it doesn’t say anything about their, their final placing in quals. But you see these same teams kind of over and over both in quals and also in finals. Um, so so much that, that it’s actually tracked somewhat formally now. So, this is uh, a website called CTF Time. There’s other ones but this one is sort of emerged as the, the predominant one and it’s, it’s sort of mostly opt-in. So, it’s you, you might not be as tracked as you want to be, but um when the results are posted publicly the teams are known and, and based on, ya know pattern matching on the IDs they’re sort of matched together. But um individuals can sort of affiliate with the team and, and opt in. But uh, um but these are tracked over time right so these teams persisted since uh, um for some it’s again a second job kind of situation. Uh there, there’s formulas, there’s there’s APIs you could use to extract the information and verify that you’re, your rankings are correct. Um, even so, still subjective right. Somebody’s still saying, oh that CTF was harder than this one by some metric right. And the, the formulas still had to be plugged in. Um there’s still interesting opportunities for cooperation and and collusion and what happens when you switch teams and all that sort of thing. Um, so how many will participate? Right. >>Woohoo! >>So we’ll go back to the uh the, the timeline. And uh, you can see the, in the in the bar graph, I don’t know if you can read the numbers very well but the, the largest bar is 414 and the smallest bar is 162. I didn’t have numbers from the, the early Kenshoto days. Uh, I don’t know how that worked. >>Yeah I don’t think we do either. [laughter] >>Uh hey I couldn’t, I didn’t have them and then I couldn’t find them so. But you can, you can see there’s a, a bit of a trend line right in the teams. And um, it’s important to know, which I think Visi was getting at, is how many teams are actually participating. Which is vastly different than how many teams registered. Right. So, I think last year there were 1500 teams that registered I think Vito said. So, 1500 teams registered 276 actually submitted something, right. Which isn’t a perfect metric, might, there might be some people that tried to play and just couldn’t figure out hack the planet, right. And they didn’t submit anything. >>Right >>It’s possible so it’s not a perfect metric. Um but these, this is a, a good approximation of how many teams are playing right. Not how many people, teams can have many people. Uh they probably have at least 8 in most cases but. >>Or like 50 if you’re School of Root >>Uh [laughter]. We don’t discriminate. >>There’s no rules in CTF >>Equal opportunity. Um so you can see a trending up um there’s, there’s sort of other caveats in that uh registering shadow teams might have some advantage depending on how uh the game is structured and how flights have values so having extra teams on your team might be useful. Um and, and so forth we can even in some cases dilute flag values strategically. So then, the, the other flip side of how many will participate. That’s how many play in open qualifiers. But how many are actually going to play when you get, uh to Las Vegas right. There’s logistics in setting up the right amount of tables and how many prizes and uh what kind of uh orchestration you need to run. So, um, when the ghetto hacker started there were 8. Right? And there was always 8. And where did 8 come from? >>Um I think we had, no it was probably Myles. >>That’s how many teams we could fit tables in the room. >>Okay. Fire code essentially. >>It was fire code and like lines of network and so we built some really nice PVC structures to run the networking cables, that’s where 8 came from. >>Yep that, my >>Did you guys >>My working theory was 8 port switch, or 8 port hub at the time. [laughter] >>There was never, th- you guys never had, I think we were, I think we ended up being the first ones to like im- implement like a team table limit. I don’t think you guys had that right? Because we, we also saw the instances of teams 8 or 10 or however many teams were in the ballroom at the time, we would see 1 table with like 3 dudes and another table with like literally people like crowded around with laptops and stuff. >>Yeah, yeah there were all starting fire code considerations there too. >>Like MUXT >>So anyway you can see that one of the things that DDTEK wanted to do as organizers is expand the competition acrossed uh several dimensions and one of them was size. So, uh, it kinda ramped up a little bit. Uh the first one should probably have an asterisk in that, there weren’t actually, 10 uh viable teams. There was one sort of deceptive team, so really, only 9 teams in the first year. And then uh the, the last year that DDTEK ran it there were 20 and that was, they were, we were sort of gently pushed because it was Def Con 20 to have 20 teams. Um, and then LegitBS maintained for a little while has then settled on, on 15 in the most recent years. Um, so how many teams will support is, is another question that you’d have to ask as uh, as an organizer. Alright, so this competition once you get to Vegas and these 10 teams or 15 teams uh get here. Um, how are you going to implement this attack defend game, right? So fundamentally uh you’re gonna be concerned about scoring. They’re, uh 4, uh I don’t know, here there’s 4 basic components but there’s going to be basic building blocks right. There’s offense, there’s defense, uh if you have defense you probably need some amount of SOA. And then there’s this concept of like other points. With like bonus points granted to the breakthroughs, or, or other stuff. So offense is going to be stealing flags or corrupting flags right you can, you can defeat and take one or you can burn one or whatever. And these have to be combined in some way to come up with a score that you can then use to declare a champion, right. So, score might be offense times defense times SLA. That has pros and cons, right. If, if any of those is zero your total score is zero so you could drive to zero real quick. So, if you have no offense do you get no points? If you have no defense do you get no points? Maybe that’s the kind of game you wanna run. I don’t know. Uh maybe you do, uh, uh more of a summation, right? If you add them all together now a zero score in any one of them doesn’t drag you to zero but it also doesn’t have that big of an effect so you can have um slightly more complex formulas where SLA maybe is the multiplier. Meaning you have to allow your opponents to play uh in order to score any points, right? If you have zero SLA it means nobody can attack you, nobody else can sort of play the game. So, you don’t, you want to encourage the game to be played. Um, uh and then uh, these need to work over time, right, we have the concept of rounds, you’re gonna to play all weekend so now this has to added up over these uh fixed, or variable length, 5 minute, 15 minute rounds, uh whatever. So now your round score is the summation of these. Uh, um uh the round score is the formula and the game score is the summation. And then you might have multiple services right, these concurrent services, these, these 10 vulnerable things that are running and your host that needs to be added in there. So maybe you end up with the formula, that looks like this down at the bottom. This is by no means the right formula or even one that will even work well but you kind of get the idea. There there’s some uh mechanics that go into devising the formula that you need to um have the score. There’s value in, in having this well-defined ahead of time. Many other methods. Um, once you even have these formulas figured out, you need to measure it all. Right? So you have uh, SLA, uh so this might be a port scanner but now they’re much more robust pulling, right. And uh, that’s very uh service specific, testing different code paths. Um, if a flag is corrupted for offense the organizers need to tell somehow. Um, there’s, there’s a lot of sort of game uh, uh strategies around uh sort of permanently overriding other people’s flags and things like that. Um, some moderate CTF’s employ custom kernels and, and hypervisors and traps and all of these things in order to detect that a override has happened. Similar protections for reads. And defense is sort of the absence of offense right? You must be doing good because nobody is successfully attacking you. So, none of this is set in stone, every year there’s a new opportunity to revise and come in with, with new scoring methods. Um, there are more questions here that, that uh um will need to be answered, right. Will there be bonus points? Uh will the scores always increase? How important is offense? So forth. >>Some of the important parts about what you’re saying, um like uh, some of the formulas he was just talking about, if I turn off my computers and start hacking, then the games over right? It’s not fun. Uh it it’s really difficult to convince security people to do blue team when they’re here for a red team contest. Uh that, that figuring out how to game things has been the evolution of the game. Really >>Absolutely >>Making it fun >>Absolutely. That’s the whole dirty tricks department. I mean we had, we had dudes basically going and like, so when we were playing our dirty tricks department was doing stuff like, we had a guy that uh did physical pen testing for a living like his, he was the guy that talked his way into data centers and did all that kind of thing. And so, he had like a high and tight, he was you know ex-military whatever. So he’d like roll around in like camo and like you know camo pants and like a, you know Def Con like, Def Con regulation black shirt or whatever. And he would actually, like never ever talk to us. We would send someone away from the table to like make drops and like meet with him. Who, he was rolling around doing things like brushing a CD off like, brushing burnt CDs off someone else’s table and he had like duct tape rolled in a loop on the bottom of his shoe and he would walk with a limp so that he wouldn’t like stick it down. But like camo guy walking with a limp, like no big deal so he would literally go by tables brush CD-ROMS off, stick them to the bottom of his shoes and walk away with them. [laughter] >>Yeah. Yeah. There’s, there’s many colors to that. >>Definitely >>Um, so these services that are running concurrently, you need to define, you need to define how much you’re going to define them. Right. Is there a spec for this? Like, how, what kind of a box are they gonna fit in? The more of a specification you have for a service, the easier it is to uh, to automate the test and the deploy, and the redeploy, and to rebuild and also to outsource it turns out. Because if you have a spec you can get, tell somebody to write against the spec if this’, if it’s an open box you might come back with something that’s really hard to, to organize around. Um, but lots of decisions about how many you’re going to run, what’s their point value and so forth. Um, need to decide how teams are going to interact with the infrastructure, so they’re connected, together, there’s some way that they’re interacting with each other. Um but they’re also going to have to, to interact with all the, the, the pie- the, the bits and pieces that you’ve assembled. Um, so will they operate their own defended host? Is it a VM? Um will they route through the table, right. Do they have an uplink that’s on a table? Do they get a tap? Is the tap delayed? So forth. There’s lots of uh, um decisions that have to be made about how the, the game is going to be represented and how it’s going be interfaced with. Um, one thing that I think, I think everybody another thing that’s continued over time is having a, a, an immense um desire to protect the integrity of the game. Right. And there’s lots of ways that you can, you can take that to heart and, and for, um for >>Specifically it needs to be just hackable enough. [laughable] >>Right, right it’s apparently, I mean it’s already a hostile network, right it’s already hostile people. Uh it’s designed to be vulnerable right. Uh but you wanna have a fair game, you want to make sure that when you’re crowning the champion you’re sure that that’s the champion. That it’s the right person, you’re confident in your scoring you’re confident in your infrastructure. Um so there’s there’s lots of different pieces. This is another slide that’s by no means exhaustive, um but you wanna take, uh I think that at least up until now [audio cuts] taken to protect the integrity of the game. Um and the non-technical side, there’s, there’s even things about table positioning, right. Is it unfair to put one table sort of like with their backs to the doors, when you’re opening the room? Uh is it unfair to give 1, 1 team a particular ID that might result in nulls in their shell code? Because of their network subnet and things like that, right. So, you really want to ha- um strive to be fair. >>Who did that exactly? [laughter] >>When we, when we designed the uh, uh that custom badge he mentioned uh unfortunately we, so we gave index numbers to people who placed in our qualifier. And PVP got first so we gave them the zero index. Uh, unfortunately their index was not exploitable on the RF network. >>So that’s one, one instance. Like everyone’s got their instance. >>Oh for sure. >>Um, so plan for failure, right. Like in this case, bring a Mac that can also do your presentation. Um but uh, and this is the kind of thing where I’m sure we can just go down the table and everybody can talk about the different failures but uh one of the failures that happened in, in this picture, I’m pretty sure this is the right picture was uh, a sort of essential scoring database had a hard disc just totally tank, right, right in the middle of the competition, right. So, you have to have, uh, uh ideally some resiliency um some like fail over, some backup plans right. Like uh the, sort of the game must go on mentality, right. Um, so what will the rules be? So, I think uh, typically historically the rule is no physical attacks right? And this is really, to keep people from physically being hurt. This is actually there and I think there’s good reasons >>Well and also >>To have that there >>Don’t cut our cables because we had to do that, like that would cost us money so don’t’ cut our stuff. >>Yeah, so there’s, there’s other things right, there’s, there’s >>Those were not the rules when we played. >>It sort of expands into things [laughter} like, don’t mess with the infrastructure right, don’t destroy personal property right, don’t toss >>Messing, messing with the infrastructure used to be like part of the game. >>That was the game >>Along time ago, that was the game. >>That was the game. >>Along time ago, we hacked >>Well this is like >>We hacked part of the scoring system and basically like I remember, I think it was DD that came up to the table and he’s like, that’s really cool, knock it the f**k off. [laughter] Like >>Did you get, did you receive bonus points for your efforts? >>Yeah, yeah definitely I mean that was always the thing is you kind of reward people for doing something novel and new. I remember uh, I remember the year that um we had uh, again sort of the de facto rule, don’t cut cables. Right? Cause, obviously you could walk around with a razor blade on your shoe and like cut cables all day long and so one of the things that we had said was you know basically denial of service isn’t actually that cool, right? So, like do something cool. And if you’re trying to do something cool and you mess it up, like okay we’ll have a little bit of forgiveness and it will be alright. So, I remember um I think it was School of Roots captain John Boss comes up to me, comes up to me and he’s like so Visi we were trying something really neat, I was standing, kind of the tables were like this and actually this was a lessons learned but we had uh, the sort of ring of tables for the uh, for the people running, you know for Kenshoto in the middle. And the tables had skirts sort of on both sides you know like the hotel does. And so, we’re standing there next to the table, you remember this story now. [laughter]. Right, so we’re standing there and this was actually the DTMF year. So, we’re standing there and he’s like so, what if a friend of mine theoretically was trying to do something really cool but kind of messed it up and now maybe a team needs a new cable? And I’m like, what’d you do? So he’s like well you see, and he lifts up the skirt of the table and there’s a dude underneath our table like right next to me and he’s like, well we haven’t been able to get him out for like 2 hours because we managed to get him under there to try and, so they had gone and bought a 900 megahertz phone and were trying to wire it into one of the other teams pots line to literally broadcast whatever tokens they were submitting over like 900 megahertz analog. So, they apparently messed up splicing it in ya know and like, like happens but then they also couldn’t actually find a window to get the guy out from under our table covertly and so he had been under there for like 2 hours. [laughter] And so like, John, John Voss lifts up the skirt. He’s like, so we’ve got this dude under your table and I’m like. That’s about it right? Something like that. Oh, there he is, yeah man. So, like, so, basically, we were like, yeah that’s, that’s actually really cool like, and well played you know, I’m getting people under our table. So, we just ran a new line and didn’t uh, didn’t cause a problem but basically that whole dirty tricks thing, like messing with the infrastructure, definitely was part of our, our game. >>On, on totally unrelated fact, when infrastructure moved to dedicated rooms I may or may not have been in balconies, the rooms locked at some point. >>But the balconies didn’t? Yeah. [laughter] Yeah, uh who was it that climbed across from one of the balconies to the other to get into the Kenshoto skybox one year? I- >>Uh, I don’t know >> I think that was School of Root 2 >>Yeah, I don’t remember. Um we’re talking about cultural influence, actually right before the talk there were some, some bits of culture slides so again this sort of an incomplete slide but um, culture is, is, is interesting in that um there’s a lot of um, of barriers right especially as this becomes an increasingly international right there’s teams from, from all around the planet that played, not only in quals but also in finals these days. And there’s, there’s clearly the language barrier, there’s clearly the popular cultures. Like uh, uh, um the hacker movies from the 80’s like some of them weren’t as internationally popular as others so like, those questions don’t resonate as much. Time zones and holidays right. Like there’s, there’s, there’s different things to try and keep in mind. Um, they tend to show up during uh, trivia and question starters and things on, on the right-hand side. Just generally when interfacing with the teams and when designing uh services that have ASKI protocols, right. The words sometimes have meanings to, to different cultures. Um, there’s also the cu- culture that’s influenced by CTF, where the community uh sort of leaves it’s mark uh in various ways. So generals and producers and actors and, and so forth uh actually come through and visit the CTF room and, and want to have sort of their own individual uh briefing and their own sort of explanation and you have no idea of the ramification of those conversations are going to be and how that sort of spirals out right? Uh so here’s uh, uh a team mentioned, uh School of Root mentioned in a, the HBO show. Um, another sort of thing to broach by the organizers is um how are you going to engage everybody right? Not only the participants but also the, the folks that are walking around and the average human attendee. So should there be ambiance, should there be a distractions, should there be attraction, like should there be music at all? How loud should the music be, should there be videos, should there be scoreboards, visual effects. >>Really, really, loud. [laughter] >>Uh there’s probably contention here about that. Um, but, but otherwise if you don’t consider these things it just ends up being a bunch of people in a fairly dark room staring at computers, right? And uh the teams don’t really have any physical interaction with each other. The attendees don’t’ really know what they’re looking at, so handouts and so forth. So, this gives way to visualization. So, uh back in uh, I- I- think this is about 2002, I don’t really know. I think this is a ghetto created scoreboard. >>Yeah, I think it is. >>Um, but we had uh scoreboards right, projected up to the, the wall. Um, here’s >>One of the, one of the problems with showing scores is uh, it can be really easy to get away with a win in the first day. And you c- you might have teams start to lose interest. This is part of you know, do scores always go up? This was part of uh, this is a zero-sum game and you can see the teams in the bottom have a little bit of a red bar and the bigger green bars are the, the top, top games. Eh, eh at some point you’re trying to keep people interested even if in reality you think there’s no point, they can’t win anymore. >>I don’t know if it was us who started it but so the first day we showed scores uh live scores, the second day we only show relative positions and finally the third day we just played back the previous >>I think there were, there were sort of a refinement process on that. We had something where like, I think for the last 4 hours we didn’t show the scoreboard, something like that too, yeah. >>Yeah, yeah, I think everybody has their own variation on that theme, I think you’re, I think you’re a little bit more extreme than has been in the past. Um, there’s some uh, some boards kind of through time. Uh, here’s uh, a Kenshoto board with the, the ordering and uh the team relative placement on a line chart over, over time. Um, a little bit later still a very similar scoreboard. Uh, even later in the Kenshoto time you still have a similar scoreboard but it’s also rotating with ones that have um, uh I guess innovative in some sense. Like showing different types of information. So, the one down in the corner is, is kind of showing what’s left on the table. These are the points that are, are still sort of up for grabs. The services that need some more attention. Um, if you open your data you get visualizations from others, so here’s some quals data that was taken by some uh, some uh sort of non-organizer folks and graphed over time and it’s sort of easy in, and different types of visualizations to see um, how the scores progress. So here you can see relatively quickly a lot of teams spike up and one sort of is, is in the lead for a long time but then there’s still a relatively dramatic come from behind a few hours before the competition, where the red line cuts up. So, this uh it’s, it’s quals so it’s a little bit different but this is an example of that come behind scoring that you like to embody in the game. Uh, if you want to encourage the availability for a come behind. >>Teams, there were also teams that were like fairly heavy into like using their lead psychologically >>Oh yeah >>To win the game, like team [audio cuts] [laughter] that spent the last 2 hours of, of CTF one year uh, very, very publicly and very sort of flagrantly having all their people just play guitar hero. Mostly just as like uh, you know you don’t even need to try anymore because we’ve got this. >>Whenever we say a certain team we mean Chris Eagle and School of Root >>Yeah pretty much >>Every time >>Just so it’s clear >>A certain team is always them. >>So there’s a new, uh new organizer took over, right, the, the scoreboard is uh, still um, relatively basic, it’s displaying the same sort of information. This is sort of one of the things that the game ends up being more important for the organizers so the visualizations end up being sort of back burnered and back burnered and then like cobbled together at the end, right? And, and that code doesn’t persist uh, generally from organizer to organizer. So, um you see this, this trend where like the new organizer that takes over, takes over and then it’s like sort of back to square one and it kind of builds back up, so uh, DDTEKs early boards looked like that. Um, then you know later on they had much more, um, appealing graphics, uh that uh displayed some of the same information but also like that’s a place holder screen on the side. And then in the end you, you sort of see the, um the the more appealing side. And some attempts at uh, trying to display types of information that are kind of hard to consume. Like the bubble chart where it shows team uh versus team action. This one is particular in the numbe- number of writes per service. So, that big blue circle is showing that like one team is like massively over writing on one particular other team instead of evenly distributing their attacks across. Um, and then you know like gadgets and stuff like that. And then another new team came, comes, takes over still relatively basic. They’ve taken a different approach where it’s sort of a graph based um thing where the, the edges display types of information but it’s still sort of basic blocks, not particularly appealing that advances over time, sort of similar information but much more appealing. >>Oh I’m saying that, that the, the blocks going across were live. That was our first iteration. I just saw the next one, it’s got much better >>Oh, yeah, yeah, yeah, that’s, that’s sort of the point. Right. It sort of evolves over time. And uh, and, and as an organizer you have an opportunity to decide how you’re going to try and convey information and what information you’re going to convey right. Because you want to engage the audience but uh, in some respect it actually also inform all the teams that are sitting there present, right? So, in some way this acts as a, a, an intrusion detection system. You can tell somebody’s attacking you and, right so that’s why you have the staged thing where it’s like live and then it’s like delayed and then it’s like gone. Or however you phase it out. But these are considerations that you have to have as a, as a organizer. Um so then there’s the, I’m just gonna rip through these pretty quick so we can get the questions. There’s the expansion into physical space, we talked about this a little bit already alright with the, uh going into the custom badges that are actually codable. Uh the, the meta game. Lock picking is super common in lots of CTFs, like as a physical aspect to get to your passwords, your keys to start the game or whatever. Um, another physical uh thing that was tried is a service that was in the game actually controlled physical things outside the game. Um, so like uh, this is the robotic chicken fight. Um that was a, a service in 2011. And uh, that was, it, it was sor- sort of an ancillary to slide, one of the teams actually went out and got like game controllers and, and pro- like, had an adapter interface to the service where they could actually automatically activate the service and then control it with their PlayStation 4 controller or something. Um, interesting thing, right? Um, so tradition, uh so future organizers, I, I think there’s a lot of aspects of tradition. If you’ve been a player or close to the organizers you get some of that, trying to like document some of that here. Um but there’s, there’s certainly a desire to keep Def Con CTF the best in the world. There’s certainly a desire to keep it fair, fun and innovative. Um there’s, there’s always desire to engage the audience. You never know what the next generation of players is going to be or where they’re going to come from. You really, really want to have that open to everybody. Um there’s logistics, there’s game banners and team banners. The winner typically gets to bring the game banner home as part of their spoils. Um, there’s swag, there’s t-shirts and stickers and so forth. Typically, announcements are happening on, on April 1st. Right that’s just the, the thing, you know when quals is going to happen. Um, fortune cookies started. I think, I think uh Kenshoto started that because they had a balcony and there was a desire to throw something off the balcony. So, there was the, the Sunday tossing of the cookies. >>That’s why we >>Oh yeah, the >>We get coins >>Fortune cookies >>They’re much bigger and they hurt worse. >>We had, we had the an- like the annual Sunday tossing of the cookies. >>Right >>Yeah. >>And so just some of those are here back in Def Con 15, some of these are DDTEK ones. DDTEK incorporated some of the challenges into some of the, um, uh fortunes. So, there’s stickers, right. Kind of a hacker thing, there’s laptop stickers and everybody’s got their stickers. There’s coins as uh, [inaudible] mentioned. Uh, now, sort of one of the longest lasting traditions. But there’s one for, for every year going back a few years. And then there’s sort of >>Holy s**t really? >>Other things, [laughter] right. >>An actual CTF tattoo. >>Uh, stress sheep and so forth. Teams bring stuff, you can’t stop them. They bring stuff, they do stuff, um, particularly like the sheep >>This slide should have been titled, will there be sheep? >>Um, there was uh, they modified the stress sheep to have the LED eyes. Um, so teams have to prepare, uh for a couple people who already brought up that, uh this is sort of a second job. It’s a multi-month, it might be an all-year round thing, like when do you start planning for next year? It’s a soon as this year's over kind of thing. Uh so there’s a lot of preparation, there’s a setting of the servers the configurations, building all the services, building the infrastructure if you don’t already have it. Um and then if you do, like the meta games stuff, you have to set up all the meta game stuff, you have to program the badges, you have to, we did uh, um DDTEK interfaced with all the human badges. So, I think if you have a human badge from DEFCON 18, you actually have CTF code in the firmware. So, that’s like a whole other layer of, of uh of working with other parts of DEFCON. >>Did you guys end up having the thing where like every year where you swore off doing it again? Like every year, we were like we are never doing this again. And then like a couple months later you’re like, well maybe. >>Well we have some like >>[audio cuts] we do this year next year it will be so much cooler. >>There’s a couple, like even now I think, eh maybe you could do it next year. And no, it’s written down, we’re done. >>Yeah, it was like the year we stopped, the year that we stopped hosting was literally the year that nobody stood up and was like, yeah, we’re doing this. Like everybody was just like, eh. >>Yeah, we we definitely got drunk at quals and like, yeah, let’s, let’s do this, we’ve got this. >>Yep >>Next morning, no, no, no. >>Not, no >>Just forget what we said. [laughter] >>So we’re talking about how much effort there is like year-round. When you get there on site, there’s an amazing amount of things that have to happen behind the scenes, and I say proportionally most of what you’re going to be looking at as an organizer is something like this. You’ll see like the empty rooms with no participants in it, you’ll be watching some of your, your buddies terminate network cables across the room as you’re trying to make sure all of your services work. >>That’s happening right now. >>And um, and this, this is the view that you get. Or you get a view in the back room, your, your infrastructure servers or something like that. Right. Uh, what the >>Also I think, I think since you’re, since you’re on the topic of preparation, it’s also pretty important to point out organizers of CTF, the entirety of the infrastructure and the code and the logistics um has always been completely something provided by the people actually running CTF. Like our actual, Kenshiro's logistics bill every year out of pocket for us was like 20 grand. >>Yeah >>Like that’s why I say don’t cut our cables, like seriously. >>Cause >>They’re our >>Because they’re our cables [laughter] >>I don’t know >>So if we’re talking >>How much it costs just to ship all the stuff out here. We had >>Yeah, yeah shipping costs, shipping costs alone. The pelican cases and all crazy nonsense. >>If we’re talking proportionally, partially what the team [audio cuts] something like this. Cause again, like the, the table are meant for fire code or due to prizes there’s only 8 black badges. Um, uh the teams are much larger than that or some teams are. And you see, uh you know extra hotel rooms with wiring all over it and you see the, um you’ll be staring at IDA screens and debugger screens and have your persistent servers that have UPSs strapped to the bottom of them so you can wheel back and forth as the game goes live. [laughter] So, so why do people play? Right? This is like, these last couple slides sort of like, make you wonder like, why? Like, right? Um so why do people play? Uh well there’s challenge, uh the, there’s some prizes especially around the world, some of the CTFs are getting like 30,000 dollar, 50,000 dollar prizes, right? It’s not quite like eGames. But um like you, you can’t make a living at it but you can make some money. Uh what if you just couldn’t get into talks at Def Con? [laughter] Right? So, you’re stuck there, right? >>I know right? It’s ridiculous. >> I’m convinced there’s some people, have logical disorder >>Glory, glory is the reason you play. >>Glory, yeah. >>So the really reason right? >>And that’s why it started, capture the flag, was to have a chance to go head to head. >>And you get the flag badge >>Glory >>It’s a black badge and a jacket. That’s why you play. >>Yep >>Yep >>Yep, oldest, oldest [audio cuts] like, around. >>So why, so why do organizers organize? Right? So this is actually the question. >>Masechism >>yeah this is the question, this is the first question I wanna, to, to hand off to the panel. Before we do that, just very quickly um, you, you roughly know who's up here but uh, to, um to introduce them sort of officially. Uh, Visi, uh wave your hand. Yeah. So Visi’s sitting >>I crowd sourced my bio because I’m s**t at writing bios. So, this is all the Twitter responses to what should my bio be. >>Yep, so he crowd sourced the bio, he was the organizer, uh sort of uh, uh main or chief organizer for Kenshoto. Uh, Chris Eagle, yeah down there on the end. Uh so he was uh, uh um a player and then a organizer and then a player again. Uh lots of CTF experience from, from both sides. Um, literally wrote the book on uh [laughter] on IDA Pro. >>Oh nice, dude hook a brother up >>I know, right? >>I got you. Don’t you worry. >>Riley >>We’re, we’re rebooting. Sorry. >>Caezar >>One second our monitors are not working. They’re giving epilepsy out. >>So Caezar is talking >>Momentary technical difficulties >>Right now. So he’s, he’s part of the Ghetto Hackers who, who won 3 times and then became the first formalized organizer and really brought it, brought it to the next level. Also known for uh, for uh, C, uh Caezar’s challenge um, sort of an annual thing at. >>21 years I have thrown a party on Saturday night because uh, behind a masochist and doing CTF wasn’t enough. [laughter] Uhm find me at any time and I will hand you a puzzle and if you can solve it, which is not very hard this year, uh if you can so, solve it then I buy you drinks on Saturday. >>Uh, Hashen. Um, uh, uh also, excuse me, 2-time champion. Uh unlike the other champions, what 2 different teams, right? >>2 different teams >>2 different teams >>First ones Meta Gods slash Team Awesome the second one's Samurai TA >>Yeah, there you go. >>Um, and um one of the current organizers, uh 2013 so we’re not really sure when they are gonna stop. >>This year, definitely. >>Yeah we’re not sure when they’re gonna stop. [laughter] Uh, and uh and Myles. Right. Um, Myles is, is uh responsible. He’s like the reason that we’re all here, right. Cause this wouldn’t have happened if he wouldn’t have started it so. Um, >>I know that our presentations not gonna do it but uh, I’m gonna- it’s time for everybody to give this guy a big round of applause. >>Specifically >>Capture the Flag >>Yeah. [applause] >>It’s Myles’ fault [applause] >>Hacking, hacking is basically a modern watchable sport. I mean there are Twitch feeds now for this kind of thing, right? And mostly it’s all because of the, the bucket that you started kicking. >>And uh, and with that I’d like to um kick it back off and I think the first question will be, so why do organizers, organizers organize and let’s go uh, in order, right? In chronological order. And uh then this room didn’t get set up uh well, we don’t’ have mics so if you have a question, uh you have to come up here and you can either take the mic or, or I can repeat it. And also I’ll put up uh, um I’ll try this thing where you can put up a URL and you can like send questions to the podium. We’ll see how that works. >>I would not trust a single person in this room for that. >>Yeah >>Alright, goes through Google stuff. >>So Myles. >>So Myles, yeah >>Why CTF? >> Why, why organize? >>Because there wasn’t a way to figure out who was best and not practice out on the live internet where the con would get shut down. [laughter] Now you, we forget that >>Who was, so you’re saying it’s harm reduction? [laughter] >>For the public >>Totally, and you forget that like at first, I- I- went behind him because I was seriously worried if my employer's heard about Capture the Flag I might be fired. And I worked as a security guy. And this last year I was in the Smithsonian as a Black Badge there, I mean this is really different. So yeah, it was harm reduction, it was head to head, it was a chance also if you had controlled the environment that you could start throwing in some stuff that explains it to the general, about why would someone stand in front of a screen for so long. Well because it’s really cool and there’s these things, it’s a puzzle that they’re solving. So, it’s clear to me that we needed something with a big more showmanship, and also chance where attackers and defenders could go head to head. And that it didn’t break the rest of the network and throw out the con. >>Yeah, uh, for, for us we came and um, we felt like we knew enough about security to not necessarily need to see every single talk so we’d catch [laughter] a couple of talks and enjoy it and then we’d kind of wander over and see Myles’ game and say, ya know, gee, so I just set down at a table. Every person at that table today is a dear friend of mine today and like we go out together all the time, we live near each other we’ve all move to, to live near each other. So, my social life is Myles’ fault. Uh the Ghetto Hackers was formed because we didn’t have any paper, cause we didn’t bring pens or paper or notebooks or, or hardly anything. Uh so we had a napkin and uh mascara and uh we wrote IP addresses that were available on the network. That was our, our group of friends came out of us sitting around each other. And, for us uh after the first year, the first year we just sort of tried and saw how it was and, and then after that there was kind of this fire of, well you know, if we just stored every exploit we know. And that w-, that was the contest back then, was like how many exploits can you bring in a searchable useable format. And uh, and i- j-, i- b-, it just became this, this passion to actually just get better. So, for, for us we didn’t necessarily try to win as much as we tried to be c-, get good. Like get good scrub. That, we were scrubs and, and we came and you kind of made a crucible and a bunch of hackers popped out. >>And then why did you run it? >>Why did we run it? Uh, we ran the contest uh, after the third year, uh somebody in one of the teams said that they hated us and we were cheating bastards and that they didn’t wanna play. >>Hackers, hello. >>We, we, we took it as a great honor, and pride? And but, but there was something uh, a, spark in, in, in, in our, among us that, um made us jump up and volunteer to take the game to a new level. And so uh we got up on stage and talked about coming back the next year and making a new contest. And Myles was part of that contest as well. Uh and, and that was how we kind of came to the uh, scoreboard and, and, and sort of all the th- some of the the trappings that are the beginnings of what these guys who won the contests that we threw and we handed the reigns to them. >>That’s actually kind of been a tradition. I know it was kind of touched on earlier but there is also a, you end up feeling this sort of sense of, of belonging and ownership to the game. Um, when you’re al- when you win it also. Um, and you have your own designed on how you want it to be or how you think it should be so, like transitioning to the reason that like Kenshoto uh decided to run CTF for a few years. I think there’s, I mean there’s a number of facets but for us watching, watching the Ghetto Hackers and watching the coalescents that it caused in the hacker community specifically. And the and the crucible effect actually is super huge, like I, we would watch people literally have a reason, have drive to learn all these new exploit mechanisms, and, and all of these like details about- part and whatever. So, for us actually I think that the biggest thing that, that caused us to actually run CTF was uh, we were concerned initially uh that it would be taken over instead of by someone that is about the hacker community, that it would be taken over by a university or a corporation or whatever like that and we were very concerned about that initially. [laughter] Turns out, ya know, off we go, modern times and all that. But um, but the other reason um, the other reason that Kenshoto stayed in the game, the reason that, not, not the first year but the reason for the years after that um, was that we really felt that it was sort of an unmeasured uh like an unmeasured resource, right? So as an example, um in our game we were sort of one of the first people to be like, don’t bring any of those exploits because they’re not going to work because it’s going to all be entirely custom software with challenges that you have to reverse engineer and exploit and land payloads against like, in that weekend. And for us that was um, it was really important for us because we felt that, that was a, that was something where people hadn’t stretched far enough yet. The, the bar wasn’t high enough and we felt like um we felt like we wanted to continue to be able to push that higher and higher. So, that eventually not everyone was making it. So as an example actually there’s also, what, what do they call it? I think they rename amateur CTF but it’s called something else now. >>Open CTF >>There’s several now >>Open CTF, right >>Yeah open, project too >>So, but like that >>Subtle >>As an example >>Subtle humble brag there >>Oh, yeah right but so, that is essentially formed out of the fact that we pushed, we tried to push the game into actually measuring the real redline of what people were capable of in a weekend. And that, that really drove us for the years after the first year that we hosted. The first year was cause we were super concerned that it was gonna become this like corporate sponsorship kind of thing. Um and after that it was because we thought that the, the, the performance that we’d seen uh, warranted further measurement. Higher and higher so. >>And that is a perfect transition to him to Mr. >>Right >>Cause we, we had a lot of the same reasons. So, we played in Myles’ game, we played in Caezar’s game, we played in Visi’s game, and uh we loved every iteration um I teach and it turned out that the game as it evolved was it, was a great microcosm of the security space. Uh in which to uh conduct uh teaching an- and learning. Uh my students got really excited about it so w- w- we loved the game as it was, loved uh, the, the game uh the last year we played. Uh with Kenshoto and when they stepped down we were also very worried that ya know, some company would come in and commercialize this thing. Uh, and it, we looked around and would say, well who's gonna to run it. We could, we couldn’t imagine who might run it. >>That’s exactly >>I think there was one other that might’ve run it, we didn’t know what they would pitch. Uh so we made our pitch. I think another team made their pitch um, and uh we basica- [off-mic comment] [laughter] >>But you lost. And >>Our, our [laughter] out, our pitch, our pitch was a lit- was basically if you don’t’ give it to us it’s gonna suck. [laughter] >>Ours was if you give it to them it will suck. [laughter] Um, uh and we really like the game and wanted to see it continue as much the same vein, of course it didn’t’ occur to us that we wouldn’t get to play for 4 years. Um, and then >>Playing is way more fun then run- >>Why we kept running it was more or less, well, we screwed that one up, maybe it’ll be better next year. Um, I think we might have got it, we got it close to right the >>Your last year >>Fourth time >>Was good >>Yeah we got it close to right the four- it actually started on time. >>It did? You didn’t delete all of our scores. >>Yeah, so um >>Chr, Chr, Chris. Before, before you, before you stop Chris, uh, how did it change your team’s experience, your school’s experience to go from playing, to running it, to playing. What d- do you play differently because you ra-, because you ran it. >>No, uh, playing to running is a really tough thing so uh, to e- it’s an entirely different mindset to become an organizer. Right, so yeah, if you’re, if it you’re doing ya know different type stuff and then now you’ve got to turn around and write it. You’ve got to write with an entirely different mindset. Uh writing a challenge is no easy task. I- it’s easy to put one bug i- in the challenge of a very specific nature. It’s a little more difficult not to put other bugs in that you >>it’s really >>Didn’t mean to >>Yeah, it’s really hard to write secure insecure code. [laughter] >>Yeah >>Just, just exploitable enough >>Yeah [laughter] and uh >>And, and >>So >>How many of you have found exploits in code you’re adapting? Cause I know I was finding exploits out there, it’s like wow that’s been vulnerable since the 90’s. >>We actually >>I’m the first one >>Had 2 challenges that, that literally just incorporated a library that we knew had an o’day in it. >>Yeah and we wrote challenges like that too. You take things from real world and you try to bake it into this challenge, uh, w- we didn’t want to drop mountains of software, like ok here’s a patch you find the o’day. Uh [laughter] which was easier, ya know, 10 years ago, but uh, eh so yeah everything was pretty stripped down and you, you try to build the bug you want people to hit and you try not to have other bugs. Um so, so from running to organizing th- that mindset was hard for some people to, to uh, uh a- adapt to. So, we did lose a lot of people actually from the, from the playing side. They, I- that’s what they want to do. They want to keep on playing. I don’t blame them. Um, but yeah, we were a much smaller group when we uh, during the time we spent running it. >>That tells into another question. This things actually working so we’re getting questions that are coming in. Uh so, uh you wanna answer HJ >>Oh sure >>And then I’ll follow >>I think I think uh >>Into the next question >>Short and sweet anything he can do, I can do better. [laughter] >>We played um TEK’s game >>Uh and you got 2 black badges out of it, uh sorry. >>We played their game, got 2 black badges and saw what they did. And knew how we were a**holes and how to beat ourselves. And decided we could make an interesting game. >>Masochism >>What was that? >>I was just gonna say it all boils down to masochism. >>Yeah, yeah, like you know what, yeah we, we made people upset about how we beat their defenses. Anyway, um so yeah same, actually same reason. Like we played their game and thought we could do better or do things differently and take an it an interest, interesting direction for the community. So, >>And, and they’ve done an amazing job and I will say that one thing as an organizer, the only thing I’ll add is the only fun you can have in organizers is trying to f**k with the players. >>Totally true, absolutely 100 percent true. >>So whether it’s a challenge you designed or some new twist, the badge challenge, the >>I really support >>Get a kick out of >>Or challenges, or challenges that were actually purposely not actually exploitable. There was a bug but it, like couldn’t be landed. >>No w- we, that’s why, our are always exploitable. [laughter] You thought there was a bug and now you want to say you were f**king the players. >>What’s that? >>Yours are always exploitable are zero. >>Oh that was an accident. >>For one team, yeah >>For one team, yeah that was th- it was exploitable. >>So the follow-on question which uh perhaps I’ll ask uh Chris first since you already started answering it. Um, how big are the organizing teams? And you can sort of take that with, with the transition from player to organizer and everybody else can answer it. >>Well we went from about 900 players down to [laughter] uh, you’d probably accord it topped out around 10. But you know, but there was you know even a subset to that, that was a little more active. But uh for the 4 years we ran it. >>We were around 10 as well, uh >>Yeah, we, Kenshoto was around 8, we had a couple outside contributors. >>Yeah but composed is funny, like people who write challenged people who organize, the guy I knows actually the head, uh he’s, he’s more like having to deal with uh, Def Con stuff. We’ve got one guy who's just an amazing network guy. No joke, he is the person that Cisco calls when they can’t answer a problem. Not even j- not even joking. >>Yeah, we were um, as many as 14 but we did ya know we made movies, we did all sorts of stuff but it was really only I think um 6 or 8 people that were the core of our team. >>You were 1. >>Yeah >>Yeah >>My- Myles was >>Army of 1 >>At, at the end was it still 1 or you drawn in a couple people at the last [inaudible]? >>It was 1 >>Yeah >>And that was why it was time to hand it on. >>Yeah, you lasted, you lasted longer, you’re a good many Charlie Brown. Um, so it turns out that this thing allows you to vote, there’s actually like most popular questions. Um, what is the most unexpected way someone solved a challenge? So, there’s like an intended path and then there’s like unintended paths, so. >>Yeah there’s a lot of those. >>I mentioned they used free BSG jailbreak [laughter] to get out of our jails. But, the we, we watched them do it and they were so inept when they go out. >>Yeah, people, people bringing o’days, people bringing o’days and then completely messing up using them correctly, would be a, that’s a good one. >>My favorite one was I had managed to let’s see, Quake was the hot game in Def Con 4, 5 so I managed to talk its software to donating Quake licenses to the Capture the Flag contest. Like Quake servers and a couple station setups so that people could play Quake and then I got free video games out of it. So, all good. Um and so someone came up to me and said, you know we’ve dialed servers but it’s not on the network. We can attack the quake server and one of the people gets so p**sed off he goes over and reboots the machine and that causes it to reboot so we can run our RC script. I said, go for it. >>We uh, we convinced a team to surrender. [laughter] Give us all their points by telling them that we were about to win even though they were ahead. And so they joined and all became Ghetto Hackers. >>Actually, I’m gonna, I’m gonna relay uh something we were talking about earlier act- an anecdote that we were talking about specifically in the Ghetto Hackers playing. So, it wouldn’t be an instance of Caezar having seen this occur as a novel way to score points to win the game, um but having executed a novel way to win the game, uh they basically convinced the CTF organizers at the time that they needed, they needed store a half rack of equipment that they had, so they had ya know, your little like uh what would it be 20, 20-ish u rack. Um, and they had a completely built out entire faceplates of real servers and stuff and they like stuffed a person into the half rack and actually got them to store it in the room with all the rest of the CTF equipment and infrastructure overnight. Because, well we need a place to put this, right? And so obviously out creeps person >>Single roots all the machines >>Single roots all the boxes >>Reboots them all >>Peace >>Puts them in single user mode, roots them all. We come back in the morning and then not prepared or worked or done anything the night before. We come in with like, I, we’re pretty been drinking all night, uh the contest is over so let’s all go home. And we got a whole bunch of points and it was, yeah it was pretty good. The contest was not over. >>Oh no I >>We did not go home >>I already mentioned the, the bashrc one which was fantastic. When we were playing, we once snuck the, the root search for one of the teams and got first blood on every service. That was during your, your game. >>Uh, this worked pretty good I can just go to the next most popular one. Um, so how do you come up with unique uh flags or challenges I think, uh I think it means challenges or services uh every year, aside from following CVE’s or meeting content from other >>That was actually a serious source of exhaustion. >>Is is >>Like >>Actually >>Like mental exhaustion of attempting to come up with services. And we used to even like, we would literally scrap 2 or 3 services every year because they wouldn’t’ get deployed correctly or they wouldn’t get done. But like coming up with like unique idea of like, uh here’s a web service that helps you make a sandwich like or whatever. >>Well and, and now with 20 years of Def Con history >>Exactly, now you wanna like do something original like >>And that’s just Def Con history that’s not talking about every other CTF >>All the other CTFs yeah, for sure. >>100 plus probably per year, how do you actually be unique, you can’t really be unique anymore. >>NO definitely now >>That’s why >>We would actually, we would actually basically, the real truth of the story how we produced our services is mostly about getting a bunch of people in a room, getting a bunch of whiskey in the room too, and like just bulls**tting until funny ideas came out. And then like codifying that list into services. And then, and then going in and deciding what kind of exploit goes into it, what kind of whatever. But every one of those services had a little >>Oh and for us, what we’ve done is we’ve really introduced multiple architectu- architectures. Uh, 3 years ago I think we had 5 different architectures uh, in 1 game and it was just XX6 and, and NIFs there was, we even had a Windows IoT arm challenges running a powerPC shell, or something like that, I don-. And this year of course I’ve mentioned we have the, the custom architecture that we built. But it really is, we’ve had to push it >>Nobody’s quite explained this. Custom architecture means >>Oh right >>A system for storing bits in electronics [laughter] >>Right >>So imagine >>Instruction stacks >>They implemented a CPU >>Processor >>Yeah we implemented a CPU >>A processor of VM all up on top of nothing. It’s, it’s um whole cloth, it’s fantastic. >>Lightening actually did it, but seemed to handle it. >>Epic >>Yeah it is pretty good, um yeah, it’s pretty much, we’ve had, and the caliber of people who are playing are just phenomenal, like it, they frighten me. People like Load Key and GeoHash, scary good. So the, the challenges, the level that we have to do for the difficulty has astronomically increased. Um the, the medium challenges back in the day are now easy challenges for everybody. >>Yeah, tools, tools have gotten so much better too you don’t want to write a, a challenge that’s gonna be auto solved by a tool, under, ya know under a minute. >>Auto solve you just, yeah >>That’s a good segway for >>You have to keep up with the state of the tools and uh try to find the weaknesses in the tools so that the, you still will get the human you just gotta do the deep dive. >>That’s a good segway into the next, actually set of questions. There’s uh several related to uh, CGC and, and sort of automation. Um lessons learned from, from Cyber Grand Challenge. It’s been a year, was it a success. >>Hackers are obsolete [laughter] >>Failure, I’m looking for one, I sorted these and now I can’t find one that I saw earlier. There’s this one, mostly for Visi, how many years until a computer wins Def Con CTF, right? >>Uh, that’s. >>So the, the thing that the, the thing the computers have trouble with right, is the dirty tricks department. Um, and so I think it will be awhile unless a game is designed specifically for the computer system, I mean that’s really, that’s really kind of where we are at like right now I think CGC was a really good example of this just massively forward in technology but at the same time you have to keep in mind that um it was a reduced instruction set and a reduced sys call set and all these other things that add a really sufficient level of complexity that I think there’s still a lot of room, a lot of wiggle room for like the human dirty tricks department kind of mindset. So, I think it’ll be a little while yet. But it depends kind of on the structure of the game, so if elements of the game require creative th- require creative thought and unique approaches um, I think that’ll continue to kind of be an arms race. And, and >>I only think it matters if they have arms. If you have to actually go walk, pick something that the computer won’t do that well. >>Well and actually that’s a really good example, because, because you guys were notorious specifically for incorporating into the CTF a whole bunch of side games, where those side games were like, every team gets issues this ancient hard drive that was like gigantic and, they’re gon- that hard drive is painted your team's color and it’s going to be out at the Def Con shoot which I think still happens, right? >>Myles >>Yeah >>Myles >>Yeah that, the incorporate Def Con shoot was mine, >>Right >>That turned out to be a really bad idea [laughter] >>Right, well so basically I think a team got a, a team got some bonus points for like having like basically punched a hole with a bullet like through the most center point of the drive or something like that. Um, computers aren’t gonna do that for a little while yet, so. >>Th, the other aspect with computers is you have to freeze the game, right chess is, doesn’t change. Chess is chess, is been chess forever and a computer knows how to play chess and if you wanna >>How, how would the computer, how would the computer deal with this new architecture, >>Yeah, so >>Right? >>You know >>Read the manual >> We wanna see something different every year, we wanna throw curve balls every year and its, unless you advertise that some amount of time in advance, like the CGC version that you guys did last year, uh it's, I think in Def Con CTF in particular I don’t think we’ll ever see it because that’s not the game we humans want to play. >>All, all we’ll ever need is 64K of RAM right? [laughter] >>So uh, it, it probably wasn’t clear earlier but 4 people up here were involved in, in the CGC competition some way and then for those that aren’t aware Def Con CTF last year was mostly more or less CGC compatible, right and the winner of CGC the, the machine was a player in the CTF competition last year which kind of ties all these questions back into the panel. So, uh it’s sort of a related question and um I won’t direct at anybody in particular. But, so it’s been a year right, CGC was last year and the CTF that had the computer was um, was a year ago. Um, was that a, a success, a failure was it sort of um, a, a >>I would say that >>History >>I would say it was definitely a f-, a success. I really am a fan of the fact that uh anger was really open sourced. Um, I think >>Pushed tools >>Right getti- th- getting these tools out to th- the general community is fantastic. The fact that anybody can go and use these and learn how to write software, write tools, to automatically RE and go towards uh exploitation is fantastic. Um, when you can m- lower the bar s- uh to entry it really makes it that much better. >>Learning uh, learning how the teams did their job last year is probably the best thing that anyone in the audience, any of us could imagine doing for their career for the next year. >>An, and to be honest even >>State of the art >>Uh, th- future of ARRI just in normal careers is gonna be automated, it’s go- software’s becoming so incredibly difficult, some amount of automation is going to be required, just to get even the low hanging fruit anymore. So incor- we’re actually, we have to incorporate this idea into CTF so that we can still be representative of the wider security community. >>Yeah, I think that’s, the mistake people make when they look at the CGC is to think that the goal was to build a purely autonomous system and that wasn’t the goal. The goal was advance the state of the art in software a- analysis, and wh- what, ya know we’ll see last, we saw last year, we’ll see it this year, is that uh, software automatons assisting humans make it, ya know making humans better at what they’re doing, is is probably where we’re going. The best chess playing systems in this world are hybrid systems that pair computers with the best players. >>Yep >>Not, not purely computers. >>Yeah, I think, I think we’ve seen that too in like, the a, even other CTFs now all with modern things like anger being applied in an automated way to catch some of the low hanging fruit or point out areas of the code that’s like, this probably like, we don’t, it can’t necessarily automatically generate an exploit for every condition yet. >>So we’ve actually had challenges that required um, anger or some automation, automated reverse engineering. That was >>Absolutely >>Like I think it was called a Thousand Cuts by Vito. Where you do, you were given a thousand binaries really fast and you had to be able to exploit them in computer speed, not human speed. Um there’s even a challenge on pwnable dot KR, called AEG where it requires you, you to download it and it gives you a random binary every time and you have to, uh auto RE it and that, write the exploit. >>So how international has CTF become? Um, when did international teams start showing up and do Americans still stand a chance? >>One of the, one of the first uh things that I remember happening was uh, Dillon Kanabraun from Hack in the Box. Uh reached out and asked uh, if he could take CTF obviously, do it yourself I mean do whatever you want but, uh that was uh taking CTF to Kuala Lumpur, Asia. Uh, in I think 2001. So it’s, it’s been coming for a long time, it’s been s- diversifying and spreading rapidly uh, it’s I think one of the most true and honest ways that hackers can really be better than each other. Because we’re not very good I think otherwise at knowing like am I good enough. And being able to say, well I’m better than that guy is a huge, huge uh foundation to stand on >>Yeah, we have uh, competitors from all over. We’ve got Korean and Taiwan, Taiwanese and Chinese and m- most major areas of, of the world are representative, like are represented and um, they actually, 2 years ago Def Corp from was it Korea, actually won our game, so it was a, one of the first times that I can remember, and outside one, they were mostly Danish um that they’ve won CTF. >>I remember actually it kind of, it kind of like goes back to some of the slides you were talking about earlier with the language barrier stuff. I think the first like fully like non-English speaking team um was, was f- was a couple of the Korean teams that were fielded early on in the Kenshoto CTF and like we actually had to really specifically sit down and think like, what are we gonna do about this? Because we don’t want to create a game, because the game isn’t about English, the game is about bits, right? And so like we actually tried to create like e-, we actually went through several iterations of attempting to create like pictographic representations of like stealing a key, key submission and things like that with like, like you know stick figures and stuff like that, because the language barriers, is so significant that, um, I remember actually one of the um, the captain of the Korean team at the time, um they did amazingly well in quals. They like whomped everyone in quals that year. They show up, uh at the actual game and they had a lot of trouble but it was mostly about understanding what was going on, and the and the actual mechanism of the game. And the, the captain of that team came up to me and was like, I think this next year we’re gonna work on hacking English. >>Nice >>Uh some of those stickers, I think that might have become Ken’s stickers later on. >>Yeah, no they were totally stickers. >>So at the top there you can see >>Oh yeah, actually, no that’s, that’s exactly some of what I’m talking about. We were like, yeah. >>So, I mean clearly you didn’t make those right then but like that became a thing for later. Um so I guess speaking of that uh, what’s with all the, the Japanese or like sort of Asian references, right? So, like when Myles ran it, it had like the big >>It’s his fault >>The, the red uh >>Did ever pick out uh, Myles >>Japanese thing, right >>No I think that was actually >>It was Caezar >>Back that up, oh ok >>We did, uh we decided that we wanted to give away a championship belt. So, we gave away the Root Fu Championsh- it, it was like a boxing belt, like >>Like a literal like WWF like belt >>Yeah, yeah, so we made up this whole concept of Root Fu. And it was gonna be, um ya know a measure of how good you could compete against other people. And it basically turned into hacking and and, and the whole thing fell down but as a theme. You know we had um, uh all the cultural references, in, in, in Blade Runner and hackers, and I mean the connection to >>CyberPunk >>Yeah CyberPunk, uh yeah Chatsubo. Uh I think I was just kind of the gestalt hacker meme at the time. Um as were a whole lot of bad ideas and things that we’re growing past. So, we don’t have to live in that world forever but uh I think it was just the way that things uh, everything grew up then. >>We kind of just followed suit. I mean Kenshoto actually the like the name is in Japanese so. For us there was that tie in but it was mostly out of the influence of that same sort of cyber punk picture that you guys had created. >>And, and one of our uh, founding members uh Praisin’ Angel uh is Korean, uh woman who uh fits in a half rack it turns out. [laughter] Uh, and she and uh Data Angel uh did all of the what we call the Ghetto News Network. We did um news videos that were kind of in um the Blade Runner if you remember the movie the Blade Runner on the screens there’s uh, some kind of Asian commercials I think over like a soda. Um >>I’ve seen it yeah. >>We kind of took that image and we just blew it out and made a whole fake news system. >>They had her basically congratulating teams on not f**king up the network as I recall correctly. >>Uh, yeah so I just, had some back up slides to show some of that, there’s the, I can’t remember what it's’ called but it’s like the Japanese for uh, um, uh something religious the red thing in the background there. And uh, certainly the Japanese stuff continues uh over time. Um, so what’s the oh I wanted to, this one. So how do you find middle ground uh, in the challenges between the ones that are too easy and the ones that are too hard? >>Everything is too easy. >>Yep >>Hackers are so much better when they’re under pressure than we are when we’re sitting around trying to be cool. >>The way our, we had to modify our scoring algorithm, w- the way we do it, is uh so for quals everything starts out at 5 points except for baby's first, and the more people solve it the less it becomes worth. So it just kind of, they were self-correcting. >>Yep >>Cause we, we’re it’s funny if you write a challenge it’s really hard to judge how hard it is when you write it yourself. >>So, so wait it’s 5 points when I get it but then if so many people follow along, then I lose my points? >>Right so its >>Yeah that’s fantastic >>We also made >>So much smarter >>We made other people in Kenshoto g- um actually do all the challenges with no prior knowledge of them and so >>Testing >>Yeah testing, essentially >>Time is the problem >>Because the problem is it’s also really it's really easy to implement a challenge where there’s some like leap of faith that you didn’t realize you made and then other people wouldn’t and aren’t part of the logical analysis progression. And so that being the case like we, we had found challenges, actually there were 1 or 2 in your guys’ game too, and there’ve been others and specifically in, usually in quals rounds. Where like there’s some like leap of faith required if you, actually I think the, I think the best examples are listen, listen to the crazy a** explanations of the Def Con badge challenges where they’re like, I realize that these dots were actually morse code of GEOlocation things that were airports that then if you arrange the airports and then reverse their letters and then just all these things that you’re like, why did you think to do that in the first place? And so, like we didn’t, we didn’t want to have challenges that were, that were, why did you think to do that in the first place. We wanted there to be this sort of thread through them and, and essentially the way we imp, enforced that was just entirely, here you go other person on Kenshoto like go break this and solve it. >>It’s also really easy to make questions that are easy for your top 1 percent. And so, it’s like no one gets it and the top 1 percent gets it in 5 minutes and then where’s the rest of the contest? >>Well but that’s, so for the, for the real game that’s what we were shooting for cause we were trying to we were basically, if anyone was topping out that R and solving all the challenges they weren’t hard enough, right? So, for us, we, we considered it a failure in any year if someone solved all the challenges. And so, I think that they like the pushing of that scale to, to being higher. We didn’t really concern ourselves in the real game with that approachability or, or middle ground but in quals we did because that’s the thing that lie everybody gets to play everybody gets to be inspired and do something, right >>One of the things that we did for quals was we really, prior to us they had things lie forensics, trivia and because those weren’t really necessary for the actual finals game, um we only did, ya know Puntables, we did like gorilla programming, things that will, you have to know in order to play, uh succeed in finals. So, forensics where you have to go and take ya know the first bit of every sector of the NTFS dri- or section ah that was terrible. That was also before you did flag open brace cl- closed brace. That’s when you were guessing. >>Yeah, we left those in because we wanted be, we wanted quals to be accessible to a large number of people. >>Yeah it was a conscious decision for us too >>That’s why we added baby's first because we wanted that same approachability without it still be you know directly uh, ac- or uh be directly applicable to the game. >>Playskool my first tp exploit >>Yeah >>Um, you guys ever seen evidence of collusion? [laughter] >>So, uh >>We literally >>I was going to say >>Colluded with teams >>Teams >>All the time >>Won via collusion >>We only won by collusion [laughter] >>Uh where’s, where’s Zeon is he here? >>So >>There he is [laughter] >>So I think at, at this stage in the game for us most people have at least begin treating it like a gentlemen’s game. Like, we genuinely want to see who is the best and other teams feel the same way. For example, uh a few years ago, there was a team for some reason who put a wiki up on the game network and that wiki had their passwords on this, and one of the, the teams actually the [inaudible] officers came up and said hey guys, we saw these guys had a wiki up you should probably tell them. And that was during finals, where they could have gone just completely wiped all their flags, and just ran the board. But it’s really become, uh, uh like I said a gentlemen’s game. So, for the most part >>Whereas like, whereas like in our game like we were owning other people’s laptops and stuff too. Like if it was connected to the network. >>So that’s sort of breaking outside of the intended path, but that’s not really collusion. Right? That’s >>No, so, but there was, there was legitimate team, there was legitimate team collusion at least in the, in the, in one of the ghetto hacker wins and, and uh, I mentioned Zeon and Evans, stuff like that from back in the day. Because it’s that’s actually really hard thing from a game mechanic to detect and so we actually had, so I mentioned the idea of like, ha, ha to mess with the teams we had services that weren’t exploitable but they were actually canary services because if someone submitted a token from that, from another team either the game mechanism was broken and they managed to get tokens they shouldn’t’ be able to or they’re colluding. So, we were very concerned about that because it’s almost impossible to totally prevent right. If you’re just like slide a couple of tokens by here and there between 2 teams to that they can like get, get way far ahead you make it so it’s a race between 2 rather than 8 or 10 right. >>Yeah for us the, the biggest problem for collusion happens in quals. So, you have less control over quals than what’s going on in the game itself because it’s not on your own infrastructure, they’re not in a room >>I actually think uh the, our game mechanic that we do where if you submit more than eh, s- submit the keys, uh that the score drops for everybody. Um you’ll see, so back when I, a- actively played I had a team myself I would go and test the keys and see if it worked. And however, that’s actually a bad move now. Because if you were to do that, you actually dropped the score for everybody, including your own team. So, if you start colluding there’s a possibility that somebody will actually pull ahead of both of you. So, >>Right >>Yeah because you end up cutting those >>Strategy, right? Because now the people that are, you could delude flags that are more important to other teams. >>Yeah >>Right >>But if, but if you >>HJ is that specific to quals? >>This is specific to quals >>Yeah >>But if you’re on top then you’re on top, right? >>Right >>So that’s not gonna affect you much. >>I mean we, we built collusion detection in the quals because we had unique cha- challenges were tweaked in a, in a small way or everybody had their ya know, their own service, their own accounts. >>Monitoring source IP addresses of submissions and >>Yeah >>People would say I can’t log into my account, that’s because you’re using someone else’s creds and you’re not supposed to see those creds unless you’ve been sharing or the keys were different per team. So, we, we did set up some services wh- that were specifically to detect teams that were sharing answers. >>It takes a massive amount of work >>I th- I think the, the conclusion is, that uh as an organizer you’re increasingly playing game theory. We’re playing a game. Everybody else is doing an activity. [laughter] We make an activity and for us it’s a game and maybe that’s why we like to >>So that’s why you organize because you want to play the game. >>Kina yeah >>Well we know the tricks for the game, so we try to beat them. >>Yeah, for sure >>Uh so here’s a perhaps interesting question so why is, uh the finals uh attack defend style, why don’t you just continue Jeopardy style? >>Oh, because that’s not hacking. >>Yeah so >>Because, because reality. >>Yep >>Cause that’s bulls**t. >>If, If, if we wanted it to be a puzzle game it would be a puzzle game. We want it to be a hacking game. >>It, there’s almost zero pressure in Jeopardy, right? >>Exactly >>You sit back >>Exactly >>And you answer questions and you’re in your living room. >>Yeah >>Attack defense, you, you there’s a reactive nature to that and that, that can cause people to >>And, and, and we’ve always wanted to make a contest that everyone could start. Everyone could start down the road to winning Def Con CTF and we needed something that’s hard enough that it actually slows people down who aren’t ready. >>So, it adds another game mechanic in terms of, nice, for example defense-like patching. I can patch my binary but there’s a chance that it will now fail poles. So. >>And the whole SOA discussion from earlier. >>We’ve also added uh a game mechanic in our game where we have a concurrency for patches, so if you patch something and all you do is add ya know 40 hex, 40 to the st- side of the stack well that’s published now so everyone can see what you just patched. >>If you take nothing else away from this conversation that is the piece of sheer genius that we’ve been missing this whole time since we started playing. >>Agreed >>Is when you add defense everyone else has an opportunity to do the same. That turns it >>Yep, I think really into a game finally, for the players because now do I accept your patches blind? >>Oh PPP, like they definitely back doored patches, like they’ll, they’ll patch something and then like, yeah go ahead use our patches. See what happens. >>Yeah and like purposely deploying patches that then cause other people to try and analyze those to figure out what was going on. Purposely deploying a harmless patch that doesn’t actually necessarily fix anything but sufficiently complicates something >>Or >>That you use, that you use resources of the other team having to try and reverse engineer, like what the hell does that patch mean? >>I’ve definitely heard of, uh patches with Q&U bugs in them. >>Yep >>Yeah y- y- you can create an obfuscated code contest sub contest as a player now while you use the system. >>Which, which really ultimately becomes like team captain leadership triage contest really because. >>Who came up with this idea? >>So it was actually uh the guys who did CGC. >>Yeah >>It was there idea. >>It came from CGC >>Thought >>Just amazing, just the most important thing that happened. >>Um, how bout, what’s the uh, uh I can’t find the exact wording, what’s the hardest question or service that each of you have ever fielded in your tenure? >>Man >>That’s hard >>Super tough, yeah >>Yeah and, that’s eh no preparation you just have to come up with it but like which one of your >>I would say every new CTF is so much harder than the old ones, that listening to anybody but Hawaii John is probably a waste of time cause the new contests are just so much harder. >>Well so I’ll give you >>Uh, I, I invented a CPU in order to write bugs on, um uh, is such a high bar. >>One of our guys uh, Solier it was- was uh, J Dub, um he wrote a MUD. H- a full featured MUD, where you had to go in and collect certain uh numbers of items from each, like from b- uh NPC’s and then that’s your shell code. So th-, in the place, then, then you actually cause, you trigger the overflow and the number of items you have of different types becomes your um, your code. [laughter] Yeah, it’d be your shellcode direct. >>So we had a MUD, we had a MUD but you could hit A L a lot of times and then hit enter and it would crash. [laughter] >>Yeah no >>See the game is not the same thing >>I- well what was really fun, he put, uh he made us all God mode characters, um so we get to walk around in it and, it was really funny because if somebody attacks you, you can’t stop attaching back it just happens automatically so eventually you realize, alright we don’t hit these guys. Because if you hit them I just die >>We, we had a, we had a MUD a couple years but it was all just about like uh the social aspect of quals it was where everybody would talk and, and hang out and stuff during quals, um but effectively we ended up like killing all the players a couple of times by accident, like some monster that one of the other Kenshoto people was creating like got loose and we didn’t, I don’t know. >>Oh it S-, it was S-, so Seer Goon the guy that wrote the MUD is also the guy who uh developed the badge that you saw up there. So that badge he designed and we floated ourselves on a hot plate. Like not even joking, put it on there, did all the solder and LE, the components and floated on a hot plate and, that was, you had to exploit it over rf, you can send them all text messages back and forth. [laughter] >>Yeah >>So it’s like, it’s like a twist question that came in from the audience. Uh what about the most elaborate challenge that you were able to dream up but that couldn’t quite pull off. >>Man >>It’s like still on the to-do list >>Uh let me make sure I can act- >>A really good scoreboard >>Actually >>Uh the >>Ours was >>Scoreboard >>The uh, ou- ours was the uh World Series of Poker printer challenge. >>Oh yeah I remember that one >>Oh yeah that’s right >>That that’s when we started to destroy that thing. You’re like, no, no, no it’s not our printer >>W- we >>Yeah, I remember that one >>I- one year at the Rio, we, we, rolled in right after the world series of poker had ended and in the back rooms were all the printers that they had used for the World Series of Poker's and we just wheeled one out and put one on every table. Everyone had an Ethernet [laughter] we ran cable to every table. And uh, yeah we wanted to work that into the game but didn’t quite get there. >>So [off mic comment] >>Y- yeah some of them got fully disassembled and never reassembled and >>Yeah >>I think >>There’s, there’s actually some significant analysis on the internet of some of those printers >>Pay for a few printers I think >>Uh Guynar you here? No. I was trying to ask if I could mention his challenge but I guess I’ll go ahead it’s in a text message. So, he came up with an awesome idea, uh GDB over phone. So, you dial in and says, this is your prompts and he actually got a decent way into the challenge, like it, I ya know press 1 for R aright. These are you registers, alright. Ya know, step. There was a [laughter} But we couldn’t, like the, the band- bandwidth and throughput just wasn’t enough for that one. >>Operator, operator >>My God that’s great, that’s great >>Real person [off-mic comment] >>Well >>And then, and then do the teams have to implement like, like audio lang- language processing to sort of automate the debugger? >>Hang, hang on, hang on yeah at Goa at NolCon uh I was a, I was really proud of one of the kids there he made a DTMF only attack for IVRs. He found a sql injection attack in the default template for IVRs running asterisk asterisk. So he had a compressed DTMF string that he could play into an IVR and get it to read out the username and password of all of the accounts allowed to edit the system. >>Awesome >>And the phone came out, it, h- h- he used his own voice so it was hard for me to understand, but uh nothing to do with our game but uh but you can make DTMF attacks. >>Yeah >>Turns out it takes only about I don’t know 50 milliseconds or something to get a digit through. >>Um, when did black badges start being awarded, was it always 8? >>I don’t know >>Uh >>Caezar >>We got the first one, um I think they were I don’t know actually when they, black badges came out [off-mic comment] >>I’m saying they were, when the team was I think the largest ever team was like th- tw >>20 people >>20 odd people and DT’s standing there with like 1 black badge, like [laughter] uh, yeah. >>Yeah so he went and grabbed and he could like spare 8 total so >>That’s hilarious >>That became 8 of us got black badges the rest of them did not >>Cause yeah, we pick 8 people for our team, we picked 8 people for our team for the first year because you guys did and you probably picked it because at some point he just had 8 badges. That’s kind of funny >>8 teams, 8 badges >>Well alright >>8 players >>That was, that was based on, that was also based on the like table layout at the time, like there were like 8 approximate seats without whatever and so we basically decided that you couldn’t have more than that at the table >>I think that’s still the same we, we, we limit the size of the table just because uh, just managing it all’s a pain. >>Which I mean doesn’t really prevent large teams but what it does do is gives large teams the same problem that large organizations in real life have. Which is left hand right hand communication problems and organizational problems because like 20 people sitting at a table can still kind of coalesce and self-organize but 20 people divided into 2 groups sitting in 2 different rooms can’t >>In the real world you can buy a fractional T3 or, or a you know a private line and have it run and have somebody put up some VPNs and buy some keys and walk them over there, all these things, here you’ve got 48 hours to end the contest from the time you know where you’re gonna be and getting a remote team connected is >>Yeah limited the effectiveness of a large team was one of our driving ideas >>Yep >>And the way we’ve done that is that we actually don’t, wh- when the game first started like, during DDTEK they would just give you all services immediately except for maybe a few handful excep- exceptions, now we’ll drop one service so sure you have 50 people, let all 50 people look at 1 service, good luck, alright have fun. >>Limiting parallel >>Limi- yeah li- limiting parallelism is what we do and then ya know maybe la- maybe later we’ll do ya know 2, 2 more services or something >>Yeah we never thought about anything like that I wish, honestly I wish we had because limiting, limiting team size was a huge motivator for us too. >>For a certain team >>2 actually >>Uh it’s >>You, you and Shellfish >>It’s sort of interesting because it also plays into the, sort of distribution int- inside of a team, right? How many rockstars are there and how, how fast does it drop off and like what is the curve >>We also started implementing what we do cal-, uh we patch a service so like real world we’ll implement the basic functionality of a service and then y- maybe mid-day on Saturday, >>Change it >>A completely new one and the original bug is patched, so now you have to pull the person who was originally the, the facto expert on that, that service and they now have to do this new thing because do you really wanna ramp up somebody else on, on the same service or, hold them? So yeah we’ve, we’ve really spent a lot of time thinking about these game mechanics and, and how to make it um, more playable, useful for the real world. >>Yep >>Uh so how many women have participated? >>I know that um, um More Smoke Leak Chicken has at least had female participation. Um unfortunately not enough. >>Yeah >>We had more than 25 percent every year that we’ve played. Uh we, we uh did not try to make a conscious inclusion of people, we picked the best people we knew and in Seattle that’s a mixed group. >>We had women with us, uh not a huge percentage but we had more and more with us every year with School of Root. >>Imagine they just didn’t want to smell us. We start stinking after a day. >>Par- participation participation-wise there’s been a lot of mixed team, um but by far it’s obviously uh been lopsided. >>What about it as organizers. >>Uh so, uh Lightening, uh eh- her name is Jewel, the person that wrote this. >>Wow, again it’s just a full stop. >>Phenomenal, yeah like custom architecture implemented by a woman for absolutely, yeah. >>We’re not looking for a winner, I’m just c- [laughter] >>No, n- >>No, no, no >>No, that’s >>We found one >>That’s >>You weren’t looking for it but we found it, yeah we found the winner of CTF >>Exactly >>And it’s up here with Lightening >>Um, I feel like just putting this up here even though it’s clearly a troll but Frank wants to know. He’s hacked the Gibson and only got a partial download of the garbage file so can you help him look through it? >>I think >>There’s a person in the audience who does not know what this means and we just found this out right before we walked to this room [inaudible] >>Hack the Planet >>Or something >>Blank, blank, blank right? >>I kind of have to do it >>Yeah >>It’s like >>Blank, blank, blank >>It’s like the only question that’s been posted that wasn’t anonymous so I felt like I had to actually put it up there [laughter] Um so here’s a curious question at least to me, uh did you do dry-runs and like how many people would participate in the dry-runs if you did do >>We never had time >>We can’t, w- time plus we can’t do dry-one, runs. >>We did cause we really only trust ourselves. >>We did dry-runs of individual services >>Okay yeah >>Right. So, the, the the idea of the dry-run thing of like here other member of Kenshoto or whatever like try to solve this challenge but other than that >>Internal >>No way >>Data testing I guess you’d call it >>Yeah, yeah unit, unit testing and not integration or systems testing. >>No >>Like systems go online at the real game >>Test them >>Yeah, exactly and we had to make changes like, like last second all the time, like every year. >>M- yeah, we did it as well, we did it for our, all of our automated polling scoring all of that stuff, I mean th- the second part of the question doesn’t make sense to me, right. How did you stop any info leaki-that’s an outside thing. [inaudible] >>Some sort of beta run or something, more people have, it sounds like >>We didn’t have >>No way >>It was only team, right >>No way >>It wasn’t like we opened it to public beta or something like that >>No way >>It, it was only our internal dev team. >>Yeah >>It’s, it’s probably one of the reasons his code hasn’t even transacted from hand to hand really. Is, is just because we all feel like, at least I think we felt like, we wanted to play Myles’ game and then once we kind of got on the inside we didn’t want to be tainted, we didn’t want to taint anybody else we wanted all of our friends to be able to enjoy the game that they loved. >>Yep, totally. >>Got 5. Oh, we’re done? Okay. Program says another 9 minutes. But we can be done if you want us to. [laughter] >>My beers empty >>Mine too >>Alright last, last >>No more beer >>Last question then >>Let’s go >>Uh, what qualities do you admire most in the teams that compete? >>Fury, like there are teams that just like throw down and they don’t do anything else. They don’t sleep. They don’t eat. >>Do you remember the mad >>Passion >>Hackers >>Yeah >>I was going to say like I admire no whining [laughter] >>Yeah, like as, as an organizer that, that becomes a big problem. I agree like that’s yeah, that’s a big issue but just the fury the passion of it. >>Like, the fact that you can go and you spend ya know 12 hour, 10 hours in front of a PC and then you go and spend all night in front of, doing the same thing. It’s pretty hard core like, like I kind of admire all of it. Yeah >>Yeah the endurance, the i- it really is at some point an endurance feat. >>You know what’s funny is thinking about that is you’re right now th- the cost entry is, well they get free entry now I don- it wasn’t when you guys, was the case but they get 8 badges per team but before when you’d have to pay >>We didn’t even get badges >>Oh yeah. When you pay, you go and you come here like especially people coming here from far away. Russia and Japan, and they spend their entire weekend in front of a computer, the exact same thing they do at home and they come in here and do it so uh >>Th, the yeah. Aside from that the thing that always shocked me as an organize-, time to solve, is the thing that I always looked at right? Some, something drops like radically faster than you expected it, >>Oh yeah >>That that’s like an instant >>That’s frightening >>Wow >>Sometimes >>Yeah, agreed, agreed. >>Yeah th- and the level it’s not just the stamina, like yeah I can drink a bunch of Mt Dew and see, I put in the 10 months getting to this >>That’s true a lot of people bring, now you have to bring custom stuff, for >>Right >>Example when h- he was mentioning that if you pull a flag and you don’t encrypt it, it’s gonna get caught, cause oh, that’s one thing we mention too is that pe- all the teams have access to the packet captures, so they see all the data coming across the network and they control what’s uh, coming in and out of the box, so, um if you don’t encrypt your, your data back and forth in your shell code, your shellcode has to implement the encryption, then you’re gonna fail. >>Alright, do you have something? >>Uh, uh just, just one of the things in time, time to crack uh, I don’t know if it was you guys uh, 1 of our routers leaked 1 bit of information and the scoreboard had a different TTL, off by 1 from th- >>I remember, no that was, that was, I remember >>Was that you? >>That was us, yeah >>Was that you guys? Aight >>Yeah >>The TTL on the >>Basically we could tell, we could tell apart the packets that were your engine testing that the service was online from everyone else. >>So they firewalled only the players >>Nobody else can get >>Percent SLA, zero percent defense >>Yeah >>An- and they did this in maybe I don’t know, maybe 7 minutes. From the time, we handed them their discs, they were. It was amazing >>We profiled TTL and your agents on your >>Yeah >>We missed that one we didn’t, yeah >>Alright, we’ve got the, the cutoff so um the, the slides will be posted and uh, th- the video will be posted so it's, uh thank the panelists and uh I suppose we’ll be around for a little bit. >>Thanks for coming >>Yeah thanks for coming [applause]