>>Umm so the purpose of the talk today is to talk about bug bounty programs, what going on.  Give you a little bit of education and then talk about the various types and what's happening.  There's a lot that's going on out there.  There are some people who saw hands that went up that are doing this stuff.  Use them as resources.  There are some folks that run bug bounty programs in this room, so definitely spend time with them as well.  My name is Jake Kouns, I am the CSO for a company called Risk Based Security. 
This Carsten Eiram he is the Chief Research Officer, he's done a bunch of bug bounty stuff so with that we're going get rolling.  The reason we got in to this stuff is that we actually do a lot of vulnerability intelligence and we've been tracking for awhile what is going on with disclosures and what's happening and how is money influencing what's going on in research.  That's what leads us here today. 
All right, so, information security.  Career decisions.  I would imagine most of you are already somehow in the security world doing some kind of work or you're here because you want to learn and work your way into a security role somewhere, right?
And there's lots of choices, right?
I think you've been hearing over and over that it's really hard to be an I.T. generalist these days , even worse an I.T. security generalist, you can do it but everything is getting so specialized that you sorta have to pick a role, right?
And so we go out there and we look you see all these words like analyst and auditor and specialist and all this other stuff doing some searches on some job hunting sites.  It's funny our favorite one was the cybersecurity hunter as a title, we thought that was pretty amusing.  Does anyone have that as a title?
[ laughter ]
No.
You're working on it.  Is that what it was?
What it comes down to typically and I hate to sort of, to generalize it this much sort of you're presented with two pills, right?   It's the blue pill or the red pill.  How many blue pill people are in the room?
Couple.  A lot of people seem to be undecided on what you're going to take, right?
But what it comes down to it most people say they're red pills the way they go, right?
And almost everyone seems to be talking about offense and that is what is sexy, et cetera.  Blue teamers seem to be really uptight these days, right?
It feels like there's too much red team attention going on the blue team guys feel very sort of taken aback , right, you know, like it's not fair to them.  I can say that because I start my career as a fire wall guy so I was a blue team guy for a long time. 
So, why do people like red teams, right?
There's a lot of reasons and typically these are some of the ones that come back, right?
Constant learning opportunities, you get to play and break things, I mean how cool is that, right?
That's what you're paid to do and you're generally well paid to do this.  And even sometimes you get to mess around with social engineering stuff which some people in the room really enjoy.  And it can lead to a lot of other fun things trying out, right?
Then at the end, there's nothing better than popping a box, getting rude, something about winning, just feels great and people love that.  When you say red team it can be sort of broader but people think of that as pentesting, right?
And it seems like everywhere you go and you talk to vendor or security vendors and companies everyone is looking for Pentesters right now.  But actually when you go out on a job search board, like Indeed, you do a search for Pentesting only sort of 17 come up, you know, jobs that are open, if you will.  But it does feel like everyone is looking for some type of Pentester these days.  When you're a Pentester there can be some things that are pretty painful, right?

So, there's a lot of fun but there's also some things that go along with it that maybe aren't so great.  And a few things are that you are usually working at some sort of company and you're dealing with some sort of political politics, those sorts of things which is no fun, right?
You have to test during specific hours, many of those times can be sort of when you do want to be sleeping or drinking or doing something else later at night, weekends, et cetera.  You end up having to write these really, really long reports and you know no one is going to read or even understand but you have to make it really, really long because it has to seem like it's valuable, right.  It's a short report, why would the customer pay all that money for the short report, so get in there and write some stuff, right?
Then you do have to deal with these things called clients, right?  So, statement of works, or present those findings conference calls, et cetera.  Those are things that just aren't as fun, right?
So there's also an option of being an independent Pentester.  That means you don't have to work for the man but there's still a break down, right?
That break down is roughly we'll say a third of it, it's the fun part, breaking and doing stuff.  Then you got another third of it's the administrative task and documentation then that last bit being a sales weasel, right, like you're going to have to find work, it doesn't show up at your door and go I'm so happy can you do this Pentest for us, right?
It's just not going to happen that way.  So, there's got to be sort of a better career choice potentially right, maybe a bit more of a tasty decision and we think that there might be, right?
And you can be this guy, right?
You like this, the quick pepper sprayer guy, right.  You can be dog or what I say is better yet, you can be this guy.  And hopefully in this sort of line up of going on the bug bounty hunter route you're 90% hopefully of your time researching vulnerabilities and what not and then maybe that little bit less time, 10% or so is working on the write-ups and all that other stuff.  All right, so with that I'm going to have  Carsten do a quick overview sort of to set the bounty stage with you guys.   
>> One of the first things we want to cover, research motivation, back in the old days before we had all these bug bounties.  Back then we were reporting vulnerabilities to the vendors, because it looked good on our resumes and it got us creating vendor advisories.  If you didn't have a job or wanted to get in to this sort of business there was a good way to actually get a job and potentially for one of these companies that you found vulnerabilities in.  If you already had a job, it could obviously lead to higher salaries or better jobs.  So why we like to go around saying that oh, back in the old school days we were so altruistic, let's be honest there was nothing altruistic about it, it looked good for us and worked out nicely.  
The problem was that it wasn't hassle free.  Reporting vulnerabilities to vendors was often very painful, often they didn't respond, if you did get a response they were often a legal threat.  So, a lot of researchers just said, you know what, fuck it, lets find some alternatives.  So, what a lot of researchers did they published it just somewhere else, like we just dropped it on a mailing list, we would trade, give it away within groups for respect and goodwill, we would use it offensively for fun and or profit or we would just kind of say, ah, found this, fuck it, store it in a digital box somewhere, move on. 
Or, there was this other option because we did have monetary options back then also, there was the grey markets, three or four letter agencies and we also had the black markets.  So, we had these options if this was something we wanted to do.  So some of the vendors and security companies kind of realized that, you know what, if we start rewarding discoveries we can actually centralize researchers come to us and report vulnerabilities to us.
In 2002 Idefense created their vulnerability coordination program where researchers can come to them, sell the vulnerability findings to them, they would coordinate, they would provide the information to their customers, make a business model out of it that way.  2004 Mozilla created their bug bounty program where they were paying $500 bucks for critical vulnerabilities.  So, we wanted do a quick pop quiz.  In what year was the first bug bounty program started?  Because a lot of people considered this to be something  pretty new, anyone else?
No?
1990?
1995, exactly.  Then do you also know who did it.  Netscape, exactly.  In October 1995, Netscape actually launched the first bug bounty program where they were paying money and what was very interesting about that program was the fact this they said we don't want to pay people to find vulnerabilities in our stable products, we want to incentivize researchers to find vulnerabilities in our beader products so before they go stable...does that remind you of any other companies that just did a bug bounty program recently?
Microsoft actually did this recently and people were talking about that's a new way of doing it but actually someone was doing it 20 years ago. 
Through 2000 to 2008 disclosure of this huge battleground between vendors and researchers, researchers have problems getting vendors to respond and taking it seriously and getting them to understand the issues, the perception was whether it was true or not that the vendors didn't take vulnerabilities and security seriously unless we it dropped on a mailing list.  And we like to say that researchers back then were hard core full disclosure the quote unquote, right way.  The focus was on getting bugs fixed that's what they were focusing a lot on.  Then in 2007 we saw the first kind of bug bounty competition, the PWN2OWN where you could win MacBook pros and ZDI was offering up $10,000 U.S. dollars also in bounties.  That competition has grown to 2010 certainly $100,000 pay on the line and this competition grows every single year.  They set the bar high for being able to win this money  but if you can do this there's a ton of money in it, there's a ton of PR.
In March 2009 we also saw some researchers coming out a bit more saying, you know what, we have this new philosophy, no more free box.  We want to get paid for our work, it's valuable, we want to get paid.  To be honest, we don't really know how much affect it had.  But it definitely sparked a debate and it also made it very, very clear to vendors what these many research expectations were.  We wanted to get compensated for the work that we were doing.  So that leads us to present day where bug bounty now all the rage, everyone is doing it, ones that aren't doing it kinda want to do it because it looks good and it's getting a lot of attention.  Now we're seeing all these different type of bounty programs.  We are seeing what are called company run bug bounties, we see third party bug bounties.  We see these competitions as mentioned.  Krausers spot programs - Which all things we'll be covering next and also see different types of rewards, we see cash, prizes like T-shirts, get a mark, get some conference attendee sentence, fame and glory appreciation, but if we want to do this as a career choice and not just be hobby hunters, cash is king, right?  End of day getting a mark, getting a T-shirt is nice for the hobby hunters but we want money, right?   That's what we want to focus on.  So by that Jake will now discuss company run bug bounties. 

11:27

>> All right.  So our definitions this is what we believe these are.  We're calling it company run bug bounty basically it's the bounties being run by the company that owns the website that you'd be going after or the software, right?
Almost all cases reporting and coordination all that sort of stuff is directly with the company and not through some sort of intermediary.  The process is simple you discover a vulnerability, you send the details to the vendor or some list that they have information on the website of what it is, hopefully the vendor can reproduce it and then they will accept your submission and get paid, right?
The number of bug bounty programs just continues to grow, every time we turn around we're finding more and more that pop up we maintain a list of bug bounty programs as we map the vulnerabilities that we're discovering, just to give you a little bit of sort of understanding of what is going on there is around 300 or so programs.  260 of those have some type of reward, about 165 provide some sort of recognition and about 75 of those have some sort of monetary reward.  Bug crowd has a really nice crowd source list publicly.  You can go to their site and check it out and look for some of the ones if you want some more details. 
I can't really talk about bounties without talking about Google, right?
Google started providing bounties in 2010 they continue to be one of the more serious players when it comes to the bounty space.  We believe they're a big reason that bounties took off with 2.7 million in prizes, August 2013 Google paid out a little bit more than $2 million for over 2,000 valid reports.  Some pretty serious money.  They also took things a bit further as well saying we're also not going to pay for things in our software but we'll pay for vulms in other software as well, right?
They really just continue to push for fixing vulms disclosing them in a timely manner, fixing them in a timely manner these sorts of things.  They have been a serious player.  Talk a little bit about Facebook.  So Facebook was -- their bounty program was founded in 2011.  They have over 1500 bounties that have been paid to, date about 600 unique researchers were paid and they paid researchers in 79 countries.  With the top countries being India, USA and U.K.  And their whole premise again is to find issues that can help make Facebook a more secure platform, right? 
Couple of interesting bits, their average bounty is in the low thousands.  They have a minimum of $500 that they pay for a bounty they don't have a maximum set.  So it all just depends, and the largest bounty that they have paid out was a little more than $33,000.  So you can get a pretty substantial payment from those guys, they have more details on that vulnerability on their website you can find out more details about the program. 
Pretty interesting to see there.  What's also interesting is they provided us some information, we actually sent out questionnaires to a bunch of vendors asking for feedback as well they provided us with some information also they blogged about it, they paid out more than $1.5 million in bounty rewards in 2013 you're talking about a fair amount of money.  So that was really interesting but couple other things that popped up that were really eye opening.  They received close to 15,000 bug submissions in 2013 that is a huge number, right?
And they said it was 246% increase in one year, the number of vulms that were submitted to them.  What was eye opening that only 687 were actually deemed valid and received some sort of financial compensation, right?  For you that is really early if you're not great with math that's not a high number, right?
It's not great.  So you want to make sure that if you're trying to do this for money that you're one of the ones in that range so whether it's following their rules which we seen problems with, they have some things where it says don't attack live profiles and people do and they still expect to get paid you're not going to.  Follow the rules, get some stuff clear to them you want to get in there so you're one of the ones getting paid not one of the ones that is getting nothing or a T-shirt. 
We did mention that a list continues to grow that is very true but there are some -- lots of vendors out there that you would expect that would have bounty programs and do not.  There's a lot of hold out.  So still room for a lot of growth there.  So again if you're trying to make this a career, it seems like more and more companies are going that path and we expect to see more coming. 
All right.  Now we're going to talk about what we call third party bounties.  This is sort of set the stage for this, this is bounties that are run by other companies that do not own the software and typically they are just looking for software things, not talking about website or site specific vulnerabilities.  The reason they do this is they use this information they buy up information to either provide to their customers as an alerting service or bill it in to their products so that their products are finding vulnerabilities that other products may not know about that is sort of reasoning incentive in most cases for doing it.  Then almost all cases reporting and coordination is directly handled with that company running the bounty and you're not talking to the direct software manufacturer company that you found the issue in. 
So, ZDI it's probably one that most people are aware of or if you're getting in this space, definitely look in to, they were founded in 2005 this is their 10th year of doing that, so kudos to ZDI for really being strong in this area.  And they focus again on software used by global enterprises, right?
The list of what they will pay for is not published anywhere so it's not super clear what you should research in many cases but you can look at the upcoming advisories and get a feel for what they have done in the past and what is coming up, to sort of understand what platforms and types of bugs that they would pay for.  Couple quick points for them, there's over 3,000 independent researchers that are registered with them, nearly 100 companies represented, U.S., U.K., India, Germany and France are the top five countries.  They didn't share with us the amount of unique researchers that have been paid.  But we -- with working with them and seeing some other things they have paid out 1,715 bounties, average bounty again they weren't willing to share.  But ZDI has paid bounty ranges from three figures to six figures they have paid some serious money for vulnerabilities as well.  And one of the things that they also do they have sort of extra rewards and extra money if you're a repeat customer.  If you are loyal to them and repeating things over and over you can get cash bonuses, you can get Balta conferences, all those sorts of things, right?
So there are extra rewards if you send a lot of things to them.  All right.  I-defense the VCP program, Carsten mentioned that as one of the first ones out there, 2002 we're not really sure if they exist anymore to be quite frank.  There has been nothing published since October 2013 timeframe and most people that we sort of talked to about if they're using them or not are starting to say, don't bother, don't waste your time.  Information about them is on their site if you want to look in to it feel free.  Share with us if you get anywhere with them.  Another one that we want to mention, Exodus intelligence they were born out of the ZDI program and to be quite frank not too much is known about this program yet as well.  They came out and announced some things about the program but we haven't seen a whole lot with it.  They talk about targeting again sort of critical vulnerabilities and issues and widespread software.  It's unknown the number of bounties they've done they weren't willing to disclose that.  It's unknown, they weren't willing to disclose the number of researchers that are participating but if you look on their site about what they're up to they say their going to intend to be competitive and they also say there is yearly bonuses for the top four researchers of 20 grand each, again, it's unclear we don't have any evidence of what's been going -- or what they have been doing but something to look in to if you're in this space as well.  Contact them for details.
>> Just some quick pointers if you want to try to call some of these third party companies a lot of these pointers actually go for the company run bounties as well.  First of all make sure you are clear on what software they are likely to accept as Jake said look at -- for instance with CDI -- look at what they publish, look at what they have upcoming to figure out.  When you report vulnerabilities to them also make sure that when you report them as cases, create a case for each separate distinct vulnerability.  Not attack basis but for root causes.  One of the reasons you want to do this is because if you sent a case where you just kind of bundle everything in that's a risk that they might not completely understand that you're reporting X amount of separate vulnerabilities so the offers that they will give to you would be for one vulnerability and then you'll have to start  discussing with them that hey, there are this many vulnerabilities shouldn't it be a higher amount?  So, if you create one per case it will make it easier for them and it will make it easier for you and it will increase your chance of actually getting paid probably.  When it comes to providing up and providing information to them make sure you include as many confirmed details as possible.  Don't do any guesswork cause that will just confuse them especially if you are very long.  Trim down to make it easy.  The reason you want to provide all this information to them is again, to help yourself.  Because the more detailed information you provide to them the clear, cleaner your PUCs and exploits are the easier it is for them once they start validating your findings to quickly do it then you get paid quicker and also less confusion about it.  Make sure to clearly list the software that you tested.  And since they are confirming your findings make sure you actually provide references -- links to where you obtained the trial software if you did, or where they can at least find it.  That will also save you time and make sure the whole process goes much, much smoother.
Next thing we want to talk about are the crowd sourced bounties.  And when we talk about crowd source bounties we're talking about all these portals we seen jump up, all these platforms that have all these researchers tied to them then companies instead of creating their own bounty program they can come and use this portal to quickly set up to their liking, to get things done.  What happens is all these bounties will be open to researchers, some of them will be open, some of them will be private, but a lot of them will be open to you.  And then the way it works is that you send your submission through this portal then it's being passed on to the companies to make sure to get stuff fixed, all the payment all that stuff will happen through the portal also.  What we are seeing is this blur between the traditional bug bounties and Pen testing, so Jake was talking earlier about being an independent Pen tester and how you have to do all this work, well some of these new platforms actually make it a bit easier, at least cutting out the sales process because you can sign up for these programs or get invited to participate in them.  Bug Crowd is one of the first ones we want to discuss it was founded in 2012.  The targets are web, mobile, client side embedded applications, the reason you also introduced Flex which is a crowd source penetration test.  Currently we wrote this they have 23 public, currently active programs.  They then had a number of private programs also that you have to be invited for.  They have had 170 programs that they completed so far and since October 2013 they have had around 57 companies participating.  From research perspective they have over 10,000 researchers that have signed up for this service, so they definitely have the largest researcher base with varying quality from different countries.  But if you look at the number of unique researchers that have been paid money it's only 231 researchers.  There might be a lot of researchers we don't know how many necessarily are very active.  The sign-up process is very easy, basically just go to the sign-up page, find user name, e-mail address, password, boom you're in you can start signing up to the different programs and start finding  vulnerabilities and then report them.  It's an easy process. 
If we looked at the number of bugs they have been handling a bit more than a thousand bugs since November 2012.  The average bounty amount is around 250 U.S. dollars
Pay out is primarily through PayPal but have been made exceptions with Western Union and some other options.  The average time to process of submission is between two to six weeks but generally seems to go pretty fast.  The largest single payout has been $13,500 so there's still some significant amounts also.  And locations with interest -- we talk again about the locations and again most of the researchers here seem to be from India, Europe and the eastern part of U.S.
They also have this leader board where you can actually see who is doing the best at the moment.  Well you can kind of see -- they provide -- there's at least different options you can get kudos points, you can get rewards, it's up to the different companies that participate in this program what they want to provide.  You can see -- some people like all these kudos points but again that's also a lot of people that aren't happy about it.  Again as we've said if you are going to do this professionally, we don't really care about the kudos we care about the cash.  There are some people that are complaining that too many of the programs that are currently active do not have actually monetary rewards.  People are going after Bug Crowd for it but really end of day they shouldn't be faulted for it because they just provide the options and it's up to the companies that use them to what they want to choose.  So it's really the companies.  And again, if you are a charity then as a researcher, yeah, I might invest some of my time to find some vulnerabilities and do it for kudos.  But if you are a security company that has just received a ton of VC money, for me as a researcher it's fucking offensive when all you want to give me is a thank you for finding vulnerabilities on your site.
[Applause]
The next one we want to talk about is HackerOne.  It was founded 2013.  Their targets are a bit different because they are different response teams handling things.  It can really be focused on whatever these response teams want to get handled, currently 63 different teams that run public programs on HackerOne then there's a lot of private programs.  That you have to be invited for.  They say they have thousands of researchers that are registered and around 800 have submitted valid findings that either lead to a bounty or some sort of recognition we don't have the exact number of people that have actually been paid.  800 have either received recognition or money.  Sign up here is also very easy, name, username,  password, boom you're in, and you can actually start working immediately in finding stuff. 
They have paid more than 1300 bug bounties.  The average bounty amount is almost $700.  Largest pay off has been $15,000 they have had multiple of those.  One of them was the incident bug bounty that they're running was paid for Heart Bleed.  We all know and heard about.  Other ones were Yahoo! have been paying out 15,000 bug bounties also, so that's also some serious cash there.  Usually bug bounties one that is pretty interesting because they cover different technologies, if you find vulnerabilities in products like PhP, pearl, Jango, you can actually report vulnerabilities.  They also have an Internet bounty where if you find vulnerabilities in other components that are considered critical to the internet like for instance Heart Bleed you can actually get paid as well.  The minimum bounty here in this case is $5,000 U.S.
Also have a different version called the sandbox escape where if you find a vulnerability in chrome, other browsers like Internet Explorer, or Adobe reader find sandbox escape you can get paid through this program.  So you can report the vulnerabilities to Adobe and pass it on to these guys also and again minimum payout is $5,000. 
Another one we want to discuss is CrowdCurity also found in 2013.  Their focus is on web applications and usually things that are Bitcoin related.  Currently they have 45 active bounties they have run around 90 programs all time and companies in the 50s to 100 range have used the platform they have 1300 researchers signed up and around 3-400 of them are actually considered active.  The most predominant research again is from India, Europe and U.S. around 100 unique researchers have been paid so far.  Signing up here is again also very easy like the other programs, quick details and you're in and you can start immediately.  They pay around 800 bucks.  Average bug bounty is $150 they do bit differently they provide package options to the customers and then they can decide what they want so we are not seeing the same big ranges that we do for some of the other programs.  The largest single payment has been $1500 here. 
CrowdCurity is also doing the hall of fame stuff but what we can talk about there is their approach is that they also show report qualities actually going for researchers and you can see the report qualities how good it is not just the kudos points.
Last one we want to discuss is Sinagh founded in 2013.  It's actually not a managed bug bounty provider as such more focusing on application vulnerabilities across web, mobile, infrastructure, really this sort of penetration test approach where as a researcher you sign up, it's not as easy we'll cove that.  And then you get paid per vulnerability you find, so instead of being paid for just doing an engagement you get paid for showing results.  They didn't share a lot of details with us so we don't know how many clients they have we don't know how many researchers they have, we know that around 40% of them are U.S. based though the rest are international. 
That is it.  The application process is challenging, it's invite only.  So when you sign up you have to provide them with a lot of information, you have to kind of make them understand why you deserve to be in this program, how many years of experience have you had, et cetera, once you fill out the process form they will actually schedule a call with you.  Like a 30-minute call where you can talk to them, explain your background so they can figure out whether you are fit for this platform.  They do a lot of research compared to the other programs.  Again number of pay outs, unknown bounty amount, average is unknown.  They have shared with us the payout range is between $100 U.S. dollars to $5,000.  There's no upper limit but that's the range of what you should expect to get paid per issue you find. 
So some pointers when you report to these crowd source programs it's actually a bit different.  Because the risk of duplicates for these programs is much higher than for some of the other programs.  So speed is a much, much bigger factor.  If you sit down and take the third party program approach for instance, where you sit down, you write up a very nice report, you want it all to be clear, you send it in to them, like I wrote this amazing report then you know what, someone just beat you to it because there are so many other people looking at it and they only pay the first person that find it so all you get out of it is kudos for your hard work.  So instead, what you should do in this case is when you find a vulnerability, don't just like collect them and then report eventually.  Get them out immediately, quick sheet quick description just send it out obviously you want some level of detail but just something quick so you can make sure that you increase your chances of being the first one that actually gets the money. 
What we really do like about many of these programs though and encourage you do, a lot of them provide a heads up on upcoming bounties so they will say we have this new one that will launch.  So you can actually sit and be ready for it.  As soon as that program goes live you can be away hacking immediately that allows you to quicker find some of the low hanging fruit also, get it reported quickly and increase your chance of getting paid.  Because if you get in late on those as we have been talking about, hundreds of researchers on most of them, then a lot of people have already been looking at it and there's a good chance they find it.  When I was using some of these programs also if you get in late just to find it you might find a vulnerability, you go oh yeah that's good.  You write it up, report, you send it off quick, the thing is someone already found it just hasn't been fixed yet.  Waste of time, 2 kudos points, yey. 
So, Jake will now cover progress, some better approaches, a different category something you might want to do and look at instead. 
>> All right.  So, bug brokers, what are we talking about.  Basically a researcher who finds a vulnerability, works with a broker and that broker will then find the best market and the best price for that information.  This could be a number of avenues including grey and black markets.  Its generally still thought of as the way to get the most amount of money possible for your research.  And almost all cases, the reporting and the coordination is directly handled with the broker, who handles everything about that transaction and details of the vulnerability are typically never published.  A lot of the other areas, the terms and agreement may vary by program but at some point you make get a little credit and go through the disclosure process, in this case usually it's not discussed at all as you can imagine. 
So one of the places we looked at that does brokering is a company called Beyond Security, has a team they call secure team, they have the secure team secured disclosure it's not a purchasing program but if you have a vulnerability of interest that is critical or something that they care about they will look at it.  They weren't willing to share a lot of information with us as you can imagine so we don't know the number of researchers, we don't know how many unique ones have been paid but we were able to find from them that they had researchers in all continents, except for Africa and most are U.S. and Europe.  Here is where it gets interesting.  They shared with us that over 100 bounties were paid last year, if you look at the average amount they're saying that the bounties were between five and 100,000.  And they shared with us and claimed that the largest single pay out that they had was over $1 million.  You're talking about some serious money.  When we shared this with several people we got a lot of people say bull shit we don't believe it, so there is that, but from working with them, they're claiming that this is legit that they have had over a million dollar pay out. 
A little bit of a different approach because again, it still does take time to report issues and disclosure process, Secunia had created a program that coordinate quite some time ago was in the end of 2011, they weren't actually willing to really pay you anything, but they would get you to a conference and that sort of thing.  But it was really just an offer to help get things fixed and be that broker for you.  But what's interesting is this service is closed for business.  Rest in peace August 2013.  And really what came out of it was just again shows that there's a lot of work when it goes in to disclosure process that they ended it basically saying that the amount of time and effort just wasn't worth it to the organization this is now no longer an option.  You just can't talk about brokering without this guy, everyone know who this guy is?  The Grook?
Right?
He's got a lot to say on this topic if you're interested, if you want to hear his thoughts he's talked a lot about it, contact him I'm sure he'd be willing to share more and talk to you about information. 


>> We could go on to discuss bug bounties, is it worth your time and some of the considerations you should make.  You wanted to do reality check before starting out.  When I eventually left Secuina and actually a couple years before I left I was toying with the idea of going full time as bug bounty hunter actually wanted to do it both myself and my family to Thailand and do it down there.  I'll cover a bit later why that was part of the consideration.  But before doing all this you want to do as I said reality check, ask yourself a couple of questions.  The first one, how much money would I need per month to stay afloat.  If you have a house a mortgage, a car, wife and kid to take care of in Denmark your expenses could be a lot higher than if you're single in Thailand.  Depending on how you spend your free time down there.

[Laughter]

The reason I mentioned Denmark versus Thailand is because location actually matters.  These are some quick stats from -- most of them are from sourced from a website called pay scale that shows average salaries a year for Pen Testers and it covers everything from like the new junior guy that just started to the very senior guys.  Obviously if you are in former category you'll be earning less than these amounts.  If you're in the latter category you might earn twice as much.  The point is that salary is also followed by expenses.  If I'm sitting in Thailand and the average salary down there is say $15,000, $10-$15,000 U.S. a year.  I don't have to find a lot of vulnerabilities to make that happen quite quick yeah. If I'm sitting in Denmark and the average salary is 100k -- takes a lot more effort.  So, this is part of the thought process also if it doesn't make sense financially for me to start doing this.  If you are in a lower income country they'll have low expenses, stay away from Iran and some of those countries they have embargoed a lot of these programs that actually won't pay you, so you move down there for no reason.  So then we kind of have the idea now we know how much money do I need to stay afloat, have a comfortable life.  Obviously we want to do this to have a comfortable life unless then it's no fun.  The next question you want to ask yourself, ok what is the combination of products, vulnerability type numbers that I look at to make it happen.  This is where we talk skill set also.  If you are a hard core reverser and you can make bad ass exploits, well then you will usually be in category where you can find vulnerabilities that can be sold for more and then you might find it easier to have these numbers that you have for yourself.  If you have a skill set that is restricted to finding process scripting on websites, well then you need to find more vulnerabilities.  What we like about bug bounties is the fact that there are options for everyone.  Just depends then of how many vulnerabilities you have to find to hit these targets.  We see some researchers that find a few vulnerabilities and some high profile people get paid a ton of money, the competition we talked about, other researchers that focus on other programs, guy that is worth mentioning is Argod, he's been using CDI for like four years or so, and in that time he's found 200 vulnerabilities.  Then you do the math there that he's been making a good living off of that and sticking to, I call it lower hanging fruit but he's been keeping clear of all the big -- not finding vulnerabilities in Internet Explorer, he's keeping it to HP products and some other IBM products that not many people are looking at and doing it very successfully. 
Part of this process is also again figuring out the numbers or to figuring out, how much will I get paid, so kind of make a plan there.  If you decide, okay, I'm going to hit these numbers and hit Yahoo! hard find a lot of vulnerabilities, you report a lot of scripting vulnerabilities to them and then they say thank you here is a $12.50 voucher for your trouble.  This happens, I see some of you laughing so you know.  This actually happened for a security company that wanted to test it, so this is also why Yahoo! actually changed the policies.  So now if you report some vulnerabilities to them you can get up to 15K right?  So, now it's interesting.  That's another one of the offensive ones. 
Now we know how much money do I need, and we also know like how do we want to make this happen.  Then your last conclusion, how much time will I need to invest in this. 
One of the great things about being a professional bug bounty hunter is of course the freedom that's involved.  So, again if you are living somewhere where you only have to find very little small number of vulnerabilities to make it happen, great.  Maybe you spent a few hours a day, a day a week you can make these numbers, right?
But if you have to find a ton of vulnerabilities to make this happen you are living in high income country with a lot of expenses if you have to work 60 plus hours a week, all of a sudden it's not very fun, is it.  Then you might want to ask yourself, well is it worth it?
Of course if you have to spend that much time to make it happen you're also increasing the risk of it not working out for you.  Again, those are three good questions.  And all the questions I was asking myself when I was considering this. 
All these questions then as I said led you to conclude bug bounty hunting, is that the career path for me or should I keep it as a hobby on the side?  Again most important part is, if you don't form some sort of plan from the beginning and figure out how you want to get these numbers, what you're going to focus on, there is very small chance it will work out for you in the long run.  Again, if you have to generate a thousand dollars a month and you're living in India, Thailand, great.  That's pretty easy.  If you have to generate 10,000 a month it gets harder. 
So, Jake will then finish off by discussing bug bounties and what we think is to come. 
>> All right.  A little pressed on time because a lot of content we want to cover so we're going to go over a few of these things.  The good news for anyone that is a bug bounty hunter or wants to be, software is still complete shit right now and so there's lots of things to find, right?
If you look at these numbers from the OSPD project where we track vulnerabilities we're seeing 10,000 vulms a year it doesn't seem to be slowing down, this also does not include any of the site specific vulnerabilities and websites, right.  You're talking about a lot of options out there.  We think that the rules and requirements will start being enhanced a bit more specifically when people are going after cloud services, right?
You start trying to do allegedly good bounty hunting on a software as a service and you wipe it out, the company is not going to be real happy about that, right.  Be real careful of that we think there will be more and more clarity that comes in that space.  Legal threats, right.  They still happen for those in the room that have been around for awhile know this you think back to 2005 the Cisco versus Mike Lynn most people say, we remember that, but things have gotten so much better.  In many ways they have, right.  But legal threats are still happening and they are still successful so you still want to keep in mind with what you're up to and be smart.  Specifically there's a big difference between a bug bounty and this thing called extortion, you want to be very careful.  For example, this guy, probable onion on Twitter he had a great thing he contacts these guys saying you have 12 hours to fix the vulnerability in your system otherwise I will take control.  You've been warned, right?  And then all of a sudden the company comes back and says we'd be happy to do a bug bounty on this.  What the heck are you talking about there?
Not really what the bug bounty program is set up for, right?
Of course, you would say this guy would probably come back, he's interested in the money, okay, let's talk.  Be careful about that because if you think about it we find out that a team was arrested, right, this guy was a team from Canada was arrested for doing things like this and swatting people like Crebs and all kinds of other things so -- the bounty stuff and vendors are now willing to work with you and pay money but let's not get in to -- be very careful about this whole extortion area, right.  From a researcher's standpoint, it's great now that companies are willing to pay money, right, so we want to let researchers know this should be something that we should be appreciating the fact that they're willing to pay and not sort of this sense of entitlement.  So many people will just go out there and pick a random website and start messing around with it and then demand payment.  When that company never asked you to do that.  It's uncommissioned work, right?
So definitely there's a lot of companies out there, there's a lot of ways we describe that are willing that have said they will pay, stick with those.  You don't need to go out there. 
We do have Google again, now they launched this new thing called project zero if you've heard about it basically creating a dream team of researchers to weed out bugs in popular software.  We believe at this point it might not have issues on websites, but on other software bits we think it's going to make it a lot harder and raise the bar for it.  And we're already seeing people trying to calculate the ROI and what is going on in that space of what's worth it. 
Wrapping it up here, more and more companies are going to jump on board with bug bounties programs, we ultimately think that kudos and karma stuff is nice but it's going to lose out to where the biggest amount of money is.  We believe that while companies are going to continue to have bug bounty programs as part of their security programs that they're going to realize that eradicating vulms early on during the SDL process still is a cheaper, better way so we're going to see some shifting. 
With that, this presentation would not have been possible with a lot of people we want to thank Brian Martin, Katie from HackerOne, Nate from Facebook, HP, ZDI, CrowdCury, Secure Team, Kasey from Bug Crowd, all the bug bounty hunters out there and future bug bounty hunters, thank you. 

[Applause]