>> so hello everyone, my name is Ryan Calabis and my presentation in titled the bieber project,so this is a 101 on digital advertising, AD technology and AD project, so I work for -- we are web cap option company we work for digital advertising I did work on executer analytics this will fold together after I go through this.  In any case let's talk about or main topics we'll talk about bus and currency of digital advertising why am I even talking about this why is this even relevant.  Then we are going to go through the [indiscernible] system I can't go through that many systems -- how to start off with foundation and stuff this is -- then after that we'll go through -- different types of fraud such as like publish, malicious objectionable content non human traffic and my pet project the BE -- BIEBER project.  Why are we even talk and about this.  Basically we are talking about this because digital advertising is big money so the total digital add spend is estimated about 60 bill dollars 2015.  Estimated also that five years from now digital -- would be one hundred million dollars.  It is a big pie and a lot of different people with like a -- would like a big chunk of the buy that's why this is very very important topic right now like in add particular and even enforced security.  So, let's talk about the currency of digital advertising.  So digital advertising has several met tricks but one of them most important and primary is lt gnomist delivered or served impression that's views so but primary problem here is not all on line address or actually -- are actually seen, probably under stand like advertisers are not interested they are not interested in paying for address that are never seen obviously, right.  So, here's the thing.  Let's go through like the different METRICSS.  When you start using an add there's an add on the left-hand side, you see the ad.  And that's the subtle impression.  Next, you have the CLICKS which is the other one then you have conversion conversion could be a purchase assign up form and stuff like that but what we are going to really talk about and focus in this talk is really a view bell impressions that is primary METRIC that we are talking about.  Primary METRIC if -- whole train would be fraudulent that is basically the currency of digital advertising.  What I'm going to go through next is like the ECO system.  So.  When I first, when I first moved to digital advertising I came from security.  When I first came like to digital advertising someone she had me this.  This is the [indiscernible] of digital advertising when I saw that oh man I don't know what to do with this.  So it's very complicated and well, we won't go through this today.  So, what we are going to go through is like a disstilled version so I actually made this but I even disstilled it more.  We are going to talk about the main entities in digit fat advertising which is advertise sir, demand sight platform, the [indiscernible] the supplier sight platform and publisher.  So this is my real 101-1101 version first to think about two end of the spectrum two ends of the spectrum is the advertise sir the ones who sell the product and publish sha who is selling the ad sideways the media space its your block like CNN, CS -- main -- like Nike forward an all of those kind of guys.  The address would have like advertising agencies like mad men stuff so like here we have an advertising agency one the bigger ones curative P and then what happens here they produce ACON at that time but in terms of digital advertising the real important one is demand sight platform VSP so this is where really the magic paps this is where the campaigns are made, it's very important because this is where you actually enter like targeting information likes who do I want this add to be served like what demographic, what location and all of that stuff.  So, in the other end you have SSP the supplier platform.  It manager's impressions that come from different publisher and it all intersect in this add exchange think of the add exchange as a marketplace an auction place of these different impressions and what happens is the DSP matches its criteria to all the impressions that are being sold.  Let me give you example.  Like forward forward wants to do like an -- FORD wants to do an add campaign for F150, they want to do end year campaign advertising agency conceptual lies it so what happens there like well what we want, we want these address to be shown at these particular times at well all devices then -- only the United States, let's say that's their criteria that's their targeting criteria.  Like for example my blog received impression or a view from let's say the MANILA it goes there goes to the auction house this impression is coming from MANILA doesn't fit criteria, no.  Like, there's no F one -- no F16789  -- for example something comes in hey this impression comes from Florida from a mobile sight it -- from sprung that's when the DSP starts the bid and if it wins the bid the ad is served so that is really like general gist of what's happening in an add exchange so I know this is pretty high level and we can go there like we can go like into more detail like after this.  I'll just be walking around here if you want to talk to me about the system but this is enough to start talking about ADD fraud.  And all that kind of stuff.  So, one thing you might want to go deeper is like how do you actually like serve an ADD so so here is a process of serving an add there's actually three things, first, make the campaign.  So, that's where the DSP part happens, right.  DSP would usually have like a campaign management system and this campaign management system will have like the ability to like add -- ADD targeting information which I will show I was little bit like some screen shots like bigger screen shots so I think it's hard to see there so with the bid admitting process, the BIDDING project the ADD exchange where the DSP and SSP link together then if the BI -- bid wins then the ad is served.  Let me give you quick screen shots of what campaign management system looks like.  So you see here you can actually define what your budget is.  Then you can like set like different demographics like what location you want this particular ADD to show up.  You can also do a lot of things you can actually pick what particular operating systems it should show up in like what particular devices it show up in like all different platforms you can actually fix.  It's very good at this targeting thrust -- just think about that.  Then you can do all sorts of stuff contextual stuff.  Key words, topics, so it's very very like, very smart.  Very smart in terms of targeting.  So next you can actually pick also what in -- like what publishers to publish on.  So, it's very very good at that.  You can actually like you can start up loading creative here you can actually like up load your [indiscernible] juror JPEGS and flash files you are probably thinking advertising, redirects, targeting, to she's, you can do, like you can do all start -- stuff with this one, like, you can do like like targeted attacks if you wanted to that's why advertising is so big now.  But we are not going to -- we are going to talk about mal veer advertising a little bit that you think that is enough for the background.  We can talk about now like the ADD fraud problem itself.  So, what ADD fraud?

>> So it is the crib practice of attempting to sell ADD that have no intention of being viewed by a user.  You are serving ADDS you are purposely serving ADDS that no one is seeing that is basically the definition of ADD fraud.  It's kind of weird you'll see that later.  So the problem there's a lot of claims about how big, the big, the extent of the problem is.  So, in some cases in the low end, they say that ADD fraud and like, fraud impressions are about 30 percent of the low end and 60 percent of the high end so what's the scans of this?  

>> If you think -- let's use low end like 30 percent so if you think about it, the total digital ADD spend is 60 billion so even 30 percent that's about seven billion I think.  So that is pretty big chunk of the pie, right.  So whose whose losing money obviously the advertise sir.  They have too much money any way that's a big chunk of change.  So, in any case who are the actor who are actually making money out of this.  So -- so they make money out of like generate tongue traffic.  And the next one obviously who makes money in advertising?

>> It's a publishers.  It's like the blog, the big websites and all of that stuff.  So how do they make money?

>> So if they purchase traffic, there's a cost there and they get money from the advertise sir so they make money off of the spread.  So in maybe two weeks from now I'm releasing another paper called economic of ADD fraud so I guess just wait for it and stuff. So, what are we doing about it?

>> So before I go through that, I want to introduce you to the interactive active advertising bureau so when I first started like when I moved to my new job in like, the digital advertising, the first meeting I went to was the an tie Malware group IAB, so IAB actually develops industry standards or on line digital advertising.  So, they are actually doing really good things but honestly it's a bit confusing.  Like I was confused at first special l with the ADD fraud -- they have this whole list about like ADD fraud here's like a quick outline and it had me badge felled some of it criss coving why is this ADD fraud what I'm going to do is disstill the -- into something a little bit sim and a little bit more straightforward.  So there are basically three main types of ADD fraud the first one is publish -- to increase impression county we'll talk about that later.  Second one is serving legal or malicious content.  Third one is using non human traffic to increase impression.  So, you see here, right, -- two of them are related to increasing impressions so directly related for generate ting money but the second one is actual allocated little bit different its serving content it's about serving the wrong kind of consent -- content.  Et cetera -- let's go for the first one first the idea of publisher tricks to increasing impression count is to make like not really one ADD impression but make a few ADD impression look more like  -- more ADDE pressings how does that happen some of the exam potentially here are hidden PIQEL one by one -- hidden PIQ -- how does that happen.  For example, like, eye view one website and I see one ADD like the red block there I assume that's an ADD.  So you see one ADD but the publisher reports that I saw three.  So that's how the publisher tricks work.  Let me give you some examples like with the hid Ken PIXELS typically you want to see this.  So, if you see this, you see three ADD, right that's legitimate, but you see this like hidden ADDs so when you look -- let's say you look at that page you see no ADDs but it is actually serving ten or more why because they are getting the ADD as in the -- so you are actually generate tongue like a ton of I will -- a TON of impression that's the idea of these hidden address the same thing with ADD stack king you stack the ADDS I guess you see this a lot in porn sites or something like for example you see one ADD then you don't know that things other ADDs there you see one but its generate and or serving ten or more or something like that.  So the thing here is like, this is this is kind of effective but its rarely seen now less than before.  Because -- because pub issue bluish her make money out of their sight and the publisher are doing this on their sight so it can be directly attributed to them so they don't want that happening so that's why this is a little less pref lent nowadays these like publisher tricks to do this.  So, let's go to, let's go to the next one.  So okay so serving objectionable and malicious content.  Sometimes ADD fraud doesn't mean increasing impressions although it could lead to that.  This case like the prominent examples are like serving Malware MAL veer advertising, exams like fraud fraudulent websites and stuff like that then non [indiscernible] when I say objectionable non brand serve these pertain to pornography, hate and stuff like that so there's also like categories regarding objectionable categories that you will see.  How does this work?

>> Well we just go back to our like how ADDS are served for example malicious advertise he is you want to send like like IRS examine or something through an ADD you just make correct campaign like campaign set up into the SP and if you win the bid you pretty much just like like get all your ADDS through the publish her which ends up April -- in the user fairly straightforward that's also what happens when you talk about mal veer advertising as well so that is the malicious content stuff.  I'll give you some examples.  When I started like doing research other than this, I was looking at an add -- dash inventory of ADD network, and well needless to say in some network the inventory the publisher inventory is very dirty so you see here like you are serving Malware there's TONS of ADD wear here's the thing some cases even if you serve like, like, like illegal content it could actually generate like more impressions such as ADD wear, like here's an interest and thing like exams and stuff and the power of like targeting   Ted advertising its too small like for example in that very left-hand side, the scroll in the lower left eye actually like loaded that in our MANILAA office I got fraudulent like examine website that's directed to the -- so you see it's in -- all the department stores there are actually like in the local area.  So it's really smart doing it this way like sending your malicious stuff through address because you can actually target specific [indiscernible] so that's wallets really cool about that stuff.  So, now, we can start talking about non human traffic.  So, most of the time when you talk about non human traffic, really it is -- but in some cases it gets lumped in like some other stuff gets lumped in some gray areas and it's not actually bought but actually like human impressions that are kind of low quality so I'll show that to you guys in a little bit.  So let's talk about non human traffic.  What is the best way to investigate this?

>> So when I started this the best way to investigate this was actually start buying traffic as well.  So that's what I did.  So, the two main questions that I wanted to answer is what is purr Chad internet traffic made of and can I buy internet traffic and get away with it.  So that's -- and that's the start of well the BEIBER project.  The project is really it's all like just a TON of blog that I made that are basically HINEY parts that's basically it.  I did not make the content by myself.  So, I just grabbed it from different sites but I had a ton of them.  So the idea here is to like buy -- purchase traffic and purposely direct those impressions in those blog.  And those blog actually have like -- and what particular types of contractor ribs and at contributes that I wanted to get and then I collected information and did analysis.  So I said collected, right, so I used a lot of Java script and you can get a lot out of a BO -- that's really what I used to figure out like what is this traffic actually is and I just [indiscernible] and just went through it.  Then next is the fun stuff.  I started buying traffic.  So initially I was thinking like oh this might be hard like where do I buy traffic because like like for example It's like hard to get like typically I thought it was going to be as difficulty as get thing like Malware but no it's really easy so there were a ton of vendors that I found like selling traffic and these are just one of the few I probably like used 30 for this experiments and there's also like traffic market places out there and you can buy all sorts of stuff so not only internet traffic not only impressions but you can buy like YouTube views, YouTube likes, actually like I spoke DEF CON kids this morning that other one, it was a little -- what I presented was a little bit different it wasn't -- so I did like an experiments so I had a Twitter account.  So I opened that like maybe a long time ago when I started like, cause I gave a talks here in DEF CON before and I think I need add Twitter account so I had a Twitter account opened to have four years I had 21 followers for four years.  And a few days ago I started buying followers and now I have four thousand followers impressive, huh.  So so I don't know about that so that's off topic.  So traffic market places.  And now let's talk about what is purchase internet traffic made of?

>> So, well bought really like most of it so how do you know as I mentioned like the clues are actually in the impression itself like you grab it using Java script and you just analyze it.  So there are a lot of clues.  And you know, right, like browser can like a ton of information about you and what you are doing and all you need is like some quick Java script I'm not a Java script G U R you or anything like that.  So anyway, so you can look for status suspicious information for example like plug ins, does it have plug ins like a lot of bugs does it have plug ins.  Does the plug in match what the browser is or any of that stuff does it have screen at contribute window at contribute how big is the view port is it giving you the correct view ports the product identifiers are all the product identifiers matching so does it look like that its actually saying the browser is what its saying it is.  [indiscernible] so it it have a frame rate an mason frame rate is it rent ring Java script there's a ton of stuff you can look at and determine whether its non human traffic.  Of course user agented as you all no like user agents are very easy to forge and stuff like that.  But, you know, oh these are just like the clues, like sometimes like like all this stuff that -- but sometimes the clues are really obvious, you know, and when I started buying like internet traffic the cheapest one I noticed didn't even care to -- that it was like -- it was plain sight in their user agent so the cheaper you get the more bots and cheaper bought you got.  Not all traffic are made equal.  So I say like this is actually like inaccurate it should be like weaker approaches and like smarter approaches.  So, what are the weaker approaches?

>> Well, traffic generate so there's a ton of traffic generators out there you can buy.  You can purchase traffic why not generate the traffic yourself so you can use like traffic generates so there's a lot out there I won't go through each one of them.  And the thing is like I made like a summary of all of this so I'll stop here.  Like the custom stuff you can do custom stuff and you its too small there but you see like a lot of these custom stuff are actually made by like through like you bought stud I don't not sure if anyone is familiar with you bought stud I don't it's an awesome software.  It's not malicious its actually a legitimate software and what you use it for is for web masters to help facilitate like ought mating stuff in their websites like logging in, doing stuff, posting stuff but you know all good things can have like a bad aspect as well and you can also use this to like create followers, alcoholic on likes, create subscribe percent and all of those stuff you can manipulate that.  But really It's like its like It's like the -- so it's really like very very useful and it has like a scripting language and all of that stuff.  So, so, you'll see a lot of the bots that they are being sold actually the made out of you pot studio.  Traffic generator, what are the important things to look for if you are buying a traffic generate?

>> So obviously it has to be able to CLICK on links and selected areas.  And then  -- you can't have it bottom barred admitting like like the web side right then obviously changing user agent is very important.  Custom -- fur not like CNN or something you can just type it indirectly in your browser for example you are like a blog like known blog and you have like suddenly you have like millions -- millions and millions of prim messing -- that is suspicious usually you would like to does a receiver from a search engine, someone found your sight and went to your sight some of the important things with traffic generator obviously proxy support is important so like you can't have your traffic coming from one IP but here's the problem also with IP proxy with the proxy stuff S like, a lot of the one of the main thing that like interactive advertising bureau ideas is like, like having checking like proxy you can scrape from a lot of like different sites so its -- whether good guys no the proxy address and versus you so that's the challenge there.  So, but sometimes it is a little bit tricky yes, sir so you can do like a more tricky yes, sir things.  So, let me tell you about [indiscernible].  So is actually a traffic exchange software and it's in Chinese but they are like tutor y'alls out there to learn what the buttons mean so I'll tell you more about it which is pretty cool.  So traffic -- so here for example I installed [indiscernible] in my computer but the thing is like [indiscernible] it won't visit nie website because if it visit my website it's just pretty much a traffic generator, right.  [indiscernible] like visits other peoples websites and when you start visiting other people's websites you generate TOKEN the more TOTENs you have the mother -- the mother other people have  enforced stalled visit your would be sight that's the idea how it works.  You see there's traffic exchange and you don't have to think about proxy because its actually coming from different computers already.  So that's the neat thing about it.  So let me show you a quick video of the [indiscernible] stuff.  So what I'll do I won't run the whole -- I'll do like manual fast forward stuff so here is -- oh it's not showing up.  How does this work?

>> Oo my oh this is going to suck.  Oh, no.  I'll drag it over.  Right.  So I'm releasing -- oh, it works.  That embarrassed me.  Where's that thing.  Thank you Molly this is going to be really difficulty for me.  Okay.  So let me go through this it's going to be a little bit difficulty so yep, so, you see there on the right that's [indiscernible] and I have like -- open so let's go through this so yes so it's running so start -- so you see like is already running I'm visiting like sites already so this is my BIEBER blog, so these are moi blog let's continue let's continue.  And you see here I'm already receiving impressions right off the bat.  Let's go.  So, [indiscernible] has started a process and it is resource intensive it eats a lot of resources here it is already context tongue to IP address you see here its -- and I have low priority actually so lots of resource -- so let's see what's happening in the network.  So it's already visiting a lot of sites.  I have stats for you at the typed of this so there you go.  So traffic is very busy.  You see there on the left-hand side -- shark It's like banking away so here's my blog continue to Google and stuff here's the interesting stuff one of the main things they told me one of the weaknesses most of the traffic actually comes from china.  But, you know, like in my most reSeptember once, no, its actually coming from all over the world.  And I don't have like actual stats but in my sampling most of -- of it it's not all china it's not majority even so getting like a lot of stuff here, that stuff so it's still working, there you go so let's -- so at this point like there's like already like a ton of [indiscernible] processes there.  It's very busy yep so let us generate some stats I'm generating stats off wire shark.  So here's the interesting thing here so what happened here like Iran it for 30 minutes just 30 minutes so for 30 minutes I generated four thousand HTTP requests from my computer alone so it's a lot and like I probably visited about 300 to 400 unique websites so it is a lot of stuff.  You might be wondering how much impressions did I get for that 30 minutes I only about 200 it's not fair, huh.  Time generate tongue way more than I am getting but still, still, like if you look at the impression they look like pretty quality impression all over the world.  So its kinds of useful.  So next -- let's move on.  Okay were done with the engining and stuff.  There of course there's Malware as well so this is like the one of the last NHT stuff that I'm going to show you guys.  So the problem here is is I have another video so next let me open the stuff for my add wear video. Okay -- ADWARE video, it's not work hadn't.  There you go.  There you go.  Here we go.  So same with the [indiscernible] stuff I'm going to forward it manually.  So I will tell you about like the correct -- corrected risk of Malware that's add fraud so let's go through here.  So one of the first things you'll notice is you'll seldom notice anything just as with all types of Malware it's very subtle.  -- subject.  Let's go through it.  Sometimes you notice if you know key stuff for example there like just a pop up from a web page came out.  So where is it coming from you probably like have an idea already.  So, the pop ups are actually coming from hid ten windows like hidden browsers running in the back are background so Malware is using your browser with your plug ins with your mind types and in working like in running it off like like in the background but just hidden so that is very smart actually.  So these are, you probably see there the hidden windows already.  The next, here's the thing though like sometimes, oh I have ten minutes so the funny thing here is in some cases like what you see is -- you see clues suddenly if the Malware makes -- forgets to like mute the volume suddenly you are working suddenly like an ADD will play you'll see the audio but there's no like ADD anywhere and there's no video anywhere so that's when you'll see there so let me make this a little bit faster so since -- since I have ten minutes already so here as I mentioned like high jacks browser very smart if you high Jack about sir you are piggy back king off the -- so it's hard to actually like fingerprint it as malicious because you are actually piggy back and off browser that's really smart.  Aside traffic generate that what's really smart about some of the user browser it actually generates scrolling behavior and mouse behavior, you see the scrolling there, right.  But the mouse behavior is not too obvious.  But that's other thing because in some cases you try to detect non human traffic bug BO checking user events and user engage try to buy pass this by adding some user movements so let me end that.  Then -- so you can detrains analysis to catch those, you can -- as I said you can do -- human in general have more purposeful pattern like they go from element to element, smoother movement mixture of events I actually had like another video but I won't be showing that I have something here like the actual like my BIEBER website track engage its able to like like what track what element you hovered in how long you hovered in the element what -- even if you highlighted a text, like what text are you highlight tongue how long you stayed there what alcoholics you are doing it actually gets way more, CLICKS you are going Malware very injury key you do a program scroll and a program mouse movement so there should be like ways to look at other stuff.  So, as I mentioned there are gray areas here not all in NHT all like traffic are made of boss so mirrors the the gray area so in some cases when you buy traffic like traffic vendors will actually put your website in other high traffic websites and serve it, pop under and pop ups.  And a lot of like verification services miss that.  They almost a lot of them will say it's all boss but sometimes it's not.  Sometimes its human but low quality.  So in this case likes for example win -- in one of the things that -- one of the vendors I looked at 70 percent of all traffic came from one PIXEL -- usually this is indicative of pop under.  Pop under a gray area pop under it pop under but difficulty to say whether it was seen or not because it takes interaction to actually see a pop under.  So can I actually buy internet traffic and get away with it.  First well if an advertise sir no what to look for, like the at contributes is about yes you will get caught but if not you will not get caught another thing that I discovered when like paying for all this internet traffic is you get what you pay for.  So, lower prizes you really get boss with higher prizes when you start buying traffic you get frames, pipe ups, pop under.  How much time do I still have so I don't have anymore time.  Oh five minutes so I will just do like a quick wrap up so, we talked about like the businesses of advertising, right, so important thing to know there is like yes it is a very important field to start going into like as I mentioned it is a 60 billion-dollar industry right now.  And just keeps on growing.  And there is like a lot of stuff that you can like, like figure out there.  The next is the ECO system.  Remember important things publishers, advertisers you had demand sight platforms where you do -- which manager's impression you have ADD exchange cheers which are basically the marketplace for these impression finally you have publish summer fraud, you have malicious content and non human traffic.  So, that diagnosis it for me.  So if you like my presentation please visit us at [indiscernible] so. I would like to mention my books totally unrelated to ADD fraud or add DD particular please check it out any way.  I'll be here if you have any questions I'll be walking around thank you very much.

>> [Applause]