00:00:00.000,00:00:04.438 >>Uh so uh we're going to get the next talk started. Uh let's give big round of applause for 00:00:04.438,00:00:09.443 them. We've got at least one new speaker. How is any [applause] 2? Well alright. [applause] 00:00:14.982,00:00:19.853 >>Hey everybody three for the price of one. My name is Gina Matthews. And I'm a computer 00:00:19.853,00:00:23.991 science professor at Clarkson University if way way upstate New York. Nosebleed New York as 00:00:23.991,00:00:29.429 we like to affectionately call it. Right across from um Canada. Uh Ronnie Bull is both a PhD 00:00:29.429,00:00:34.501 student with me at Clarkson and also faculty member at Utica College. And Kaitlyn Trumbull uh 00:00:34.501,00:00:38.038 is a senior undergraduate who has been helping us run all these attacks that we're going 00:00:38.038,00:00:42.509 to tell you about. Particular we're going to talk about V-land hopping and arch poisoning and 00:00:42.509,00:00:46.813 man in the middle attacks in visualized environments. And this is a bit of a follow on to 00:00:46.813,00:00:52.152 the presentation we did last year at DefCon. [pause] So here's a little road map of what 00:00:52.152,00:00:57.157 we're going to do. We're going to talk about the context of the problem. For later 2 network 00:00:57.157,00:01:00.694 security in visualized environments. The problem in multi-tenant environments 00:01:00.694,00:01:05.699 especially in cloud services. We're going to talk about the test platforms that we used. And 00:01:07.734,00:01:13.540 all the different virtual uh the hypervisors and the virtual network uh de devices that we 00:01:13.540,00:01:17.778 tested. And we're going to talk about the specifics of the attack. We've got some uh great 00:01:17.778,00:01:23.083 videos and demos to show you. Specifically of the uh the V-land hopping and arch 00:01:23.083,00:01:27.554 poisoning this time. We also have videos and many many details of the map 00:01:27.554,00:01:31.625 [indiscernible] DHCP attacks that we talked about last year if you want to look those up. 00:01:31.625,00:01:36.630 And finally some conclusions. So um the key question is when you have virtual machines that are 00:01:39.866,00:01:45.572 hosted in a multi-tenant environment [pause] how safe are they? What can they do to each 00:01:45.572,00:01:51.178 other over the network? Um there is essentially connected to a virtual version of a physical 00:01:51.178,00:01:56.483 networking device. And the question we're an trying to answer is the layer 2 network 00:01:56.483,00:02:01.154 attacks that typically work on physical devices do those work on their virtual counterparts as 00:02:01.154,00:02:07.894 well? And if so what are the implications especially in a multi-tenant environment. Cloud 00:02:07.894,00:02:11.365 services that rely on these virtualized environments could be vulnerable. And that can 00:02:11.365,00:02:16.570 include data centers, hosting, mission critical or sensitive data. This is not the only class 00:02:16.570,00:02:21.608 of attacks from collated virtual machines. It's kind of an old story. You're vulnerable to 00:02:21.608,00:02:25.479 those that are close to you but in uh public kind of multi-tenant environment you're 00:02:25.479,00:02:31.351 not quite sure who you land next to. And um we've done some other work in the past of the you know 00:02:31.351,00:02:36.223 they types of things that virtual machines can do to each other. As have many other people 00:02:36.223,00:02:41.228 but in particular today we're talking about the attacks over the network. So here's a pretty 00:02:43.296,00:02:49.736 a simple diagram of the setup we're worried about. We have a physical host here. And on it we 00:02:49.736,00:02:55.108 are hosting a bunch of virtual machines. One of them that's a nasty malicious one. It's going 00:02:55.108,00:03:01.181 to do things to the other uh victims. And they're you can see that they're sharing a virtual 00:03:01.181,00:03:08.021 switch or bridge. And that's passed through to a physical network interface. And to kind 00:03:08.021,00:03:14.227 of jump ahead the bottom line is that we do see in our research that virtualized network devices 00:03:14.227,00:03:19.533 have the same ability to be exploited as physical devices. Sometimes we see more 00:03:19.533,00:03:23.904 vulnerabilities in physical devices. Sometimes in certain cases we see less in a physical 00:03:23.904,00:03:28.508 device and we're going to show you that. Uh for the attacks that we're talking about today. 00:03:28.508,00:03:33.780 Another investing thing that we've seen is that um some of these attacks allow have the 00:03:33.780,00:03:37.317 ability to leave the virtualized environment and affect the physical networks they're 00:03:37.317,00:03:42.856 connected to as well. So that's another interesting bonus for you. What are some of the 00:03:42.856,00:03:49.830 consequences if a malicious VM successfully launches a layer 2 network attack in a multi-tenant 00:03:49.830,00:03:53.834 environment like this? What are some of the things that they could do? Well they could 00:03:53.834,00:03:58.738 capture all the network traffic coming from the victims that are co-located with them. They could 00:03:58.738,00:04:03.276 re-direct traffic. They could perform man in the middle attacks. Denial of services 00:04:03.276,00:04:07.447 attacks. They could gain unauthorized access to restricted subnets. And they 00:04:07.447,00:04:12.452 could affect performance which is I guess uh a sub example of denial of service. [pause] So 00:04:14.621,00:04:18.758 here are some of the test scenarios and results that we're going to be talking about. We're 00:04:18.758,00:04:23.697 gonna give you a little update on mac flooding. Uh especially because we've changed over our 00:04:23.697,00:04:28.535 testing platform completely since we spoke here last year. We're gonna talk about v-lan 00:04:28.535,00:04:34.741 hopping in quite a bit of detail. Also arch poisoning. And some man in the middle attacks. 00:04:34.741,00:04:39.379 Here is a lovely picture of the test environment that we were using last year. It was pretty 00:04:39.379,00:04:43.817 much built from pieces and parts that we could salvage. Uh rest in peace you served us well. 00:04:43.817,00:04:50.223 Thank you very much. Um here are for uh complete details uh some of the hardware specs and the 00:04:50.223,00:04:54.995 hyper visors and the virtual uh networking devices that we were testing when we spoke to you 00:04:54.995,00:05:00.000 last year. All those details are on the DefCon 20-3 CD. Um we have nice new shiny hardware uh 00:05:04.671,00:05:09.609 thank you to co college for that. Um and it allowed us to basically repeat all the 00:05:09.609,00:05:15.081 experiments we did before and just you know a much more um controlled environment than our 00:05:15.081,00:05:20.086 pieces and parts. And also extend the work. So here is some of the gorey details of the test 00:05:23.390,00:05:28.595 environment that we used this year. You can see a lot of different hyper visors. A lot of 00:05:28.595,00:05:33.200 different virtual networking devices. And you can see some details here of the hardware 00:05:33.200,00:05:38.138 specs. Again all of the gorrey details are available for your viewing enjoyment um on the 00:05:38.138,00:05:43.143 DefCon docs. Okay so to kick things off and also just to provide a good um transition 00:05:45.812,00:05:49.316 between what we did last year and this year and give you a little proof update on the mac 00:05:49.316,00:05:54.888 flooding stuff. First you know here's a a simple diagram of the scenario in which we were doing 00:05:54.888,00:06:00.927 mac flooding attacks. Again we have 2 VMs attached to a virtual switch. We have a victim a 00:06:00.927,00:06:05.799 target VM and we have an attacker that's patched through to a physical interface that 00:06:05.799,00:06:10.804 connects to a physical switch and the internet beyond. And the target VM is just doing a set of 00:06:14.074,00:06:18.378 regular pings and we're going to see what happens to the performance of those pings when 00:06:18.378,00:06:24.351 the attacker launches a mac flooding attack. And here in this uh graph which is something 00:06:24.351,00:06:29.356 we showed you last year you can see from like 0 to 65 or 70 or so what was happening when it 00:06:31.958,00:06:37.530 wasn't under attack. So low latency pings. No trouble. You can clearly see when the attack 00:06:37.530,00:06:43.069 get long gets launched. And you can see just after 240 when the attack stops. You can see clear 00:06:43.069,00:06:48.074 impact on the network performance for the the victim VM. And this was done on gen 2 00:06:50.410,00:06:54.881 [indiscernible] bridged interface. And this gives you just a sense of you know we're 00:06:54.881,00:07:00.787 just doing a lot more systems and platforms this year. So this is the same test performed 00:07:00.787,00:07:06.926 across not just um that gen 2 uh zen bridge but also gen open um excuse me zen open v switch 00:07:06.926,00:07:13.300 centrix zen server hyper v standard switch hyper v nexus 1000 v the proxmox bridge VM 00:07:13.300,00:07:18.305 ware E X X i and a control physical switch a cisco 29 50. And um so some of those when the 00:07:23.043,00:07:27.147 attack starts don't aren't affected. It's a little difficult to read from this this 00:07:27.147,00:07:32.118 graph. Maybe it's a little easier to read here. Still not incredibly easy. We didn't we we 00:07:32.118,00:07:36.289 realized at the last minute that there might have been a better way to do the one. But um from 00:07:36.289,00:07:41.227 this ay if you look clare carefully you can see that the hyper v standard the hyper v 00:07:41.227,00:07:46.232 nexus um VM or X X i were not vulnerable. Um and uh I'd like to make the point before we head 00:07:51.805,00:07:54.941 in to some of the rest of the things that all of these layer 2 vulnerabilities that we're 00:07:54.941,00:08:00.213 discussing were targeted toward the virtual networking devices not so much the hyper visors 00:08:00.213,00:08:04.951 themselves. What we're testing is what happens with the virtual networking devices. So it's 00:08:04.951,00:08:09.956 interesting you see a mix right? Not everything is vulnerable. Not everything is safe. And 00:08:09.956,00:08:14.694 you're going to see that trend continue across all of our tests. It wasn't uniform which 00:08:14.694,00:08:19.199 things were vulnerable and which things weren't. This point I'm going to pass it over to Ron. 00:08:19.199,00:08:24.504 He's going to tell you about the v lan hopping. [pause] >> Okay so we've performed a few v lan 00:08:24.504,00:08:27.974 hopping experiments on the environment as well to see what we could do um about getting 00:08:27.974,00:08:32.612 frames onto unauthorized networks. Uh we fond some interesting results. So some 00:08:32.612,00:08:37.484 basics about v lan hopping. It attacks um just allowing um unauth unauthorized access to 00:08:37.484,00:08:41.855 another virtual lan a packet switch network. Um 2 methods of doing this currently uh switch 00:08:41.855,00:08:45.992 spoofing and double tagging. Where switch spoofing is Cisco proprietary um using Cisco 00:08:45.992,00:08:52.465 protocols. Uh and double tagging is an exploitation of the 8 0 2 dot 1 cube protocol. [pause] Uh 00:08:52.465,00:08:57.036 so basic virtual internet information we have a basic frame. Uh ethernet frame here. 00:08:57.036,00:09:01.808 And what we do to get v lan going is uh we stick 4 byte um tag in there. Or v lan pattern 00:09:01.808,00:09:05.612 with 32 bits of extra information. Um to support the v lan ID to show where that frame 00:09:05.612,00:09:11.017 is supposed to go on the correct networks. So switch spoofing. Here's a CBE listed um about 00:09:11.017,00:09:15.088 Cisco switches. They they support 8 0 2 dot 1 x security allow attackers to bypass port 00:09:15.088,00:09:21.194 security and gain access to v lan uh via spoof Cisco discovery protocol messages. So a little 00:09:21.194,00:09:24.898 bit about the Cisco discover protocol. Basically it's a proprietary protocol that Cisco 00:09:24.898,00:09:29.269 has on their switches that allows 2 or more switches to identify each others operating 00:09:29.269,00:09:34.674 systems. Automatically negotiate connections things like that. So we can have an vulnerability in 00:09:34.674,00:09:38.978 that protocol and actually exploit it we can get on a different v lan. [pause] Couple 00:09:38.978,00:09:43.450 more CBEs and vulnerabilities for switch spoofing. Uh Cisco catalyst 29 hundred virtual lan 00:09:43.450,00:09:48.121 switches allow remote attackers to inject 8 0 2 dot cube frames into another v lan just by 00:09:48.121,00:09:53.660 forging the ID tag. Um and a quote directly from Cisco from one of their white papers about 00:09:53.660,00:09:58.932 a dynamic trunking protocol basically if it's enabled um you can send a fake BT packet out 00:09:58.932,00:10:03.670 and switch that port from auto into a trunking mode. And we're going to see that is pretty 00:10:03.670,00:10:07.006 effective across some of the environments. Um and it's important to note that on most 00:10:07.006,00:10:09.976 Cisco switches out there at least some of the earlier ones that are still in production 00:10:09.976,00:10:16.783 today um DTP auto is is the default setting. So you'll wanna make sure you uh harden that. 00:10:16.783,00:10:21.588 Okay so again this is just another Cisco proprietary layer 2 protocol. Um along you do that 00:10:21.588,00:10:26.526 automatic con uh configuration of trunking connections between 2 switches. Um pair this with 00:10:26.526,00:10:31.331 CDP and you're pretty much have uh automatically configured network. So some of the 00:10:31.331,00:10:35.869 consequences of this? Um basically the attacker can have a full trunk connection to the 00:10:35.869,00:10:40.640 switch. They can make themselves into a trunk. Uh spoof themselves but as a switch which 00:10:40.640,00:10:43.910 allows them to have 2 way communication on that v lan. They place themselves directly 00:10:43.910,00:10:47.847 on that v lan. Um allowing them to eavesdrop on the track. They can send messages to and from 00:10:47.847,00:10:54.420 systems on that on that v lan. Okay so here is um spoof switching demo. Uh basically we 00:10:54.420,00:10:59.959 do this in the E S X i environment. Um the setup is we have a virtual machine um within 00:10:59.959,00:11:04.464 the E S X i environment. And it's going out through the virtual switcher bridge. E S X i 00:11:04.464,00:11:09.168 is more like a bridge than a virtual switch uh the standard switch on it. We have that 00:11:09.168,00:11:15.275 connected to a physical Cisco 29 50 switch. Um and the port is set to DTP auto. Just default 00:11:15.275,00:11:19.746 configuration. And then you see um there's another system on the other end that's connected uh 00:11:19.746,00:11:24.384 via trunk link with a v lan 20 and one support over that trunk link going into another hyper 00:11:24.384,00:11:31.224 visor environment there. So let's take a look at this uh this demo here. [pause] Now all 00:11:31.224,00:11:34.994 these videos are on You Tube. The links are on the white paper on the DefCon CD. So if you want 00:11:34.994,00:11:39.832 to watch them later. [pause] Okay so on the left hand side we can see the configuration for 00:11:39.832,00:11:45.939 Cisco 29 50 switch. Um on the right hand side we see the E S X i uh network settings for the 00:11:45.939,00:11:50.243 cali virtual machine that's on that system. We're gonna first take a look at the uh interface 00:11:50.243,00:11:55.982 settings for the um E S X i server. In this case it's called luminarra. Um you can see it's 00:11:55.982,00:11:59.686 connected it's just set to v lan 1. So it's just done the default v lan with s switch. Nothing was 00:11:59.686,00:12:04.257 changed besides the defaults. Um it's not set to a trunk port or anything. Uh we're just gonna 00:12:04.257,00:12:09.262 verify the trunk status on that. [pause] And we can see that it's using 8 0 2 dot one cube 00:12:11.965,00:12:16.736 encapsulation. It's not trunking and the mode is set to auto. So it's just default configuration. 00:12:16.736,00:12:20.406 Nothing was changed. The goal for onto the virtual machine oh we're going to take a look at 00:12:20.406,00:12:23.910 the network interface here. We can see that it's just the basic network adapter. It's on the 00:12:23.910,00:12:29.449 isolated v lan test environment we created. It's a separate network for that. Um which is 00:12:29.449,00:12:36.322 over there on the switch on net fort. [pause] So if we dig a little deeper into these details 00:12:36.322,00:12:38.958 [pause] um we can take a look at that network and we can see down there in the bottom we've setup 00:12:38.958,00:12:43.162 that v lan test network and the cali virtual machine is the only system on that network. Um and 00:12:43.162,00:12:49.235 if you look at the properties of that we can see that this uh this connection has no v lan ID 00:12:49.235,00:12:54.774 site to it. So it's just on the default v lan for the E S X i virtual switch. Um now we're 00:12:54.774,00:12:59.779 going to go over to the virtual machine itself. Grab console access. [pause] And we're going 00:13:01.881,00:13:04.784 to start the attack. In this case we're gonna just use the your [indiscernible] program. So 00:13:04.784,00:13:11.791 we're just going to type your siennia minus i to get into interactive mode. [pause] Gotta 00:13:11.791,00:13:16.796 bear with me a little bit. [pause] Okay and then we're going to select the default 00:13:21.000,00:13:26.005 network interface on this. So just hit enter. [pause] And then with the launch the attack. So 00:13:28.307,00:13:33.746 we're just going to go up and select the uh DTP or the dynamic trunking protocol. [pause] And 00:13:33.746,00:13:38.751 then launch the attack. [pause] Press 1. [pause] And we can see it successfully went into trunk 00:13:41.954,00:13:46.759 mode. So the uh dynamic desirable effect was effected over there on the switch side we 00:13:46.759,00:13:50.530 see the line m protocol went down. It came back up again. And if we check the status on that 00:13:50.530,00:13:55.535 trunk port we'll see it indeed went into trunking mode. [pause] We see it's now instead of v lan 00:13:58.805,00:14:03.743 1 it shows the trunk. [pause] And if we take a look at the full status [pause] We can see 00:14:10.283,00:14:14.320 that it's d set for trunking and auto [indiscernible] encapsulation. So the system was 00:14:14.320,00:14:18.725 effectively put on as a trunk now we can access any v lan we want that's associated with that 00:14:18.725,00:14:23.262 trunk. So in this case I need a system that was on v lan 20 or whatever. So that was the switch 00:14:23.262,00:14:28.267 spoofing attack. You can see it was affected or effective in the E S X i environment. And let's 00:14:30.703,00:14:36.409 see [pause] Okay so results across the board here. We can see that uh the physical cali 2 00:14:36.409,00:14:40.213 point 0 control system we setup the physical switch. We tested it with a control to make sure 00:14:40.213,00:14:44.484 that it worked first. That way before we proceeded into he virtualized environments um you 00:14:44.484,00:14:48.755 can that it pretty much worked under every system that that used bridge on networking 00:14:48.755,00:14:52.158 instead of a virtual switching networking. So if you're using an environment that has a 00:14:52.158,00:14:58.164 bridged network uh virtual switch um this is gonna work. And it worked in uh ah o open 00:14:58.164,00:15:01.968 source zen with linux bridging. It worked the vm ware E E S X i environment as you saw. And it 00:15:01.968,00:15:06.205 also worked in proxmox which also supports bridging as well. So any environment that uses the 00:15:06.205,00:15:10.676 virtual switch it it did not work because basically those that DTP packet when it was sent 00:15:10.676,00:15:16.582 through it was blocked. Uh for getting to the physical device. [pause] Okay so mitigation. Um 00:15:16.582,00:15:20.153 in the physical world you just disable unused switch ports. Don't give access to people who 00:15:20.153,00:15:25.591 can actually get connected to those ports. Um disable CDP and DTP. Or use it on a as need 00:15:25.591,00:15:29.695 basis for those ports that you really need it setup on. Um restrict the amount of trunk 00:15:29.695,00:15:33.299 ports. You don't want to just have ports setup to be trunks if you don't need them. Although 00:15:33.299,00:15:36.903 most all the hyper visor environments if you read their their documentation they 00:15:36.903,00:15:42.375 specifically say that for best practices setup a trunk port for connection to the the virtual 00:15:42.375,00:15:45.645 switch. So that you way you can actually support v lans into your physical environment and 00:15:45.645,00:15:50.449 other virtual environments. Um so really just limit these the access but in the virtual world 00:15:50.449,00:15:54.086 it's really hard to do this. So if you're using bridged environments realize that this 00:15:54.086,00:15:59.659 could be affa affecting your network. [pause] Okay let's talk about double tagging now. Um 00:15:59.659,00:16:03.796 here's the CVE for double tagging. The 8 0 2 dot 1 cube v lan protocol allows remote 00:16:03.796,00:16:08.467 attackers to bypass network segmentation spoof v lan traffic via message with two 8 0 2 dot 1 00:16:08.467,00:16:13.472 cube tags. So we say before that we have um a normal [pause] oh basically this is the uh normal 00:16:16.275,00:16:20.580 traffic this is the the scenario we used here where we have 2 switches. Um a trunk connection 00:16:20.580,00:16:24.851 between them supporting v lan 1 2 and 3. You can see 2 machines on one side connected to that 00:16:24.851,00:16:29.522 first switch. One's on v lan 1. One's on v lan 2. And on that second switch we have v lan 1 2 00:16:29.522,00:16:34.193 and 3. So the goal is to get some frames across between the 2 switches. Um from that system on 00:16:34.193,00:16:40.933 v lan 1 and try to get it over to uh v lan 2 or 3 instead of v lan 1. [pause] So here we can 00:16:40.933,00:16:45.838 see the normal flow of traffic for just using a single tag. Um where it goes through v lan 1. 00:16:45.838,00:16:49.008 It goes through to that first switch. Goes across the trunk connection still with that v lan 00:16:49.008,00:16:53.346 1 tag. And then it goes into that system that's v lan 1 on the other side. So basic normal 00:16:53.346,00:16:58.384 flow of traffic. And we can see that the other v lan 2 v lan 3 machines do not get that that 00:16:58.384,00:17:04.123 frame. It just gets blocked. [pause] So here's a comparison of frame types. Um the first one 00:17:04.123,00:17:08.694 is a standard ethernet frame without v lan support. The second one is that standard v 00:17:08.694,00:17:13.366 lan tag with one uh 8 0 2 dot 1 cube tag in the middle or header in the middle. And third one 00:17:13.366,00:17:17.503 down in the bottom is doing uh 8 0 2 dot 1 cube double tagging. So um this is where we're 00:17:17.503,00:17:21.874 actually doing uh a frame within the frame or er a v lan tag within the frame and we're doing 00:17:21.874,00:17:27.380 multiple ones. [pause] So in this case here we can see that um the effects of double 00:17:27.380,00:17:32.485 tagging. We have v lan 1 and v lan 2 on the one side. That tag has 1 and 2 in it so we have 2 00:17:32.485,00:17:37.423 headers in that in that frame. It goes in to that first switch. It doesn't get passed on to that 00:17:37.423,00:17:41.627 first system ah connect to that first switch because we're still reading that v lan 1 tag. We're 00:17:41.627,00:17:45.898 not seeing the v lan 2 tag. As soon as we go across that trunk connection that v lan 1 tag is 00:17:45.898,00:17:50.903 stripped off passing over the the frame with the with the tag of 2. And then when it gets to 00:17:50.903,00:17:54.807 the other side it goes directly to that system on v lan 2 and doesn't go to the v lan 1 or v 00:17:54.807,00:18:01.614 lan 3. So. Here we can see the same thing with 3. Just a different example. [pause] Okay 00:18:01.614,00:18:05.751 so the consequences. An attacker can send packets to any targeted v lan as long as it's supported 00:18:05.751,00:18:11.924 over that trunk connection. Um and as long as they have access to that native v lan. Um the 00:18:11.924,00:18:16.228 target on the access line is isolated. And um it's in a native v lan broadcasting. This 00:18:16.228,00:18:19.598 is not a good attack for eavesdropping. Because basically it's a one way attack. So you're 00:18:19.598,00:18:24.437 sending it a frame to another v lan you're not getting anything back to you. So it's good for 00:18:24.437,00:18:27.974 dos attacks. It's good for maybe a one way covert channel to another system on another 00:18:27.974,00:18:33.846 network. Um but that's about it. [pause] Okay so we have 3 different scenarios. We ran with 00:18:33.846,00:18:37.316 this. The first one was a control scenario where you had the 2 physical switches. Which 00:18:37.316,00:18:41.787 was this was known to work in. Um we had a physical attacking system connected on the native v 00:18:41.787,00:18:45.958 lan. And what queued that first trunk link to the second switch and we were able to access the v 00:18:45.958,00:18:50.162 lan um on the other side on v lan 20 for this. Due to the sake of time I'm not going to show 00:18:50.162,00:18:53.232 this video right now. It does work here. You guys are more than welcomed to go click the 00:18:53.232,00:18:56.268 link. Um but we have 2 more videos that I'd like to show that are a little more effective 00:18:56.268,00:19:01.640 than this one. [pause] Okay so the second scenario we did was 2 virtual switches. So we took out 00:19:01.640,00:19:06.312 that uh that second 29 50 and instead we're going from one virtualized environment through 00:19:06.312,00:19:10.316 the physical switch into another virtualized environment. So in this case we have an attacker 00:19:10.316,00:19:15.421 VM. It's on no v lan. Um just on that native v lan on that virtual um environment. Going 00:19:15.421,00:19:20.459 through that virtual switch connected through a trunk link via best practices. Um on 1 and 00:19:20.459,00:19:26.198 20 into the Cisco 29 50. Though another trunk link connected to another virtualized environment 00:19:26.198,00:19:31.570 which is going into a virtual switch there on an access uh VM on v lan 20. So in this case in 00:19:31.570,00:19:36.442 this demo we have attacker systems on zen server. And the target system's on proxmox. 00:19:43.783,00:19:46.819 [pause until 19:43] Okay so again on the left hand side we can see the switch configuration 00:19:46.819,00:19:52.291 for the 29 50. It's in the middle of these two uh two systems. Port 31 is uh the 00:19:52.291,00:19:57.296 centrix zen server. Port 29 is the proxmox system. Yes I do like Star Wars. Uh the middle 00:19:59.432,00:20:04.437 system is zen server. And the the far uh right system is the cali system on proxmox. So the 00:20:04.437,00:20:08.674 proxmox system is our target. The middle system is our zen server attacker. What we're 00:20:08.674,00:20:13.946 going to do is take a look at those interfaces just to check out the settings. [pause] So 00:20:13.946,00:20:19.452 port 31 is our attacking system. Separate trunking. Notice the mode is on. I didn't do auto 00:20:19.452,00:20:25.024 because I just specifically set this to trunking mode for best practices. [pause] And we can 00:20:25.024,00:20:30.029 see the 8 0 2 dot 1 cube encapsulation. [pause until 20:35] So what we'll do here now 00:20:36.569,00:20:40.940 is check out the the settings for the the the attacking system. And we can just see I 00:20:40.940,00:20:44.643 have it just setup on a basic network interface port. There's nothing special about it. 00:20:44.643,00:20:48.080 There's no v lan tags on that system or not. It's just a straight connection in to that 00:20:48.080,00:20:54.420 that server on that VM. [pause] Switch. The proxmox system you can see it's tagged for v lan 20 00:20:54.420,00:21:00.659 for that client. Um on that system. So we're going to run the attack from the centrix zen 00:21:00.659,00:21:06.765 server VM I'm cali there to the proxmox system which is also cali. Um on the proxmox system 00:21:06.765,00:21:10.803 we're just gonna run a simple TCP dump command. And we're gonna [indiscernible] ICPM 00:21:10.803,00:21:14.974 traffic to just try to pull out that ping packet we're gonna try to send through. Um so just 00:21:14.974,00:21:19.979 basic TCP [indiscernible] with some veracity. And graphing for ICMP. [pause until 21:27] And 00:21:27.586,00:21:31.223 then over on the zen server system we're gonna setup your [indiscernible] to launch the 00:21:31.223,00:21:36.228 double tagging attack. [pause] And selecting the default interface again. [pause] This 00:21:39.765,00:21:44.770 time we're going to select 8 0 2 dot 1 cube protocol. And then we have to do some setup on the 00:21:44.770,00:21:51.710 bottom there. So um down on the bottom you can see there's a tags for in input v lan and the 00:21:51.710,00:21:56.482 the target v lan. There's also uh the source IP address and destination IP address. So we 00:21:56.482,00:22:01.086 wanna go from v lan 1 to v lan 20. And then we wanna go from a source IP address it doesn't 00:22:01.086,00:22:07.660 matter so we spoof the address of 1 90 2 1 68 5 dot 5. Um and then the target system is 1 90 2 00:22:07.660,00:22:12.665 1 68 1 dot 11. So we have to just input that in there for the settings. [pause until 22:19] 00:22:19.405,00:22:25.010 And we're setup there. So we can see up on top we're using warp 1 90 2 1 68 1 dot 11. Um so we see 00:22:25.010,00:22:30.182 the aarp protocol there. It actually went through. We're gonna launch the attack. Sending 00:22:30.182,00:22:35.187 out that ICMP protocol. Or ICMP frame. [pause] And we sometimes you have to keep launching this 00:22:38.657,00:22:42.394 attack a little bit. What I found was when I started doing these this testing nothing 00:22:42.394,00:22:45.631 happened. And I'm like okay it's not going to work. Not going to work. Then I sat there for a 00:22:45.631,00:22:48.934 little while longer and just kept pressing the attack button pressing the attack button and 00:22:48.934,00:22:55.808 then finally what happened is we'll see here in a second. [pause] Took a little bit of 00:22:55.808,00:22:59.979 time. So if you're doing this at home boom we see all the traffic. So there was a little 00:22:59.979,00:23:03.515 bit of latency going from one virtual environment through the physical switch into another 00:23:03.515,00:23:07.353 environment. But we can see that the frame was actually successfully forwarded from v 00:23:07.353,00:23:12.157 lan 1 through a virtual switch through a physical switch through another virtual switch 00:23:12.157,00:23:18.097 to v lan 20 on the other side. So you can see that the uh spoofed address of 1 90 2 1 68 5 00:23:18.097,00:23:22.334 dot 5 going to the cali 1 host name which obviously E T T host file on that cali system 00:23:22.334,00:23:27.306 resolving the local IP address the host name um on that system. So this was successful. We were 00:23:27.306,00:23:30.809 actually able to do the double tagging of tags through a virtual switch through a 00:23:30.809,00:23:35.814 physical switch into another virtual switch. [pause] [applause] Thank You. [applause] 00:23:43.722,00:23:48.227 Okay so we have one more here. Um in this case we just decided hey if we can do it with 2 00:23:48.227,00:23:52.564 virtual switches let's take it out. Let's go through one physical switch into another 00:23:52.564,00:23:56.335 virtual switch and just see if that second virtual switch will will successfully do that double 00:23:56.335,00:23:59.738 tag. So we don't have 3 switches. Now we're down to 2. One's a physical one's a 00:23:59.738,00:24:04.476 virtual. Um in this case the physical cali system is the attacker. And we're going into a 00:24:04.476,00:24:09.381 Microsoft hyper v guest on a Cisco Nexus 1000 V. Has anybody ever tried to setup a Cisco 00:24:09.381,00:24:14.386 Nexus 1000 V? How fun is it? Right? Oh yeah. Okay. So let's take a look at this one. [pause] 00:24:21.860,00:24:26.865 Same pretty much setup. We have um on the left hand side the the physical 29 50. And there's a 00:24:28.934,00:24:33.572 port 1 which is connected to that physical uh cali system which is in the middle. And we 00:24:33.572,00:24:38.577 see it's just on v lan 1. Uh no trunking going on there. So this has a trunk connection between 00:24:42.915,00:24:48.687 the physical switch to the Cisco uh Nexus 1000 V the hyper V environment. Um so we'll check 00:24:48.687,00:24:53.692 out that interface as well. [pause until 25:00] And we can see it's set for trunk as for 00:25:05.137,00:25:10.142 best practices. [pause until 25:11] And 8 0 2 dot 1 cube encapsulation again. [pause 00:25:20.185,00:25:25.057 until 25:20] Uh we had v lans 1 2 and 10 supported on here. And 20. Um now we'll go over and 00:25:25.057,00:25:29.461 check the configuration for the Nexus 1000 V which supports the same Cisco commands pretty much 00:25:29.461,00:25:35.567 for identify trunk connections. Um so here you can see we have ethernet 3 1 is the trunk 00:25:35.567,00:25:40.873 connection connected to the Cisco 29 50. And we have V ethe 1 which is that cali system 00:25:40.873,00:25:45.878 connected on v lan 10. In this case. And that's connected to that Cisco Nexus 1000 V. [pause] 00:25:49.515,00:25:53.619 So in the middle here we have our um attacking system which is a physical system. And then on 00:25:53.619,00:25:58.624 the far right we have our um hyper V VM. And again we're just gonna run TCP dump. A little bit 00:26:00.626,00:26:02.628 of [indiscernible] traffic. [pause] Forward this a little bit so we can speed it up. 00:26:02.628,00:26:04.630 [pause] And then on the attacking system we're gonna run your [indiscernible] again. 00:26:04.630,00:26:06.632 We're pretty systematic in our approach these this testing to make sure we did it across the 00:26:06.632,00:26:10.402 board and it was the same everywhere we did it. Um now our physical cali system is multi 00:26:10.402,00:26:15.407 home so we have 4 network interfaces on this so we can do different testing on different 00:26:19.945,00:26:24.950 networks. So in this case we choose ethernet 2 because that's the one that's actually on that 00:26:32.257,00:26:36.161 v lan test network. Um so we just went in there and changed the the configuration the 00:26:36.161,00:26:41.166 interface. [pause] And then we're gonna go up and choose the 8 0 2 dot 1 cube protocol again. 00:26:45.804,00:26:50.809 [pause] Setup that attack down in the bottom. Already seen that so I'll fast forward through 00:26:53.512,00:26:58.517 that a bit here. [pause] okay [pause] Just verifying the IP so where we're going to that uh IP 00:27:03.689,00:27:09.161 of 1 92 1 68 1 dot 1 99 in this case is the virtual machine the hyper V. Um and we're gonna 00:27:09.161,00:27:14.800 start launching this attack now. So here same thing. Just launching it. We can see the uh 00:27:14.800,00:27:19.805 spoofed IP address going to the uh the destination IP of 100 wasn't 199 was 100 8 0 2 dot 1 00:27:22.107,00:27:28.647 cube and the protocol is ICMP. And if we just start launching that thing again took a little 00:27:28.647,00:27:35.020 bit of time to get this thing to successfully go. [pause] But in the end we see that it actually 00:27:35.020,00:27:39.458 works. So it went through the physical switch from a from a physical attacker through that 00:27:39.458,00:27:45.964 physical 29 50 into the Cisco Nexus 1000 V on to V lan 10 um on the the target virtual 00:27:45.964,00:27:49.668 machine. So it actually worked. So we're seeing that we can actually double tagging switch 00:27:49.668,00:27:54.673 spoofing attacks in virtualized environments which is pretty scary. [pause until 28:00) okay 00:28:01.680,00:28:07.386 So overall results of this um the first set of charts there on the left hand side show the uh 00:28:07.386,00:28:12.524 open V switch Linux bridging. Pretty much everything worked um for going in. So this is this 00:28:12.524,00:28:17.062 isn't coming from this is trying to get into um attack a virtual machine on another v lan inside 00:28:17.062,00:28:21.967 of of virtualized network. So it worked on every single environment except for the 00:28:21.967,00:28:26.071 standard V switch for hyper V. Uh we're finding this is because when you're launching your 00:28:26.071,00:28:30.342 [indiscernible] you're actually spoofing Mac addresses on the attacker side. And what we saw 00:28:30.342,00:28:34.079 last year when were were discussing our Mac flooding attacks those also didn't work 00:28:34.079,00:28:39.184 in the standard V switch for hyper V because the standard V switch has some protection from 00:28:39.184,00:28:44.222 Mac spoofing against it and and the server 2008 2010 environments. Um so that's the 00:28:44.222,00:28:49.328 only thing that protected it against this attack was that Mac spoofing um safety mechanisms. 00:28:49.328,00:28:52.431 So if that wasn't there or if you could do this attack without spoofing Mac addresses you could 00:28:52.431,00:28:56.501 probably get away with it in that environment. Over on the other chart we see um what was 00:28:56.501,00:29:01.640 successful what could we launch the from? Uh which environments could we could we actually send 00:29:01.640,00:29:05.777 something out from it. In this case um we saw it worked in the bridging environments. So 00:29:05.777,00:29:09.548 anything that was a bridge except for uh E S X i in this case worked. We also saw it 00:29:09.548,00:29:13.785 worked in in uh zen server with open V switch and just a a standard zen environment with 00:29:13.785,00:29:17.322 open V switch. So pretty much all the open source software was affected by this attack. 00:29:19.625,00:29:24.029 Mitigation techniques standard ones for physical devices is limit your access to V lan 1 or 00:29:24.029,00:29:28.734 just eliminate V lan 1 all all together. Um maybe label it something different so it's not 00:29:28.734,00:29:33.872 easily identified. Um restrict access to switches by Mac address which can be spoofed. So 00:29:33.872,00:29:37.342 that's a little bit difficult. It's kind of a double edged sword there. Um but the heart of 00:29:37.342,00:29:41.213 this attack is having access to that native V lan as we saw it. And even the native V lans 00:29:41.213,00:29:45.283 within the virtual switches themselves which may not be identified. Um so you really 00:29:45.283,00:29:48.553 need to look at hardening these these uh these switches. [indiscernible] production 00:29:48.553,00:29:53.558 environments. Okay so we're pass on to Kaitlyn now for our ARP ARP spoofing. [applause] >> Okay 00:29:57.996,00:30:03.001 our spoofing. It is ARP is an address resolution protocol. It's a layer 2 network protocol 00:30:08.774,00:30:13.912 that maps the physical Mac address to the logical IP address. Each system on a 00:30:13.912,00:30:18.817 network has an ARP cache and it stores the information from discover nodes on the network 00:30:18.817,00:30:25.223 into the ARP cache. Uh the common entries in our cache is the debug gateway and the local 00:30:25.223,00:30:30.228 DNS. [pause] The process for ARP is an initiating system sends out a broadcast broadcast 00:30:35.133,00:30:41.640 request to the entire network. Is asks who has a certain IP address? A node on that network 00:30:41.640,00:30:46.611 who has that IP address responds with its Mac address and the information is stored into the 00:30:46.611,00:30:51.616 ARP cache. [pause] I don't know [laugh] [pause until 31:06) Okay so this is just a demo of what 00:31:08.200,00:31:15.040 we did. Um the left hand screen is the router or the default gateway. In the center is the 00:31:15.040,00:31:20.812 target machine. And on the right is the attacking machine. On each of them we had ARP cache 00:31:20.812,00:31:27.619 displayed and you can see the IP address of the attacker and the target over on the default 00:31:27.619,00:31:33.925 gateway. And you can see the Mac address displayed with it showing each separate entry. And 00:31:33.925,00:31:39.598 then over on the attacking machine we're going to run our ARP poison script which is just 00:31:39.598,00:31:44.603 a basic python script. And that's going to run [pause] It's going to go through and poison 00:31:46.805,00:31:53.278 the ARPs. So we're gonna go over onto the target and the default gateway and check our ARP cache. 00:31:53.278,00:31:59.017 And you can see as it comes up and displays with the 2 IP addresses that the Mac addresses 00:31:59.017,00:32:05.257 are now the same. So on the default gateway both the target IP address are showing that the 00:32:05.257,00:32:10.262 Mac address is now the same. So that was spoofed into tricking the Mac address. [pause] Um then 00:32:18.370,00:32:23.608 we're going to go and we're gonna run a ping. Just [indiscernible] just to get some 00:32:23.608,00:32:30.115 packets flowing. Get some traffic. And that's gonna finish up the script and run through 00:32:30.115,00:32:35.120 there. [pause] Once the script is finished it's gonna restore the target machine. So we're 00:32:38.323,00:32:43.328 gonna go over and stop the ping. And restore the ARP cache and check that over. [pause until 00:32:51.703,00:32:56.408 32:51] And then you can see on the target and on the default gateway that the ARP cache is 00:32:56.408,00:33:01.346 now displaying as it was in the beginning. So the target was restored. And then we also ran a 00:33:04.583,00:33:11.323 TCP dump script. So that's gonna show the traffic that came through. And then you're going 00:33:11.323,00:33:16.328 to be able to see Google's IP address showing traffic to the IP address showing that we 00:33:19.030,00:33:24.035 captured packets from Google. [pause until 33:38] Okay. And then for the results you can see 00:33:40.852,00:33:45.857 it worked on all the platforms. [pause] Um ARP spoofing mitigation. On Cicsco switches 00:33:51.296,00:33:57.002 you can make use of the [indiscernible] dynamic ARP inspection validation. Um it's 00:33:57.002,00:34:03.375 not supported on the Cisco Nexus 1000 V. And then there's also ARP watch which runs as a 00:34:03.375,00:34:10.248 service on a Linux system and monitors the network in changes for ARP [indiscernible]. [pause] 00:34:10.248,00:34:15.253 [applause] >>Okay so to kind of wrap this up and to put it into some context. Um we can see that 00:34:28.033,00:34:33.171 they're the results do show that virtual networking devices can pose the same sometimes even 00:34:33.171,00:34:38.209 greater risks than their physical counterparts. So that's an important thing for every um 00:34:38.209,00:34:42.914 although a users of a multi tenant environments to understand. Which systems were 00:34:42.914,00:34:48.119 vulnerable varied pretty widely across the tests. There was no one best system. And there were 00:34:48.119,00:34:53.124 some uh tests like the ARP one that everything was vulnerable to. That lack of sophisticated 00:34:55.327,00:35:00.231 security controls similar to what is available on enterprise grade physical switches greatly 00:35:00.231,00:35:06.771 increases the difficulty in securing virtual switches against attacks like these. So 00:35:06.771,00:35:11.409 what is the bottom line impact then to somebody running in a multi tenant environment? A 00:35:11.409,00:35:16.014 single malicious virtual machine has a potential to sniff all the traffic passing over a virtual 00:35:16.014,00:35:21.586 switch. Can pass through the virtual switch and potentially affect the physically connected 00:35:21.586,00:35:26.391 devices as well um traffic from other parts of the network to be sniffed as well. So it's a very 00:35:26.391,00:35:30.261 significant threat to confidentiality integrity and availability of data that's 00:35:30.261,00:35:34.966 passing over a network in a virtualized multi tenant environment. And there's a lot 00:35:34.966,00:35:38.970 of very mission critical stuff being housed in virtualized multi tenant environments these 00:35:38.970,00:35:43.975 days as we know. So what can users of these systems do? Well educated users can question 00:35:46.611,00:35:50.348 their hosting providers. Which virtual switching implementations are being used? 00:35:50.348,00:35:55.120 Which attacks are they vulnerable to? Are they doing additional mitigation strategies 00:35:55.120,00:36:00.358 against attacks like these? They can also audit the risk of the workloads they're choosing to 00:36:00.358,00:36:05.030 run in the cloud with this enviro with these you know with this understanding in mind. They 00:36:05.030,00:36:10.235 could consider extra security measures on their own or request additional security measures 00:36:10.235,00:36:14.072 from their hosting provider. Things like increased use of encryption. Additional service 00:36:14.072,00:36:19.778 monitoring. More detection of traffic. Attack traffic that could be present. Alerting. 00:36:19.778,00:36:25.116 There's a lot of things that could be done. What are some of the next steps for us? So this 00:36:25.116,00:36:29.120 is uh last year was our first time at DefCon. So this is our second time. Um we are a pretty 00:36:29.120,00:36:33.458 small team. Uh we made uh improvements this year. We did more attacks then we did before 00:36:33.458,00:36:37.896 on more regular hardware and we're we're really happy with that. But there's a lot more 00:36:37.896,00:36:43.201 we'd like to do. We'd really love to setup kind of an institute for apples to apples 00:36:43.201,00:36:46.938 testing of virtualized environments. We have kind of a long history of this. Um we've 00:36:46.938,00:36:51.576 also done what kind of malicious things can a virtual machine that's co-located do with it 00:36:51.576,00:36:57.916 hammers the disk. Or it you know hammers the CPU or it allocates a ton of memory or all sorts of 00:36:57.916,00:37:02.620 malicious things. We've done apples to apples testing of different kinds of um virtual 00:37:02.620,00:37:06.925 machine migration technologies. And you know what's going in the clear? And what's not? And how 00:37:06.925,00:37:12.163 long it takes? And when can get it it can it gets stuck? And we think there's a real place for 00:37:12.163,00:37:17.869 that kind of apples to apples independent testing of those things. If you're in a situation 00:37:17.869,00:37:21.506 of wanting to deploy some of these technologies both proprietary and open source it's 00:37:21.506,00:37:26.377 really hard for you to bring all of them in house and run through all of these kind of tests. So 00:37:26.377,00:37:31.282 we've seen a lot of great uh feedback about oh thank you for doing all of these tests and you 00:37:31.282,00:37:35.920 know pointing out where the problems are. We've also seen vendors fixing things. We'd love 00:37:35.920,00:37:40.191 to see you know industrial partners to participate in an institute of this kind. Whether 00:37:40.191,00:37:44.496 it's places that have technologies they'd like tested and would like to see our 00:37:44.496,00:37:48.466 results and have a chance to say okay yes this is the results in the default environment but if 00:37:48.466,00:37:52.103 you turned on this thing [laugh] then the results will be completely different. We would 00:37:52.103,00:37:55.907 love to partner with people like that. We'd also love to partner with people who are using these 00:37:55.907,00:38:01.579 technologies in production environments. And would be interested in these results. Um 00:38:01.579,00:38:05.049 we have leads. We would really just like to do more testing in a production environments 00:38:05.049,00:38:10.288 because we're basically using best practices out of the box proprietary and open source 00:38:10.288,00:38:15.560 technologies. Over time we've seen some of the work that we've done result in vulnerabilities 00:38:15.560,00:38:21.099 that are then patched. So we're really proud of that. Um but we are we understand that in 00:38:21.099,00:38:25.570 production environments many times there's additional secret sauce or additional monitoring 00:38:25.570,00:38:29.641 or additional conditions that are put in place. We actually got some great leads last year 00:38:29.641,00:38:35.113 after our talk. That we still need to follow-up on but this is us. [laugh] pretty small team. 00:38:35.113,00:38:39.484 And the the uh you know the biggest bottleneck we have is the need for more great students 00:38:39.484,00:38:44.956 like Kaitlyn. Um to be funded to do this kind of testing. Um so you know if anyone would like to 00:38:44.956,00:38:49.961 help us sponsor a few students uh we promise it's good educational value. Um so this is 00:38:52.797,00:38:57.802 emails uh for Ronnie and I. Also um all the details if if some of the typing and the demos you 00:39:00.471,00:39:04.442 didn't quite get you can slow them down. Zoom in on the videos. They're all there. All 00:39:04.442,00:39:09.848 the gorey details. And um uh all of the publications presentations demos are all at 00:39:09.848,00:39:16.154 Ronnie's website. Um also liked to give a shout out to Nick Moranti who uh helped us I get 00:39:16.154,00:39:21.159 our AV going uh after some last minute uh troubles. And I guess now we'll take some questions. 00:39:34.572,00:39:39.577 [pause] [applause] [pause] >> Hi so I see you did a bunch of testing on the hyper visors and 00:39:41.679,00:39:46.184 the virtual switches that come with the hyper visors. Did you try this with anything like open 00:39:46.184,00:39:51.522 daylight or VM ware NX6 [indiscernible] solutions that sit on top of they hyper visors? 00:39:51.522,00:39:57.028 >> Not yet but we'd love to right and >> Um I did start looking at some SDN stuff but 00:39:57.028,00:40:00.531 we're seeing that there's a lot of there's a lot of vulnerabilities in the 00:40:00.531,00:40:03.901 controllers out there. Like where you can actually take control of the controllers. So 00:40:03.901,00:40:07.538 by implementing SDN we're looking at implementing more security flaws into these 00:40:07.538,00:40:12.610 environments when we're trying to reduce that footprint. Um I looked at with the DHTP stuff we 00:40:12.610,00:40:16.447 did last last year we presented on that. I've found a python script just by using simple 00:40:16.447,00:40:20.418 python in [indiscernible]. Um you can write a simple script because demonized that will find 00:40:20.418,00:40:26.224 you know real DHCP servers on the network and fix that. So I think by using smaller scripted 00:40:26.224,00:40:29.627 demons that we can actually deploy without putting more complexity into these networks 00:40:29.627,00:40:34.432 would be a better solution than trying to throw the monster of SDN on top. But I mean it is a 00:40:34.432,00:40:38.770 many people are using SDN so it is a good place for us to start exploring. But that's a whole 00:40:38.770,00:40:43.775 nother research track. >> Right on thank you. [pause] >> Great talk. Um I can see you 00:40:46.244,00:40:52.517 [indiscernible] have you tried against Amazon inside the Amazon environment? >> Uh no we haven't 00:40:52.517,00:40:58.556 tested within Amazon. I think that is not in there uh uh terms of service. But uh we did have 00:40:58.556,00:41:04.595 some leads within Amazon uh which we'd love to follow-up on. I'm not sure uh yeah we haven't 00:41:04.595,00:41:10.401 done that yet. We'd love to. >> So have you done any uh cam table overflow attacks against 00:41:10.401,00:41:14.772 the virtual switches as well? >> Uh that was last year's talk. So if you look on >> Oh okay >> 00:41:14.772,00:41:19.811 DefCon server it's all there demos and everything. >> Perfect. Thank you very much. >> 00:41:19.811,00:41:24.916 Uh so along with uh you were talking about the 29 50 uh physical switch uh so that's 00:41:24.916,00:41:30.521 running I'm assuming like uh IOS 12 point 0 4 or something like that. Were you able to sort of 00:41:30.521,00:41:35.526 do that attack on like uh a newer version of the IOS like uh like a 15 dot something? >> Yeah 00:41:37.862,00:41:42.467 um let's go back up to the [pause] [off mic comment] >> Thank you >> No I don't wanna go 00:41:42.467,00:41:47.138 to the details here I wanna go to picture. I'm not exactly sure what model that switch is. But 00:41:47.138,00:41:52.143 if you look up here on the top of the rack [pause] I can get there again on the back there's 00:41:54.445,00:41:59.450 2 switches up top there. Those are brand new Ciscos. I accidentally tested it against 00:41:59.450,00:42:03.688 those and it did work. So I I don't recall which environments but yeah it did I was able to 00:42:03.688,00:42:07.492 switch proof out of those as well. Um the double tagging didn't work because it's not 00:42:07.492,00:42:11.329 using the 8 0 2 dot 1 cube encapsulation. >> Okay >> So if a switch doesn't support that 00:42:11.329,00:42:14.365 your double tagging's not going to work. There's other encapsulation pipes for trunking 00:42:14.365,00:42:20.004 that that are supported now on the newer devices. >> Awesome. Thank you. >> Yep >> Hello I'm 00:42:20.004,00:42:26.744 from Brazil first of all. Thanks thanks for the great presentation. Was wonderful. Uh 00:42:26.744,00:42:32.483 [off mic comments] Have you got the first part? Uh I'm from Brazil and first of all thanks 00:42:32.483,00:42:39.023 for the presentation to us. Great. Uh I I have worked with a lot of companies in Brazil. A 00:42:39.023,00:42:45.430 lot of them are environments from small to large. And I have never seen any any one of them 00:42:45.430,00:42:51.836 with ARP spoofing preventions implemented. None of them. And sometimes large environments 00:42:51.836,00:42:56.841 with flow balancers and to flow applications uh they are encrypted from the the client in 00:42:59.043,00:43:03.948 to the load balancer and from the load balancer to the backend. They they usually take 00:43:03.948,00:43:08.953 off the the encryption and create confirmation going on. And based on our experience uh 00:43:12.223,00:43:17.228 have you seen uh in the [indiscernible] you talked about this. Have you seen what kind of 00:43:19.864,00:43:26.037 solution have you seen if you have implementing the the the environments that you have 00:43:26.037,00:43:31.042 tested or worked with? >> Uh let's go to the ARP slide here quick. See if you if you guys 00:43:35.279,00:43:39.317 look at the slides online. We actually have um images that didn't show up here because of 00:43:39.317,00:43:43.554 this. But there's some good stuff in there for the ARP. Um so basically you're looking at 00:43:43.554,00:43:48.960 ARP watch. Cause the Ciscos have this uh this dynamic ARP inspection in DHCP snooping that 00:43:48.960,00:43:52.663 will allow you to to look for rogue [indiscernible] you know different ARP problems going on 00:43:52.663,00:43:56.400 there. But ARP watch is a limited utility you can demonize run the server on your network 00:43:56.400,00:44:00.271 and can actually look for you know malicious ARP activity on any network. It's about the only 00:44:00.271,00:44:03.641 thing we've really looked at and experimented with right now but it does work pretty well. >> 00:44:03.641,00:44:08.646 Okay but uh uh I'm I'm not pretty sure if I was clear. What of these kind of ins uh of 00:44:10.815,00:44:15.119 mitigations have you seen implemented in actually implemented in the companies. >> 00:44:15.119,00:44:19.190 Oh in the real world. >> And that's where we'd really like to do a lot more. We really haven't 00:44:19.190,00:44:22.994 done much work in production environments. We're we are >> Okay >> just looking at vanilla 00:44:22.994,00:44:26.531 out of the box stuff according to best practices. And then looking at what the mitigations 00:44:26.531,00:44:31.302 could be. We don't have a lot of visibility into how often people are doing these mitigations in 00:44:31.302,00:44:34.872 production environments. For us it's interesting to hear you say that you've seen lots of 00:44:34.872,00:44:38.943 environments and nobody's doing that. So >> No >> We'd love we'd love your email and we'd love to 00:44:38.943,00:44:43.214 follow up that would be really helpful for us. Uh >> Okay >> I see that we have a line of more 00:44:43.214,00:44:49.320 questions and we've got the kind of yank sign from the goons. Uh we'd be very happy to talk more 00:44:49.320,00:44:53.991 with folks. Um I'm not sure where the right place to go and meet people is. Uh I guess if 00:44:53.991,00:44:58.930 you come up here you can go where we go and we can follow up. Yes? >> Okay. Thank you so 00:44:58.930,00:45:01.132 much. >> Okay. Thanks everybody.