Uh, so, uh, we're gonna get this next talk started. Uh, let's get a big round of applause
for them. We've got at least one new speaker. How, is anybody, two? Yeah? Alright.
Hey everybody, three for the price of one. My name is Gina Matthews and I'm a computer
science professor at Clarkson University in way, way upstate New York. Nosebleed New York,
as we like to affectionately call it, right across from, uh, Canada. Uh, Ronnie Bull is
both a PhD student with me at Clarkson and also a faculty member at Utica College. And
Caitlin Trumbull, uh, is a senior undergraduate who's been helping us run all these attacks
that we're gonna tell you about. In particular, we're gonna talk about VLAN hopping and ARP
poisoning and man in the middle attacks in virtualized environments. And this is a bit of a
follow on from the presentation we did last year at DEF CON. So here's a little road map of
what we're gonna do. We're gonna talk about the context of the problem for layer two network
security in virtualized environments. And then we're gonna talk about the context of the
problem in multi-tenant environments, especially in cloud services. We're gonna talk about the
test platforms that we used and all of the different virtual, uh, the hypervisors and the
virtual network, uh, devices that we tested. And we're gonna talk about the specifics of the
attacks. We've got some, uh, great videos and demos to show you. Specifically of the, uh, VLAN
hopping and ARP poisoning this time. We also have videos and many, many details of the map
flooding and DHCP attacks that we talked about last year. If you wanna look those up. And
finally some conclusions. So, um, the key question is when you have virtual machines that are
hosted in a multi-tenant environment, how safe are they? What can they do to each other over the
network? Um, they're essentially connected to a virtual version of a physical networking
device. And the question we're trying to answer is the layer two network attacks that
typically work on physical devices. Do those work on their virtual counterparts as well? And if
so, what are the implications, especially in a multi-tenant environment? Cloud services that
rely on these virtualized environments could be vulnerable. And that can include data centers
hosting mission critical or sensitive data. This is not the only class of attacks from
co-located virtual machines. It's kind of an old story. You're vulnerable to those that are
close to you. But in a public kind of multi-tenant environment, you're not quite sure who you land next to. So, uh, we're gonna talk a little bit more about, uh, the
things that virtual machines can do. And, um, we've done some other work in the past of the, you
know, the types of things that virtual machines can do to each other as have many other people. But in
particular, today, we're talking about the attacks over the network. So, here's a pretty simple
diagram of the setup we're worried about. We have a physical host here and on it we are hosting a
bunch of virtual machines. One of them that's a nasty malicious one that's gonna do things to the other
uh, victims. And they're, you can see they're sharing a virtual switch or bridge. And that's
passed through to a physical network interface. And to kind of jump ahead, the bottom line is that we do see in our
research that virtualized network devices have the same ability to be exploited as physical
devices. Sometimes we see more vulnerabilities in physical devices. Sometimes, in certain cases, we
see less than a physical device and we're gonna show you that, uh, for the attacks we're talking
about today. Another interesting thing we've seen is that, um, some of these attacks allow, have the
ability to leave the virtualized environment and affect the physical networks that they're connected
to as well. So that's another interesting bonus for you. What are some of the consequences if a
malicious VM successfully launches a layer 2 network attack in a multi-tenant environment like
this? What are some of the things that they could do? Well, they could capture all the network
traffic coming from the victim's network. They could capture all the network traffic coming from the
user's network. They could create connections that are co-located with them. They could redirect
traffic. They could perform man-in-the-middle attacks, denial of service attacks. They could gain
unauthorized access to restricted subnets and they could affect performance which is, I guess, a
uh, a sub-example of denial of service. So here are some of the tests scenarios and results that
we're gonna be talking about. We're gonna give you a little update on Mac-flooding, uh,
especially because we've changed over our testing platform completely since we spoke here last year. So, uh, the most important thing, is that you won't have any
we're gonna talk about VLAN hopping in quite a bit of detail, also ARC poisoning and some
man-in-the-middle attacks. Here is a lovely picture of the test environment that we were
using last year. It was pretty much built from pieces and parts that we could salvage. Uh,
rest in peace, you've served us well, thank you very much. Um, here are, for, uh, complete
details, uh, some of the hardware specs and the hypervisors and the virtual, uh, networking
devices that we were testing when we spoke to you last year. All of those details are on the
DEF CON 23 CD. Um, we have nice new shiny hardware, uh, thank you Utah Co-College for that.
Um, and it allowed us to basically repeat all of the experiments we did before and just, you
know, a much more, um, controlled environment than our pieces and parts. And also extend the
work. So here is some of the gory details of, uh, the, uh, the, uh, the, uh, the, uh,
the test environment that we used this year. You can see a lot of different hypervisors, a lot of
different virtual networking devices, and you can see some details here of the hardware specs.
Again, all of the gory details are available for your reading enjoyment, um, in the DEF CON docs.
Okay, so to kick things off and also just to provide a good, um, transition between what we did
last year and this year, I'm gonna give you a little, uh, update on the MAC flooding stuff. First, you
know, here's a simple diagram of the scenario in which we're doing MAC flooding attacks. Again, we
have two VMs attached to a virtual switch. We have a victim, the target VM, and we have an attacker
that's patched through to a physical interface that connects to a physical switch and the
internet beyond. And the target VM is just doing a set of regular pings and we're gonna see what
happens to the performance of those pings when the attacker launches a MAC flooding attack.
And here in this, uh, graph, which is something we showed you last year, you can see, um, from,
like, 0 to 65 or 70 or so, what was happening when it wasn't under attack. So, low latency pings, no
trouble. You can clearly see when the attack gets launched and you can see just after 2.40 when
the attack stops. You can see clear impact on the network performance for the, the victim VM. And
this was done on Gen 2 in a Zen build. So, um, you can see, um, you can see, um, you can see, um, the
bridge interface. And this gives you just a sense of, you know, we're just doing a lot more
systems and platforms this year. So, this is the same test performed across not just, um, that
Gen 2, uh, Zen bridge, but also Gen Open, uh, excuse me, Zen Open vSwitch, Citrix Zen Server,
Hyper-V standard switch, Hyper-V Nexus 1000V, the Proxmox bridge, VMware ESXi, and a control
physical switch, a Cisco 2950. And, um, so, um, the, the, the, the, the, the, the, the, the, the, the,
so, some of those when the attack starts don't, aren't affected. It's a little difficult to read
from this, this graph. Maybe it's a little easier to read here. Still not incredibly easy. We
did, we, we realized at the last minute there might have been a better way to do this one. But,
um, from this, uh, if you look clear, carefully, you can see that the Hyper-V standard, the
Hyper-V Nexus, um, VMware ESXi were not vulnerable. Um, and, uh, I'd like to make the point
before we head into some of the rest of the things that all of these Layer 2 vulnerabilities that
we're discussing were targeted toward the virtual networking devices, not so much the
hypervisors themselves. What we're testing is what happens with the virtual networking
devices. So, it's interesting, you see a mix, right? Not everything's vulnerable, not
everything is safe, and you're gonna see that trend continue across all of our tests. It
wasn't uniform, which things were vulnerable and which things weren't. At this point, I'm
gonna pass it over to Ronnie, who's gonna tell you about the VLAN hopping.
Okay, so we performed a few VLAN hopping experiments on the environments as well to see what
we could do, um, about getting frames onto unauthorized networks. Uh, we found some
interesting results. So, some basics about VLAN hopping attacks, um, just allowing an, uh,
unauthorized, unauthorized access to another virtual LAN on a packet switch network. Um, two
methods of doing this currently, uh, switch spoofing and double tagging, where switch
spoofing is Cisco proprietary, um, using Cisco protocols, uh, and double tagging is an
exploitation of the 802.1Q protocol.
Uh, so basic virtual LAN information, we have a basic frame, uh, Ethernet frame here, and what we
do to get VLAN going is, uh, we stick a 4 byte, um, tag in there, or VLAN header with 32 bits of
extra information, um, to support the VLAN ID to show where that frame is supposed to go on the
correct networks. So switch spoofing, here's a CVE listed, um, about Cisco switches, that if
they support 802.1X security, allow attackers to bypass port security and gain access to the
VLAN, uh, via spoofed Cisco discovery protocol messages. So, uh, this is, uh, this is, uh, a
little bit about the Cisco discovery protocol. Basically, it's a proprietary protocol that Cisco
has on their switches that allows two or more switches to identify each other's operating
systems, automatically negotiate connections, things like that. So if we can have a
vulnerability in that protocol and actually exploit it, we can get on a different VLAN.
Couple more CVEs and vulnerabilities for switch spoofing, uh, Cisco Catalyst 2900 virtual LAN
switches allow remote attackers to inject 802.1Q frames into another VLAN just by forging the ID
tag, um, and, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh,
uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh, uh,
uh, uh. Quote directly from Cisco from one of their white papers about dynamic trunking
protocol. Basically, if it's enabled, um, you can send a fake DTP packet out and switch that
port from auto into a trunking mode, and we're gonna see that is pretty effective across some of
the environments. Um, and it's important to note that on most Cisco switches out there, at
least some of the earlier ones that are still production today, um, DTP auto is the default
settings so you want to make sure you uh harden that. Okay so again this is just another Cisco
proprietary layer two protocol um allowing you to do that automatic uh configuration of
trunking connections between two switches. Um pair this with CDP and you pretty much have a
automatically configured network. So some of the consequences of this um basically the attacker
can have a full trunk connection to the switch. They can make themselves into a trunk uh spoof
themselves as a switch uh which allows them to have two way communication on that VLAN. They
place themselves directly on that VLAN um allowing them to eavesdrop on the traffic, send
messages to and from systems on that on that VLAN. Okay so here is um switch spoofing demo.
Uh basically we did this in the ESXi environment. Um the set up is we have a virtual machine um
within the ESXi environment and it's going out through the virtual switch or bridge. ESXi is
more like a bridge than a virtual switch uh the standard switch on it.
We have that connected to a physical Cisco 2950 switch um and the port is set to DTP auto. Just
default configuration. And then you see um there's another system on the other end that's
connected uh via trunk link with a VLAN 20 and one support over that trunk link going into
another hypervisor environment there. So let's take a look at this uh this demo here. All these
videos are on YouTube. The links are on the white paper on the DEF CON CD so if you want to
watch them later. Okay so on the left hand side we have the uh the uh the uh the uh the uh the
uh essentially there's a lot of things in here. So in the left hand side we can see the
configuration for the Cisco 2950 switch. Um on the right hand side we see
the ESXi uh network settings for the Kali virtual machine that's on that system. We're going to
first take a look at the uh interface settings for the um ESXi server, in this
case it's called Luminara. Um and you can see it's connected and it's just set to VLAN 1 so it's
just on the default VLAN of that switch. Nothing was changed besides the defaults. Um it's not set
to a trunk port or anything. Um we're just going to verify the trunk status on that. O'Hare
and we can see that it's using 802.1Q encapsulation, it's not trunking and the mode is
set to auto, so it's just default configuration, nothing was changed. If we go over onto the
virtual machine, uh, we're gonna take a look at the network interface here, we can see there's
just the basic network adapter, it's on an isolated VLAN test environment we created, it's a
separate network for that, um, which is over there on the switch on that port. So if we dig a
little deeper into these details, um, we can take a look at that network and we can see down
there on the bottom we set up that VLAN test network and the Kali virtual machine is the only
system on that network, um, and if we look at the properties of that, we can see that this,
uh, this connection has no VLAN ID set to it, so it's just on the default VLAN for the ESXi
virtual switch. Um, now we're gonna go over to the virtual machine itself, grab console
access, and we're gonna start the attack. In this case we're gonna just use the Yersinia
program, so we're just gonna type Yersinia minus I to get into interactive mode,
gotta bear with me a little bit. Okay, and then we're gonna select the default network
interface on this, so just hit enter, and then we'll have to launch the attack. So we're just
gonna go up and select the, uh, DTP, or Dynamic Trunking Protocol, and then launch the attack,
press 1, press 2, press 3, press 4, press 5, press 6, press 7, press 8, press 9, press 10,
and we can see it successfully went into trunk mode. So the, uh, dynamic desirable effect was
affected over there on the switch side, we can see the line protocol went down, it came back up
again, and if we check the status on that trunk port, we'll see it indeed went into trunking mode.
You can see it's now, instead of VLAN 1, it shows the trunk. If we take a look at the full status,
we can see it's set for trunking and auto in 802.1Q encapsulation. So the system was effectively put on as a trunk,
now we can access any VLAN we want that's associated with that trunk, so in this case any system that was on VLAN 20 or whatever.
So that was the switch spoofing attack, you can see it was affected, er, effective in the ESXI environment.
And let's see. Okay, so results across the board here, we can see that, uh, the physical Kali 2.0 control system, we set up
physical switch we tested it with a control to make sure it worked first that way before we
proceeded into the virtualized environments. Um you can see it pretty much worked under
every system that that used bridge uh networking instead of a virtual switching networking.
So if you're using an environment that has a bridged network uh virtual switch um this is
gonna work. It worked in the uh open source zen with Linux bridging, it worked in the VMware
ESXi environment as you saw and it also worked in Proxmox which also supports bridging as
well. So any environment that used the virtual switch it it did not work because basically
those that DTP packet when it was sent through it was blocked uh for getting to the physical
device. Okay so mitigation. Um in the physical world you just disable unused switch ports,
don't give access to people to actually get connected to those ports. Um disable CDP and
DTP uh or use it on an as need basis for those ports that you really need it set up on. Um
restrict the amount of trunk ports, you don't just want to have ports set up to be trunks if
you don't need them. Although mo- all the hyperfabricated ports that you really need to
be able to connect to the virtual switch. So that way you can actually support VLANs into your
physical environment and other virtual environments. Um so really just limit these this access
but in the virtual world it's really hard to do this so if you're using bridged environments
realize that this could be affecting your network. Okay let's talk about double tagging now.
Um here's the CVE for double tagging. The 802.1q VLAN protocol allows remote attackers to
bypass network segmentation and spoof VLAN traffic via message with two 802.1q tags. So we saw
before that we have um a normal, basically this is the uh normal traffic, this is the the scenario
we used here where we have two switches, um a trunk connection between them supporting VLAN 1,
2 and 3. You can see some two machines on one side connected to that first switch, one's on
VLAN 1, one's on VLAN 2, and on that second switch we have VLAN 1, 2 and 3. So the goal is to
get some frames across between the two switches uh from that system on VLAN 1, 2 and 3. Um so
here we can see the normal flow of traffic, we're just using a single tag, um where it goes
through VLAN 1, goes into that first switch, goes across the trunk connection still with that
VLAN 1 tag, and then it goes into that system that's VLAN 1 on the other side. So basic
normal flow of traffic. And we can see that the other VLAN 2 and VLAN 3 machines do not get
that, that frame, it just gets blocked. So here's a comparison of frame types. Um the first one is a
standard Ethernet frame without VLAN support, the second one is that standard VLAN tag with one
uh 802.1Q tag in the middle or header in the middle. And then the third one down in the bottom
is doing uh 802.1Q double tagging so um this is where we're actually doing a frame within the
frame or a VLAN tag within the frame and we're doing multiple ones. So in this case here we can
see that um the effect of double tagging, we have VLAN 1 and VLAN 2 on the one side, that tag has
a 1 and 2 in it so we have two headers in that, in that frame. And then we can see that um the
VLAN 1, it goes into that first switch, it doesn't get passed on to that first system on,
connected to that first switch because we're still reading that VLAN 1 tag, we're not seeing the
VLAN 2 tag. As soon as we go across that trunk connection, that VLAN 1 tag is stripped off,
passing over the, the frame with the, with the tag of 2 and then when it gets to the other side
it goes directly to that system on VLAN 2 and doesn't go to the VLAN 1 or VLAN 3 so. Here we
can see the same thing with 3, just a different example. Okay so the consequences, an attacker
can send packets to any of the VLANs that are connected to the VLAN 1 or VLAN 2. So here we can
see that we've targeted VLAN as long as it's supported over that trunk connection. Um and as
long as they have access to that native VLAN. Um the target on the access is isolated and um
it's in a native VLAN broadcast domain. This is not a good attack for eavesdropping because
basically it's a one way attack so you're sending a frame to another VLAN, you're not getting
anything back to you so. It's good for DOS attacks, it's good for maybe a one way covert
channel to another system on another network um but that's about it. Okay so we have 3
different scenarios we ran with this.
The first one was a control scenario where we had the 2 physical switches which was, this was
known to work in. Um we had a physical attacking system connected on a native VLAN and went
through that first trunk link to the second switch and we were able to access the VM um on the
other side on VLAN 20 for this. Due to sake of time I'm not going to show this video right now,
it does work here. You guys are more than welcome to go click the link um but we have 2 more
videos I'd like to show that are a little more effective than this one. Okay so the second
scenario we did was 2 virtual switches. So we took out that uh second 2950 and then we took out
that virtual switch and instead we're going from one virtualized environment through the
physical switch into another virtualized environment. So in this case we have an attacker VM,
it's on no VLAN um just on that native VLAN on that virtual um environment. Going through that
virtual switch, connected through a trunk link via best practices um on 1 and 20 into the Cisco
2950, through another trunk link connected to another virtualized environment which is going
into a virtual switch there on an access uh VM on VLAN 20. So in this case, in this demo we have
the attacker systems on ZenSource and we're able to access that virtual environment. So in this
case, we have the attacker systems on ZEN server and the target systems on Proxmox. Okay so
again on the left hand side we can see the switch configuration for the 2950 that's in the
middle of these 2 uh 2 systems. Port 31 is uh the Citrix Zen server. Port 29 is the Proxmox
system. Yes I do like Star Wars. Uh the middle system is Zen server and the the far uh right
system is the Kali system on Proxmox. So we're able to access that virtual environment and
so the Proxmox system is our target. The middle system is our Zen server attacker. What we're
gonna do is take a look at those interfaces just to check out the settings. So port 31 is our
attacking system. Set for trunking. Notice the mode is on. I didn't do auto because I actually
specifically set this to trunking mode for best practices. And we can see the 802.1Q encapsulation.
So what we'll do here now is check out the the settings for the the attacking system. And we can
just see I have it just set up on a basic network interface port. There's nothing special about it.
There's no VLAN tags on that system or not. It's just a straight connection into that that server
on that VM. And that switch. The Proxmox system you can see it's tagged for VLAN 20 for that
client. Um on that system. So we're gonna run the attack from the Citrix Zen server VM on Kali
there to the Proxmox system which is also compliant. So we're gonna run the attack from the
Citrix Zen server VM on Kali. Um on the Proxmox system we're just gonna run a simple TCP dump
command and we're gonna grep out for ICMP traffic to try to just pull out that ping packet we're
gonna try to send through. Um so just basic TCP dump with some verbosity and grepping for ICMP.
And then over on the Zen server system we're gonna set up Yersinia to to launch the double tagging
attack.
Alright so we're gonna run the attack from the Citrix Zen server VM on Kali. So we're gonna run
the attack from the Citrix Con rectifier line and and Mental
klik. That's it all we've done. Just selecting that default interface again. This time we're gonna
select 802.1q protocol and then we have to do some setup on the bottom there. So um down on the
bottom you can see there's uh tags for input VLAN and then the target VLAN. There's also uh
the source IP address and destination IP address. So we want to go from VLAN 1 to VLAN 20 and
then we want to go from a source IP address, it doesn't matter so we spoof the address of
192.168.5.5 um and then the target system is 192.168.1.11 so we have to just input that in
there for the settings. And we're set up there so we can see up on top we're using ARP 192.168.1.11
um so we see the ARP protocol there and it actually went through. We're gonna launch the
attack. Sending out that ICMP protocol or ICMP frame. And we sometimes you have to keep
launching this attack a little bit. What I found is when I started doing these, the
testing, nothing happened. And I'm like okay, it's not gonna work, not gonna work. Then I sat
there for a little while longer and just kept pressing the attack button, pressing the attack
button. And then finally what happened is, we'll see it here in a second, took a little bit of
time. So if you're doing this at home, boom. We see all the traffic. So there was a little bit of
latency going from one virtual environment through the physical switch into another environment
but we could see that the frame was actually successfully forwarded from VLAN 1 through a
virtual switch, through a physical switch, through a physical switch, through a virtual switch. So we see the
another virtual switch to VLAN 20 on the other side. So you can see that the uh spoofed address
of 192.168.5.5 going to the Kali 1 hostname which obviously ETC host file on that Kali
system is resolving the local IP address of the hostname um on that system. So this was
successful. We were actually able to do the double tagging attack through a virtual switch,
through a physical switch into another virtual switch. Thank you.
Okay so we have one more here um in this case we decided hey if we could do it with two
virtual switches let's take it out, let's go through one physical switch into another
virtual switch and just see if that second virtual switch will uh successfully do that
double tag. So we only have three switches now we're down to two, one's a physical, one's a
virtual. Um in this case the physical Kali system is the attacker and we're going into a
Microsoft Hyper-V guest on a Cisco Nexus 1000V. Has anybody ever tried to set up a Cisco
Nexus 1000V?
How fun is it? Right? Oh yeah. Okay so let's take a look at this one. Same pretty much
set up we have um on the left hand side the physical 2950 and there's a port one which
is connected to that physical uh Kali system which is in the middle. Like you see it's
just on VLAN one. Uh no trunking going on there. So this is a physical Kali system.
This has a trunk connection between the physical switch to the Cisco uh Nexus 1000V on the
Hyper-V environment. Um so we'll check out that interface as well.
And we can see it's set for a trunk as per best practices.
And 802.1Q encapslation again. Uh we have VLANs 1 2 and 10 supported date here and 20.
Um now we'll go over check the configuration for the Nexus 1000V which
supports the same Cisco commands pretty much for identifying trunk connections.
Um so here you can see we have Ethernet 31 is the trunk connection connected to the Cisco
2950. And we have VEETH 1 which is that Kali system holding the valores that we have in the
connected on VLAN 10 in this case and that's connected to that Cisco Nexus 1000B. So in the
middle here we have our um attacking system which is a physical system and then on the far
right we have our uh hyper VVM and again we're just gonna run TCP dump, a little bit of
verbosity and grep for ICMP traffic. Just forward this a little bit so we can speed it up.
And then on the attacking system we're gonna run Yersinia again. We're pretty systematic in our
approach to these, this testing to make sure we did it across the board and it was the same
every way we did it. Um now our physical Kali system is multi-home so we have four network
interfaces on this so we can do different testing on different networks. So in this case we
choose Ethernet 2 because that's the one that's actually on that VLAN test network. Um so we
just went in there and changed the configuration of that interface. So we're gonna run
Yersinia and then we're gonna go up and choose the 802.1Q protocol again. And set up that
attack down on the bottom. I've already seen that so I'll forward through that a little bit
here. Okay. Just verifying the IP so we're we're going to that uh IP of 192.168.1.199 in
this case is the virtual machine in Hyper-V. Um and we're gonna start launching this attack now.
So here same thing just launching it we can see the uh spoofed IP address going to the uh the
destination IP of 100. Wasn't 199 it was 100. Uh 802.1Q and the protocol's ICMP. And if we
just start launching that thing again took a little bit of time to get this thing to
successfully go. But in the end we see that it actually works. So it went through the physical
switch from a physical attacker through that physical 2950 and then it went through the physical
switch from a physical attacker into the Cisco Nexus 1000V on to VLAN 10 um on the the target
virtual machine so it actually worked. So we're seeing that we can actually do double tagging and
switch spoofing attacks in virtualized environments which is pretty scary. So overall results
of this um the first set of charts there on the left hand side show the open vSwitch Linux bridging
pretty much everything worked um after the uh after the uh the first set of charts on the left side
um, for going in. So this is, this isn't coming from, this is trying to get into, um, an attack,
a virtual machine on another VLAN inside of a, a virtualized network. So it worked on every
single environment except for the standard vSwitch for Hyper-V. Uh, we find this is because
when you're launching your Xenia, you're actually spoofing MAC addresses on the attacker side.
And what we saw last year when we were, we were discussing our MAC flooding attacks, those
also didn't work in the v, uh, the standard vSwitch for Hyper-V because the standard vSwitch
has some protection for MAC spoofing against it in, in the server 2008, 2010 environments.
Um, so that's the only thing that protected against this attack was that MAC spoofing, um,
safety mechanism. So if that wasn't there, or if you could do this attack without spoofing MAC
addresses, you could probably get away with it in that environment. Over on the other chart, we
see, um, what was successful, what could we launch the attack from? Uh, which environments
could we, could we actually send something out from? In this case, um, we saw it worked in the
bridging environment. So anything that was a bridge except for, uh, ESX, uh, ESX, uh, the
XI in this case worked. We also saw it work in, in, uh, Zen server with open vSwitch and just a
standard Zen environment with open vSwitch. So pretty much all the open source software was
affected by this attack. Mitigation techniques, standard ones for physical devices is limit
your access to VLAN1 or just eliminate VLAN1 all, all, all together. Um, maybe label it
something different so it's not easily identified. Um, restrict access to switches by MAC
address, which can be spoofed, so that's a little bit difficult. It's kind of a double-edged
sword there. Um, but the heart of this attack is how do we, how do we, how do we, how do we
have access to that native VLAN as we saw? And even the native VLANs within the virtual
switches themselves, which may not be identified. Um, so you really need to look at hardening
these, these, uh, these switches if you're going to be using these in production
environments. Okay, so we're going to pass it on to Caitlin now for ARP, ARP spoofing.
Okay, ARP spoofing. It is, ARP is an address resolution protocol. It's a layer 2 network
protocol that maps the physical MAC address to the logical IP address. Each system on a
network has an ARP cache and it stores the information from discovered nodes on that
network into the ARP cache. Uh, the common entries in ARP cache is the default gateway and
the local DNS. The process for ARP is an initiating system sends out a broadcast, broadcast
request to the entire network. It asks who has a certain IP address. A node on that network who
has that IP address responds with its MAC address and the information is stored into the ARP
cache. I don't know.
Okay, so this is just a demo of what we did. On the left hand screen is the router or the default
gateway. In the center is the target machine and on the right is the attacking machine. On each
of them we have the ARP cache displayed and you can see the IP address of the attacker and the
target over on the default gateway and you can see the MAC address displayed with it showing each
separate entry. And then over on the left hand screen is the router and the default gateway. This is the
on the attacking machine, we're going to run our ARP poison script, which is just a basic
Python script, and that's going to run, it's going to go through and poison the ARPs. So
we're going to go over onto the target and the default gateway and check our ARP cache,
and you can see as it comes up and displays with the two IP addresses that the MAC addresses
are now the same. So on the default gateway, both the target and IP address are showing that
the MAC address is now the same, so that was spooked into tricking the MAC address. Then
we're going to go and we're going to run a ping just out to Google, just to get some packets
flowing, get some traffic, and that's going to finish up the script, and then we're going to
go ahead and run through there. Once the script is finished, it's going to restore the target
machine, so we're going to go over and stop the ping and restore the ARP cache and check that
over. And then you can see on the target and on the default gateway that the ARP cache is now
displaying as it was in the beginning.
So the target was restored, and then we also ran a TCP dump script, so that's going to show the
traffic that came through, and then you're going to be able to see Google's IP address showing
traffic to the IP addresses showing that we captured packets from Google. So that's going to
be added to that long string as well.
Okay. And then for the results you can see it worked on all the platforms.
ARP spoofing mitigation. On Cisco switches, you can make you use the DHCP snooping and dynamic
ARP inspection validation. It's not supported on the Cisco широком. It used only Cisco SW-iTX.
go Nexus 1000V. And then there's also ARP Watch which runs as a service on a Linux system and
monitors the network in changes for ARP activity. Okay so to kind of wrap this up and put
it into some context um we can see that there are the results do show that virtual networking
devices can pose the same sometimes even greater risks than their physical counterparts. So
that's an important thing for um all the users of multi-tenant environments to understand.
Which systems were vulnerable varied pretty widely across the tests. There was no one best
system and there were some uh tests that everything was vulnerable to. That lack of sophisticated
layer 2 security controls similar to what is available on enterprise grade physical switches
greatly increases the difficulty in securing virtual switches against attacks like these. So
what is the bottom line impact then to someone running in a multi-tenant environment? A single
malicious virtual machine has the potential to sniff all the traffic passing over a virtual
switch. It can pass through the virtual switch and potentially affect the physically connected
devices as well allowing traffic from other parts of the network to be
stopped. So that's an important thing to keep in mind when you're looking for a virtual
switch to be sniffed as well. So it's a very significant threat to confidentiality,
integrity and availability of data that's passing over a network in a virtualized multi-tenant
environment. There's a lot of very mission critical stuff being housed in virtualized
multi-tenant environments these days as we know. So what can users of these systems do? Well
educated users can question their hosting providers. Which virtual switch implementations are
being used? Which attacks are they vulnerable to? Are they doing any additional mitigation
strategies against attacks like these? They can also audit the risk of the workloads they're
choosing to run in the cloud with this environment, with these, you know, with this understanding in
mind. They could consider extra security measures on their own or request additional security
measures from their hosting provider. Things like increased use of encryption, additional service
monitoring, more detection of traffic, attack traffic that could be present, alerting. There's a
lot of things that could be done. What are some of the next steps? Well the first step would be to
do some of the next steps for us. So this is, uh, last year was our first time at DEF CON so this is
our second time. Uh, we're a pretty small team. Uh, we made, uh, improvements this year. We did
more attacks than we did before on more regular hardware and we're, we're really happy with that.
But there's a lot more we'd like to do. We'd really love to set up kind of an institute for
apples to apples testing of virtualized environments. We have kind of a long history of
this. Um, we've also done what kind of malicious things, kind of virtual machine that's
co-located do if it hammers the disk or it, it, you know, if it, you know, if it, you know, you
know, hammers the CPU or it allocates a ton of memory or all sorts of malicious things. We've
done apples to apples testing of different kinds of, um, virtual machine migration technologies
and, you know, what's going in the clear and what's not and how long it takes and when can
get, it can, it gets stuck. And we think there's a real place for that kind of apples to apples
independent testing of those things. If you're in a situation of wanting to deploy some of these
technologies, both proprietary and open source, it's really hard for you to bring all of them
in-house and run through all of these kind of tests. So we've seen a lot of great, uh, feedback
about, oh, thank you for doing all of these tests and, you know, pointing out where the problems
are. We've also seen vendors fixing things. We'd love to see, you know, industrial partners to
participate in an institute of this kind, whether it's places that have technologies they'd like
tested and would like to see our results and have a chance to say, okay, yes, this is the results
in the default environment, but if you turned on this thing, then the results would be
completely different. We would love to partner with people who would like to test these things.
People like that. We'd also love to partner with people who are using these technologies in
production environments and would be interested in these results. Um, we have leads, we would
really just like to do more testing in production environments because we're basically using
best practices, out of the box, proprietary and open source technologies. Over time we've seen
some of the work that we've done result in vulnerabilities that are then patched, so we're
really proud of that. Um, but we, we understand that in production environments, many times
there's additional secret sauce or additional monitoring or additional conditions that are put
in place. We actually got some great leads last year after our talk that we still need to follow
up on, but this is us. This is the three of us. We're a pretty small team. And the, the, you
know, the biggest bottleneck we have is the need for more great students like Caitlin, uh, to be
funded to do this kind of testing. Um, so, you know, if anyone would like to help us sponsor a
few students, uh, we promise it's good educational value. Um, so, this is, this is, this is
emails, uh, for Ronnie and I. Also, um, all the details, if, if some of the typing in the demos
you didn't quite get, you can slow them down, zoom in on them, the videos, they're all there, all
the great details. And, um, uh, all of the publications, presentations, demos are all at
Ronnie's website. Um, also like to give a shout out to Nick Moranty who, uh, helped us, uh, get
our AV going, uh, after some last minute troubles. And I guess now we'll take some questions.
hi, so I see you did a bunch of testing on the hypervisors and
the virtual switches that come with the hypervisors. Did you try this with anything like
OpenDaylight or VMware NSX? These kind of SDN solutions that sit on top of the hypervisor.
Not yet but we'd love to. Um I did start looking at some SDN stuff but we've seen that
there's a lot of there's a lot of vulnerabilities in the controllers out there like where you
can actually take control of the controllers so by implementing SDN we're looking at
implementing more security flaws into these environments and we're trying to reduce that
footprint. Um I looked at with the DHCP stuff we did last last year we presented on that I
found a python script just by using simple python and scappy um you can write a simple
script you just daemonize that will find you know rogue DHCP servers on the network and fix
that so I think by using smaller scripted demons that we could actually deploy without putting
more complexity in these networks would be a better solution than trying to throw the monster
of SDN on top but I mean it is a many people are using SDN so it is a good place for us to start
exploring but that's a whole another research track. Right on, thank you. Great talk. Um I
can see you uh tested against Zen uh you tried to get a little bit more of a
against Amazon inside the Amazon environment. Uh no we haven't tested it within Amazon I
think that is not in their uh uh uh terms of service but uh we did have some leads within
Amazon uh which we'd love to follow up on I'm not sure uh yeah we haven't done that yet we'd
love to. So have you done any uh cam table overflow attacks against the virtual switches as
well? Uh that was last year's talk so if you look on the DEF CON media server it's it's all there
demos and everything. Perfect thank you very much. Yeah.
Uh so along with uh you were talking about the 2950 uh physical switch uh so that's running I'm
assuming like uh iOS 12.04 or something like that. Were you able to sort of do that attack on
like uh a newer version of the iOS like uh like a 15 dot something? Yeah um let's go back up to
the. The details. Thank you. No I don't want to go to details here I want to go to that
picture. I'm not exactly sure what model of that switch is but if you look up here on the
top of the rack. I can get there again. On the back there's two switches up top there those are
brand new Cisco's. I accidentally tested it against those and it did work. So I I don't recall
which environments but yeah it did I was able to switch proof out of those as well. Um the
double tagging didn't work because it's not using the 802.1Q encapsulation. Okay. So if a
switch doesn't support that your double tagging's not gonna work. There's other encapsulation
types for trunking that that are supported now on the newer devices. Awesome thank you. Yep.
Hello I'm from Brazil. First of all thanks thanks for the great presentation. It was
wonderful. Uh have you got the first part? Uh I'm from Brazil and first of all thanks for the
presentations was great. Uh I I have worked with a lot of companies in Brazil a lot of
environments from small to large and I have never seen any any one of them with ARP spoofing
preventions implemented. None.
None of them. And sometimes large environments with load balancers and to offload the
applications uh they are encrypted from the the client until the load balancer and from the
load balancer to the back end that they usually take off the the the encryption and create
confirmation going on and based on your experience uh have you seen uh in I had uh you talked
about this.
Have you seen.
Yeah.
What kind solution have you seen if you have implemented in the the the environments that you
have tested or work with?
Uh let's go to the ARP slide here quick. So yeah if you if you guys look at the slides online we
actually have uh images that didn't show up here because of this but there's some good stuff in
there for the ARP. Um so basically you're looking at ARP watch cause the CISCOs have this uh this
dynamic ARP inspection and DHCP.
snooping that allow you to look for rogue DHCP servers and you know different ARP problems
going on there. But ARP watch is a Linux utility you could daemonize, run a server on your
network and actually look for you know malicious ARP activity on that network. It's about the
only thing we've really looked at and experimented with right now but it does work pretty
well. Ok but uh uh I'm not pretty sure if it was clear. What of this kind of uh of mitigations
have you seen implemented in actually implemented in the companies? Oh in the real world.
And that's where we'd really like to do a lot more. We really haven't done much work in
production environments. We are just looking at vanilla out of the box stuff according to
best practices and then looking at what the mitigations could be. We don't have a lot of
visibility into how often people are doing these mitigations in production environments. For us
it's interesting to hear you say that you've seen lots of environments and nobody's doing
that. So we'd love we'd love your email and we'd love to follow up. That would be really
helpful for us. Ok. I see we have a line of more questions and we got the kind of
Oh.
Yank sign from the goons. Uh we'd be very happy to talk more with folks. Um I'm not sure
where the right place to go and meet people is. Uh I guess if you come up here you can go
where we go and we can follow up. Yes? Ok. Thank you so much. Ok. Thanks everybody.