Good morning. Sound off, Jonathan. It's great to see so many folks. Great to see so many folks here, standing remotely in the back. So we're here from the government, and we're here to help. My name's Jonathan Mayer. I'm chief technologist for the Federal Communications Commission's Enforcement Bureau. I'm joined by wonderful colleagues throughout the federal government. We have Laurie Cranor, who's chief technologist of the Federal Trade Commission, Eric Mill, who's a senior engineer at 18F in the General Services Administration, and Alan Friedman, who's director of cybersecurity initiatives at the National Telecommunications and Information Administration. All right. Let's be a little louder. And with apologies to the folks who are near speakers. And Alan Friedman down at the far end, who's director of cybersecurity initiatives at NTIA within the Department of Commerce. So it's a great cross-section of different technology policy roles within the federal government. We have working on cybersecurity in the main component of the executive branch. We have delivering services throughout the government. We have an independent agency working on security and privacy issues, a real thought leader within the government. And then we've got the FCC. So I'm going to ask each of our participants to say a few words about their agency and what the agency's been working on in technology policy and the role in the agency. And then for the most part, this is going to be an extended Q&A session. It's up to you to lead the conversation. There are microphones at the left and the right of the room. Please line up and have your questions ready. And my game plan is to just alternate, and we'll do our best to answer. Okay, so Laurie, could you start? Hi, good morning. So the Federal Trade Commission focuses on consumer protection. And in the security and privacy space, we're very much interested in protecting consumers from having their private information breached and leaked. We're interested in protecting them from unfaithfulness. Fair and deceptive practices, scams, fraud, also spam, robocalls, these are all things that we are very interested in investigating and in finding ways to protect consumers. We also do outreach to consumers about how they can protect themselves and avoid falling for some of these scams. We are very interested in talking to researchers. And if you come to our session at 1 p.m., we will get into more detail about that. But we would like to work with the researcher and hacker community to identify vulnerabilities that we need to be aware of, to identify tools that we can use, that consumers can use. And so we encourage you to come talk to us. We've also set up an email address, research at ftc.gov. Where you can send us the things that you've discovered that you would like us to know about. We also have a number of events that are coming up this fall that we are going to be interested in having researchers attend. They're going to be open to the public as well as webcasts. So we have a workshop coming up on drones. We have one coming up on smart TVs. We have one coming up on disclosures. So privacy notices are one form of disclosure. But other kinds of disclosures as well. A bunch of things. And then in January, we will have our Privacy Con event. You can read about all of these things at ftc.gov slash tech. Thank you. I'm Eric Mill. And I'm with a group, a relatively new group in the federal government called 18F. It's like one eight F. And we're about two years old. And we're housed in the most excitingly named agency in the federal government. The General Services Administration. And we, so we're about a couple of hundred people. Mostly not in DC. And we are trying to do technology right in the federal government. So we actually have dozens of engineers, product people, UX people, design folks, security people. And we are trying to make sure the government can do well for itself on technology. And to make sure the government can do things the way that we all want to do things. We can embrace the cloud, use open source, do agile development. We can bake in privacy and security from the beginning of the development process. So we try to lead through implementation and delivery. We do some of the things that are maybe most relevant to this crowd. So we're currently working on getting a bug bounty program started on a number of our public facing systems. Shout out to DOD who just did the first government bug bounty program fairly recently, hacked the Pentagon. Definitely learned a lot from them. We're also trying to hire, and the government does in fact employ information security professionals. We're trying to do it a little bit differently and hire people that are senior technical implementers that don't require security clearance and put them at work on a variety of government wide systems that make the country work better. And then in particular, part of my work there is focused a lot on encryption. And in particular, 18F has been a really animating force on web encryption, on HTTPS in particular. There actually is a federal policy mandate right now in the executive branch to move everything to HTTPS only with strict transport. That's something that our organization has animated and put a lot of energy into. Something I personally work my tail off on. And that's gotten me the chance to meet a lot of the good folks here. And hopefully, we're making the government a better place. I'm Alan Friedman. I'm with the U.S. Department of Commerce in the National Telecommunications and Information Administration. We're part of the administration. We are the president's advisor on telecom and internet policy. You may have heard of my boss, Assistant Secretary Larry Strickling, who's trying to keep the internet free and open with the ICANN-IANA transition. Thank you. And in fact, that's a large part of what our organization does, is represent the equities of a free and open internet, both inside government discussions, as we sort through policy, as a giant, complex government, and also across the private sector. In fact, we take this notion of multi-stakeholder engagement quite seriously. Too often, there are policy issues where, if we wait around for legislation, it's going to take too long, and it may not be great, because there's going to be lots of complexities, and legislation is a pretty big hammer. Regulation also takes a while, and when Jonathan isn't writing it himself, it's not always optimal. So what's the tool we have left? And our tool is we try to get the right people in the room and say, guys, let's solve this, among ourselves, with the right engineers in the room, from all the different relevant stakeholders, so that we can demonstrate that this is a solution that can be solved by coordination and collaboration, rather than waiting around for long, drawn-out legal processes or regulation. We have two ongoing initiatives that might be of interest to you guys right now. One is on everyone's favorite topic, vulnerability disclosure. We know that this is not a new issue, but on the other hand, the cliche that everyone is now a software vendor really is true, and there are a lot of organizations that have never had to work with researchers before, so we're bringing together security researchers, vendors, middlemen, everyone possible, and saying, hey, how can we equip companies and organizations around the country and even around the world to know what to do when someone knocks on their door and says, hey, there's a big problem in your system, and we can help you solve it. How do we get people along that path? The second initiative, which we've just announced, is around everyone's buzzword favorite, IoT security. It's sort of universally recognized that, one, everything is going to be connected sooner or later, and two, security is a giant flaw. No one's really building it in right now. How do we start that process? How can the government promote a better marketplace for that? So we're starting with a small debate and say, hey, patching seems like an important issue, but there isn't really a universal definition for what it means to be patchable. So let's get tech engineers, people who make products, consumer representatives, security researchers in a room and say, what are the different dimensions of patchability? Here are all of the technical details, here's the user experience, here's the connectivity issues. Let's build a taxonomy and then try to collapse that down to a small set of definitions. There is no one-size-fits-all. And from those definitions, have a couple of words that we can tell consumers, don't buy a smart widget without this on the box. Voluntarily. This isn't the government saying don't do this. This would be Consumer Reports or Budge or someone else saying, hey, look for these words. But these words are backed by a couple of paragraphs of technical specifics and specifications. And by the way, if you lie about what's in your box, we have some colleagues in government who know how to take care of that. So if you're interested in talking about IoT security or vulnerability disclosure, we're very happy to have you engage. And we hope you do because when we mean it, we say it's multi-stakeholder. Whoever shows up gets a voice to weigh in and make sure that everyone else can hear what you have to say. Thanks. So let me touch on the FCC briefly and then, again, open it up for questions from the microphones. So the FCC is the Federal Regulatory Agency for Communications Infrastructure and Services, and that includes communications, security, and privacy. So in the U.S. legal system, the FTC is sort of the closest we have to a catch-all data regulator, but there's a lot of sector-specific regulation. So, for instance, our colleagues at the Department of Health and Human Services deal with medical security and privacy. FCC does communications. They deal with security and privacy. And it's an independent agency in two senses. The first is we're not within one of the cabinet departments, and the second is the commissioners are nominated by the president and confirmed by the Senate, but they don't report to the president. So if the president wants to provide input on an FCC proceeding, he writes a comment to the agency, just like any of you can. So the independence is very real. The FCC's core function is independently proposing, enacting, and enforcing rules. So it's a little bit of a blend of the three branches of government. We say, here's what we think the law should be on this issue, and then put it out for comment. And any of you can write in and say why we're right or wrong. Then the FCC finalizes those rules, and then ultimately it becomes a job for where I sit primarily, the Enforcement Bureau, to make sure those rules are followed. So the FCC's authority in communications covers a range of technologies, radio and all sorts of RF emissions, television, whether broadcast or cable or satellite or fiber or whatever's next, telephone, of course, and the agency's recent focus has been especially on broadband internet. So you may have seen the term net neutrality somewhere or other. I'm just gonna guess this crowd's heard that one before. So the FCC proposed strong rules to protect the open internet. And just a month and a half ago, the DC Circuit concluded that those rules were consistent with federal law, and net neutrality is the law of the land. So much of the net neutrality proceeding focused on the kind of economics of innovation online. But at the time, the Commission said, we know we're gonna come back and look at security and privacy, do more rules on security and privacy. We've left that open for another day. And so earlier this year, the Commission proposed rules for ISP security and privacy, saying that ISP should be transparent about their practices, should have reasonable security protections in place for your data, and that you should usually have choice, an opt-in choice, if your ISP wants to repurpose your data for advertising or anything else. We've also been vigorously enforcing security and privacy protections that are already on the books. So for instance, earlier this year, we settled with Verizon for tampering with their customers' internet traffic to insert unique identifiers that made them trackable online. There was an over-a-million-dollar fine, but more importantly, Verizon agreed to make the practice opt-in for any of these headers going out to third-party businesses. We've also done data breach cases against AT&T and Cox. And just earlier this week, you may have seen, we reached a settlement with TP-Link, a router vendor, over selling some routers that could be modified to create radio interference. But an important part of that settlement was TP-Link committing to working with the open-source community and chipset manufacturers towards bringing Linux support, custom firmware support, onto their routers. So even when we're kind of operating in one of our kind of classic areas, we're trying to make sure to promote innovation and make sure that sort of the freedom to tinker is protected. The freedom to lawfully tinker, my boss likes to point out. Let me close by touching on some of the exciting work in progress we have. So just recently, the commission set up the sort of licensing infrastructure for upper microwave spectrum. Now, upper microwave spectrum, I knew nothing about before coming to the agency. Turns out the technology is now there to make this very useful spectrum, and it's widely believed to be an important component of 5G wireless technologies. And so the commission set out its security expectations for the spectrum and plans to address 5G more fully soon. Those expectations include that there be a routing security and for voice calls and text messages, security from one communications device to another communications device. We think that's what the use of the spectrum should look like. And if you'd like to hear more about it, one of my colleagues, Admiral Simpson, is going to be doing a presentation at the Internet of Things workshop here at DEF CON. We've also done quite a lot of work recently to address robocalls. The chairman sent out letters to the major telecom firms saying he expects immediate action. AT&T has taken up the charge. They're leading a new multi-industry working group to deliver actionable results, including new deployments of call authentication standards, new efforts to make sure phone numbers like the IRS mainline can't be easily spoofed, and efforts to build compatibility interfaces so folks can bring filtering, like spam button technology, into the phone system. And then I mentioned earlier, the agency's been working on security and privacy rules for ISPs. We proposed those in March. The comment period closed recently, and so that remains work in progress. So that covers what I wanted to cover for the FCC. And again, this is your session. It's going to be mainly Q&A. So there's a microphone there and a microphone there. And by all means, line up. And we'll take your questions. Thanks. Start over here. Yes. I imagine this is for the FCC. I'm wondering, what is your timeframe for the telecoms to harden their System 7 vulnerabilities? We work closely with the telecoms to implement better protections across their networks, including SS7. The Commission hasn't put out a firm timeline on that particular issue. But an important part of the 5G communications work that the Commission's doing is saying, here's the way we think the world has to look going forward. Obviously, we're not going to tell companies how to build their networks, but we're going to set expectations, and we're going to work with them to make sure they meet those expectations. And for now, that's an ongoing conversation. But the Commission does have regulatory authority and can always be firmer if that becomes necessary. Over on the right. Hi. This question is for the FCC. What are some bits of advice you can give to private citizens so that we can be impactful during the request for comment stage? Lately, it's becoming an increasingly politicized event with large corporations lobbying excessively hard. And we don't have the monetary resources to have our voices heard. And we as technologists know that some of the things that they're doing have led to stagnation of broadband in rural areas, increasingly nasty behaviors, like with Cox trying to do the opt-in service for additional privacy. And it seems that it's getting worse in some ways. How can we have our voices heard? Thank you. So let me start with the FCC component of this. Then I'm going to hand it off to Alan and Eric to address getting your voices heard and the processes they work on. So the FCC's usual process for doing a rulemaking is we issue something called an NPRM, a Notice of Proposed Rulemaking, where we say, here's what we think the law should be in this area. And then there is a usually about 30-, 45-day comment period and another equally long reply period or reply-comment period. Then there's some period of internal decision-making. Stakeholders can continue to come in and meet with the Commission, continue to write letters to the Commission. And then ultimately, the Commission proposes final rules. Then usually someone sues. And then finally, after judicial review, the matter is settled. So that's the process. As we're making sure your voices are heard, we'll have to be careful not to comment on any ongoing proceeding. I think it's fair to say that I've been really heartened to see how the process works being in the agency. Smart comments get noticed. And if you come to the conversation with something new to say, and especially if you have some real data to bring to bear, it gets noticed. And so, the best advice I can give on how to contribute to the debate is make sure that what you write is not duplicative, ideally doesn't use curse words at us, or something like that, and gives us some really constructive input. Those comments get singled out. Let me also add, just briefly, just as a purely procedural matter, make sure you're commenting on the right proceeding. Every so often, folks will file comments in the wrong place. And the system at FCC has recently gotten a lot better for filing comments. We have a whole new online comment filing system. But make sure you file in the right docket. And make sure the issue you're writing in about is appropriate for that docket. So sometimes folks will have really smart things to say, and really great data, but it's just not germane to the specific issue in front of the agency. By all means, call that to our attention. Feel free to kind of reach out to who you think is the appropriate contact at the agency. But it's easy for it to get buried in a docket if it's not germane, because someone will review the comment and say, it just doesn't bear on this particular proceeding. So that's a kind of procedural note. Okay, so now over to Alan and then Eric. So as an example of a comment process that impressed me how effective it was, a few months after I joined the Department of Commerce last year, I get a call from one of my colleagues in a different part of commerce called the Bureau of Industry and Security. He says, hey, we're about to release a proposed rule based on this arms control agreement known as VASNR. So we had some discussions and we helped prepare them for the fact that they were going to get some strong responses. And we did get a lot of responses. And many of those were really helpful. This was a case where industry and the security community were on the same side. But they brought two very different perspectives that was very helpful. It's challenging because often people were commenting based on news stories that were based on other news stories. And so by the time they filed their comments, it wasn't something that was directly related to the regulation because a lot of the stuff is quite technical. So as Jonathan said, you know, make sure, have as much preparation as you can. But this is an area where we got the comments and they were overwhelmingly negative. I think there was one comment in favor out of over 200. And so the U.S. Department of Commerce worked with our government colleagues and has gone back to Vassanar to try to renegotiate. And so that I think is an example of feedback from the security community driving policy in the direction that it should. And so as you are preparing to engage, it helps to talk to other people. If you have colleagues or friends who are engaged in the policy network, they'll be able to give you a little bit of background. If you're curious, at least in our case, I don't know if the FCC can do it, but in commerce, we'll talk to you about what we're looking for so that you can tailor your feedback to give us the insight that we need to make good decisions. There are lots of organizations out there that are engaged in a lot of these issues, whether it's EFF or I am the Cavalry. We need more advocates for security as a unique value. So please try to engage and learn as much as you can and then give us as much feedback as possible. So I'll just briefly add on. It's actually a bit of an outside perspective. So I'm not in a regulatory agency now at GSA. Before this, I was at an NGO, a nonprofit called the Sunlight Foundation that does open government and transparency work for about five years. And I worked a lot on trying to make the regulatory process more accessible to people because I watched many different times people leave the opportunity on the table to comment on a regulation and I'll tell you that the people who will always comment on a regulation that affects them are affected businesses or the private sector. Not very often, comparably, do you get real, public, constructive input on things. And it's not always well known that, and this is distinguishing from a lot of other countries in the world, that in the U.S., executive agencies that are issuing regulations must respond to every unique comment they get. They have to at least acknowledge it in some way. And I've read many final regulations that went down and addressed all the different groups and notable comments that they got and changed their minds on small and large things as they went. You don't always get your way, but when you participate, showing up really does matter. And that was my personal experience as an advocate and open government lobbyist sometimes, working on these issues, that showing up is everything. So I really do encourage you to, you don't literally, I mean, the Federal Register if you go to federalregister.gov, they actually have added in the last few years a number of really great alerting and feeds systems for you to follow things more easily. It's actually a really great team that built federalregister.gov. They were invited by OFR to do it after they did an app contest as an outside group of developers trying to reimagine what federal regulation and commenting should look like. And there are other services that will help you do that. And I just strongly encourage you to take that seriously. I'll just add that at the FTC, we often are looking for public input. Usually when we announce that we're having a workshop, there are opportunities to comment both before and potentially get on the agenda, as well as after the workshop. And we are very much interested in people who bring us data. We want data, we want empirical results, not just the opinions, which are nice, too. But if you are a researcher who can bring us data, that's something that we are going to be very interested in seeing. Let me amplify that point. Let me amplify that point before moving on to the next question. We hear a lot from lawyers in the government. We don't hear so much from technical experts. And so that sort of input is incredibly valuable and it gets noticed. Hi. So you mentioned that the DOD now has a bug bounty but for sort of an opposite perspective, one of the things that I do is run census.io and other scanning for security things. And five years ago when we started, the DOD sent us a very strongly worded email saying you'd better stop scanning us. That means we can't participate with them, we can't tell them about vulnerable TLS implementations. So how do you engage with the DOD beyond just submitting to their bug bounty? That's a difficult question given that none of us are from the DOD. We're probably not going to be able to give you the answer that you're looking for. But, you know, in general, the closer you get to communicating with subject matter experts inside different agencies, the more you get answers that make sense and creative solutions to different problems. The DOD Hack the Pentagon program was started by the Department of Defense Digital Service, which is a relatively new team inside DOD. It's part of the US Digital Service, which is a White House initiative that has created digital service teams in a few different agencies. But that's about the best I'm going to hire. Maybe anybody here is going to be able to give an answer to that. I think just large organizations are not monolithic. And so, as we said, the closer you can get to the people who engage, the better. In the private sector, we work with large companies inside our process on vulnerability disclosure, who are trying to figure out how can we work with researchers, even as, you know, their general counsel's office is writing comments about how we need to bring back DMCA controls on their products. So the trick is to find the allies in any organization that you can. I think this panel probably is a great way to start to find the right people. And so good luck, and thank you for reaching out. And thank you also for running census.io. So at GSA, 18F uses that data in our work all the time. I personally use it in my work to understand the government surface area and to report things to other agencies as necessary and to tell people when they're falling out on something and then to work with them to fix it. So really, like, big thank you to you for that. So I'm a student who's going into my senior year at high school, and I was just wondering how did you guys get into the federal government, and how could a prospective student also get in? Thank you. Go down the line. Uh, that's... So one, I think there... I'm going to speak for everyone and say we desperately need smart, passionate, technically aware people in government. Desperately need them. And, you know, the advice I would give is it is fairly easy right now to go from the technical world into a policy track. My background is in computer science. Wasn't very good at it, so I have my PhD in policy. And when you're meet... So policy means I'm a mediocre economist and a mediocre coder. And when you're mediocre at that many things, you end up in Washington. And, you know, I was an academic, and then someone talked me in. But I think the advice I would give is stay on the technical side as much as possible. But engage in policy in your spare time. And eventually, you'll find an issue where you can find the right person and weigh in, and they'll say, we need you on our team. Yeah, I mean, so as somebody who went... Primarily my background is in software engineering. I have a CS degree, but I work a ton on policy day to day now. And it's really as simple as becoming an expert in something and being willing to talk about it publicly, privately, to leadership without fear and have confidence in what you say and really develop your skills as a communicator. Right? So, like, being a good writer is just a universal skill that will make you more effective at bringing people into your way of thinking, projecting that you know what you're talking about. And that's something that, you know, even if it's not going to be for... You know, even if you don't end up working on policy for some amount of years, like, take the time to keep exercising those muscles to keep writing and to keep getting feedback on that and to keep becoming a good communicator. Yeah, so I started my career working in AT&T and was doing research on privacy mostly. And I actually presented research to the FTC 20 years ago. I went to their workshops. And when FTC staff said, can someone explain again how third-party cookies work? I would, you know, take time from my day to call them back and to explain it yet again, right? And basically became known to them as someone who was willing to explain these technical concepts in plain language. I then became a professor at Carnegie Mellon and have steered my students and their research to trying to make our research relevant to some of the policy needs and submitting our results to government agencies. And so right now I'm actually on leave from Carnegie Mellon and the chief technologist position at the FTC tends to be an academic who comes in for a year or two. The other point I want to make for our high school student friend is that if you know that you're interested in government service, there are scholarship opportunities for you. So scholarship for service, basically if you are a U.S. citizen and have technical interests, you can get the government to basically pay your tuition in exchange for you then committing to do some work for the government. And so it's a great opportunity. To amplify something Laurie said, it's about explaining things to other people. I mean, the community that we're all a part of here, this conference is tremendously huge. Even just this room is filled with people. This is a large, amazing community. And you could spend years, you could spend your entire career communicating to and within this community and go very far. But there are certain kinds of things and certain kinds of impacts that require you to speak outside this community and to make your work accessible and approachable to a larger set of people. Because even a lot of people who aren't professional information security folks, professional privacy folks, have an interest in that, aren't dumb, and are intellectually curious and are willing to apply and integrate that stuff into their work. So it's something to remember too that even though you may not ever have to, you may never be confronted in your life with a time when you have to communicate to the broader community, it's that there are certain kinds of work that you really should do that. So I'm also a loner from academia. I'm on loan to the FCC from Stanford. Go Bears! Go Bears! That's inevitable. You know, we can't all go to school at a country club. So... So... I'm in a different stage of my career from Laurie, of course. I hope to be faculty in the not too distant future, but I'm just rotating out from grad school. And so I want to note there are opportunities absolutely at that stage of your career coming out of academia. If you don't know what you want to do next, you're going to take a little gap between what you're doing in academia and whatever comes next. The government has great roles there. There are a bunch of great opportunities straight out of college, straight out of grad school. There are programs to support that. More programs are coming online all the time. There are also wonderful internship and fellowship opportunities to explore. Even with a six-month or one-year stint in government, you can have a tremendous amount of impact. Or a summer internship, which we actually have three summer interns at the FTC in technology roles this summer. And I really want to emphasize Eric's point about communicating with folks in government. I think having worked on both sides, and I guess I should come clean, I'm also a lawyer, the way in which folks communicate in the hacker community is very different from the way folks communicate in government. And for better or for worse, but learning how to sort of speak Washingtonese is really, really important. That's something you can learn in advance of coming to the government, and it's a great skill set you can pick up if you spend some time inside the government. Yeah, over on the left. Great, thanks. Dan Tynan for The Guardian. I have a question for all the panel members, and it's kind of a general one. There's been a lot of speculation lately, given the hacks for the DNC and Hillary Clinton's campaign, that the actual election could be hacked, in particular by a certain nation state whose name begins with R. So I'm going to ask you to rate, on a scale of one to ten, one being not a big deal, 10%, and 10 being holy shit. How worried you are about this happening, and if so, what worries you most? So remember when I was talking about learning how to speak Washingtonese? Yeah. No comment? I mean... Yeah, it's... I mean, I don't think... It's not really any of our... It's certainly not my area of expertise here. So I'm going to use this as a pivot, which is the other aspect of engaging with policy is to know when to say, that's a great question, but I don't know. Let's bring in actual experts. Yeah. And fortunately, since 2000, there's been a lot of great research on security of electronic voting machines, and I don't know, or some of Avi's people here, but there are a couple of great professors out there. The other lesson I would take away that's highly policy relevant is if you really are interested in this, go and volunteer for your local elections board. You will be the only person there under 70. The 70-year-olds are wonderful, and it is a great way to learn how complex the technology and the bureaucracy and the ideal high-level goals of democracy all work together. So if you are interested in understanding the security of the election system, get some on-the-ground experience while you're hacking your election device as well. Yeah. I've been an election judge in Pittsburgh for the past 10 years, and it's a really interesting and eye-opening experience. I definitely recommend that. All right. On the right. Cool. First off, thank you all for coming here today. I can't imagine it's exciting to be told you're going to be at DEFCON. It is representing the feds, but thank you for coming. Appreciate that. And that being said, I had two questions mainly for the FTC. Where do you see the breach insurance industry going, and do you see that's going to drive private sector upping their cybersecurity game because we know legislation ain't going to do it? And is that a growing, stagnating industry? So that's my first question. The second question is, you said a minute ago, you want smart and passionate people. But the government culture tends to bring out a least performance necessary attitude. Is there anything being done at the executive level to change that culture? Yeah. So on breach insurance, yeah, once again, I have to say that I'm not an expert in breach insurance, and I'm not really sure. On the issue of getting smart people to want to come to government, I think that, you know, the administration has made a number of pronouncements about wanting to do this, saying that, like, you can wear T-shirts and jeans to work. You know, it's a good start, but that's not enough, you know, just along those lines. I think that, you know, within our agency, we're an agency that's mostly attorneys, and it's set up to work the way attorneys work. And as we are hiring more technical people, we're saying, wait, we may need to do things a little bit differently for our technical folks so that this becomes the kind of place that they want to work and where they can thrive. And I think the leadership is very much open to that. I want to add something on the culture change. So 18F is a new office. We're about two years old in the GSA, and one of our missions there and in the rest of the government is to work on that cultural problem to attract people to government and also to make it a great place to work for people. I'm actually, I really enjoy my job at GSA. It's actually the nicest, most humane place I've ever worked in terms of remote work, in terms of, you know, having, being in the cloud for email and docs and calendar, for having really nice people to work around me, to have computers to deploy things to, etc. And that is, that's a really valuable thing. There's something that's really dangerous, though, that I know we have encountered and I have encountered, is that it's very tempting to talk about culture change as people change and to talk about problems that you perceive in the government as problems with the people. But it's really not the case and the government turns out to be filled with a lot of really smart, well-meaning people in some really terrible incentive structures with a lot of fear that drives executive level decisions like fear of being criticized, fear of being punished, fear of being hauled in front of whoever. And that is, it's that thing that you have to attack through transparency, through a little bit of courage, through changing incentive structures as necessary to reinterpreting or rewiring rules around hiring and all those things. And yes, those things are all being worked on at the executive level and at the rank and file level in different ways. It is just a big problem. The US government is the largest organization in the history of mankind and it's very decentralized. But it is being worked on all over. The only thing I would add to this as someone who's quite new to government is some advice that was given to me when I was first approached is your first boss is really helpful and I'm lucky, and I think many of us are lucky to have fantastic supervisors who recognize that doing Meet the Feds and stuff like that is really important to the missions of the policy that we're trying to change. And so if you are contemplating joining government, think a lot about your supervisor and what that relationship is going to look like because a great supervisor just makes your job a lot more fun. Hey, my question is what kind of metrics or data points do you guys capture to make sure that your organization is safe or secure or on the right track? Sorry, could you repeat the question or Jonathan maybe? Yeah, so what kind of metrics or data points do you guys capture to make sure that your organization is safe or that you guys are on the right track? Right. . About what? . Yeah, sure. So, I mean, it varies, right? So, in terms of monitoring your own systems, people use all sorts of different scanning tools, people use all sorts of different metrics about the kind of costs that are incurred on those systems. I know that one of the things I work on is measuring encryption presence and quality around the government and around 18Fs and GSAs systems especially. And, you know, using all the same tools that you all probably use, things that are pasted Libcurl, things that are pasted on Google things that are based in SSLIs, we use data from ZMap scans of the internet. You know, we're running in Unix-based environments and doing that same sort of work. And so, you know, we use the same tools that you all do and use that to improve our work. So, I believe we're getting the signal from the goons that it's time to wrap up. So, thank you all for your questions. We're going to stick around for a few minutes to allow additional questions outside, if you're not inside. Please go out that exit door on that side of the room. All right, thanks again. Thank you.