Step into Northbridge at RSAC Conference 2025, where Andrew Carney and Dr. Kathleen Fisher reveal how the AI Cyber Challenge is driving real-world advances in securing critical infrastructure through the power of AI and autonomous systems. Discover how competition, collaboration, and open innovation are shaping a more resilient digital future.
During RSAC Conference 2025, Andrew Carney, Program Manager at DARPA, and (remotely via video) Dr. Kathleen Fisher, Professor at Tufts University and Program Manager for the AI Cyber Challenge (AIxCC), guide attendees through an immersive experience called Northbridge—a fictional city designed to showcase the critical role of AI in securing infrastructure through the DARPA-led AI Cyber Challenge.
Inside Northbridge: The Stakes Are Real
Northbridge simulates the future of cybersecurity, blending AI, infrastructure, and human collaboration. It’s not just a walkthrough — it’s a call to action. Through simulated attacks on water systems, healthcare networks, and cyber operations, visitors witness firsthand the tangible impacts of vulnerabilities in critical systems. Dr. Fisher emphasizes that the AI Cyber Challenge isn’t theoretical: the vulnerabilities competitors find and fix directly apply to real open-source software relied on by society today.
The AI Cyber Challenge: Pairing Generative AI with Cyber Reasoning
The AI Cyber Challenge (AIxCC) invites teams from universities, small businesses, and consortiums to create cyber reasoning systems capable of autonomously identifying and fixing vulnerabilities. Leveraging leading foundation models from Anthropic, Google, Microsoft, and OpenAI, the teams operate with tight constraints—working with limited time, compute, and LLM credits—to uncover and patch vulnerabilities at scale. Remarkably, during semifinals, teams found and fixed nearly half of the synthetic vulnerabilities, and even discovered a real-world zero-day in SQLite.
Building Toward DEFCON Finals and Beyond
The journey doesn’t end at RSA. As the teams prepare for the AIxCC finals at DEFCON 2025, DARPA is increasing the complexity of the challenge—and the available resources. Beyond the competition, a core goal is public benefit: all cyber reasoning systems developed through AIxCC will be open-sourced under permissive licenses, encouraging widespread adoption across industries and government sectors.
From Competition to Collaboration
Carney and Fisher stress that the ultimate victory isn’t in individual wins, but in strengthening cybersecurity collectively. Whether securing hospitals, water plants, or financial institutions, the future demands cooperation across public and private sectors.
The Northbridge experience offers a powerful reminder: resilience in cybersecurity is built not through fear, but through innovation, collaboration, and a relentless drive to secure the systems we all depend on.
___________
Guest:
Andrew Carney, AI Cyber Challenge Program Manager, Defense Advanced Research Projects Agency (DARPA) | https://www.linkedin.com/in/andrew-carney-945458a6/
Hosts:
Sean Martin, Co-Founder at ITSPmagazine | Website: https://www.seanmartin.com
Marco Ciappelli, Co-Founder at ITSPmagazine | Website: https://www.marcociappelli.com
______________________
Episode Sponsors
ThreatLocker: https://itspm.ag/threatlocker-r974
Akamai: https://itspm.ag/akamailbwc
BlackCloak: https://itspm.ag/itspbcweb
SandboxAQ: https://itspm.ag/sandboxaq-j2en
Archer: https://itspm.ag/rsaarchweb
Dropzone AI: https://itspm.ag/dropzoneai-641
ISACA: https://itspm.ag/isaca-96808
ObjectFirst: https://itspm.ag/object-first-2gjl
Edera: https://itspm.ag/edera-434868
___________
Resources
The DARPA AIxCC Experience at RSAC 2025 Innovation Sandbox: https://www.rsaconference.com/usa/programs/sandbox/darpa
Learn more and catch more stories from RSAC Conference 2025 coverage: https://www.itspmagazine.com/rsac25
___________
KEYWORDS
andrew carney, kathleen fisher, marco ciappelli, sean martin, darpa, aixcc, cybersecurity, rsac 2025, defcon, ai cybersecurity, event coverage, on location, conference
______________________
Catch all of our event coverage: https://www.itspmagazine.com/technology-and-cybersecurity-conference-coverage
Want to tell your Brand Story Briefing as part of our event coverage? Learn More 👉 https://itspm.ag/evtcovbrf
Want Sean and Marco to be part of your event or conference? Let Us Know 👉 https://www.itspmagazine.com/contact-us
Inside the DARPA AI Cyber Challenge: Securing Tomorrow’s Critical Infrastructure Through AI and Healthy Competition | An RSAC Conference 2025 Conversation with Andrew Carney | On Location Coverage with Sean Martin and Marco Ciappelli
Please note that this transcript was created using AI technology and may contain inaccuracies or deviations from the original audio file. The transcript is provided for informational purposes only and should not be relied upon as a substitute for the original recording, as errors may exist. At this time, we provide it “as it is,” and we hope it can be helpful for our audience.
_________________________________________
Andrew Carney: [00:00:00] Welcome to the A ICC, uh, experience at RSA. Perfect welcome. Woo-hoo. Hello. Alright, so A ICC is a public competition to find and patch vulnerabilities and open source software, particularly in critical infrastructure. And we thought it was important that as part of sharing the mission and the message here, that we level set and get folks into a head space.
Uh, where they can really focus on the competition and the stakes. And so this is a bit of a level set, uh, to get you kind of in the mindset of getting into our fictional city Northbridge. So we're gonna take a bit of a train ride, uh, and then walk, uh, get into the cybersecurity city of the future.
Marco Ciappelli: It can remind me how you walk into some Disney attraction.
Yeah. Right. It's kind of like that mindset. So you get into Yep. Immersive, right? Immersive. All right. I'm excited. And
Andrew Carney: let's,
Marco Ciappelli: uh, let's go. Let's do it.[00:01:00]
DAPRA Train: It's for real
welcome. You are currently in transiting North, an immersive city showcasing the AI cyber challenge and embodying the promise and challenges of AI for cybersecurity. My name is Kitty. I'm the virtual host of Northbridge, and we'll do my best to guide you on your business. While you're here, you can learn about the AI competition designed to.
Teams competed custom cyber reasoning systems to automatically identify and fix your vulnerabilities in an inside software mode. Discover the high stakes of securing critical infrastructure by analyzing cyber attacks. Though the past and very, very present and be part of the cyber code, people dedicated to changing the.
Cybersecurity. A dream [00:02:00] naturally has a few end. This
a cyber secure world, not as long time here. Welcome to Northridge, a city of broken dreams.
The train has arrived. We're happy you're here. Cyber connected. If a cyber attack happens while you're here, we're depending on you to help us understand and stop it. For now. As you exit the train, a guide will be waiting to orient you on your journey.
Andrew Carney: So we've arrived in North Bridge. I am a little nervous, be honest with you.
Marco Ciappelli: Stuff is happening already. I guess the door is open for us. Yes. Yeah, the door. The doors. The door still work. Alright, that's good. Hello? Pardon
Andrew Carney: us.
DAPRA Train: Welcome to Northbridge.
DARPA Docent: Thank you. On behalf of all the spaces just coming through and meet Brian, be Be Fish. Hi. [00:03:00] Welcome in. Excellent. Welcome in, welcome in.
Come on into North Bridge. Fantastic. We have our group here. Welcome in. You wanna talk in here? Fantastic. Sure. There you go. Excellent. Welcome into North Bridge. We're gonna, we get to explore the DARPA LED AI cyber challenge. So you get to go through this interactive city to see how we are protecting and securing and defending our infrastructure.
So you'll see, uh, healthcare, you'll see medical, you'll see water, you'll see cyber. This is what we want you to go through. You're gonna follow a simple line through the outskirts of our city here. A lot of interactions, a lot of folks that can answer some tech. Nickle questions about the challenge as well.
I really want you to dive in and learn as much as you can about this DARPA led challenge. Follow me through this way here. We'll meet with Dr. Kathleen Fisher. Fantastic committee. Got our first group heading over this way. We're here from Dr. Fisher on the challenge. Good started journey. Welcome for joining.
How are
Dr. Kathleen Fisher: you? Can I give you this? Can we record it? Sure.
DARPA Docent: Okay. Um, so gather round so that everyone, [00:04:00] um, can both see and hear. Um, Dr. Fisher is going to provide an overview and help form context for what you're about to experience in our City of Northbridge. The ICC
Dr. Kathleen Fisher: is a competition that is designed to bring together the best and the brightest from DARPA's broad performer community to pair generative AI with cyber reasoning systems, program analysis techniques.
To automatically refine and fix vulnerabilities in new open source. The genesis of the program came about partly because there was a.
And it showed that Chad GPT and systems like that were in fact able to find in fixed vulnerabilities in a kind of baby scale. That that's the kind of thing that where DARPA loves to see the opportunity and, and pour in lots of resources and get lots and lots of people involved to see if like that genesis, like [00:05:00] the core tiny seed of an idea could grow up into something that would be just massively disruptive, massively game changing with a ICC.
C, we were able to get the foundation companies, Google, Andro, OpenAI, and Microsoft to make available their state-of-the-art models to the competitors so that we are guaranteed to have the best foundation models available to the teams that want to compete. So the semi-finals has kind of two purposes.
One is evaluating the technical hypothesis, and the other piece is giving decision makers and everybody who attends the. Semifinal is a visceral appreciation of how important it is to make our software more secure because of the impacts, not just in the information space, but also in the physical space of attacks on infrastructure software.
A ICC is working on the real software that modern society runs on, and as competitors [00:06:00] find and propose suggested fixes. Yeah, those suggested fixes are relevant for the real software that people are using every day. A ICC is contributing to the advancement of AI and cybersecurity by putting a whole lot of gasoline on the fire of pairing AI and cybersecurity.
We just see with the growing incidents of ransomware, a vulnerable vr, we're, if we can develop a technology that cannot just. Point out that we have a problem, but if we can also say, and here's how to fix it, that could be game changing. Awesome.
DARPA Docent: Thank you guys for listening to this critical message. So your experience is gonna actually continue this way to my left.
You're gonna meet two of my really good docent friends out front of white, two Kates, and they're gonna give you a. Sense of just how high the stakes are for a cyber secure future. So please head this way and make sure you keep an eye out for that cyber down drill. The rat all.
Andrew Carney: Thank you generative ai.
Thank you very much. I wanted to take a second to [00:07:00] love stickers. He loves stickers, so we want too.
Marco Ciappelli: Thank you. That
Andrew Carney: we had at DEFCON this past August. Um, we had 42 teams. Uh, competing with fully autonomous cyber reasoning systems. So, uh, they developed their systems. They handed them over to, uh, us, the competition organizers, and we ran those systems against real open source software projects that are used everywhere.
Um, we had a Linux kernel as one of our challenge problems. We had, uh. SQL Light, uh, just extremely widely used software. And, uh, we had synthetic vulnerabilities added to, uh, synthetic forks, uh, of that software. Um, and then had the teams try to discover those synthetic vulnerabilities just to see, uh, if. If they could find them, the teams were able to find, uh, almost half of the vulnerabilities, 22 outta the 59 that we put in.
They also found a vulnerability that we did not put in. They found a zero day in SQL light that we reported to the maintainers and [00:08:00] had patched, uh, and they were able to. Patch. The teams, once again fully autonomously, were able to patch most of the vulnerabilities that they found. Very, very exciting. Um, and this was an effort where we saw a lot of collaboration from tech leader, uh, uh,
Dr. Kathleen Fisher: commercial
Andrew Carney: leaders in this space.
Uh, Google, Microsoft and Anthropic and Open AI have all been actively collaborating with us to develop. The competition, challenges and infrastructure. Uh, and it was very exciting to see the Google teams after the semi-final event kind of take a resurged interest in the combination of, uh, uh, frontier models and securing, uh, source code.
Um, uh, it's very, it's been fantastic partnership to date and very productive. Uh, so we're very grateful for it. Um, and now we're, uh, kind of on our way towards finals. In August at Defcon this year where the top seven teams will compete against many more challenges, uh, uh, many more open source software [00:09:00] projects, um, many more classes of vulnerability.
Uh, so I'm very excited to sort of see how they're able to do that. I, I will also say in semifinals, the teams were fairly resource constrained. They had a relatively small amount of compute. They had four hours of wall clock time and they had, uh, a hundred dollars in LLM credits. So when they found vulnerabilities and patched them, we're talking tens of dollars per discovery and patch.
Um, which is just fantastic. So we've given them more resources as we've upped the difficulty and scale of the challenge for finals. Uh, and we're very excited to see what they're able to do with, you know, even more, uh, even more resources.
Dr. Kathleen Fisher: Yeah. Um, do you guys have any questions?
Marco Ciappelli: Well, no, I, I just have a comment because I love the idea of the healthy competition and at the same time became collaboration.
Yeah. And that's how you actually advance. Yes. Right?
Andrew Carney: Absolutely. The, the, another exciting thing about this competition is that after the competition is [00:10:00] over, all the teams will be releasing, uh, their cyber releasing systems with a permissive open source license. So. Uh, after they receive their prize money, uh, the community will benefit from that technology.
It'll sort of, the ri the rising tide will lift all those. Um, and we're very excited, uh, for, for kind of that transition into a, you know, col a more, even more collaborative Yeah. Kind of space and
Marco Ciappelli: healthy competition.
Andrew Carney: Yes.
Marco Ciappelli: I love that. Love. Yeah.
Sean Martin: Can you tell me a little bit about the teams? Who, who are they?
Are they from organizations? Are they white hat hackers? Black Haters? Who's the,
Andrew Carney: so I don't know what color hats they, they wear. You know, that's, that's their choice. We'll
Marco Ciappelli: never know.
Andrew Carney: Um, uh, but they come from an interesting mix of university affiliated teams, uh, and, uh, uh, uh, small companies. Um, uh, some of them are affiliated with larger companies, uh, as official sponsors.
Um, some of the university teams are in fact multiple universities, all kind of collaborating [00:11:00] together. Uh, it's a really interesting mix that kind of runs the gamut from, uh, yeah, like I said, commercial like research, uh, uh, companies and, and academic kind of groups. Um, yeah. Very cool. It's very exciting.
All right, let's keep moving. So as we move through the space, uh, we have, uh, a number of different attacks, real world attacks documented, um, uh, over the last, you know, 20 ish years, uh, as well as some, uh, lesser known kind of facts about, uh, critical infrastructure. The, the, the real cons. Consequences of critical infrastructure being vulnerable.
Um, so we have, uh, a sort of simulated version of water infrastructure being attacked and how, uh, you know,
Dr. Kathleen Fisher: these are, these are the
Andrew Carney: sorts of vulnerabilities we do not wanna find out about the hard way. Um, because once they've been exercised, once they've been, um, once they've had that impact, uh, you know, the cleanup will cost orders of magnitude.
The, the, the resolution will cost orders of magnitude more in terms of time, money lives, unfortunately. Um, then, [00:12:00] then a, a, a, an early remedy. Um,
Dr. Kathleen Fisher: uh, yeah, and we really
Andrew Carney: wanted to bring that sense of, uh, um, the gravity of the situation. We wanted to make that really apparent to the audience. 'cause it's, it's one thing I think to talk about, you know, especially in the security space.
We're inundated with how vulnerable we, we almost become accustomed to like how, uh, fragile our, our kind of position is in the world and our infrastructure is, and, and even as we search more resilient, kind of more secure spaces. Um, and I just think it's important to remember that, you know, the, we are, we are in some respects lucky that, that the worst has not happened.
Um, but it's still very much on the table, unfortunately. So we would love to avoid that, you know, um, uh,
Marco Ciappelli: overload of information.
Andrew Carney: Yes.
Marco Ciappelli: Yeah. Which is always a risk, right? When you talk about communicating and creating, uh, awareness. Yes. There is that, that kinda like tipping point [00:13:00] where where is too much and where can you focus to really educate and I feel like also to get the budget, the government, the cities.
Yeah. A federal state and, and all of that to actually be aware Yeah. That we don't need to wait for things to happen because that would not be a good idea. Absolutely.
Andrew Carney: Um, we've been very fortunate, I think, uh, uh, our efforts, um, in, in preparing for the open source release of the tools, we've been working with other government agencies across the federal government, um, and private companies that have an interest in critical infrastructure and sort of preparing them.
Okay. How. Once this is available, how will you take advantage of this? How will you use this? How can we help you use this? There's, um, that's, that's a space that DARPA and arpa h um, and our other government partners we're all invested in. Uh, it's very much, you know, a big tent kind of affair. Yeah, where we, we know that we have to, um, support both our, [00:14:00] uh, sister agencies at the federal and state level, as well as private entities that, you know, manage or, or develop, uh, some of these critical, uh, tools and patients.
So
Sean Martin: there are a couple of extremes, right? So there's the fear angle as part of this, and then on the technical side, it can get pretty complicated of what's, what's actually running and what vulnerabilities are there. Absolutely. And. How do you bridge that gap to help state, local and federal agencies understand what the real risk is without being fearful, and then also connecting communications policies and to bringing teams and operations together, and then obviously individuals to help pull all this.
Andrew Carney: You know, I wish I had a simple answer to that very good question. It's not a simple question, but, but I mean, you, you're, you hit the nails on the head, right? Like all of those. Kind of ducks need to be in a row for us to be successful. And, um, I think to the credit of all the stakeholders [00:15:00] we've engaged, everyone is willing to come to the table on this, whether they're, you know, private industry, um, uh, they're a government agency that has, uh, perhaps authority, but maybe limited budget or resources.
Um, you know, uh, EPA cyber is responsible for, uh, a lot of water security in the US and they are, uh, historically under. Staffed, underfunded effort. I mean, there are 53,000 water filtration and processing plants in the US
Dr. Kathleen Fisher: That's
Andrew Carney: a lot of sites to maintain. Um, and that, that, that hardware, that investment in that infrastructure is typically made on multi-decade timelines.
Dr. Kathleen Fisher: So you
Andrew Carney: can imagine, right, a world where that becomes network connected, where we're trying to leverage, um, any flavor of, uh, automation. Everything kind of, that complexity starts stacking up really quickly and then understanding that, making sure that that operation. Continue seamlessly. Um, that's very hard.
Uh, and so, and not to mention all the different, you know, state, local, uh, and [00:16:00] federal entities that you may need to align. So we're, we're working on that. Um, but it is a challenge for sure. Are you working with the ISAC and the Is We, we are actually, if we want to maybe move, uh, so we've engaged, uh, uh, we've engaged some of the ISACs through our partnerships with like cisa, the sector coordinating councils.
Um, uh, uh, and then, and then even found key stakeholders in, um, critical infrastructure kind of, uh, maintenance and development, uh, to, uh, just once again offer, offer our assistance, like where the government we're here to help. Um, and also ask them right where the challenges, what are they seeing? What do they feel is the biggest challenge to securing their own infrastructure?
'cause oftentimes, right, they oftentimes, sometimes they've already thought about this a lot and they already know what they need. Or think they know what they need and have pretty good idea, but they're not able to, you know, like close that gap entirely. And that's where we can help. Um, we can provide expertise, we can provide funding potentially, um, which, [00:17:00] and leveraging this technology, the, the economy of scale, like it just comes much more kind of the, the value proposition is very strong at that
Marco Ciappelli: point and sharing information.
If somebody's successful, then why not sharing so that the people after that other entity, other government agency, they can also. Void. Uh, I mean, it's a budget. Yes. That works out pretty well, right? Yeah. One, discover the other. Adopt it.
Andrew Carney: Yeah. I, I think, I mean, the ISAC model, um, is so interesting because that's where you get people that compete in business to collaborate for.
Mutual kind of security benefit. And I think, uh, I think, uh, the software supply chain, even their non open source, uh, components, there's a lot still of sort of, uh, mutual benefit from collaborating on securing that environment. Absolutely. Um, especially given how interconnected a lot of institutions are, um, thinking about financial institutions particularly, but this is true across the board.
Um, so once again, [00:18:00] mutual kind of like shared benefit, um, if we can just agree to collaborate. Ah, I like this. Yep. Um, a different community. Yeah, a different way of looking at a community. I, I think a, a, uh, a good ex. Another good example of that is healthcare, um, where you have a very diverse vendor environment and a very complex technology stack where any of those could potentially be.
A way in to compromise a hospital for a ransomware attack or disclose patient records. So, um, if we walk over to the Northbridge Medical Center, I'm, I'm gonna take them.
Sean Martin: Sorry. You, you check in and you don't check out. Yep.
Andrew Carney: Doctor? Doctor. Um, no copay. No copay. It's, uh, uh, so, you know, one of the things we wanted to highlight, I, I think, uh, once again, coming from an enterprise security space, it's easy to forget how complicated these environments can be.
Um, and so, uh, as we look at the different kind of, uh, you know, uh, [00:19:00] threats and insecurities and how the different pathways they open up through these systems, I mean, hospitals are extremely complex and there's no hospital that runs a single vendor kind of stack. They're running devices from tens of vendors, all that may have different network connectivity requirements, all that have different kind of, uh, uh, proprietary protocols that may be hard to introspect, but from an attacker's perspective might provide amazing lateral movement mechanisms.
Right? Um, so securing these environments is really challenging. Um, we have 6,000 hospitals and clinics in the US today. All of them have pretty unique network configurations. Many of them have, uh, health at home and, and telehealth, uh, presence. So they're expanding their network, uh, posture into people's homes.
Remote clinics, uh, teleradiology is very common. So once again, massive interconnected network. Any gap in that tech stack, you know, makes this whole ecosystem vulnerable. Um, and so just kind of trying to provide a [00:20:00] small mapping of, of how these vulnerabilities propagate and manifest. Mm-hmm. Um, and then the, even the potential where if we had this very rapid, I think I mentioned earlier, the semi-final round had, they had a four hour window to analyze hundreds of thousands of lines of source code.
And find and patch vulnerabilities. And they were successful, you know, with uh, uh, uh, um, many of them. And so the idea that we could have, you know, a brain dead integration into our, like CICD pipeline, for example, where the patches are not only being pushed very rapidly, but those are high assurance patches or the evidence of their, uh, uh, uh, utility non interference.
The patches will not kind of prevent the application they're patching from. Uh. Will continue to function as needed. And then also how do we start to ensure that the entire network, entire environment will not be negatively impacted by the changes to that one device. I mean, that's a problem that A ICC is touching on, but, uh, there's other [00:21:00] work at DARPA at ARPA H that continues to look at this.
How do we. How do we know that the changes we make, whether it's a patch or um, a configuration change, how do we know that the system will both remain performant and improve its security and safety?
DAPRA Train: Yeah.
Andrew Carney: Um, I mean I realize it's a hard problem that I think everyone in this at RSA is cognizant of.
Marco Ciappelli: Yeah.
Andrew Carney: Um, but I think it's good to concretize it right in these use cases that aren't necessarily known.
Marco Ciappelli: Well, I think this is a great representation for people that don't really have a big picture. 'cause it is almost like. Uh, like a digital grid with a electric switch, how do you bypass the problem? Yeah. And how do you make it resilient?
Yes, yes. That's a, that's a big thinking. Yeah, it's like 360. It's hard. It's hard. It's really hard.
Andrew Carney: And you know, today it's, I think. I would love to get to a world where doctors never have to take cybersecurity training. Right? Oh, yeah. That, that doesn't, don't we all right. Like I don't want clinical staff to have to think about [00:22:00] cybersecurity.
Marco Ciappelli: Right. Well, that's the goal of cybersecurity will be that you don't see it.
Andrew Carney: Yep.
Marco Ciappelli: And, uh, it doesn't affect your everyday small mom and pop business as well as, of course, big organization and, and organization and save lives. So, exactly. I think that people that will come here will definitely have a very good people.
Sure. And vision of what it really is. Still complex. Still complex, but yeah. Better understanding.
Sean Martin: Yes. We, we can't forget that the healthcare system itself is also complex. Yes. Yeah. Well, of course. And so every little part of that whole workflow and, and patient engagement and, and healthcare providers and payers, the whole thing.
It's complex with a bunch of systems, tons of data. This, this paints a picture of an overlay of threats and vulnerabilities, which
Marco Ciappelli: we also have. And you know, Sean, it it really make you think when you start seeing around a hack alert everywhere, right? Yeah.
Andrew Carney: Yeah. I mean, thinking about change healthcare, right?
Right. Like, that was a, I I [00:23:00] don't think, uh, I don't think enough people were clocking how, how disruptive that would be until we had experienced it. Yeah. Um, and, and those sorts of, you know. Catastrophic, sort of, um, either supply chain kind of like choke points or dependencies are really hard. I mean, you know, that's just one, one large component of this complex system.
Right. I think, I think we're
Marco Ciappelli: getting there to realize that the complexity, it's even, you know, beyond what we could have imagined with fantasy and fiction. Yeah. Like reality has become. The future, the present is the future nowadays. Got it. Yeah. And I, and I love the idea of creating this simulation and having this, this experience.
Yeah. That I think is really gonna open up mind.
Andrew Carney: Yeah. And so our last, uh, last, um, major area here is. Talking in more detail about semifinals, and actually it's running the [00:24:00] competition. This is showing the teams how they're interacting, the, uh, they're fully automated, no human in the loop here. Their cyber reasoning systems, how they're querying the LLMs.
Um, and you can sort of see kind of the, the patterns from some of them. At this point we're in round four, so, uh, looking at SQL light, um, that grid up there represents the synthetic vulnerabilities that we inserted into these synthetic forks of these projects. Um, and sort of the, the globe, like the combined performance of all the teams, uh, is represented there.
Um, but once again, this was just a few hours and a hundred dollars of, of LLM credits, um, for each team and they were able to do pretty remarkable things. Incredible.
Marco Ciappelli: It's incredible. Yeah. It
Sean Martin: very good. Yeah. Alright. Um, yeah, I think for me, I, I don't know if there's a chance to talk with some of the team members at some point through maybe.
Maybe we look to the future between now and Defcon and maybe get an update from you on what the finals look like. [00:25:00]
Andrew Carney: Absolutely. Our teams are hard, like nose to the grindstone working on, uh, developing their systems for finals. Um, uh, but we do actually, I think two of them are here at RSA exhibiting in one way, uh, in, in some way or another.
Um, and we will have the teams, uh, at Defcon, um, uh, uh, we wanna give them an opportunity after we talk about a, after we, the results are released for them to talk very candidly about Yeah, their experience and how they're per, how their CRS is performed. Um, the, the how is. Such an interesting question. Um, we have actually visualization experts just dedicated to trying to help us explain how the cyber reasoning systems, these collections of program analysis tools are leveraging the LLMs.
Uh, and, and those workflows are, once again fascinating but very hard to untangle. Um, so, so the how is gonna be really interesting to continue to explore, I think too. And it's
Marco Ciappelli: definitely a work in progress. Yes. Never gonna end, never gonna end. Once you [00:26:00] finish this, you can start another one. Right, because, 'cause there's an evolving risk,
Andrew Carney: abs and I think so re releasing the tools, open source and our infrastructure, open source, that's the other component of this.
So we want to, we're gonna release the data and the infrastructure such that someone could rerun the competition on their own if they wanted to, or start testing other tools or models in that same competition environment. So we're really trying to make this investment, um, the value to the public at large, uh, as high as possible.
Yeah. Um, and I think, uh, 'cause that that's sort of like fully automated, um, uh, uh, minimally biased or unbiased data sets, like, I feel like getting those out of these tools is also just super valuable. Yeah. Um, and for
Sean Martin: me, the inside the mind of the teams and the individuals. If, how they think about this relative to how bad actors think about it.
And if they had unlimited resources, how might they change [00:27:00] their tactics and their techniques to, uh, to approach this?
Andrew Carney: Yeah, yeah. Our, we've, we've up. The, so their budgets are increased to thousands of dollars per, per project, uh, for this, for our final round. So we're hoping to get a glimpse of that. But yes, I think that's an interesting question to keep asking.
Yeah, for sure. Yeah.
Marco Ciappelli: Well, thank you for the tour. Really, really appreciate it. I think it's amazing what you're doing. Thank you. And, uh, we'll do our part to share it with everybody as well and get an update from you when the time comes.
Andrew Carney: I mean, come out to Defcon. We'd love to Oh yeah. We'll be there.
Awesome. We'll be there. Sure. Yeah. Right. Thank you. Thank you.