The player is loading ...
Sign up to get updates from us
Episode 63: In this episode of Critical Thinking - Bug Bounty Podcast we welcome back Jason Haddix (From Episode 12) to talk about some updates to his The Bug Hunter's Methodology, as well as his own personal life and hacking journey. We talk about the start of his new company, and then venture into topics such as using threat intelligence and buying credentials from the dark web, recon techniques, and ways to integrate AI into your workflow (or target list).
Follow us on twitter at: @ctbbpodcast
We're new to this podcasting thing, so feel free to send us any feedback here: info@criticalthinkingpodcast.io
Shoutout to YTCracker for the awesome intro music!
------ Links ------
Follow your hosts Rhynorater & Teknogeek on twitter:
https://twitter.com/0xteknogeek
https://twitter.com/rhynorater
------ Ways to Support CTBBPodcast ------
Hop on the CTBB Discord at https://ctbb.show/discord!
We also do Discord subs at $25, $10, and $5 - premium subscribers get access to private masterclasses, exploits, tools, scripts, un-redacted bug reports, etc.
Sign up for Caido using the referral code CTBBPODCAST for a 10% discount.
Today’s Guest:
Resources:
Dehashed
Flare
CSP Recon
https://github.com/edoardottt/csprecon
Timestamps:
(00:00:00) Introduction
(00:05:37) Updates to The Bug Hunter's Methodology
(00:14:46) Red Teaming
(00:21:29) Bug Bounty on the Dark Web
(00:36:19) FIS hunting
(00:47:59) New Recon Techniques
(00:58:32) AI integrations and bounties
Justin Gardner (@rhynorater) (00:01.079)
Yo, what's up Jason? Welcome to the pod, man.
Jason Haddix (00:03.709)
Hey, thanks for having me, guys. It's good to be back.
Justin Gardner (@rhynorater) (00:05.595)
Of course, yeah. Thanks for coming back on. I think, I wanna say, you know, Nagli, I think, has made a couple of appearances, but I think you're our first two-time solo guest. So, you have the honor, you have the honor, my good sir. Dude, this is looking like a pretty packed document. Where do we start, where do we start, where do we start? Well, recently you actually switched over to full-time working for yourself, right?
Jason Haddix (00:17.749)
Oh, that's awesome. That's a good title to have. Nice.
Jason Haddix (00:35.581)
Yes, I did. Yeah.
Justin Gardner (@rhynorater) (00:36.707)
That was, that was what? Couple months ago?
Jason Haddix (00:39.301)
Yeah, two and a half months ago, about three, two and a half, three months ago, yeah.
Justin Gardner (@rhynorater) (00:42.796)
Nice man. How's that been going?
Jason Haddix (00:45.449)
It's been going great. So I, you know, like I was at a place and I've always worked for other people my whole life. And I've always felt like, you know, I've worked with great teams and stuff like that. But, you know, when I started doing the training last year, it just became more and more apparent that, hey, I could probably do this for myself. And, you know, my friends have been pushing me to start my own consultancy for a long time. Like Dan, Dan Measler and like, you know, Clint Gabbler, like those guys.
Justin Gardner (@rhynorater) (00:56.719)
Mm-hmm.
Justin Gardner (@rhynorater) (01:09.241)
Mm. Yeah.
Jason Haddix (01:13.621)
have always just been like, dude, you should have your own consultancy. Like there's no reason you should be working for anybody else. And so I finally took their advice and so I started Arcanum Information Security, which if you're a fantasy nerd, it comes from the book, Name of the Wind. Arcanum is the, this kind of the art of magic in that book, the Patrick Rothwick books, yeah, so.
Justin Gardner (@rhynorater) (01:32.807)
Oh sweet. I love the logo for that. What is it called again?
Joel Margolis (teknogeek) (01:34.83)
It's a great series of book.
Jason Haddix (01:37.029)
Yeah, it's a great series of books, yeah. It's called The Name of the Wind is the First Book. Yeah.
Joel Margolis (teknogeek) (01:39.912)
the name of the wind.
Justin Gardner (@rhynorater) (01:41.579)
Name of the Wind, all right, that's interesting. I love the logo you have for this company. It's super cool, and I guess it fits with that whole sort of dark magic naming convention there. That's great. And so you've been doing full-time that in content creation, and you've been running the Bug Bounty Hunter's methodology classes. Those have been, I've just seen so much good feedback for those classes, man, all over my Twitter feed.
Jason Haddix (01:46.069)
Thank you.
Jason Haddix (01:54.42)
Yeah.
Jason Haddix (02:08.465)
Yeah, it's been great. So I started those solo and then now they're attached to the company and we do consulting and we're having a whole bunch of other classes come out this year. So, but yeah, I mean, the Bug Hunters Methodology course has surpassed even my wildest expectations and the feedback has been fantastic from students. Just, I mean, I knew because it was a public talk for a long time and I knew that there was a lot of good response to it.
Justin Gardner (@rhynorater) (02:27.554)
Mm.
Justin Gardner (@rhynorater) (02:33.025)
Mm.
Jason Haddix (02:36.905)
But the community around it, the Discord, the contributions from the students, I mean, we have people dropping scripts in there all the time to do cool things. And just every class is also different too, like different student questions come up. And so like we go off on these awesome tangents and since they're recorded, you just get like a lot of value out of the course. We've had guests come on to do like small little sections. It's been amazing. I sincerely enjoy teaching this course.
Justin Gardner (@rhynorater) (02:57.807)
Mmm.
Justin Gardner (@rhynorater) (03:07.823)
One of my favorite things with having a content community of sorts is when you are talking about a specific subject or something like that and then someone from the community just drops something in the Discord or something like that just blows your mind and is so well developed and not only are they clearly understanding the content that you're talking about but then they've taken it a step further and actually implemented it or exploited something with it. It's just the best feeling ever.
Jason Haddix (03:22.801)
Oh yeah. Mm-hmm.
Jason Haddix (03:33.565)
Yeah. It is. When people find a bug in the Discord, I am like doing a happy dance all the time. Like I'm just like, yes, go everybody! Yeah. You guys have JD in your Discord, XSSDoctor, who's been... He's smart as F and his story is also amazing. So, yeah.
Justin Gardner (@rhynorater) (03:39.152)
Hahaha
Yeah, man.
Justin Gardner (@rhynorater) (03:46.368)
Yeah, yeah.
Justin Gardner (@rhynorater) (03:51.367)
Yeah, dude, I was just talking to him the other day. He's awesome. And he's been explaining some of the bugs that we were talking. And actually, that's exactly who I was thinking of, which is hilarious. When you when you said that. That's that's great, dude. Yeah. So so I guess.
Jason Haddix (03:58.901)
Oh, that's funny. Yeah, yeah. Him and I are good buddies. Yeah. So
Joel Margolis (teknogeek) (04:04.214)
Yeah, I'm always blown away when I see the stuff that's going on in the discord. Cause there's, I mean, I always feel like there's no bugs. And so when I see people like posting all these bugs all the time, like, okay, there's a lot of bugs.
Jason Haddix (04:12.422)
Mm-hmm.
Justin Gardner (@rhynorater) (04:12.536)
Yeah.
Jason Haddix (04:16.609)
There's bugs. It just takes eyeballs, right? It takes different perspectives and eyeballs, yeah.
Joel Margolis (teknogeek) (04:19.478)
Exactly.
Justin Gardner (@rhynorater) (04:21.563)
It sure does. And yeah, so I guess bringing back around to the Bug Hunters methodology piece, this piece of content, super legendary. I remember reading this when I was just getting started in Bug Bounty. And one of the cool things that really makes this one of the top pieces of material in the industry is the fact that you're constantly updating it. So
let me ask man, what kind of stuff has been sneaking in? You don't have to give us like, you know, the sauce from like your, you know, now very much deservedly paid course, but maybe give us just a little hint of what kind of stuff is in this latest, you know, evolution of the talk.
Jason Haddix (05:03.965)
Yeah, so, okay, so the new stuff, this cohort, right? So I update the class between each cohort, which is usually, I do a cohort every quarter, right? So I'm not teaching the class all the time. It's not on demand video, anything like that. So, but for this cohort, we pushed really hard on a JavaScript analysis section because we had some before, but I was, you know, like I was talking to a lot of the students from the last cohort and there is this difference between
Justin Gardner (@rhynorater) (05:11.655)
Mm.
Justin Gardner (@rhynorater) (05:23.368)
Nice.
Jason Haddix (05:33.129)
like pen testers and security auditors who are a little bit younger and a little bit older. And it's nothing to do with like skill or age or anything like that really, but what it has to do with like kind of the web you grew up in, right? A lot of the methodologies for testing websites are in a lot of the training out there. It's based around, first of all, like, you know, the structure of the data being passed in, you know, parameter value format and then...
Justin Gardner (@rhynorater) (05:39.249)
Mm.
Justin Gardner (@rhynorater) (05:57.632)
Mm-hmm. Yep.
Jason Haddix (05:59.917)
And then, you know, pretty much like websites where resources were statically linked and stuff like that. And that is not how the modern web works, right? The modern web is a single page applications, heavy JavaScript, everything is in that JavaScript for functionality site, sometimes lazy loaded, you know, resources, and then you have, you know, now you have cloud protections protecting, you know, every sites and, and so there was this kind of
gap that I wanted to help those people who grew up in the old version of kind of the web get to the new version. And I took inspiration from you guys too. You've been talking a lot about JavaScript bugs in the last like six episodes, right? And so the thing I thought there was a gap for is just that there is an understanding for web pen testers and bug hunters. And so I started at the beginning. I'm like, okay, first, how do you find JavaScript both inline and files, right? Okay. So what do the
Justin Gardner (@rhynorater) (06:34.011)
Hmm. Yeah, we have. Hmm.
Jason Haddix (06:55.901)
Let's talk about them. Let's talk about the hurdles that they present for testers, right? So lazy loading, highly minified JavaScript. Okay. How do we overcome those? Right? Okay. How do we get the lazy loaded application, um, you know, to either show up in our proxy or, you know, raw, we can get to the files or something like that. And then is there any way that we can deal with minification and obfuscation of JavaScript? And what are we looking for in the JavaScript specifically? And so I built out a whole section.
around that and I scoured the internet for the best de-office skaters and the best tools and I even built ghetto strings analysis, cheat sheets, stuff you can just control F4 and then I talked about putting JavaScript into VS code and building regex, more advanced regex for finding stuff that you want to look for in files. I talked about, obviously, if you have
Justin Gardner (@rhynorater) (07:47.791)
Hmm.
Jason Haddix (07:55.073)
and JS Loose by Tom Nom Nom, I talked about that. You know, and how pathing is so weird in JavaScript, how you can build paths, you know, dynamically, and so you can miss them if you're not looking right or you're not building them or constructing them right. So those are the types of things I talked about in the JavaScript section, and it was fantastically received, like amazing. So that was that section. And then I did a cheat sheet for all of the JavaScript front end.
Justin Gardner (@rhynorater) (07:57.242)
Yeah.
Justin Gardner (@rhynorater) (08:14.033)
Mm.
Jason Haddix (08:21.253)
engines and what's, you know, what, you know, what are the common vulnerable? Yeah. So what are the common, what are the common issues with that framework? How does it handle certain special characters? Where, um, where are the main app files hosted? Where is the database connection string hosted? So in case there's some kind of access control vulnerability is set up on those files and then, you know, like the bigger, the biggest one being like, you know, dangerous to set inner HTML or something like that, like how that leads to.
Justin Gardner (@rhynorater) (08:23.919)
Dude, no way. That's awesome.
Jason Haddix (08:47.338)
can lead to issues. And so I started to build those cheat sheets for the students too to have to take home.
Justin Gardner (@rhynorater) (08:54.575)
but yeah.
Joel Margolis (teknogeek) (08:56.374)
Do you, how often do you find that while you're writing these that it's, it's more of like a, like a re-learning process for yourself more than like a writing for everybody else? Because I find when I do this sort of writing, I'm like, I'll, I'll write something and I'll be like, actually, is that true? And then I'll go back and I'll check it. And I find myself going down a big rabbit hole where I'm learning a ton more about that. And then I circle back and I write that down. Do you find that is the case a lot?
Jason Haddix (09:17.429)
Yeah, oh yeah.
Jason Haddix (09:23.749)
100%. In fact, some of the stuff I do in the class, I know that I have a big, big blind spot for certain sections of tech. And I, so I put it in the syllabus and I promise it to the students. And then that forces me to create the content and be an expert on it. And so like, oh yeah, I'll just be like, and it's like this ADHD thing that I do all the time. And it's like, okay, now it's out there. And they're expecting Jay Haddix level content from you.
Justin Gardner (@rhynorater) (09:40.519)
Dude... Savage. That is savage.
Joel Margolis (teknogeek) (09:47.767)
Ha!
Jason Haddix (09:51.717)
in this section, so you better kill it. And I'm like, okay, cool, like, let's go. Yeah.
Justin Gardner (@rhynorater) (09:53.679)
Dude, that's such a hack because I feel like a lot of people, the traditional wisdom is like, don't post about your goals until you've achieved them or something like that, because it like steals the dopamine or something like that from you and you like get the dopamine ahead of time. Or so I don't know, I've heard that before, but I'm like total opposite of that. I'm like, let's, you know, promise it and then force yourself to deliver on it. I feel like that's such a good way to motivate yourself.
Jason Haddix (10:05.81)
Yeah.
Jason Haddix (10:16.606)
Yeah.
Yeah, it's worked really well for me.
Joel Margolis (teknogeek) (10:22.288)
There's this blog that I love called Wait But Why. It's written by this guy, Tim Urban, and he did a TED Talk about procrastination. And one of the things that he talks about is this concept of a chaos monkey that lives in your brain. And it only really comes out if there's fear of embarrassment or really strict deadlines. And that is like...
Justin Gardner (@rhynorater) (10:36.834)
Haha
Justin Gardner (@rhynorater) (10:42.069)
Oh my gosh.
Joel Margolis (teknogeek) (10:44.03)
Exactly how I operate, which is that there's this chaos monkey that never comes out until like I've set some sort of really bad parameter or like guardrail in place that if I hit it, it's going to be a disaster. And then I'll actually get the thing done.
Justin Gardner (@rhynorater) (10:52.936)
Mmm.
Justin Gardner (@rhynorater) (10:58.423)
Yeah. Now, dude, that's great. I was just gonna say, you know, with regards to, I can't relate as much to the ADHD, you know, approach to things, but I can definitely say that, like, trying to motivate yourself to...
Jason Haddix (10:58.545)
Yeah. Yeah, I think. Oh, God, sorry.
Justin Gardner (@rhynorater) (11:14.011)
accomplished monumentally huge tasks is you definitely need some hacks for that if you're going to do it on a regular basis. And I'm really impressed that you went through and actually started working and got so far in this cheat sheet because that requires so much dev as well, man. When I hear about you saying, like, ah, I'm going to take all these frameworks and figure out how all of the special characters are handled in the parsing and that sort of thing, it's like, oh my gosh, that hurts. But really, when you get through
Jason Haddix (11:30.208)
Yeah.
Jason Haddix (11:40.65)
Yeah.
Justin Gardner (@rhynorater) (11:44.021)
where that data lives for each one of these individual, you know, framework types that are getting so common. And I think, you know, with that, you can really use that to write better regex and write tools that will better parse those JS files to pull out the paths, et cetera, which is really valuable.
Jason Haddix (12:00.197)
Yeah. You know, one thing I noticed doing, um, doing the cheat sheets specifically was that, um, was that there was this common thing around all of kind of the new frameworks, uh, whether they're JavaScript or not. And it's that you cannot use the default functions to protect, uh, you know, to do input encoding or output or output encoding or input validation on certain sections of the app. Like it just doesn't work. Right. And like a lot, not a lot of people think of this,
Justin Gardner (@rhynorater) (12:24.022)
Mm.
Jason Haddix (12:29.517)
right? So it's like, let's say you're building a webhook, right? Like, you know, to integrate with another site, right? Well, it's very hard to implement those functions across, like, an integration piece of your application, right? And so, like, people just don't do it. They're like, they have to, you know, somehow take special characters of certain sorts into the integration backend and hook it up via webhook. And then, so there's like common places, and I've put them in a section that I call heat mapping.
Justin Gardner (@rhynorater) (12:56.408)
Mm.
Jason Haddix (12:56.621)
We're like, it's just really hard to integrate those protections. And so I say check those first, right? Because like, you know, there's bound to be bugs there. So yeah.
Justin Gardner (@rhynorater) (13:06.223)
Yeah, absolutely. So moving a little bit away from the JavaScript stuff, because we may have bored our listeners over the past six weeks on JavaScript stuff, because I don't know, I've just been loving that shit lately. But the Bug Bunny 100's methodology has sort of evolved a little bit, like you mentioned, and now there's a bunch of different aspects of it, including Red Team related stuff that I think you've broken out a little bit.
Jason Haddix (13:14.052)
No, I'll get it.
Justin Gardner (@rhynorater) (13:33.011)
And then we, like we were talking about right before we went live, there's this aspect of some of these more red team techniques sort of sneaking their way into bug bounties nowadays. So you want to talk a little bit about that.
Jason Haddix (13:41.969)
Yeah. So I try to keep the class at two days. And so what happened was I had a whole bunch of red team content in there from my last year and a half of like diving headfirst into modern red teaming. And so I had to spin it out. And I still give it right now in the format it's into the students as like an extra section after time is over for the two days. But, you know, that section is a lot more for
Justin Gardner (@rhynorater) (13:56.067)
Mm.
Jason Haddix (14:10.489)
you know, hardcore red teamers, all my tips and tricks. And so we talk a lot about things like, you know, what is the best stage zero C2, right? So like we use, you know, we use an implant, right? As our stage zero, you know, implant. And then we have a stage two implant. And then we talk about like modern techniques for internals. Yeah. Okay. So there is this whole community of security, right? I mean, like we're very used to bug bounty and AppSec. That's where I lived for many years. But when I started,
Justin Gardner (@rhynorater) (14:28.719)
Can you define those terms for me? I'm not as familiar with Red Team stuff, yeah.
Joel Margolis (teknogeek) (14:31.574)
Yeah, I'm not familiar with it either.
Jason Haddix (14:40.501)
testing, I was a red teamer. And back then red teaming meant Metasploit, you guys know about Metasploit, right? Yeah, yeah, cool. Yeah, yeah, absolutely. So, I mean, so like Metasploit was kind of our first, like in Cobalt Strike, we're two of our first C2s, command and control servers. Right? And so the idea is, you know, in red teaming, you get initial access, and you can get initial access through a whole bunch of ways. You can fish somebody.
Joel Margolis (teknogeek) (14:47.126)
Yeah. Yeah, yeah.
Justin Gardner (@rhynorater) (14:48.247)
Yeah, of course.
Justin Gardner (@rhynorater) (14:57.283)
Mm.
Jason Haddix (15:06.173)
Which you know you can fish them for credentials or you can fish them to have them open a file and actually get a shell on their box And so back in the day You know like this wasn't super hard like red teaming like you could you could win on a very simple technique of sending a VBA Macro or something to somebody and them opening it and then you were you were good But modern red teaming is much more complex because people have EDR software on their box. There's way more email filtering There's just a ton of stuff
Justin Gardner (@rhynorater) (15:12.525)
Mm-hmm.
Jason Haddix (15:36.045)
that is meant to stop actual adversaries, of which if you are a red teamer, you're trying to emulate an adversary. So the first thing you have to decide on when you're building your red team practice is kind of your implant, which is once you exploit someone, how are you gonna gain control over memory on their box? And then pivot from there, and then make sure that if...
some security software or a reboot happens that you can regain your connection. And that's the goal of what a C2 server is, is a command and control server. And so you have to choose, and there's hundreds of C2 servers, open source, closed source, paid, not paid. And so we talk about the decisions that I made in building my Red Team infrastructure over the past year and a half. And so one of the things we used as a stage zero implant was NIMPLAN.
Justin Gardner (@rhynorater) (16:06.022)
Mm. Right.
Justin Gardner (@rhynorater) (16:22.743)
Mm.
Jason Haddix (16:27.881)
which is the NIM programming language, which is harder for EDRs to parse and basically report on. And so NIMplant was our stage zero. But as far as like a usability tool, like you can build the implant and then a user, you know, you fish them or they click on a file or something like that. And then you have control over their box. As far as usability, NIMplant leaves a little bit to desired. So then you have a stage.
Instead of a stage zero, you have a stage one, which is a better, more modern framework where I can log into a web GUI and control somebody's computer. And so we use a different C2 after an implant. Yeah.
Justin Gardner (@rhynorater) (17:00.119)
Mm.
Justin Gardner (@rhynorater) (17:03.251)
And how are you sneaking that? So, okay, so let me get this. Because this is bringing me back to my consulting days a little bit. Man, I wasn't Red Team stuff, but I was doing pen testing stuff, and we absolutely were using Cobalt on a regular basis and that sort of thing. And the go-to approach, I guess, back then was, having some sort of obfuscation tool that would, or writing something custom yourself to get the connection back to Cobalt. But so, you land on the machine with an implant.
Jason Haddix (17:10.301)
Yeah, yeah, yeah.
Yeah, for sure.
Jason Haddix (17:26.077)
Yes. Yeah.
Justin Gardner (@rhynorater) (17:32.387)
And then you're escalating, you're using a different, I guess, sort of C2 to have more interactability of sorts. How are you sneaking that second payload past whatever restrictions are on the device?
Jason Haddix (17:32.416)
Mm-hmm.
Jason Haddix (17:47.769)
Usually DLL sideloading is the main thing. So we went from NIMPLANT as stage one, C2, to SHARP C2 as our stage two. And so you can sideload in an executable straight into memory, you can sideload in a DLL, you can run custom PowerShell, you can run custom ShellScript, you know, like if you're on a Linux box. The only problem with SHARP C2 was that it had ended up on some...
Justin Gardner (@rhynorater) (17:49.966)
Okay.
Jason Haddix (18:15.413)
like anti-virus lists, and they have basically blacklisted the function names in the implants. And so you have to do this intermediate step using this tool called Confuser EX, which will statically rename all of the function names to different words, and then you can bypass most EDR these days.
Justin Gardner (@rhynorater) (18:21.903)
Yeah.
Justin Gardner (@rhynorater) (18:33.854)
It's so funny. That was the case back in the day as well when I was doing it. You just go in there and you open it up and just change the name a little bit and then it's crazy man. The stuff that they're detecting off of for that.
Jason Haddix (18:41.065)
Yeah, yeah, yeah.
It's really interesting because like... Go ahead, Joel, sorry.
Joel Margolis (teknogeek) (18:47.502)
It's always super interesting. I was going to say it's always super interesting how there's these huge suites of tools that exist for every sort of section within security. I haven't heard of any of these except for Metasploit and Cobaltstrike. It's really interesting to hear about all these very specific sort of tools that exist for these specific problems that are common, but in bug bounty, we never have to deal with this.
Justin Gardner (@rhynorater) (19:00.428)
Mm.
Justin Gardner (@rhynorater) (19:12.39)
Hmm.
Joel Margolis (teknogeek) (19:15.106)
detected by antivirus, it's just not something that's ever been on our radar.
Justin Gardner (@rhynorater) (19:18.683)
Mm.
Jason Haddix (19:19.621)
Yeah, it is very much a discipline that could consume your whole brain, honestly. And I wouldn't even consider myself an expert in it, right. But I knew enough to get going. And so, yeah, so in the red team section, we basically spun it out into its own course now. And one of the things that I see, you know, I saw as like actually a defender, or I guess you want to call me a defender, like I see so, right. During my life was that
What you get in a red team or a pen test report these days is good, it's valuable. There's often very good findings in there, but it doesn't actually mimic what real adversaries are doing in the lattice of, or the order of operations they go through to try to breach you, which has a lot to do with just buying creds on the dark web and the underground, which Jocelyn or Justin and I were talking about before the show. So if we wanna dive in, we can.
Justin Gardner (@rhynorater) (20:15.715)
Yeah, dude. Yeah, no, 100%. That's what I was gonna ask you is like, I'm wondering, you know, we talked about some of this red teaming stuff. How can we apply those things that we've learned from red teaming to bug bounty in a way that they're gonna get accepted?
Jason Haddix (20:31.857)
Yeah, so one of the things is like kind of threat Intel based findings, right? And there are a couple hunters that exist out there that have been doing this for a long time. But it's not really a talked about topic, I would say a ton. And this is the idea of finding working domain credentials on the dark web and submitting them as a bug bounty report. And so in my experience, it's about 50 50. So the first thing is you have to make sure that the cred is actually working. Right. So it's a domain credential.
Justin Gardner (@rhynorater) (20:50.84)
Mm.
Jason Haddix (21:01.853)
you're able to get access to somebody's domain account, possibly through a VPN concentrator, a piece of IT infrastructure hardware, or just into their Microsoft Azure account or something like that as a, you know, as a kind of like a, to prove that you have data that's private and access that is, you know, a big deal. And so, and so you have things like have I been pwned, and then some like middling stuff like de-hashed, which is a service that is like have I been pwned, but.
basically gives you the actual passwords. And that would be like what I call level one threat Intel data and is most often already patched. By the time it gets to those services, like have I been pwned and de-hashed? I mean, not patched, but the credentials have been changed, the cookies have been rolled, all that kind of stuff already. So it's not really gonna be that useful for you. But they're like a level two, three, and four of how deep you get into the dark web forum ecosystem.
and more these days the Telegram ecosystem of people selling credentials that are fresh. And so like five out of my last six Red Team engagements, I have gotten from gathering these credentials from Telegram and Discord channels of actual threat actors. Yeah.
Justin Gardner (@rhynorater) (22:12.431)
Dang, dude, that's nuts. So, and then, and are you buying the credentials, just the credentials you need in that scenario? Or do you end up having to like drop a grand on like some sort of, you know, big giant list of credentials?
Jason Haddix (22:27.957)
So it depends on how the seller sells them, right? So basically what happens is these adversaries are career fishers or career operators of cred stealer malware. So like Redline or something like that is a very common one that people get hit with. And so they're not actively targeting a company, they're targeting large swaths of people in their campaigns and then they sell the result of their campaigns. In order to prove to the community that the campaign is fresh
Justin Gardner (@rhynorater) (22:35.82)
Mm.
Jason Haddix (22:58.109)
and has good data in it, they give what's called a Steeler log sample in the telegrams. And so you can parse those and look for your company or your bug bounty targets domain inside of the Steeler log samples. And then if you see them, you can download them and grab the creds out of those samples.
Justin Gardner (@rhynorater) (23:18.763)
Are these like a standardized format that they're dumping these in Telegram? Or are you just kind of grepping through and just do like, you know, anything that says, you know, dot uber dot com, you know, or whatever.
Jason Haddix (23:25.587)
Yeah.
Jason Haddix (23:29.329)
it's not standardized all the time because every different info stealer malware formats the output of what they take from your machine differently. So like, you know, sometimes there's a passwords file. Sometimes there's like, you know, it's embedded like three folders deep and not called passwords. The format's different and how they pull it from the browser. But when you buy the packs, I think your original question was, if you can go to the forum ecosystem and find a cred for, let's say we're doing Uber, right? Or something like that, right? Like,
Justin Gardner (@rhynorater) (23:37.179)
Mm.
Jason Haddix (23:57.173)
So you can look for at uber.com and some of the forum ecosystems and immediately buy those. And you can buy them for as cheap as $10 for a cred. Now, there is no warranty on the forums as to if that's going to be valid or fresh. But that is a lot of times what happens. And then...
Justin Gardner (@rhynorater) (24:16.517)
What about like 2FA in that scenario?
Jason Haddix (24:19.025)
Yeah, so a lot of times you'll get 2FA, but what happens with these cred stealer malware, like Redline, they don't just steal your creds, they steal your cookies too, which, because they're system level malware, right? So they steal your cookies right out of Chrome, right out of Brave. And so if I can find the domain in the cred on the forum or either in an InfoStealer, I know that they'll have the cookie too somewhere for the browser. And so then I can grab the cookie and just inject it and walk in. And so that is...
Justin Gardner (@rhynorater) (24:22.754)
Mm.
Justin Gardner (@rhynorater) (24:32.079)
Savage.
Justin Gardner (@rhynorater) (24:42.993)
Mm.
Jason Haddix (24:48.873)
That is the beauty and the beauty and the hate of kind of the underground, right? It's like they're starting to sell more than just cookies. I mean, they dump so much stuff with the cred stealer malware now, it's pretty insane. But yeah, so, and then there's other people who sell large packs and you look at the preview and those can be thousands of dollars. And so it just depends on, you know, if I'm on our red team engagement, how much did they pay me for the red team engagement? How much am I willing to invest in success, right?
Justin Gardner (@rhynorater) (24:53.826)
Mm.
Justin Gardner (@rhynorater) (25:16.855)
Yeah.
Jason Haddix (25:18.353)
There's also some legality around it too, like buying stuff off the dark web. So...
Justin Gardner (@rhynorater) (25:21.407)
Yeah, I was gonna ask about that. I mean, is there like some, is there legal considerations or I don't know, maybe even ethical considerations with like, all right, I'm sort of funding this industry and or giving money to career criminals, you know?
Jason Haddix (25:37.073)
Yeah, no, it's been a topic of debate on Twitter, I think. So when we go into a Red Team engagement, we talk to our client about this, right? It's not like we just do this off the bat and we're like, hey, these are your credentials and your employees. So you can give us explicit access to access, you know, the StealerLog data. And we ask them if they would like us to purchase it or get an intermediary to purchase it. So that's one conversation you have to have. The other one is usually we don't have to have
Justin Gardner (@rhynorater) (25:46.913)
Mm.
Jason Haddix (26:07.017)
conversation with them and in a bug bounty hunt too, is if they're already being disclosed in the preview that's free, right? So if you find a cred in the preview that's free, that's already on the public internet basically. And so those we just we tend to just grab and tell them about and use them in the operation or report to the bug bounty. So yeah.
Justin Gardner (@rhynorater) (26:25.682)
Mmm, yeah.
Joel Margolis (teknogeek) (26:26.734)
Are you indexing them at all or is it just sort of case by case? Because I've had some discussions and I have a little bit of a pipe dream at some point where I'd like to download basically every public breach dump that I can. And there are some tools out there. One that I found that's pretty good that's by SensePost is called Frac. It's designed for indexing breach data.
Jason Haddix (26:45.193)
Yep, Frac is amazing. Yep.
Joel Margolis (teknogeek) (26:49.182)
Um, and searching it in like an efficient way, but do you do any of that with the data that you're buying or do you do that any path like passively for other engagements and stuff or.
Jason Haddix (26:58.757)
Yeah, so I cheat a little bit. So when I was at, when I was at BottoBot building the Red Team capabilities, I bought a company, like I did a ton of research into companies that did this already, right? They indexed and they crawled and they could get to levels that I didn't want to put in the work to do, right? Like you have to have sock puppet accounts sometimes to get to this level of data. Like you have to be very careful about your OPSEC. And so there's several companies that are at this like level three cred.
Justin Gardner (@rhynorater) (27:22.843)
Hmm.
Jason Haddix (27:26.865)
intelligence exposure level. And so I bought Flair when I was at, when I was at Botto. And so I found them to be the best. And so I used them in my Red Team ops. And then luckily, when I left, they were like, Hey, we really enjoyed that relationship. Would you like to come be part of our company? And so like full disclosure, I'm like, I'm not shilling. Yeah, I'm not shilling Flair because, because I work, I like work there part time. I'm shilling them because they're fucking amazing. So yeah.
Justin Gardner (@rhynorater) (27:44.003)
Heck yeah.
Justin Gardner (@rhynorater) (27:52.167)
Yeah, I love that order, you know, of like, I love the product, I use the product, and now I work for the product.
Jason Haddix (27:57.245)
Yeah. Well, I mean, it's like you guys with Kaido, right? I mean, it's amazing. Yeah. So Flare has a database and a very SIM-style dashboard to look up all the dark web creds and grab them. And so I used Flare pretty much exclusively. But it is a product that you have to buy. And so in the course,
Justin Gardner (@rhynorater) (28:01.579)
Yeah. Yep.
Jason Haddix (28:24.409)
I talk about some repos that have lists of all of the underground Telegram channels, all of the underground forums, and a whole bunch of torrents for the latest public breaches so that you can build your own. And so let's see, we can give that to the listeners for free. Let's see here. Yeah.
Justin Gardner (@rhynorater) (28:32.179)
Mm, mm.
Justin Gardner (@rhynorater) (28:36.865)
Mm.
Justin Gardner (@rhynorater) (28:41.867)
Yeah, that'd be great if you could drop it in the Discord or in the Google Doc. That'd be really cool.
Jason Haddix (28:47.953)
Yeah, yeah, absolutely. So one is, let's see here. You guys are having me pull up my notes, my bad. Oh yeah.
Justin Gardner (@rhynorater) (28:58.275)
Yeah, it's a lot of work to get all of that data all cleaned as well. So it's tricky and I wonder, I imagine that service has got to be pretty expensive. I've never seen it.
Jason Haddix (29:02.398)
Yeah.
Jason Haddix (29:12.865)
So I don't think they were thinking of doing it, but me and a couple of other Red Team consultancies saw, they were originally marketing towards CTI teams, right? And businesses. But then I think me and a couple of other very forward thinking Red Teaming companies came to them and were like, well, yo, we would pay less than kind of what your enterprise offering was, but we'd use this in our Red Team engagements. Do you have a model for that? And I think what was really cool was that most-
Justin Gardner (@rhynorater) (29:15.48)
Hmm.
Justin Gardner (@rhynorater) (29:19.224)
Hmm.
Jason Haddix (29:39.837)
CTI companies would be like, no, F off. That's not something we wanna do. And Flare was like, yeah, sure, let's figure it out. And so there is like a pen test shop slash red team shop package that Flare does. And it's still not like trivial, but it is less than kind of an enterprise deployment. So.
Justin Gardner (@rhynorater) (29:51.392)
Oh really? That's really cool. Huh.
Justin Gardner (@rhynorater) (29:58.159)
Does it, do they have their stuff public for the prices on that? I wanna look it up right now.
Jason Haddix (30:04.101)
I am very, very new and I am only part-time. So I'm still learning about the code. Yeah, yeah, we'll go look at.
Justin Gardner (@rhynorater) (30:07.655)
Okay, gotcha. We'll have to go suss that and we'll link it down in the description afterwards. But I could definitely see that being a worthwhile investment. And I have heard of some, bringing this back to Bug Bounty, I have heard of some people in the community absolutely crushing it with the credential thing. Like, I don't know, man. That feels a little bit, it feels a little bit meh to me from a bounty perspective. But like you were saying before the call, if you've, at the end of the day, Bug Bounty...
Jason Haddix (30:24.213)
Yeah.
Justin Gardner (@rhynorater) (30:37.359)
can be reduced down to POC or GTFO. And if at the end of the day, you're logging in and you're leaking data, that's bad and that needs to be fixed.
Jason Haddix (30:40.138)
Right.
Jason Haddix (30:45.213)
Yeah, I think from a customer point of view also, I mean, I think that it's something I talk about in the class I've ranted on a lot this weekend, was you have to think about it from a customer point of view too, right? So like, let's say that a bug hunter comes to you and says that they have a working domain credentials that walked into your VPN, right? And you didn't explicitly put that stuff out of scope. What are you gonna do? Are you gonna out of scope that dude who could blog about it and say,
Justin Gardner (@rhynorater) (30:52.406)
Mm.
Justin Gardner (@rhynorater) (31:06.225)
Mm.
Jason Haddix (31:15.229)
you know, Uber and like just like an intent. No, you're not. You're gonna pay the P1 and be like, cool, thanks for the report. And either you're gonna decide that something you wanna continue to do and put it in scope. It's gonna be something that's a gray area. You're still not gonna put it in scope but maybe accept it, you know, piece by piece. Or you're gonna take it explicitly out of scope for the bounty. And so, you know, when bug bounty platforms like Hacker One and Bug Crowd and Integrity.
Justin Gardner (@rhynorater) (31:17.106)
Yeah.
Justin Gardner (@rhynorater) (31:34.54)
Mm.
Jason Haddix (31:41.309)
you know, they start a bounty with people. These are the types of things that they talk about in the kickoff calls. Like, do you want creds coming in? Do you care about creds that are leaked on GitHub? You know, and so they have a very defined list of out of scope things. And you'll notice that sometimes people explicitly nuke things from that list because they don't want to explicitly point the bug hunters at these types of loans, but they also want to be able to take it in case it happens because it's a risk, right? It's a, you know, a risk of some sort. So bug bounty sometimes is a game.
and you have to figure out how to play the game.
Justin Gardner (@rhynorater) (32:12.555)
Yeah, there's such a sweet spot there of like, it's not in the out of scope so they accept it but it's not in the in scope so that they incentivize looking for it. And that, if you can find that margin of stuff that's not obvious to other hunters but you know the client is vulnerable to and willing to pay for, that's some money right there. That is some serious money.
Jason Haddix (32:17.729)
Mm-hmm.
Jason Haddix (32:22.578)
Yeah.
Jason Haddix (32:33.649)
Yeah, I mean, you're like greenfield hunting at that point, right? So, yeah, I know of at least three hunters who do this kind of cred intel hunting, like almost exclusively, and they don't talk about it very much. Yeah, yeah.
Justin Gardner (@rhynorater) (32:36.875)
Yeah, 100%.
Justin Gardner (@rhynorater) (32:47.339)
Yeah, and I wonder why. I wonder why they don't talk about it very much.
Joel Margolis (teknogeek) (32:49.314)
Yeah.
Yeah, I think most a lot of programs don't count credential stuffing as like a valid bonus. Even like, even if it is like one set of creds, I think a lot of programs will say, well, this there's not a systemic issue here. This is like at most some sort of like misconfiguration or a gap or like a lapse in sort of like the external monitoring side more than like, there's nothing to fix, right? There's no code change or anything like that. So much as like maybe a policy change. And so I think a lot of programs just say, you know, thanks for letting us know. But, you know,
we're not gonna pay for one set of creds. But I guess it really depends on, like if you can do, if you could show a lot of impact, then I think depending on the program, some will accept them, some won't.
Justin Gardner (@rhynorater) (33:25.883)
Mm.
Jason Haddix (33:33.705)
So if you find a domain credential and you're working against a modern organization that has a VPN login and the creds work on a VPN login because they're tied up to the Azure authentication system, then I found most people are like, yeah, we'll pay for that. It really also depends on the history of security of the company itself I found. So.
Justin Gardner (@rhynorater) (33:57.241)
Yeah.
Jason Haddix (33:57.397)
Companies that have been breached by credentials being leaked online really fucking care about it, right? And company... Yeah, yeah, yeah. And then companies who are, who have never had a breach via this method, they don't care. And you know what? I mean, I think that that's fair. I also think that they haven't been through that pain and understand it. And I think that, you know, if they do get hit with it eventually, guess what's going on in scope? And guess what?
Joel Margolis (teknogeek) (34:03.075)
You know, we keep mentioning Uber here. I don't know.
Justin Gardner (@rhynorater) (34:05.654)
Oh no.
Joel Margolis (teknogeek) (34:09.91)
Hahaha
Justin Gardner (@rhynorater) (34:25.825)
Mm.
Jason Haddix (34:27.133)
I mean, I don't want to leak anything crazy, but there are at least two programs either existing right now or being designed right now for specifically threat intel data in three of the bounty platforms. Or so there's two and two of the bounty platforms right now. And so this is about to be a thing. Like this is about to be a new skillset that maybe we do have to develop. And some people have been doing it for a while and other people will have to figure it out. Yeah.
Justin Gardner (@rhynorater) (34:52.975)
Dang, man. That's gonna be intense. Get your telegram scraper scripts ready because that's definitely gonna be the direction that the industry's moving in, I think. So I guess we'll leave that topic there, and I wanna move into some of the bounty hunting you've been doing lately. I've seen you hitting FIS, and I've heard from other hackers that FIS is a great target. Can you give me the rundown of that?
Jason Haddix (34:58.709)
Yeah.
Jason Haddix (35:21.125)
Yeah, so I mean really was inspired by you guys, I think, doing the Discord community, like the kind of like access only or paid only version of the Discord community. And you know what I'm good at is like I'm good at a lot of things, but I'm really good at Recon. I'm you know, I don't have any qualms in like saying that I'm you know, like that is one of my specialties. And so my idea was that as part of the Discord subscription, every month we would
Justin Gardner (@rhynorater) (35:37.292)
Mm.
Jason Haddix (35:50.545)
And I would kick off doing recon with, you know, like I have all of the paid services, I subscribe to them all. I'm doing all of the methods in the presentation, which are the extra 10 to 15% stuff that nobody else will do. And I'm finding Apex domains that nobody else has found. And so we did FIS this last month. It was our first or second target, I believe, as part of the discord. And so a whole bunch of hunters found bugs.
Justin Gardner (@rhynorater) (35:57.557)
Mm-mm.
Jason Haddix (36:19.813)
I found some bugs, recon-based bugs basically. But we also learned a lot about enterprise deployments of security products and kind of changing our recon methodology a little bit, depending on what was going on. And we implemented some really cool new stuff that went directly into the class that students had dropped in the channel for things that had successfully found them hosts. And yeah, so we learned a lot. One of the first things we learned was about like,
Justin Gardner (@rhynorater) (36:33.379)
Mm.
Jason Haddix (36:50.462)
an aggressive deployment of a cloud protection like Akamai or something like that. Yeah, for sure, yeah. And so what we found was, we were doing recon for FIS, we were getting 403 all over the place with the Akamai and Cloudflare basically inline WAFs. And so what we learned was,
Justin Gardner (@rhynorater) (36:55.823)
Yeah, dude, that's such a pain in the ass.
Jason Haddix (37:16.929)
There were a whole swath of their sites, their websites, which eventually we wanted to get to, right? Where they just blacklisted VPS providers. Like they blacklisted DigitalOcean, they blacklisted AWS. Their whole thinking was that, hey, yeah, hey, if you're using this website, you should be a user only and you should be coming from a residential IP block. Yeah.
Justin Gardner (@rhynorater) (37:29.339)
Classic.
Justin Gardner (@rhynorater) (37:36.491)
It's not a bad idea, you know, it's, there's no reason why these VPS is, should be actually hitting your website.
Jason Haddix (37:43.869)
Yeah, I mean, unless you have some kind of integration where, you know, a partner of yours is coming off of one of those things, that's why it's a little bit more dangerous with like Amazon, but they definitely blacklisted the hell out of DigitalOcean for sure. And so, and so, you know, that's my preferred VPS. So I noticed right away, I was like, why is this happening? And, you know, I'm working with a couple other people doing the initial recon. And it's like, OK, well, they're blacklisting anything but residential IP addresses. So then I had to take my whole recon.
Justin Gardner (@rhynorater) (37:54.671)
Mm-mm.
Jason Haddix (38:12.593)
like set up and move it to a Kali box in a, you know, in a virtual machine on my home network, right? So that I could hit a lot of these sites. And then I'll, I do.
Justin Gardner (@rhynorater) (38:21.531)
Oh man. Jeez, I hope you got a strong internet connection. I hope you got Joel's, you know, what was it, 10 gigabit? That was crazy.
Jason Haddix (38:28.229)
Yeah, yeah, yeah. So I mean, I have good internet, so it was fine, but then you run the risk of blacklisting your house if you send attack traffic or if you send too many requests in a row. So it was really trial and error. And so eventually what we had to get to was using HTPX to pull down the data that we wanted. We had to like run from my house and we had to rotate user agent strings and we had to bring it down to 15 threads. Yeah.
Joel Margolis (teknogeek) (38:28.631)
Yeah.
Justin Gardner (@rhynorater) (38:37.846)
Mm-hmm.
Justin Gardner (@rhynorater) (38:55.603)
No way dude, that's so much. Holy crap.
Jason Haddix (38:59.089)
So it took us a week and a half to do a full scan across all of their Apex domains. And so that was just like the learnings that we've had. And then we've moved into next month, we're gonna do T-Mobile. And they have some of the same configuration. So luckily we learned these things early. And so that led us into like these discussions of like, okay, well, like how can we do this without being that ghetto, right? Just like bringing it back into my home.
Justin Gardner (@rhynorater) (39:16.013)
No.
Jason Haddix (39:25.741)
And so we're looking in right now into, I just finished my interview with them a week and a half ago, is Bright Data. Have you guys ever heard of Bright Data before?
Justin Gardner (@rhynorater) (39:32.795)
I've not heard of this now.
Jason Haddix (39:34.705)
Okay, so Bright Data is a company that basically is a proxy company for people who want to do web scraping. And so they have several different types of networks that you can subscribe to, and you basically set up this proxy agent, and then every request you make through the proxy agent, you set it up the same way you would set up a SOX proxy, goes from a different IP basically. And so they have...
Justin Gardner (@rhynorater) (39:54.663)
Oh nice. It's like that thing in Amazon, what is it? The Firefox, whatever Firefox uses on the back end, I think it's like API Gateway or something like that.
Jason Haddix (39:59.469)
Uh...
Joel Margolis (teknogeek) (40:00.523)
at fireprox.
Jason Haddix (40:01.981)
Yeah, fireprox, yeah.
Yeah, it's API gateway. Yeah. So a lot of API gateway stuff has been even blacklisted now. So bright data pays people to run their agent on their local computer, kind of like we used to do folding at home and stuff like that and SETI. And so, yes. Yeah. So, um, so I just got approved. And so.
Joel Margolis (teknogeek) (40:05.558)
Yep.
Justin Gardner (@rhynorater) (40:11.223)
I'm sure.
Justin Gardner (@rhynorater) (40:18.531)
Oh dude, that's sketchy man. I don't know. It's like botnet as a service or something like that.
Joel Margolis (teknogeek) (40:21.694)
This is mega sketchy because there's a lot of companies who do this stuff for like Instagram proxies that are not asking you to. Yeah.
Jason Haddix (40:33.873)
to you have to you have to take an interview with them and now I can proxy all my recon scans through bright data's Their residential proxy. So these are people's homes. So for this kind of protection FIS will not be able to block me anymore And they will be able to they were like, you know I'm not I'm not scanning them with like nuclear or anything like that I am literally just doing the same kind of work that scrapers do so I want to pull back with HTTPS. I want to pull back
Justin Gardner (@rhynorater) (40:34.497)
Mm. Ha ha ha! I love that.
Justin Gardner (@rhynorater) (40:57.175)
Yeah. Mm.
Jason Haddix (40:59.593)
The ASN that it resolves to, the title is really important. The FabCon hash, yeah. I'm not doing screenshotting in my first round of Recon. I usually do it like in my second round, but yeah, I mean, mostly I'm just pulling metadata about the site, yeah, to get us kicked off. But yeah, that's a thing that we're doing.
Justin Gardner (@rhynorater) (41:06.062)
Mm. Are you doing screen chatting?
Justin Gardner (@rhynorater) (41:11.687)
Mm.
Mm.
Justin Gardner (@rhynorater) (41:19.235)
Dude, I think that's so funny, man. We were like, well, this is super sketchy. Then Jason's like, anyway, I signed up. And so, no, that's cool, man. And I think, you know, it's a valid way to get around it. You know, these poor people are gonna have Akamai bands on their houses, but that's what you get when you sign up for Bright Data or whatever to be an agent.
Jason Haddix (41:27.249)
Yep. Yeah.
Joel Margolis (teknogeek) (41:28.28)
Hahaha
Jason Haddix (41:39.904)
Yeah.
Jason Haddix (41:43.729)
Well, I think that rotation happens every request, right? And so the network's big enough so that I don't think that individuals will get banned by the cloud proxies if we keep the threads down. I don't plan to blast like, you know, like 6,000 threads or something like that. Right. I'm yeah. Yeah.
Justin Gardner (@rhynorater) (41:47.668)
Oh, that's good. Yeah.
Justin Gardner (@rhynorater) (41:57.884)
Mm. Yeah. And hopefully they're rate-limiting that too from their side because those proxy endpoints are their, that's their thing. So I imagine they're doing something like that from that level. But it's definitely a creative way to get around the restrictions that you run into if API Gateway is out of the question.
Jason Haddix (42:07.185)
Yeah.
Jason Haddix (42:17.117)
Yeah, yeah, for sure. So yeah, that was some of the stuff that kind of came out of that, shook out of that. We also realized that they were geolocating or locking like regionalized domains. So like, you know, FIS.UK or FIS.JP or something like that. You couldn't get to them without being from one of those. So we've started to build some stuff that can get around that, like some, you know, mostly SOX proxy, just stuff to get around that. We don't have to do the bright data stuff for that.
Justin Gardner (@rhynorater) (42:32.576)
Yeah, yeah.
Justin Gardner (@rhynorater) (42:41.613)
Mm.
Yeah, get some like little localized provider or something like that, spin up a little VPS over there, some data center in, you know, yeah, that makes sense. No, that's good, and this is the kind of thing we talk about all the time on the pod, which is like, if you go that extra mile to get access to something that's really, really annoying to get access to or costs any amount of money at all, there's so much less competition in there from a bug bounty perspective. And so, getting...
Jason Haddix (42:46.749)
Yeah, exactly. Yeah, yeah. Yeah.
Jason Haddix (43:09.949)
Yeah.
Justin Gardner (@rhynorater) (43:13.619)
access to geolocation stuff, getting access to, or even like attacking targets. I have a mentee that I'm working with and I told him, you know, you should attack programs that you have access to from your, from your country that only you can sign up for. So he went after this banking app.
and he's found a bunch of really awesome bugs there. And it's because no one else has the, you know, whatever government registration number that you need to sign up and actually get an account on this thing and actually pop it. So, geolocated stuff is huge.
Jason Haddix (43:46.025)
I think that this is part of a discussion we have in the class I have with the students about going deep. There's a lot of people, including you guys, who talk about going deep. And sometimes people don't talk about, yeah, getting intimate, yeah. But sometimes people are like, just go deep. And you're like, what the fuck does that mean? What do you mean, bro? What do you mean go deep? Yeah, what do I do with my hands? Give me, you know, and so.
Justin Gardner (@rhynorater) (43:55.244)
Mm. We're getting intimate with the application, as we like to call it here. Ha ha ha. Ha.
Justin Gardner (@rhynorater) (44:08.094)
What do I do with my hands?
Joel Margolis (teknogeek) (44:11.362)
Uh-huh-huh-huh-huh.
Jason Haddix (44:13.041)
So one of the things is paid access to paid functions, right? So like, not many bug hunters are willing to pay for some of the pro versions of the applications and get access to the functionality that is, cost the money as an investment and that can be super green field. Geolocation are places that you can only access with certain types of data. So one being like you said, like maybe a regionalized driver's license number, a regionalized social security number, regionalized insurance number, like...
Justin Gardner (@rhynorater) (44:14.806)
Oh man. Right.
Jason Haddix (44:41.589)
hitting those functions that make you have to that. Maybe things that require KYC, right? Like some bug hunters will see KYC and just completely stop and they'll be like, cool, not doing that, you know? Yeah.
Justin Gardner (@rhynorater) (44:47.084)
Mmm, yeah.
Justin Gardner (@rhynorater) (44:53.547)
Yeah, it's scary, man. It's scary. I've put my social security number into multiple websites for bug bounty hunting. And it's like, okay, this is probably not the best idea. But also, you know, the payoff can be huge. And it was funny, a couple events back, this is probably two years ago, I was talking to Ben and Ben had like signed up for a credit card and opened a bank at this account at this company. And I was like, dude, did you really like do all that?
Jason Haddix (44:59.869)
Yeah, yeah, yeah.
Jason Haddix (45:07.925)
Huge.
Justin Gardner (@rhynorater) (45:21.911)
And he's like, yeah, man, like, you know, a lot of people do this for the signing bonus. And I, you know, I'm about to pop like a 10 K crit on this thing. And I'm like, wow, yeah, that's much better than the signing bonuses you get for like opening a bank account somewhere.
Jason Haddix (45:29.49)
Yeah, yeah.
Yep. Yeah. And one of the other things we talk about is like many companies will have a developer program, right? And you get extra access and early access to features, right? So you can join the developer program. You can, you know, depending on what type of site it is, there's, you know, reseller programs that get access to whole portions of the application that nobody, you know, has seen before as a bug bounty hunter. So there's just all these different ways to get extra functionality to hack. So we talk a lot about that in the course. So, you know.
Justin Gardner (@rhynorater) (45:39.319)
Mm.
Justin Gardner (@rhynorater) (45:43.822)
Yeah.
Justin Gardner (@rhynorater) (46:00.231)
Mm, yeah. I think...
using your own position, whatever it is, using your position as an American, using your, you know, position as a developer, using your position as, this makes me think of the episode with Mane's, how he used his understanding of the digital marketing industry to apply that to his hacking and how much success he got out of that with the Yahoo program. So you know, each one of us has our own eyes and we talk about that all the time on the podcast and through Pug Bounty Media, but you kind of got to ask yourself when you're
What do I have advantage on that somebody else doesn't have? And how do I apply that to these targets, these list of targets that I've got to pick one that I have the best chance of popping? It's a big thing right there.
Jason Haddix (46:43.569)
Yeah, yeah, yeah. So I mean, there's a whole bunch of other stuff in the doc. I mean, I know we're running pretty long, but you know, like we're good.
Justin Gardner (@rhynorater) (46:51.243)
No, no, we are we are mad dude what I want to see next but I want to see next is these new recon techniques Cuz you cuz like this one says Reverse the D mark and I have not seen this before so hit me with that
Jason Haddix (46:56.242)
Okay, alright.
Mm-hmm. All right, okay. So like some of these are really like new and kind of exclusive to the course, but I love you guys. So I will drop them in the pod. Okay, hold on. All right.
Justin Gardner (@rhynorater) (47:08.703)
Yeah. Ah, dude. Okay. All right. Well, you don't have to go into too much detail, but yeah.
Jason Haddix (47:14.657)
Okay, so the first one is reverse DNS, right? And so the idea of forward DNS lookup, right? Like is very useful. It's like an interview question we get in tech. It's like, how does DNS work, right? Like when you go to google.com, what happens, right? But a lot of times right now, what you can do is you can ask the reverse question of, what domains lie in this IP address? And if they have pointer records, you can find out what Apex domains live on.
Justin Gardner (@rhynorater) (47:17.615)
Mm-hmm.
Justin Gardner (@rhynorater) (47:24.602)
Right.
Justin Gardner (@rhynorater) (47:28.621)
Mm.
Jason Haddix (47:42.353)
and subdomains live on certain IP addresses. And so this can be really useful if you're looking at, if you have the IP address of, you know, an ASN for a target of yours, right? So you know their autonomous system numbers, you know they own this IP space, and you can run pointer record lookups on them. So there's one tool that does this. Yes, this is, yeah, this is the tool that we use. Yeah, Hack RevDNS does this, right? So Hack RevDNS does this. And then a whole bunch of other websites can do this for you through their GUI, right? And so this is one technique that is kind of well known.
Justin Gardner (@rhynorater) (47:52.176)
Mm.
Justin Gardner (@rhynorater) (47:57.131)
Yeah, I think Hack Luke has a tool, like Hack Rev DNS or something.
Justin Gardner (@rhynorater) (48:08.791)
Mm.
Jason Haddix (48:12.085)
But it works really well. It works amazing to find subdomains that are kind of hidden in these pointer records. And so that's a cool thing to find. The more newer kind of edge case is that, is that, you know, there's a lot of companies like Security Trails and Netlist, and these companies that are spiders of infrastructure on the internet, like Shodan and things like that, right? And so these companies who sell, you know,
Justin Gardner (@rhynorater) (48:19.608)
Mmm.
Justin Gardner (@rhynorater) (48:35.981)
Mm-hmm.
Jason Haddix (48:40.129)
To me, it's recon data. To other people, I don't know what they do with it, but to me, it's recon data. For these, yeah, yeah. So for these companies that sell that, they all have different issues that they play in. And so one is, whoisxml API is a pretty well-known one. It's a website that you can go to and they are for pay. And so I have to invest a little bit into doing this technique. But what they have the ability to do is a reverse name server lookup. And so what we found with FIS was that they're using
Justin Gardner (@rhynorater) (48:44.725)
No, it's absolutely recon data.
Jason Haddix (49:09.945)
Akamai as their name server fronting, right? And so what you can do is they have like six, six name servers if you just dig them right for their name servers. And you're like, cool, these are the Akamai name servers that they're using. Well through is who is XML API and one of the search and one of the API queries you can do with them. So you can say what other servers use this name server? And you would be surprised about the records that come back. It's usually around 6,000 to 8,000 hosts come back using an Akamai.
Justin Gardner (@rhynorater) (49:37.148)
I'd imagine so.
Jason Haddix (49:38.997)
using an Akamai fronted name server, but with some simple grep for knowing your customers, like for FIS, I look for FIS and some of their other Apex domains, using some simple grep through there, we found some Apex domains that we did not find anywhere else. Like literally like 12 Apex domains that belonged to FIS that were being used that no other hunter had found by using this query, yeah. And so we had Greenfield.
Justin Gardner (@rhynorater) (49:45.007)
Hmm.
Justin Gardner (@rhynorater) (50:01.391)
Dude, no way. Wow.
Jason Haddix (50:06.289)
And then you run those through subdomain discovery, right? And then, yeah, and so that was, that reverse DNS technique was really valuable.
Justin Gardner (@rhynorater) (50:13.871)
That's something that I've thought of before, but I would have thought was just too bloated. Like, you know, especially in modern cloud infrastructure, I'm impressed that you even ran it against Akamai related stuff. Cause I would, you know, maybe I would say if we had, if this company had their own, you know, name servers that they have, you know, that they're on metal or something like that. Yeah. But even on Akamai.
Jason Haddix (50:28.817)
Yeah. Oh yeah, it works great for that. Yeah, yeah, it works great for that. But no, I mean, it's all an amount of, so I call some of these methods in the later part of the classes, I call them lossy methods. Like if you think in, like if you're an audio file, right, and you think about audio, sometimes, you know, some types of audio are lossy, right? And they require, you know, a lot of work to, you know, to get gold out of them. But sometimes they pay off.
Justin Gardner (@rhynorater) (50:41.325)
Mm.
Justin Gardner (@rhynorater) (50:55.852)
Yeah.
Jason Haddix (50:57.457)
a lot and so they're all in the course right and so this one is this one you know with some simple gripping honestly for FIS or fidelity or whatever we found some amazing stuff um yeah
Justin Gardner (@rhynorater) (51:08.923)
Dude, when you said audio file, I just, I'm just seeing myself as like, I know, I know, but I'm not an audio file, so. No, no, so I'm seeing myself as like a.mp3 or whatever. I'm like, I'm like, if I'm an audio file, what are you? That's like, the audio files, you know, here on the podcast with me are like, what is wrong with you, Justin? I understand now. I, I'm sure.
Joel Margolis (teknogeek) (51:13.002)
That's a very common term.
Jason Haddix (51:17.048)
Oh, okay, all right, yeah.
Joel Margolis (teknogeek) (51:21.983)
Ehh
Jason Haddix (51:27.506)
Yeah No, no like a person who's really in the who's really into like the fidelity of audio. Yeah Yeah, yeah Yeah, yeah, yeah
Joel Margolis (teknogeek) (51:36.846)
You know, flak, over 320 Kbps. We just gotta throw some terms out.
Justin Gardner (@rhynorater) (51:38.703)
I'm sure Joel knows exactly what lossy, you know, means.
Justin Gardner (@rhynorater) (51:45.584)
Oh my gosh. Mmm. Yeah.
Jason Haddix (51:46.773)
So those are the two reverse DNS methods. And then I put in the doc reverse demark, and this was one that XNL Hacker gave to me. And so, oh dude, XNL is amazing. I don't know if you guys are gonna have him on your show at some point, but he's fantastic. Okay, I'll put in a good word, he's a homie. So reverse demark is another one of these reverse type lookups, and so basically there's a few websites.
Justin Gardner (@rhynorater) (51:56.899)
What a guy.
Mmm. We'd love to, yeah, for sure.
Ha ha.
Jason Haddix (52:16.601)
which categorize all the demark records on the entire internet. They have spiders that go off and grab demark records. And so you can just look at a company and their demark record and see what other domains those are applied to. And so we found some amazing Apex domains from that. And the same thing happens with CSP. So you can look at a CSP policy. There are several websites out there that you can do a reverse lookup on a CSP policy for a domain to see if it matches one-to-one. And then you can find Apex domains.
Justin Gardner (@rhynorater) (52:22.157)
Mm.
Jason Haddix (52:45.521)
or subdomains using that method. So those are three of the newer ones that we've been running.
Justin Gardner (@rhynorater) (52:47.693)
interesting.
I'm looking here at the demark policy. Okay, I see where the domain is defined here. Yeah, that's something that I hadn't really thought of. But man, these sort of fringe techniques, dude, you gotta really just automate them and then blast them all out there, you know, because there could be some crazy stuff that comes back from, you know, reverse demark. And that's the one thing you need to get access to, especially, like you said, things that give you access to Apex domains.
You've identified a new apex domain and then you have a clear set approach to expanding your scope once you have that apex level domain. So that's really valuable. And the CSP thing too, I've seen many times before.
Jason Haddix (53:31.709)
Yeah, yeah. So for the, yeah, yeah. I mean, Corbin actually talked about it on Twitter like last week or something like that. So the DMARC record stuff, we use a site called DMARC.live, if anybody wants to know. That's what we do the reverse DMARC analysis with. And then CSP Recon is a command line tool that you can do for the reverse.
Justin Gardner (@rhynorater) (53:42.512)
Mm.
Justin Gardner (@rhynorater) (53:48.381)
Mm. Okay.
Jason Haddix (54:00.053)
for the reverse CSP lookup.
Justin Gardner (@rhynorater) (54:01.987)
Whoa, dude, check this out. This is, okay, this is a lot more domains than I thought if on demark.ly. I'll put this in the, I'll put this in the doc. This is pretty impressive.
Jason Haddix (54:08.353)
Mm-hmm.
Joel Margolis (teknogeek) (54:11.246)
What was the CSP one again?
Jason Haddix (54:13.444)
Yeah, a CSP tool is called CSP recon.
Justin Gardner (@rhynorater) (54:16.903)
CSP recon. That's cool that there's something that sort of does it automatically because what I've kind of done in the past is just grepped through all of my history files and pulled out every content security policy header that I've seen and kind of grepped against those, but something to automatically parse that is really valuable, I think.
Jason Haddix (54:32.949)
Yeah.
Jason Haddix (54:37.533)
And so I think you hit on one thing that I want to respectfully disagree with. And I feel like it's, it's a competitive disadvantage is that, is that you were like, well, I need to just automate this stuff to get it out of the way. Right. And that has not been my experience at all. When I manually understand the naming conventions for a company or an apex, and I go through these websites or these tools manually, I find way more stuff than if I'm just trying to automate it and get it off my plate. I care more about these esoteric techniques that nobody else is doing. And.
Justin Gardner (@rhynorater) (54:41.599)
Mm. Mm, mm, hit me.
Justin Gardner (@rhynorater) (54:48.381)
Mm.
Justin Gardner (@rhynorater) (54:59.463)
Mmm.
Jason Haddix (55:07.785)
by caring more and putting more time into looking at maybe 6,000 direct domains, even though I'm eyeballing it, or I'm using like ghetto search or some easy regex, I find more than if I were just to dump it in a file on my command line and try to like automatically do it. Yeah, it's part of understanding. Yeah, it's part of understanding your target, right? And so that's where I succeed where a lot of other bug hunters.
Justin Gardner (@rhynorater) (55:17.356)
Mmm.
Justin Gardner (@rhynorater) (55:22.263)
and just grip it. Okay, so you're using the eyes there.
Jason Haddix (55:32.609)
They really want to get out of recon really fast. And I know we talked about this in the last episode, right? Like recon is defined stuff to hack, right? That is the purpose of recon. Recon doesn't find you bugs. But also if I can find that last 10% of attack surface and it's greenfield, it makes hacking way easier for me. Yeah.
Justin Gardner (@rhynorater) (55:34.788)
Mm.
Justin Gardner (@rhynorater) (55:39.725)
Mm.
Justin Gardner (@rhynorater) (55:49.491)
such a balancing act, man, because I've really leveraged, even Joel just asked me earlier today, what are you using for word lists on this target or whatever, and I'm like, man, I don't run any brute force on this target that I've been looking at for the past three weeks, because it's just not as much on my radar nowadays. But I really do think that there is, especially with what you said in the beginning, which is a lot of data that we need now to do hacking can be found in the JavaScript files on the front end.
Jason Haddix (56:02.505)
Yeah?
Justin Gardner (@rhynorater) (56:19.011)
Which is great, but it's not everything.
And so I think there's definitely a balancing act that needs to be done there. And I think an application for AI could also be implemented here. And not to push against your manual eyes approach to it, because I think that there's a lot of merit to that. And I think that that's something that's big. But also being able to say, okay, here's FIS, here's a bunch of words associated with FIS. And do any of these domains seem tangentially related to FIS?
Jason Haddix (56:29.822)
Yeah.
Jason Haddix (56:50.429)
Oh yeah. Yeah, it's actually a fantastic one. I think out of the class, I think people are always deving code and drafting scripts. I think people talked about this idea too, and they were gonna try to do this. We haven't done it yet, but hey, that's the next section for the next class. It's like, can we achieve what I get with my methodology by just using my eyes and simple search and staying on the websites that offer this data, can we get it from AI, which could be pretty, pretty cool. So yeah, yeah.
Justin Gardner (@rhynorater) (56:50.653)
Great application for LLMs.
Justin Gardner (@rhynorater) (56:55.03)
Mm.
Justin Gardner (@rhynorater) (57:01.079)
Mm.
Justin Gardner (@rhynorater) (57:11.499)
Mm.
Justin Gardner (@rhynorater) (57:18.928)
Have you integrated a lot of AI stuff into your workflow?
Jason Haddix (57:22.982)
Yes, I have. Yes.
Justin Gardner (@rhynorater) (57:25.003)
I figured I'd just toss that one up for you, man. I tossed that one up and just...
Joel Margolis (teknogeek) (57:27.895)
So wait, I want to ask about this now. Because like for me, this is like, this is also like a big area of concern. And like I feel the same way as you were like, I would much rather look through something manually than start to automate it. Because when I start to automate, I have to make generalizations and assumptions. And that just like, either I'll get lost down rabbit holes trying to figure out whether my assumption or generalization is 100% true 100% of the time.
Jason Haddix (57:44.597)
Yeah.
Justin Gardner (@rhynorater) (57:46.808)
Mmm.
Joel Margolis (teknogeek) (57:54.866)
I find that I have to like, just sort of cut that as a loss and be like, okay, I'm going to miss certain things and that's fine. And I don't like either of those. And AI is like the same thing for me where like if I'm using an AI model to like make these determinations, either I have to go back and manually check and be like, did the AI actually do this right? Or I have to say, okay, it might miss stuff and I'm going to miss that. And that's fine. That's an accepted loss. And again, I don't like either of those situations. So how do you sort of balance that?
Jason Haddix (58:03.371)
Mm-hmm.
Justin Gardner (@rhynorater) (58:15.423)
Hmm. Mmm.
Joel Margolis (teknogeek) (58:24.786)
with like not wanting to do so much automation, but also wanting to use AI in your flow.
Jason Haddix (58:29.985)
I think it depends on how much time I feel like I have. Like if I feel like I have a ton of time, like we spend a month on each Apex domain, right? I feel like, and I start two weeks before I give the recon data to the Discord. So I feel like two weeks is a long time to do a recon, right? So I do most of it manually, but if I get a program invite from HackerOne, Integri-T, or BugCrowd, right? And I just need to do quick recon, I will trust and-
Justin Gardner (@rhynorater) (58:44.516)
Mm.
Jason Haddix (58:57.833)
I will trust in some automation. I will not trust in some other automation and I can use AI to kind of make that a little bit quicker judgment. And I accept that I might miss some stuff on like a new program invite usually. So that's kind of where I think I draw the line. But I mean, just in general, right? Like not with parsing data, but in general with AI, I've built some really, really impressive bots to help me just in my hacking in general. And so my bots are...
all the time. They're up right now in my other window to help me with hacking.
Justin Gardner (@rhynorater) (59:28.571)
Hmm. You know, I haven't applied it as much to hacking as I have to other arenas, the AI related stuff. And so, I mean, they excel at sort of breaking down all of this, or, you know, or correlating various concepts even that seemed a little bit tricky to do from like an actual, we had to understand what the words actually mean perspective. What do you, what do you, what?
Jason Haddix (59:35.073)
Mm-hmm.
Justin Gardner (@rhynorater) (59:54.867)
areas have you applied it to beyond like source code analysis and that sort of thing that we hear about all the time.
Jason Haddix (01:00:02.113)
So I built a bot called SecGPT. It's in the GPT store. And really, I learned a whole bunch of prompt engineering tricks from honestly watching YouTube, basically trial and error myself. There's a bunch of really good academic papers on prompt engineering that are out there on several sites. And so I built this bot called SecGPT.
And it's probably one of the best tools, I think, that I've ever made in my life. And I've made a lot of cool hacker tools. Yeah, and so secgpt is loaded. I mean, I feel like any prompt or any LLM bot, the success of it is based around how much context you give the bot, right? And it's system instruction, right? And so a lot of people are using just generic chatgpt, first of all, to try to do some of the security work in one-shot prompts, which is...
Justin Gardner (@rhynorater) (01:00:32.163)
Wow, dude, that is a strong statement.
Jason Haddix (01:00:57.097)
Like it's not going to give you great stuff, right? But if you create your own GPT bot or you use the API, you can get infinitely better output from some of these bots. And so I wrote a two-page system prompt basically on how this bot should act, how it should, what specializations it has. And it's mostly based around AppSec hacking, but also some NetSec stuff. How it should respond in certain scenarios,
Justin Gardner (@rhynorater) (01:01:14.627)
Wow.
Jason Haddix (01:01:26.721)
When a user inputs X, Y, and Z and they want to know about this, I want you to give them fully formed attack strings with different options based on this knowledge. I fed it with a couple of big sections of research that I like around polymorphic payloads and how to basically bypass simple filters, simple regex filters. You can talk to this bot like it were a senior at your pen test company, and it will give you really, really great output.
So I ask, I dump whole JavaScript into it sometimes and I'm like, okay. And in the context of the system instruction is if raw JavaScript gets pasted to you and the user asks about, you know, this JavaScript in any way, shape or form, here's the things you're supposed to do. I want you to dynamically try to build any path thing with, you know, full source domain. I want you to look to see if you recognize any secrets in the patterns. I want you to identify these.
functions that are dangerous within certain JavaScript frameworks. I want you to highlight all of these functions inside of the JavaScript and try to tell me what they're doing. And I want you to give me an overview of what you think the application is doing just from this snippet of code that they've given me. And so it knows exactly what to do and you put some JavaScript in it and it does it really well. And so I also have it build you curl requests for API calls and pull out all the API paths and yeah, it's pretty dope.
Justin Gardner (@rhynorater) (01:02:38.202)
Mm.
Justin Gardner (@rhynorater) (01:02:52.707)
Yeah, that sounds much better than what I'm doing with this sort of thing. And I'll have to give this a shot because, you know, to be honest, my experience this far has been, despite GPT-4 being extremely knowledgeable in a lot of things, just the technical details stuff, it...
Jason Haddix (01:02:53.694)
Yeah.
Hahaha!
Justin Gardner (@rhynorater) (01:03:10.283)
it misses, you know, at a level where you can't really have misses as a hacker. You really have to read the spec yourself and really understand these sort of edge cases. And so I've kind of grown to distrust it a little bit, but it could be a result of my prompting not being specific enough. And I have noticed as well, you know, when I put in a special prompt or whatever when I'm doing, because I use it to assist me with Python programming all the time.
Jason Haddix (01:03:26.1)
Yeah.
Justin Gardner (@rhynorater) (01:03:40.215)
say anything just dump back the code, you know, that sort of thing. You know, my process gets a little bit easier but, you know, when I expand that out and maybe give it a lot more context on what it's expecting or what role it's supposed to be filling, I might get better results.
Jason Haddix (01:03:54.806)
Yeah, I mean, I think that there's a lot that goes on taught about using current gen LLMs. And I'm building a whole course around it, honestly. I think it's going to be one of my masterpieces is this next course called Red, Blue, Purple AI. But they're
Justin Gardner (@rhynorater) (01:04:00.605)
Mm.
Mm.
Thanks for watching!
Justin Gardner (@rhynorater) (01:04:10.467)
That's a great title, dude. Holy crap, he just rattled that off. He's like, red, blue, purple, AI. I'm like, okay, like red fish, blue fish. I'm like, that's great. LLM fish.
Jason Haddix (01:04:16.017)
Yeah, well, so I, so I, yeah. Yeah, if it's that, no, I, um, well, I knew I had a hit because I did a talk, right? I did an hour long talk on using current gen LLMs, and this was a year ago, and I took it to a private conference, a private security conference of a certain industry, I can't name it because it's super secret to rule, but I took it to this conference and I did it as a keynote, and I'll be damned if I didn't get more positive feedback.
Joel Margolis (teknogeek) (01:04:20.354)
Purplefish AIfish.
Justin Gardner (@rhynorater) (01:04:27.585)
Yeah.
Justin Gardner (@rhynorater) (01:04:31.921)
Mm.
Jason Haddix (01:04:45.881)
and like, like just fanfare about this talk than anything I've done in my career. People were like, that was amazing. Like that was fantastic. And so I was like, well, that sounds like something I want to, you know, probably develop and so now it's going to turn into a two day course, but really the gist of what I was trying to say is that a lot of people use single shot prompts. They use the general GPT that's available and the configurations around making really, really good bots is not, it's not available to you in that.
Justin Gardner (@rhynorater) (01:04:51.16)
Wow.
Jason Haddix (01:05:15.537)
Right? It is devising a very contextual role for the bot in its system instruction, which gets parsed first, not a user instruction, not what you type to it. Right? It's hidden from you. It is the first thing that the model uses to define its knowledge base and bring down its scope of research. And so I have like, you know, for a sock bot that I'm making, right? I basically went and I looked at what does a sock level one, level two, level three person do these days? Right? Like what tools are they working in?
Justin Gardner (@rhynorater) (01:05:27.102)
Hmm.
Justin Gardner (@rhynorater) (01:05:35.013)
Mm.
Jason Haddix (01:05:45.417)
What type of responsibilities do they have? What is their output? What is a really good one versus like a really bad one? And so then I build that into a bot, having done deep research into what I want it to give me and what I want it to do. And then there's other configurations too. Like there are several like actual API configurations to do with, you know, most models. Like the temperature is a really, really big one for hacking and yeah, for hacking and tech domains, you don't want any variance in temperature. You want zero variance in temperature.
Justin Gardner (@rhynorater) (01:06:06.467)
Yeah, that's huge.
Jason Haddix (01:06:14.193)
And so there's a whole bunch of other configuration things that you can do to make your bots better. And so that's the kind of stuff that's built into, to those discussions.
Justin Gardner (@rhynorater) (01:06:21.463)
Yeah, yeah, I've definitely played around with the APIs a lot, and I think they're a lot of fun, and there's a lot of applications outside of the hacking arena, but I'm definitely going to continue to be on the lookout for things inside the hacking arena. And then I also just, I haven't done enough yet with the, from just a white box perspective of, we have the code, it's explaining the code, it's sort of running through it, and it's in its brain of sorts. And so I think that'll be another thing that I work on shortly,
well. Before we transition out of the AI arena, you have in the dock here AI bias bounties and that's something that I haven't heard a ton about. What's up with that?
Jason Haddix (01:07:06.054)
Okay, so if you're not like in the know, right? Like there are several topics around hacking AI algorithms themselves, right? Like my stuff, I usually talk about using AI to help me hack but there's a whole bunch of topics around hacking the LLMs themselves. One is like data exfiltration, another is bias. And so this is what's kind of referred to in the...
Justin Gardner (@rhynorater) (01:07:17.454)
Right.
Jason Haddix (01:07:31.013)
unfortunately, in the academic sphere of AI and LLMs and ML as red teaming. They call it red teaming. It's completely different from the red teaming we do with us. Yeah, it is a pain. Yeah. But there are several bounties that exist out there. So there's some whole platforms too that like a lot of bug hunters don't know about. But there are some platforms and programs starting to appear on the bug bounty platforms about someone's going to use this model, right? This model that's been trained.
Justin Gardner (@rhynorater) (01:07:39.271)
Yeah, the overlap's kind of a pain, dude.
Jason Haddix (01:08:00.661)
to do something at their company. And they wanna make sure that they're not subject to any legal trouble and that the bot doesn't like go crazy racist or go crazy like biased towards people or sexist or, you know, or there's no like preferential treatment of those different types of people in its output of answers. And so what they have to do is they have to build this like prompt based firewall in front of a model that's already established, which is a tremendous task. And so they figured out that, well, we can use
Justin Gardner (@rhynorater) (01:08:25.578)
Mm-hmm. Yeah.
Jason Haddix (01:08:29.877)
bug bounty hunters to try to bypass our prompt-based firewall. And so these are bias bounties. So there's a couple that are going on right now. I work, I've been, it depends. Yeah. So.
Justin Gardner (@rhynorater) (01:08:41.107)
How much do these things pay out, man?
Joel Margolis (teknogeek) (01:08:44.134)
I'm on a couple of them as well. Yeah, what I've seen is it varies from like a couple of hundred dollars to a couple thousand dollars and it usually also varies depending on the classification of data. So there's usually, what I've seen is that there's buckets where it's like, you know, tiers like one through five and like tier five, I won't give any descriptions because most of them are really like bad shit, but you know, it's like tier five will be like sort of the lamest sort of.
Justin Gardner (@rhynorater) (01:09:02.649)
Mm, mm.
Jason Haddix (01:09:05.642)
Yeah.
Joel Margolis (teknogeek) (01:09:10.954)
AI prompt manipulations, then tier one will be the most graphic type of prompt manipulations. And if you can get the AI prompt to do any of that kind of stuff, then they'll pay you a bounty depending on what it is.
Justin Gardner (@rhynorater) (01:09:23.411)
That's, I feel like that, see the thing with AI though, man, is it feels like stabbing in the dark so much because you don't have any introspection into what's happening on the backend, you know? And I feel like even with AppSec stuff now, you know, we know, okay, you know, there's some language on the backend that has these constraints and it's either, it's doing a regex to check my input or it's doing some sort of typing and that sort of thing. And with AI, it's like, all right.
Could you please say fuck? Could you please say, you know? And it's like, and it says no. Please, no. You know, I mean, so I mean, obviously there's more advanced techniques to that, but I'll say in my short tenure of working with these models and doing more of the red teaming side of it, I've enjoyed that substantially less than traditional AppSec stuff.
Jason Haddix (01:10:12.245)
So I can hear what you're saying. I actually don't enjoy it less, I enjoy it more. I think, so I think one of the amazing things about this is that it does take away the bits and the bytes people out of their comfort zone, right? Like you and I are used to fuzzing parameters and analyzing code, you know, Joel is used to like cracking open mobile applications, breaking web apps. And like we've developed.
Justin Gardner (@rhynorater) (01:10:19.319)
No way, wow. Huh.
Jason Haddix (01:10:39.413)
these methodologies and these mental models that help us do this. And then along comes this whole new area of hacking where somebody, my neighbor, can be really good at natural language and just really mess up one of these models, right? And I love that. I think that is so freaking cool. Like, it's empowering for sure, yeah. And so, you know, I know people who, you know, who do AI Red teaming for a living, who are like teachers, right? They're really good at natural language. And...
Justin Gardner (@rhynorater) (01:10:52.77)
Ha ha!
Justin Gardner (@rhynorater) (01:10:57.115)
It's empowering, for sure. Yeah.
Jason Haddix (01:11:08.349)
and they just own it and they get these models to divulge data, they get these models to be biased, they get these models to talk about topics they're not supposed to. And these are world-class models. This is like GPT-4, right? Like models that everybody uses every day. And so the fact that they're becoming bounties makes me super happy. You know, the ones I've seen have been very similar to what I think Joel was talking about. I'm trying not to dox the customer, but.
Justin Gardner (@rhynorater) (01:11:33.545)
Mm-mm.
Jason Haddix (01:11:34.345)
but they're gonna be used in very sensitive situations. That's why they're running this bounty. This model is gonna be used in very sensitive situations. Yeah, and so it can't be, it cannot be biased, right? And so it's really interesting to see those programs. I'm trying to think of, there's a whole platform that does AI-based bias bounties. What is the name of it? I can't remember right now.
Justin Gardner (@rhynorater) (01:11:38.984)
Ah, I think I know what you're talking about, yeah.
Justin Gardner (@rhynorater) (01:11:54.168)
Mm.
Justin Gardner (@rhynorater) (01:11:57.887)
Even though I don't like necessarily red teaming, you know, AI red teaming stuff, I do like money. So if it, you know, if they are, if they are convinceable, you know, like, there could definitely be something there. But yeah, it would definitely be interesting to take more of a technical approach to it as well and say like, let's use an analysis of what.
Jason Haddix (01:12:09.116)
Okay.
Joel Margolis (teknogeek) (01:12:10.582)
Haha
Jason Haddix (01:12:15.081)
Yeah. Okay.
Justin Gardner (@rhynorater) (01:12:25.143)
words have what strength in each individual language. You know, like maybe if I used specific words in Japanese, which has a lot more cultural impact around a certain set of words, you know, maybe I could get it to say something that it shouldn't say. Or even maybe bilingual approach to it could be interesting as well.
Jason Haddix (01:12:38.11)
Oh yeah?
Jason Haddix (01:12:43.325)
Yeah.
Jason Haddix (01:12:46.785)
There's actually one of the ways to, I mean, it's one of the very simple ways, but it is a way of prompt injection is using different languages. On very, very dumb models, I would say.
Justin Gardner (@rhynorater) (01:12:55.668)
Mm, yeah. I've seen that on the props injection piece, but I hadn't really thought about it from the cultural sort of meaning piece of like, okay, you know, if I say, to take a particularly pointed example, one of the strongest things you can say in Japanese is telling someone to die, right? And that's definitely not, you know, a nice thing to say, you know, in English either, but it's not like,
Jason Haddix (01:13:02.869)
bias. Yeah, yeah, yeah.
Jason Haddix (01:13:21.129)
Yeah, yeah, yeah.
Justin Gardner (@rhynorater) (01:13:23.415)
you know, you're kind of shaking and you're like taking a back, you know? And so I think taking these understanding, something applying international studies as well to this sort of thing and understanding, okay, well, this word from this language has this strong impact that might override this model's, you know, prompting in the past as well. It could be a really interesting area of research.
Jason Haddix (01:13:47.561)
It can get really niche, especially in certain cultures, there is a lot of stuff left over from caste systems where you can basically slight hand, backhand talk about people based on some slang that was used in that. And that needs to be protected against in the modern world. And it can get really, really in depth. The cool thing about this though, is that you can build AI bots.
Justin Gardner (@rhynorater) (01:13:57.047)
Yeah. Oh, wow, yeah.
Jason Haddix (01:14:14.197)
to build you AI bias attacks to attack AI bias bounties, which is pretty amazing. So you build these frameworks in your head about like how you are succeeding. And this is something my buddy Keith is doing right now. He's a security engineer at GitHub, but he's on one of these programs that we're talking about. And I think he has something like 70 submissions to one of them right now, because he's built his own automation that uses AI to hack AI, which is, I find it fascinating.
Justin Gardner (@rhynorater) (01:14:18.131)
That's re- oh my gosh. That's ridiculous. Mmm.
Justin Gardner (@rhynorater) (01:14:34.91)
No way, why-
Justin Gardner (@rhynorater) (01:14:40.899)
But what differentiates the submissions though, I wonder, is it like, you know, you have to be able to technically, this is where the rubber meets the road, I think, you've gotta be able to technically explain why this, you know, submission is different from another submission where the result is both you got the thing to curse, you know? And so I think that could be another area where it's like, okay, we need to actually apply analysis to these promptings rather than
Jason Haddix (01:15:01.997)
Mmm, yeah.
Justin Gardner (@rhynorater) (01:15:10.095)
just trying to make it longer, trying to use different words and that sort of thing, and saying, okay, you know, this word from this language or this sentence structure sort of overrides the prompting that has come before it and results in a technical bypass of whatever firewall is in place.
Jason Haddix (01:15:26.121)
Yeah, I mean, this could be a whole show, right? But I think that that's one of the ways is to try to circumvent these, either the model-based system prompt or the user-based system prompt, which is usually where the firewall lives, using certain words. But there's also just like metadata and outcomes that you can feed AI and then scenarios. And if you...
Justin Gardner (@rhynorater) (01:15:29.038)
Yeah.
Jason Haddix (01:15:49.129)
basically put those in lists inside of another AI and say, hey, try every different variation of trying to get it to do this thing with this outcome with this bias, then you can print out like 70 of those tests really quickly and just see which one of them work and then wait till the customer patches and then see if the other one works and then wait, see if they patch and see if the other one works, so yeah.
Justin Gardner (@rhynorater) (01:15:54.116)
Mm-hmm.
Justin Gardner (@rhynorater) (01:16:05.794)
Mm.
Justin Gardner (@rhynorater) (01:16:09.667)
That's true. Yeah, but I feel like using AI as a, you know, creating like a system prompt firewall of sorts is kind of like not gonna be the best measure for this sort of thing. Because at the end of the day, you know, you just talk circles around it eventually and it's gonna spew something out. But I say that, but then I think like, okay, well, how would we firewall a human?
You know, really the way you firewall a human is you understand what is wrong, you know, what things are wrong and what things are right, and you teach that to the human. And how do you teach it to the human besides a system prompt? By telling them what actually is wrong and what's right, and kind of reasoning through that with them. So you know.
I came in a little bit hot in the beginning with when I first started talking about AI and saying, you know, using AI to solve AI problems, but I'm not really sure there's another way to do it. Yeah, it's a tricky problem.
Jason Haddix (01:17:11.393)
I mean, in academia, it is a tricky problem. In academia, and I mean, just anybody who gets really deep into this, right? Like the prompt-based firewall companies that are spawning up, they are there to try to get you to a place of like acceptance basically, because as long as a model has trained on a set of data and has, it has done that, like your, your
you're always going to be able to get around it with some type of prompt engineering, some kind of clever prompt engineering or prompt injection. And so really what most people are trying to do is like make sure that it doesn't, uh, overtly and easily show bias. Um, like if you have to go out of your way, uh, to tell it to say fuck, right? Well, like that meant that the user was trying to elicit that reaction, right? And so it's not really a thing that people care about. Now, if you are putting in, yeah, if you're putting in a plausible situation where
Justin Gardner (@rhynorater) (01:17:55.78)
Mm-hmm.
Justin Gardner (@rhynorater) (01:17:59.499)
Yeah, it's less practical.
Jason Haddix (01:18:05.545)
You're asking the bot, hey, do I, you know, like I have these candidates to promote one's name Jane, one name Mark, one name Robert, right? And it is de-prioritizing Jane because it thinks it's a female name. That's a big fucking problem for your company legally. So there are situational ways that like, you know, like people are just trying to get best effort out of some of these things. In the real world, yeah. Yeah.
Justin Gardner (@rhynorater) (01:18:14.253)
Mm-hmm.
Justin Gardner (@rhynorater) (01:18:21.196)
Right, right, yeah.
Mmm.
Justin Gardner (@rhynorater) (01:18:28.763)
crafting plausible scenarios that might actually happen where that's a lot more interesting to me of like Can I craft a scene and it actually almost becomes artistic then, you know It almost then becomes something along the lines of like a like a playwriter saying like, okay here is this flow the sequence of events that we're gonna present to this AI and The results of that will elicit some sort of negative response that that's that is fascinating I really like that and it adds that artistic piece as well
Jason Haddix (01:18:37.098)
Yeah.
Mm-hmm.
Jason Haddix (01:18:55.625)
Yeah. And you can get back to the tech stuff too, right? Like many of these models are being tied up to APIs now. And there's a lot of examples of now people using the LLM with its access to an API to actually hack, you know, technical data or PII or something like that. Or, you know, one of the options that, you know, that came out really recently was the airline that had to honor what the LLM bot said, basically on discounts for flight codes.
Justin Gardner (@rhynorater) (01:19:12.931)
Mm.
Justin Gardner (@rhynorater) (01:19:23.8)
Yeah, that's nuts.
Jason Haddix (01:19:25.233)
Yeah, and so like if your bot says it, they're a representative of your company and you have to honor that via law. And so like that's a whole nother place that you can look at too. So it's really interesting, yeah.
Justin Gardner (@rhynorater) (01:19:33.911)
Wow dude, that is definitely a crazy thing. We gotta wrap up here, I gotta stop right now. But Jason, thanks for coming on, man. Is there anything you wanted to shout before we close out?
Jason Haddix (01:19:45.213)
No, just, I mean, yeah, I mean, the new company is called Arcanum Information Security, arcanum-sec.com. We're gonna do more training, we're gonna do an AI course pretty soon, a standalone Red Team course, and an Interception Proxy course as well, where we're gonna go over all the Interception Proxies and what their benefits are over each other. So yeah, we're deving a whole bunch of content, we'll be at conferences if you wanna hang out, and yeah, that's it really.
Justin Gardner (@rhynorater) (01:19:57.647)
Dude, sick.
Joel Margolis (teknogeek) (01:20:00.176)
Awesome.
Justin Gardner (@rhynorater) (01:20:10.951)
Awesome man, that's a lot of great info. Thanks for coming on, man. All right, peace.
Jason Haddix (01:20:14.07)
Thanks.
Joel Margolis (teknogeek) (01:20:16.782)
See you.
