The player is loading ...
Sign up to get updates from us
Episode 48: In this episode, joined by the spectacular Sam Erb, Google Security Engineer and DEFCON Black Badge winner. We talk about the importance of understanding how systems work to find vulnerabilities, and how his engineering background influences his hunting style and methodologies. Then we jump over to his Career Development and his work with Google, and then chat about some of the recent Google Vulnerability Programs.
This episode is sponsored by Wordfence! Wordfence recently launched a game-changer of a bug bounty program with ALL WordPress plugins over 50k installs are in-scope. They are currently paying 6.25x their normal bounty amounts, and have agreed to give CT listeners a 10% bonus on top of that! Head over to https://ctbb.show/wf for more info and keep an eye on the CTBB Discord for inspiration/collabs.
Follow us on twitter at: @ctbbpodcast
We're new to this podcasting thing, so feel free to send us any feedback here: info@criticalthinkingpodcast.io
Shoutout to YTCracker for the awesome intro music!
—— Links ——
Follow your hosts Rhynorater & Teknogeek on twitter:
—— Ways to Support CTBBPodcast ——
Sign up for Caido using code CTBBPODCAST for a 10% discount.
Hop on the CTBB Discord
Discord premium subscribers get access to private masterclasses, exploits, tools, scripts, un-redacted bug reports, etc.
Today’s Guest:
And
Google Device Vulnerability Reward Program Initiatives
Transcripts
(00:00:00) Introduction
(00:02:50) Hacker Methodology with Sam Erb
(00:12:20) Balancing Bug Hunting and Personal Life
(00:15:53) Deep Diving on a program and using automation.
(00:27:00) Optimizing Bug Hunting and Understanding Attack Vectors
(00:39:22) Collaboration and Boundaries
(00:45:42) Career Development and Entrepreneurship
(00:55:13) Winning Black Badges at DEFCON
(00:58:02) BufferOver
(01:09:11) Working at Google
(01:19:23) Google Bug Bounty Programs
(01:31:41) BONUS Cool Bugs
transcript.txt
Justin Gardner (@rhynorater) (00:01.489)
Alrighty, we got a super special guest today. Mr. Sam Erb. Welcome to the podcast, man.
Sam Erb (00:07.314)
Hi, long time listener, first time caller. I think it's the right term.
Justin Gardner (@rhynorater) (00:10.389)
Yay, dude. That's great. Well, thanks for coming on, man. And I appreciate you listening. I do know that you do listen, because you'll ping me from time to time and talk about stuff from the pod. So definitely, definitely appreciate that.
Joel Margolis (teknogeek) (00:11.015)
the
Sam Erb (00:23.262)
Yeah, yeah, I, you know, this is on my rotation, you know, and I'm like doing dishes or showering, I don't know, listen to this.
Justin Gardner (@rhynorater) (00:31.61)
Showering. Yeah, the best time.
Joel Margolis (teknogeek) (00:35.251)
That's true. You got to get those shower thoughts in, you know?
Justin Gardner (@rhynorater) (00:37.853)
Yeah, man. Yeah, it's actually been really cool to you know that because we've I think Uh, I almost said shopify spotify has been coming out with those. Uh, you know like
Sam Erb (00:38.024)
Oh yeah.
Justin Gardner (@rhynorater) (00:48.993)
was it, 2023 wrapped, yeah, thing, and people have been like, oh, you know, we listen to Critical Thinking all the time, you know, top whatever percent listeners, so definitely, definitely appreciate that. So I guess normally, at the beginning of this podcast, we will go into like little history of Sam, you know, and that sort of thing. But I think we're gonna skip that this time, and we're gonna come back to it later, and we're just gonna jump right into some hacker methodology.
Joel Margolis (teknogeek) (00:50.413)
Yeah, they're wrapped. Yeah.
Sam Erb (00:51.76)
Oh yeah.
Justin Gardner (@rhynorater) (01:18.097)
stuff. Okay. So just for, okay, I'll do a little context. Sorry, a little context.
Joel Margolis (teknogeek) (01:23.746)
Tell the people who we're talking to.
Justin Gardner (@rhynorater) (01:24.809)
All right, so this is Sam, currently Google employee, hacker extraordinaire, one of my top respected hackers, and one of the most influential hackers in my life as far as methodology and like, I guess hacking techniques that I respect as far as that goes. So Sam, really happy to have you here. And I kinda wanna dive a little bit into what makes you.
so special as a bug bounty hunter and a hacker. And my theory is that is largely coming from a background that is not related to web security, right? So talk to me a little bit, because you were a low level, I guess you were doing low level development, right? In the beginning, or was it always security? I know you were working with stuff that was not web related.
Sam Erb (02:13.49)
Yeah, so it's always been web tangential, I think is the best way to describe it, and security tangential, but I have spent a decade as a developer. So I worked at Cisco for a few years doing crypto hardware integration.
Justin Gardner (@rhynorater) (02:16.879)
Mm-hmm. Yeah.
Justin Gardner (@rhynorater) (02:30.593)
Yeah, so this is exactly the shit I'm talking about. Say like, crypto hardware integration? You know, like a question. Okay, so I guess, I wanna hear about that, and we're gonna hear about that. But what I really wanna get to is why...
Joel Margolis (teknogeek) (02:35.186)
Hehehehe
Sam Erb (02:36.403)
Yeah.
Justin Gardner (@rhynorater) (02:49.409)
Is this whole concept of going wide and versus going deep and I categorize hackers by this right and you hear me talk about This on the pot all the time. You are very much a go deep sort of hacker by my estimation and so I want I'm wondering is that a do you think that's a function of You know having spent so much time at a lower level that you're more comfortable at the lower level so you try to get down lower and lower and lower and lower or You know, is that an intentional decision made by you?
Sam Erb (03:16.398)
That's a really good question.
Sam Erb (03:20.626)
I think there is a certain amount of, like, I am comfortable going and reading an RFC and then, you know, applying that somewhere and trying everything that I can find in the RFC. But at the same time, it's, you know, having been a developer, it's like you kind of...
Justin Gardner (@rhynorater) (03:31.308)
Mm.
Sam Erb (03:39.294)
you know what mistakes, like to me anyways, you know what mistakes developers make and then you can go and find those mistakes in the wild. And the security engineering side of that, it's like how do you prevent those mistakes from happening or build frameworks that don't allow them. But yeah, I mean, you've seen it. It's, you know, there's been a lot of instances where, like I think that there should be a bug there and, you know, I will spend.
Justin Gardner (@rhynorater) (03:47.753)
Mm.
Justin Gardner (@rhynorater) (04:00.551)
Mm.
Sam Erb (04:08.294)
of 10 hours going off and reading an RFC, which is just like, you know, completely... it's not the healthiest thing in the world, but it's like once you get fixated on something might be vulnerable, then like it's hard to think about other things.
Justin Gardner (@rhynorater) (04:09.413)
Thanks for watching!
Joel Margolis (teknogeek) (04:22.701)
Yeah, for sure. Do you think that also your engineering background has pushed you to be more of a technical hunter rather than that black box hunting? I think a lot of the hacking that I've seen you do is diving into how does the source code for this work or reading through the more open source approach. And do you feel like your engineering side has really pushed you in that direction as well?
Sam Erb (04:50.474)
Yeah, absolutely. I want to understand how the systems work to break them. The story that I tell actually a lot is like, this was at one of the US government live hacking events. Somebody from the US government was like, asked me and a few other hackers, is like, how do you guys avoid the dark side? How do you stay good hackers? And I was personally, I was so offended by that, because all I want to do is go and figure out how the systems work and break them. It's like, I don't...
Justin Gardner (@rhynorater) (05:00.972)
Mm.
Justin Gardner (@rhynorater) (05:09.43)
Hahaha Right
Yeah
Sam Erb (05:19.147)
what your social security number is, you know? It's...
Justin Gardner (@rhynorater) (05:20.776)
Yeah, yeah, no that makes sense. Classic.
Joel Margolis (teknogeek) (05:23.129)
How do you not, yeah, how do you not commit credit card fraud? What's going on? Look.
Sam Erb (05:27.198)
Yeah, yeah. And I'm like, maybe that's what she wanted to do, and like, that's, you know, but that's not why I'm here.
Justin Gardner (@rhynorater) (05:31.397)
Yeah. No. And I think that speaks largely to.
Joel Margolis (teknogeek) (05:34.158)
It's
Justin Gardner (@rhynorater) (05:37.209)
the curiosity motivated aspect of, and I think that's one of the things that makes you a really great researcher and a really great bug bounty hunter, but it does come with that downside, like you mentioned of like, all right, I'm gonna read this thing till my eyes bleed, you know, and like, and try to figure out, you know, never letting go of a bug, especially, you know, if you can't figure out, if you can't explain everything in the situation. Yeah.
Sam Erb (05:39.479)
Mm-hmm.
Sam Erb (06:00.822)
Yeah, I love the term accidentally secure because there's things that I'll come across where like I know things, well, right, yeah. I know things are broken in some way, but in bug bounty context, you can only report things that are valid. You can't report like half a bug really. Well, you could, but you'd get negative reputation or whatever.
Justin Gardner (@rhynorater) (06:04.745)
Yeah, I hate that term. I love it, but I hate it, you know?
Justin Gardner (@rhynorater) (06:17.282)
Yeah.
Joel Margolis (teknogeek) (06:21.461)
So, so along that line, like, how do you go about sort of thinking about bugs when you're when you're approaching a problem and you want to like really deep dive and like find sort of the most impact? What is like what is your thought process?
Sam Erb (06:37.494)
I honestly, and I think you guys have seen this at some recent events, it's like I'm really interested in looking for locations where...
Sam Erb (06:49.294)
uh you can get the highest CVSS score i think is the best way to describe it like looking for those unauthenticated places with um
Justin Gardner (@rhynorater) (06:53.932)
Mm-hmm.
Sam Erb (06:59.742)
you know, network's not authenticated, no privileges required. You know, aim for that CVSS 10.0 and avoid the places where like you, you can't possibly get that. You know, I had to say it, but like mobile apps come to mind. But now at the same time, like mobile apps should have a separate, my opinion anyway, should be rewarded differently. Like.
Justin Gardner (@rhynorater) (07:03.689)
Mm.
Justin Gardner (@rhynorater) (07:07.852)
Hmm.
Justin Gardner (@rhynorater) (07:12.546)
Yeah.
Yeah.
Joel Margolis (teknogeek) (07:16.601)
Wow, wow, shots fired. So...
Justin Gardner (@rhynorater) (07:22.105)
Yeah.
Joel Margolis (teknogeek) (07:23.477)
Yeah, no, totally. I totally agree. We actually had this problem recently as well, where we were trying to award a mobile vulnerability and try and figure out how to score it. And if you go traditional CVSS, you run into all these problems where adjacency is physical or local. It's really very, very difficult to score in the same way. But do you actually approach it from a CVSS perspective? Purely? Do you just go sort of like, OK, if I were to think about this from CVSS,
Justin Gardner (@rhynorater) (07:38.871)
Yeah.
Sam Erb (07:39.904)
Yeah.
Joel Margolis (teknogeek) (07:53.341)
what parts of the app should I avoid or is it more sort of like high level?
Sam Erb (07:59.526)
I mean, to be fair, that's where I'll start. From there, I'll kind of work my way down. But it's, yeah, at the end of the day, it's like we're all trying to, as bug hunters anyways, we're trying to maximize reward. That's how we do it.
Justin Gardner (@rhynorater) (08:03.112)
Mm-mm.
Justin Gardner (@rhynorater) (08:15.969)
Yeah. And it's like, this kind of reminds me of a talk I gave a while back on.
open source methodology when I was talking about the Grafana bug at NahumCon. It's this concept of like, okay, I've got this goal. And I need to work back from where, let's say I wanna submit a 10.0 CVS bug, right? Which we all do in Bug Bounty and also specifically in live hacking events. Then you've gotta start naturally from the spots where it's possible for you to get a 10.0 CVS bug, right?
more sense to focus on the unauthenticated context. And actually I had a little bit of a question here. It seems to me like in the life hacking event that we collaborated on, I'll say it but we're going to have to bleep it,BLEEEEP, that you focused a lot on authentication there and you focused a lot on authentication at the last couple of life hacking events that I've seen you on. Would you say that's your bread and butter? Where does that play a role in your testing?
Sam Erb (09:22.294)
You know, I always say it's where I'm gonna start. And I just often never move on. That's a lot of the truth to it. It's like, I'll spend like, you know, next thing I'll know is it'll be like 20, 30 hours later. And I'm like, I'm still trying to figure out all the pieces and understand how they all mesh together.
Justin Gardner (@rhynorater) (09:25.282)
Really.
Hahaha! That's great. Yeah.
Justin Gardner (@rhynorater) (09:41.25)
Yeah.
Joel Margolis (teknogeek) (09:41.413)
So when you find yourself down a rabbit hole like that, do you find it's more that just like, you've sort of stumbled into the fact that the app is more complicated than you initially realized, or do you keep coming up with new, more creative scenarios that you have to sort of suss out and see if they're viable vulnerabilities?
Sam Erb (10:02.962)
I mean, it's a little bit of both, but like, you know, listening to kind of going back to like listen to this podcast while I'm like doing other things, like there's a lot of times where, you know, I'll spend like an entire night just looking at like a single, like a single piece of the application and then, you know, go and brush my teeth and be like, oh, I'm an idiot. You know, I need to go back there and try these things. I didn't quite realize this is how they all fit together. And
Justin Gardner (@rhynorater) (10:09.986)
Mm.
Justin Gardner (@rhynorater) (10:23.947)
Mm-hmm.
Joel Margolis (teknogeek) (10:24.76)
Mm.
Sam Erb (10:31.222)
Yeah, that's often led to my best bugs.
Justin Gardner (@rhynorater) (10:33.348)
So it's been, you know, it seems like.
Joel Margolis (teknogeek) (10:33.485)
Mm-hmm.
Justin Gardner (@rhynorater) (10:35.825)
You know to use the technical terms is funny because my wife always gets on my case You know for using technical terms for like non-technical things, but it's almost like you know you've got like a you know like a Like what's the screen? You know like in the terminal? You know like it's running in the back of your brain You know just like all right. Let me just screen detach from that right and it's just running back there like processing and And and I think you see that with a lot of top bug bounty hunters is like even if you're walking away
you know, from the screen, when you're doing stuff like brushing your teeth or like changing your clothes or like, you know, you're thinking you're thinking you're in that mode. And it gets overwhelming, right? Like that's something that really takes a lot of a lot of the parts of your life. And as somebody who's not a full-time bounty hunter, somebody who has a full-time job, I'm wondering how you manage that, you know, with your full-time job and with bug bounty and like,
Justin Gardner (@rhynorater) (11:35.279)
to bug bounty from these other things to, because it kind of like tries to take over sometimes, I think.
Sam Erb (11:41.146)
I mean, it's a really good question. To be honest, I don't do it. I don't do too much about bounty work anymore. I used to participate in every life hacking event and like, you know, as, you know, my partner was off finishing off her PhD for a while and that just gave me a little bit more free time because she was off finishing that. And so, you know, really since then I've had...
Justin Gardner (@rhynorater) (11:46.614)
Yeah.
Yeah.
Justin Gardner (@rhynorater) (11:58.555)
Mm.
Sam Erb (12:08.714)
significantly less time from that. I will say though, it's like, for me anyways, it was most challenging to not associate time spent elsewhere with like a dollars per hour.
Justin Gardner (@rhynorater) (12:10.823)
Mm.
Sam Erb (12:23.742)
Like, you have to be really careful, like, once you start doing this with any consistency, you start getting paid with any consistency, that you don't associate your time with an hourly rate. Because all of a sudden, like, and honestly, to be completely honest, I still have a lot of trouble just sitting down and watching TV. Like, that's just a challenge of mine, because like, I'll start thinking, like, oh hey, like, you know, I actually, I call this, um...
Justin Gardner (@rhynorater) (12:24.132)
Yeah.
Justin Gardner (@rhynorater) (12:32.162)
Yeah.
Justin Gardner (@rhynorater) (12:41.174)
Yeah, totally.
Sam Erb (12:51.462)
I call it, it's almost like a metronome in the back of my head that's like, you know, tick, tick. So I call it like a tick almost, like why are, you know, you should be doing this, like why are you hanging out with your friends? You should be going making money, like, and so like to have, you have to, to me anyways, it's important to not even try and calculate that because once I start going down that route it's, it can really drive me to negative outcomes. Yeah, exactly.
Justin Gardner (@rhynorater) (12:55.841)
Yeah. Yeah, yeah.
Justin Gardner (@rhynorater) (13:00.701)
Mm-hmm. Yeah, yeah, yeah.
Justin Gardner (@rhynorater) (13:12.609)
Mm, yeah. What? It's a, it's a slippery slope for sure. You know, we talked about it, you know.
on the episode a little bit with Douglas, with Douglas Day with Archangel. And he was saying, yeah, now I'm trying to find it hard to justify going out with my friends instead of staying home and hacking. When you have this essentially unlimited access to trade your time for money that's not really limited. And sometimes it's huge amounts of money too.
Sam Erb (13:41.65)
I mean, to be fair, it's a good problem. It's a very privileged problem to have.
Justin Gardner (@rhynorater) (13:44.617)
It is, it is, and I'm grateful for it, but when you swing it the other way, it's a little bit, it becomes destructive. And so it's hard for bug bounty hunters to regulate that for sure. Definitely a pitfall to be aware of. So jumping back to the hacking methodology thing, I'm wondering, being a deep diving hacker, I kind of get crap all the time from the community for like, yeah, I'll spend like three weeks, four weeks on a program before I move on. I'm wondering,
Joel Margolis (teknogeek) (13:45.615)
Hehehe
Sam Erb (13:52.822)
Yeah, they could easily come, yeah.
Justin Gardner (@rhynorater) (14:14.471)
I'm looking for some validation here. No. How long do you spend on a program and how long do you, how deep do you dive? You know, and also, again, let me caveat that question though, because I know that, you know, the brain works a little better for Mr. Sam or compared to me. So it may not take you quite as long to get through a target, but.
Sam Erb (14:35.978)
I have, I don't know. So when it comes to life hacking events, especially, I will look at that target, nothing but that target for that time period, and then just stop completely. For other targets, I will actually come back to them, especially targets that I'll submit a bug and they'll react to it quickly. Like that's been my test. It's like, if I submit a bug, they react to it quickly. Like, okay, like this team isn't gonna have like too many duplicates sitting around for me to find. They're quick to respond,
Justin Gardner (@rhynorater) (14:40.89)
Mm.
Oh yeah. Yeah.
Justin Gardner (@rhynorater) (15:02.75)
Mm-hmm.
Sam Erb (15:05.886)
I can go and submit other things and not be worried that they're going to like, I don't know, maybe not pay me or something, for example. But honestly, there have been very few product targets that I've really stuck with. Airbnb was one that I think we actually participated in that life hacking event together. That's that trophy behind me.
Justin Gardner (@rhynorater) (15:13.487)
Sure.
Justin Gardner (@rhynorater) (15:25.025)
Yes, we did, Sam. Yes, we did. And I took first at that event, but somehow somebody else got the belt. So love that. No, we'll dive into that later. I'm sorry. Continue, continue.
Sam Erb (15:39.326)
Um, that's yeah, sorry. Yeah, sorry. Um, but, uh, it may be was one where it's like, I, I found that I really liked how their application worked and, um, you know, so I did the extra step there. I even automated everything that I could. Um, and I really stuck with that for years until they made some like pretty high level, they actually wrote a whole engineering blog post on, on this and it was like more of like an engineering change, but it also resulted like a really positive security outcome where like,
Justin Gardner (@rhynorater) (15:50.339)
Mm-hmm.
Sam Erb (16:07.614)
all of their security became much more centralized and less bug-prone.
Justin Gardner (@rhynorater) (16:09.954)
Really?
Justin Gardner (@rhynorater) (16:13.837)
Dude, can you talk a little bit about that? Like now that the blog post is out, can you talk about some of the details of what they changed and how? Yeah. Of course. Yeah.
Sam Erb (16:20.554)
Well, this is entirely from an outside perspective. I don't know any of the insider details of any of this, obviously. But they, without trying to not talk about bug reports, sorry, they centralized, seemingly centralized a lot of their, sorry, what?
Justin Gardner (@rhynorater) (16:34.883)
You're good.
Justin Gardner (@rhynorater) (16:41.57)
Yeah.
Joel Margolis (teknogeek) (16:42.649)
Well, for what it's worth, if you want to talk about some bug reports, we can always cut it out and bleep it later. So that just makes it easier.
Justin Gardner (@rhynorater) (16:47.317)
Yeah, if it's a problem.
Sam Erb (16:47.894)
No. Oh no, it's all good. I really think what it comes down to is they really centralized their auth management. And that really putting that all into one place and centralizing control of all of their API endpoints really led to, it seems like it was more of an engineering decision, it probably led to good engineering outcomes. But the side effect was you didn't have teams going off and making their own APIs with the same mistakes over and over again.
Justin Gardner (@rhynorater) (17:05.8)
Mm.
Justin Gardner (@rhynorater) (17:11.557)
Sure.
Sam Erb (17:17.582)
Um, so, you know, that after that, I kind of stopped.
Justin Gardner (@rhynorater) (17:21.374)
So what does that mean, Sam? If we were to take it another layer clearer, is that saying, okay, they implemented some middleware that sits on front of all the API endpoints that implements their access control stuff and all of the idors disappeared? Or are we talking about auth? I mean, is that really what we're talking about? Okay, yeah, so like, read through that one, didn't I?
Sam Erb (17:37.678)
I mean, yeah, roughly. Yeah, yeah, yeah. I don't want to get into bug specific, sorry. Yeah.
Justin Gardner (@rhynorater) (17:47.617)
No, I mean, like, we can say that Airbnb had idors, you know, that's not the end of the world. No, and that's big, you know, and so when you see a giant change like that, you know, that changed the way that you interacted with that program substantially.
Sam Erb (18:03.134)
Yeah, I mean, it's a good security outcome for them. You know, we're obviously, you know, we're both simultaneously like security driven, but we're also monetarily driven. So it's like, yeah, obviously it resulted in me getting paid less money, but it also means everybody being more secure. Oh, I mean, I automated everything I could, probably 50 to 100 hours.
Justin Gardner (@rhynorater) (18:14.905)
Yeah.
Justin Gardner (@rhynorater) (18:19.702)
So how long did you spend on that program? What?
Justin Gardner (@rhynorater) (18:27.785)
Okay, and is that over the course of six months? Is that over the course of a couple weeks? Is that over a few months? Okay, and what does automation for that sort of thing look like?
Sam Erb (18:32.643)
No, a few months, yeah.
Sam Erb (18:37.306)
Everything was custom. I was writing custom parsers for their JavaScript to pull out API endpoints, testing those API endpoints.
Justin Gardner (@rhynorater) (18:44.337)
Mm-hmm. Were you automatically testing the API endpoints, or were you alerting on them and then manually testing them? Yeah. Yeah, yeah.
Sam Erb (18:50.034)
Well, usually just manually learning to be fair, but there was some automation there. Then there was also... I've gone through a series of revisions of my own recon architecture, some of which I ended up open sourcing, but that was all pointed at that. I don't actually know if I ever... Well, I mean, part of it was. That was the whole DNS prep thing.
Justin Gardner (@rhynorater) (19:03.554)
Mm-hmm. Yeah.
Justin Gardner (@rhynorater) (19:08.653)
I don't think I saw that. Is that on your GitHub?
Justin Gardner (@rhynorater) (19:18.138)
Ah yeah yeah, DNS crap, yeah of course.
Sam Erb (19:20.546)
Yeah, so that's just like, you know, you can take advantage of DNS as hierarchical. You can search it really easily based on that, as long as you pre-sort it.
Justin Gardner (@rhynorater) (19:26.309)
Mm.
Justin Gardner (@rhynorater) (19:31.309)
Yeah.
Joel Margolis (teknogeek) (19:32.665)
Yeah, I was going to ask so like two things on that one. You mentioned that you wrote like very custom automation. Obviously that's a little bit easier when you have a software engineering background, but it was that something sort of like. That you decided to do because you were spending a lot of time on Airbnb or do you do that for every program or like what made you do that?
Justin Gardner (@rhynorater) (19:41.861)
Hehehe
Sam Erb (19:51.487)
Oh, it...
Justin Gardner (@rhynorater) (19:52.485)
This guy freaking probably wrote it in C. Like...
Joel Margolis (teknogeek) (19:54.957)
Ha ha!
Sam Erb (19:57.293)
To be fair, it's usually in four languages, but yeah, no C in there.
Justin Gardner (@rhynorater) (20:00.225)
What? Why are you using- Get out! Get off the pod! Not allowed! Oh jeez.
Joel Margolis (teknogeek) (20:02.185)
I'm not I'm not I don't even want to know
Sam Erb (20:06.37)
Sure. No, I just, you know, I try to use the right things for the right purpose. Like, you know, I just string everything together with bash, but like where I need to, I'll drop into like a lane to just get things faster. I'm not quite at rest yet, but I'll get there eventually. Yeah.
Justin Gardner (@rhynorater) (20:19.18)
Yeah.
Justin Gardner (@rhynorater) (20:23.073)
Yeah, well, few of us are. No, that makes sense, and actually JS monitoring is something that we've been talking about a lot in the critical thinking community lately. And there have been some really cool solutions that are coming up around it, and it's exciting to see. And you were kind of ahead of the curve on this industry, doing this back in what, I mean, when did we compete at that event? Was that 2018? Yeah, 2019 maybe.
Sam Erb (20:48.328)
It's been 2019. Yeah.
Joel Margolis (teknogeek) (20:49.133)
Yeah, something like that.
Justin Gardner (@rhynorater) (20:53.225)
So, you know, what was that? I mean, let me just try to break it down technically. So you would reach out to the, you know, HTML page, pull out the JavaScript file hash, go to that JavaScript file. Would you beautify it or would you just run regex on it or were you doing diffs of the code directly or were you doing diffs of like the strings in the code? How did that look?
Sam Erb (20:59.736)
Yeah.
Sam Erb (21:16.814)
Um, if I recall correctly, it was like, Oh, I think my pack is the right term to be using their, their side, reach out to Airbnb, pull out the web pack. Uh, they followed. Once again, this is pushing the limits of my job script knowledge. Um, they would, yeah, that too. Yeah. Um, they would, uh, follow a very common format for their API endpoints in that web pack, even though it was somewhat minified. Um, which you could then.
Justin Gardner (@rhynorater) (21:21.069)
Yeah, yeah, yeah.
Justin Gardner (@rhynorater) (21:25.281)
Mm.
Justin Gardner (@rhynorater) (21:29.185)
Yeah, it's a little bit, it's a little farther back too, so yeah.
Justin Gardner (@rhynorater) (21:38.198)
Mm-hmm.
Mm-hmm. Yeah.
Sam Erb (21:46.446)
pretty straightforward to extract that. And yeah, I mean, from there, it's all just like Discord notifications, like, hey, there's this new thing. And, you know, it's a really easy way to, when you know a program well, it can become a very easy way to find bugs. Because as soon as something launches, you can just be like, okay, I'm just gonna go test it, you know, and you're off to the first one there.
Justin Gardner (@rhynorater) (22:02.531)
Yeah.
Justin Gardner (@rhynorater) (22:07.088)
Yeah.
I mean, this is what the pros are doing when you're going long-term, going deep on a program. And that actually makes me think a little bit more. Obviously, you were sort of taking notes with code in this scenario, right? You were saying, okay, I've noticed this structure in the JS file on how these API endpoints are done. I'm gonna write some codes automatically, parse all this, and help me see this in a better way. But I'm wondering what kind of notes do you take for yourself in general? Are you like a, you know, whatever?
Sam Erb (22:22.805)
Hmm?
Justin Gardner (@rhynorater) (22:38.199)
asdf.txt sort of guy or you like I've got this whole postman built out with their whole API spec sort of guy We're on the spectrum
Sam Erb (22:44.89)
Um, yeah, I did, I did hear that in the last episode. That's wild. For, for me, I actually take, I have like a pretty standardized format at this point for recon. I think all my recon is up in his own folder. Everything is very, very neat there. I can even diff it between like runs, um, to see what's new. But when it comes to like looking at the application itself, now I, yeah, I'm a readme.txt guy.
Justin Gardner (@rhynorater) (22:48.065)
Uh huh. Yeah, it's nuts.
Justin Gardner (@rhynorater) (22:54.977)
Mm.
Justin Gardner (@rhynorater) (23:02.143)
Mm-hmm, great.
Justin Gardner (@rhynorater) (23:11.026)
Yeah, sure sure.
Sam Erb (23:12.668)
Next time, next time I hack on this.txt.
Justin Gardner (@rhynorater) (23:15.489)
Yeah, I mean, like, what do you think about that, though? Like, just out of curiosity, you know? We all have the things that we do, you know, because that's just how we do it. But do you think it'd be more optimal for you to take time to write more thorough notes, or do you think that would interrupt the creative process, or what?
Sam Erb (23:23.583)
Yeah.
Sam Erb (23:33.462)
I think it really depends. I think I like, to use the RDB case again, I think if I were to spend more time on that program, then yeah, I would absolutely have taken much more structured notes on that. I have a lot of respect for the folks who do. I don't know, that's something I wish I should get better at.
Justin Gardner (@rhynorater) (23:37.335)
Mm.
Justin Gardner (@rhynorater) (23:43.734)
Mm.
Justin Gardner (@rhynorater) (23:51.405)
Yeah, yeah, there's definitely room for growth.
Joel Margolis (teknogeek) (23:54.137)
Yeah, I mean, I thought I had structured notes and then even still I've seen some other people who have like way more structured notes and I'm like, I'm looking at theirs and I'm like, dang, I need to make some improvements. And then every time I go back to mine, I'm like, it would be a lot better if I had, you know, X, Y and Z. But this is one of those things that is sometimes it's a real big time investment and depending on like what your workflow looks like, like if it's not like a bump in your workflow, then oftentimes it's just better to just.
Sam Erb (24:00.363)
Yeah.
Justin Gardner (@rhynorater) (24:12.73)
Hmm
Joel Margolis (teknogeek) (24:23.745)
You know, keep it simple and...
Sam Erb (24:26.01)
Yeah, I think it was the interview with Franz, which, I don't know, I feel kind of honored to go in after him. Like a little starstruck, to be honest. Yeah, right? But he talked about how he was taking notes of parameters used and stuff. And I almost wish I had even a whole other layer of structured recon. Because usually I'd be like,
Justin Gardner (@rhynorater) (24:29.793)
Mm.
Joel Margolis (teknogeek) (24:33.913)
Hahaha
Justin Gardner (@rhynorater) (24:35.01)
Big shoes to fill.
Justin Gardner (@rhynorater) (24:47.235)
Mm-hmm.
Justin Gardner (@rhynorater) (24:51.406)
Mmm.
Sam Erb (24:53.062)
For the most part, anyways, I tend to stop with subdomain enumeration a little bit past that, but not too much. And I really think that there's so much that could be out of there that would, at the end of the day, just make the bug hunter's life easier, so to speak.
Justin Gardner (@rhynorater) (24:57.084)
Hmm
Justin Gardner (@rhynorater) (25:08.664)
Yeah.
Yeah, there's a lot of extra layers you can go down deep and when you start getting diminishing returns, I don't know, and it actually makes me think of this topic, this sort of hypothesis that I've been kind of thinking about lately. Because us three, we've all been successful in bug bounty and when you start building a Discord community like we are at Critical Thinking, you get a lot of people being like, hey, this is what I wanna do, can you help me get there? And we don't have time to answer all the questions, but it has made me
Sam Erb (25:15.98)
Yeah.
Justin Gardner (@rhynorater) (25:39.055)
think more about how to frameworkize what I do, right? And how to like take away life lessons from something that I just kind of do intuitively, right? And one of the things that came to my mind is I wonder whether bug bounty hunting can be minimized to
Sam Erb (25:49.887)
Mm-hmm.
Justin Gardner (@rhynorater) (26:00.289)
or simplified, I guess, to number of attack vectors tried. Like, I wonder if there was a way for us to optimize our hunting for just raw number of attack vectors tried, whether that would be close to the optimal path to produce bugs as the result. Like, am I off base yet? I'm clear. Oh, okay, okay. That's great. No, that's good. Good, good, I love it.
Joel Margolis (teknogeek) (26:24.596)
I'm not gonna lie.
Sam Erb (26:25.966)
Well then like, an Apache zero day would just dominate that chart, right? Or something, you know? Like an Apache zero day, or like an intadex zero day. Like...
Joel Margolis (teknogeek) (26:30.933)
Right.
Justin Gardner (@rhynorater) (26:32.732)
The what would dominate that chart? Say it again.
Oh yeah, no, I mean like, well what do you mean by that? Because I mean like, okay, you know, I'm gonna try if I can do this Eidore on this one endpoint. No, okay, you know, I'm gonna try to do this, you know, C-surf over here, you know, like, the more attack vectors I'm trying, the more likelihood one of them is gonna work, right? And so, but I guess this is sort of like a brute force approach. All right, Joel, you know, all right, if you're gonna give me that face, all right, hit me, hit me, hit me, hit me.
Joel Margolis (teknogeek) (26:37.345)
Right. Yes.
Sam Erb (26:45.58)
Yeah.
Joel Margolis (teknogeek) (26:58.297)
Well, no, no
Justin Gardner (@rhynorater) (27:10.485)
No, but it's not though. Like, it's not really.
Joel Margolis (teknogeek) (27:17.198)
Well, abstractly, yes. There's no like, every vulnerability has an equal chance of occurring. Depending on the context and you with your intuition as a hunter, you can determine what things are more likely, right? And so I think that's like the real nuance is that the more that you've been hacking, the more you're able to narrow, like quickly narrow down your scope of vulnerabilities that you're testing based on what you're seeing. Whereas...
Justin Gardner (@rhynorater) (27:26.281)
Right. Yeah. That's what I'm talking about.
Joel Margolis (teknogeek) (27:41.381)
Theoretically, there's infinite possibilities, right? Like you could test infinite variations of I-doors and vulnerabilities in all different parameters and everything and never hit anything because you're sort of blindly testing the wrong things when you should be doing what you're doing, which is narrowly targeting it down.
Justin Gardner (@rhynorater) (27:48.221)
and cookies and yeah.
Justin Gardner (@rhynorater) (27:53.96)
Mmm.
Justin Gardner (@rhynorater) (27:57.58)
Mm.
Sam Erb (27:57.91)
Yeah, and so I've actually, this has a lot of parallel with binary reverse engineering. And the thing I found there anyways, is like, I'm not great at this myself, but like, once you figure out, this is from, I probably saw this on Twitter at some point, but once you understand what a printf function looks like, you see it and you can cast it aside as like, this is a printf function, I don't need to look at this again.
Justin Gardner (@rhynorater) (28:03.553)
Mm.
Justin Gardner (@rhynorater) (28:08.877)
We're gonna get some Sam Erb shit right now, dude. This is about to be... Ha ha.
Justin Gardner (@rhynorater) (28:19.809)
Mm.
Justin Gardner (@rhynorater) (28:26.595)
Yeah.
Sam Erb (28:27.894)
That absolutely happens in this industry. I know what this framework looks like, or I know this is a WordPress site, just gonna cast it aside or look at extensions and that's it, for example. But that first time you see something, it's like, yeah, you have to put in the time to understand it. And going back to what you were saying, looking for the, trying to understand what the quickest way is to find bugs.
Justin Gardner (@rhynorater) (28:39.272)
Mm-mm.
Sam Erb (28:57.094)
I really think having a way to search or categorize your past reports would go a long way towards that but then you have to have that catalog of past reports. It comes up with me actually a lot with XXE type bugs, like that whole, any variant of an XXE.
Justin Gardner (@rhynorater) (29:05.281)
Mm.
Sam Erb (29:18.606)
over time I've kind of built up like a catalog of like past reports and it's like it's very easy to just go in there and find working exploits. But if you search on the internet it's you know they might work, they might not work, might only work in certain contexts. And so you kind of develop this catalog of things you found and therefore can just leverage that to go find again.
Justin Gardner (@rhynorater) (29:24.271)
Mm.
Justin Gardner (@rhynorater) (29:37.613)
Yeah, yeah, that makes sense. And specifically on targets, like you mentioned with the Airbnb thing, I mean, some targets are just gonna be vulnerable to the same thing over and over and over again, just by nature of their software development life cycle and their inner culture decisions on how to do authentication and authorization and that sort of thing, right? And so that's definitely good advice. And I guess what I mean a little bit, just going back to the whole volume of attack vectors sort of vibe,
I don't mean to... I still think this methodology could be helpful for beginners because it's so easy to get...
lost in all the things you need to study, and all the things you need to understand. But I really liked your scenario of like, we all needed to test a WordPress site a couple times before we knew that the vulns are gonna be in the extensions. And so, I don't know, it's kind of a hard thing to normalize because I feel like there should be some way to get the 80-20 rule out of this.
Sam Erb (30:34.081)
Yeah.
Justin Gardner (@rhynorater) (30:47.269)
For us to be really thorough on a target like we are in the live hacking events, you've got to really go 100%. But in order to find 80% of the bugs, with 20% of the effort, there should be some sort of algorithm that should lead us down that path. But it's not optimizing for attack vectors, I can see. That does make sense, Joel, that you could just endlessly test a bunch of random things. So I don't know. It's hard to nail down.
Joel Margolis (teknogeek) (31:14.785)
Yeah, it's definitely a tricky thing. I think a lot of different automation stuff has tried to capture that, where whether it's the DNS, a lot of that, I'd say 80%, maybe not 80%, but maybe 60% is DNS recon, subdomain takeovers, simple XSSs, vulnerable software that's been deployed that is online. There's a lot of that stuff that's just easy.
sort of low hanging fruit that all you have to do is find it in order to exploit it. And then the rest is like, sort of like authenticated testing, like having to go deep, having to like actually understand the relationships between different parts of the application and identify how you can leverage that or exploit it to pop, oftentimes the same types of bugs, but just in a different context.
Justin Gardner (@rhynorater) (32:04.505)
Hmm. Yeah, I'm sorry, Sam. We're getting a little bit away from the topic now. But but you know, it does it does make me.
Joel Margolis (teknogeek) (32:09.721)
But.
Justin Gardner (@rhynorater) (32:18.069)
Yeah, oh jeez. Well, let me let me while you fix that, let me let me play around with this. Yeah, it does make me think though, you know, like, yeah.
Joel Margolis (teknogeek) (32:24.389)
I probably should have run out of storage space.
Joel Margolis (teknogeek) (32:29.837)
Wait wait wait. It's not recording.
Justin Gardner (@rhynorater) (32:33.514)
It's recording on my side.
Joel Margolis (teknogeek) (32:35.509)
I know, but his browser ran out of storage space, so it's not recording. Yes.
Justin Gardner (@rhynorater) (32:38.933)
His recording has stopped.
Justin Gardner (@rhynorater) (32:44.817)
Okay. Gotcha. Mm-hmm. Now you're good.
Justin Gardner (@rhynorater) (33:03.361)
I don't know man, I just think there's something to be said for volume as well. You know? Like there's something to be said for, okay, you know, I'm gonna just go and try all the things I can think of. But maybe that's not the exact, exact approach. Yeah, that seems to be working better now.
Sam Erb (33:16.619)
Okay.
How's that?
Joel Margolis (teknogeek) (33:19.769)
Yeah, yes, we're good
Sam Erb (33:22.152)
Okay, hopefully you didn't lose any recording there.
Joel Margolis (teknogeek) (33:24.713)
No, no, it'll put you in as basically like a separate person will stitch it in.
Sam Erb (33:28.503)
Okay.
Justin Gardner (@rhynorater) (33:28.553)
Yeah.
Justin Gardner (@rhynorater) (33:32.297)
Alright, well let's move along from a classic Justin tangent there. I'll continue to keep on noodling on how to... Dude, it's frick, he knows what's coming, man. Yeah, you know, I don't know. I feel like there should be some way, but Joel, we'll save this for a time when we can really hash this one out. So one of the things that I wanted to ask...
Sam Erb (33:38.231)
Thanks for watching.
Joel Margolis (teknogeek) (33:39.525)
I know I'm about to have a 30 minute debate with Justin about this after- hh
Justin Gardner (@rhynorater) (33:58.985)
I guess, sort of tangentially related to the debate that we just had there on how to optimize for bugs. Do you think you, and this is kind of a gut feeling I've been wanting to ask a bunch of the people that come on the pod, how much of your time do you think you spend thinking about the bug versus actually testing something? So let's say you've got a scenario where you've got like a bug you're trying to exploit, right?
Do you often just sit back, stare at the ceiling, think about the bug? Or are you constantly iterating and trying new things and brute forcing this and that sort of thing? If you had to guess, what would your ratios be between those two?
Sam Erb (34:43.847)
It's really terrible, honestly. So I spend far too much time throwing everything I think of at bugs. Like I know I'm in trouble when I start going into burp and like using the hex fields to send every valid character through to something like that means I'm probably at a little bit too deep. But, but then like I'll more often than not, like when things aren't like super obvious, I'll find bugs when I'm not, when I'm staring at the ceiling, you know, or equivalent.
Justin Gardner (@rhynorater) (34:50.279)
Mm.
Justin Gardner (@rhynorater) (34:57.372)
Yeah.
Joel Margolis (teknogeek) (34:57.765)
Mm-hmm.
Justin Gardner (@rhynorater) (35:01.216)
Yeah, yeah.
Justin Gardner (@rhynorater) (35:13.239)
Mm-hmm.
Sam Erb (35:14.003)
Um, like that stepping back for me anyways, and thinking about how the system works and like, you know, thinking about the system as a whole, often leads to some of the better findings that I've had. Um, you know, you're, at the end of the day, it's like you're trying to, you're trying to outsmart a lot of other people who are thinking about the security and like, you're not, you have your own unique perspective, but like that's...
Justin Gardner (@rhynorater) (35:27.276)
Mmm.
Justin Gardner (@rhynorater) (35:34.626)
Yeah.
Sam Erb (35:42.283)
That's also something that other people are going to consider potentially. So you have to find your angle, but also find a way to find something unique.
Joel Margolis (teknogeek) (35:45.806)
Yeah.
Justin Gardner (@rhynorater) (35:45.921)
Mm.
Joel Margolis (teknogeek) (35:56.109)
So on that topic, do you have any sort of tips that you like to use when? So the way that I'm thinking about this is basically, if I was hitting a dead wall or a brick wall, oftentimes what I would try and do is figure out what is the backend running? Or how do I identify more parts of this infrastructure? Do you have any things that you like to do, like just sort of your go-to things that you like to try and throw in there to see if you can figure out more about the stack?
Sam Erb (36:06.474)
Mm-hmm.
Sam Erb (36:24.179)
Yeah, and one of my favorite things to try and do is to try and understand how their development environment works. Like, I'll go and find a development environment, find their origin servers, like, see if I can get access to those and poke around and see if things can get reflected into prod or vice versa. Just understand more about their deployment environment and then also their development environment, and even search on GitHub, for example.
for not, not for secrets, so to speak, but just for like development related things. Yeah, yeah, yeah. Just anything you could find that would like, you know, you find somewhere like on GitHub, like a random IP address that happens to be the origin server and like that, that can be huge. Like that could be all you need for certain bugs.
Justin Gardner (@rhynorater) (36:56.217)
Data. Yeah, for knowledge.
Justin Gardner (@rhynorater) (37:05.145)
Yeah.
Justin Gardner (@rhynorater) (37:08.621)
Yeah, dude, see, I love this so much. This is great because listen to the way Sam talked about that, right? He said, I wanna learn more about the development environment, so I go and find the dev servers, right? And that's not, Sam is not looking at those dev servers as like a separate entity of sorts, right, in this whole system. He's looking at it as a part of the whole, right? And you can use that. That.
that extra piece to give you knowledge about prod. Or you can find this little piece of data on GitHub that can, when you actually read the source code and you look at the open source libraries that they have that they're no doubt using in their own platform, can give you all the little pieces you need to get deeper and deeper and deeper. That's such great advice, Sam.
Sam Erb (37:54.451)
Well, yeah, and that's also been how you'll find a lot of shared secrets between development and production. And development won't be as locked down, so you just get access to the secrets more often than not. Or there's also been bugs where it's like, kind of like what you're talking about, you're using something open source, but that open source library happens to use a static secret somewhere. That's very common.
Justin Gardner (@rhynorater) (37:58.87)
Yeah.
Justin Gardner (@rhynorater) (38:03.575)
Yeah.
Justin Gardner (@rhynorater) (38:12.649)
Yeah. Yeah, I actually, man, where was it? I saw recently you found something that had a static secret. It was like a GitHub security issue that I saw pop up somewhere. I was like, where'd it go, Sam? But I'm trying to remember what product it was on, but I'll look it up afterwards and link it. Sam's like, yeah, you're gonna have to be more specific, Justin, that happens to me every day.
Sam Erb (38:26.479)
Yeah, yeah, um, I don't
Oh god. Okay, yeah, I know exactly what you're talking about. I don't recall exactly what I said though.
Sam Erb (38:38.841)
No, it's earlier this year and I just don't recall the product. Oh, that's terrible.
Justin Gardner (@rhynorater) (38:41.117)
Yeah, no, you're good. Yeah, so I guess the last thing that I wanted to kind of talk to you about in sort of the hacker methodology section.
Justin Gardner (@rhynorater) (38:52.193)
Collaboration, right? What role, I mean, obviously we've collaborated together and collaborating with Justin, special experience there. But talk to me about collaboration, how it works in with your methodology, how it motivates you or how it makes you perform different in a live hacking event or even just in a bug bounty context.
Sam Erb (39:16.655)
Yeah, like the people that I collaborate with, more often than not, it's like.
I don't know, I end up putting like, higher expectations on myself than the other person. It's probably outside to describe it. I don't want to collaborate unless I know I can give like 110% to that target. Well, so I just won't even recommend it. Like if I'm like, I don't know, if I'm like off doing something else, I'll be like, yeah no, sorry, I just can't. And it's not because I don't want to collaborate with you. It's...
Justin Gardner (@rhynorater) (39:29.125)
Mm-hmm. Yeah.
Justin Gardner (@rhynorater) (39:35.873)
I've noticed that you do that. I love collaborating with you, Sam. Ha ha ha.
Justin Gardner (@rhynorater) (39:44.437)
Yeah.
Joel Margolis (teknogeek) (39:47.717)
I really respect that you have that boundary though. For me, I have this really bad habit of just saying yes to everything. So here I am, it's December, I currently have about 15 side projects going on, and I'll be doing three of them today.
Justin Gardner (@rhynorater) (39:53.462)
of getting nerd sniped.
Sam Erb (39:54.099)
Oh yeah.
Sam Erb (39:59.24)
Yes.
Justin Gardner (@rhynorater) (40:03.515)
Yeah, dude, Sam, it seems like you've got some healthy boundaries, man, in place with Bug Bounty, which is cool. And yeah, well, can you talk a little bit about that? Because to be fair, I think we had a conversation about this a little bit. You've been in this tech world a little longer than some of us hackers have. And you've told me some stuff that I...
Sam Erb (40:08.539)
It used to be less healthy. Yeah, I'll be honest.
Sam Erb (40:22.871)
I am old. Relatively.
Justin Gardner (@rhynorater) (40:28.989)
you know, that I appreciated and that, you know, you've learned from, from the, the wisdom of being in the industry for a little bit longer. How have you, I guess, gotten Bug Bounty to a point where you have a better work life balance with it and, you know, a better relationship with Bug Bounty and are able to control those sort of obsessive tendencies, right? Because it doesn't seem like you're, and maybe it's, it's just by, by virtue of the fact that you're not in it every single day because you...
you know, have a full-time job and you kind of distance yourself from it in that regard, but I know you also have, you know, a family and that sort of thing and you have other responsibilities there. And I'm just wondering how it all gets balanced or if there's any tips you've learned along the way.
Sam Erb (41:13.399)
I mean, for me, the biggest thing is like I've stopped looking at a lot of random programs. I think it's probably the best way to describe it. Like I used to, you know, every invite I used to be like, oh, I'll poke at this. And now it's like, you know, you get that invite and it's like, well, okay, like if I have time for this, you know, or like, but then like, I will try and like, when I do choose to do like a live hacking event or like, you know, especially some of the, um,
Justin Gardner (@rhynorater) (41:19.905)
Mm.
Sam Erb (41:41.439)
like US government really about the value stuff. It's like, when those come up, it's like, okay, like I'm gonna make a point to like set a time side for this, like, you know, let folks in my life know essentially like I'm gonna be spending these nights like doing this thing and like.
Justin Gardner (@rhynorater) (41:51.125)
Yeah, especially when you're married. Yeah, you've got that level of communication you've gotta get through or else it's not gonna work.
Sam Erb (41:59.975)
Yeah, yeah, like you're like, I'm gonna work all day and then do this all night. Like you probably won't see me too much, you know? It's like, yeah, it's like that. You know, that's not sustainable.
Justin Gardner (@rhynorater) (42:03.445)
Yeah. Yeah, exactly. Yeah.
Or maybe you shouldn't say it like that. Maybe you should say, hey, sweetie, you know, what did you think if I tried to earn a little extra money? Yeah. No. Yeah.
Joel Margolis (teknogeek) (42:14.949)
I know you wanted a new car, so.
Sam Erb (42:18.183)
Yeah, right. And I mean, like, I don't know, you also, like, I remember the first bounty I got, and this was like the first like hack the Pentagon thing, and it was like $150. I was like, this is wild. Like, I've never seen so much money before in my life, you know? And now it's like, ah, yeah, you know, I guess I'll respond to this thing because it's gonna earn me another $500 or something. I'm like, that's, it's, you become very...
Justin Gardner (@rhynorater) (42:28.461)
Mm.
Yeah.
Justin Gardner (@rhynorater) (42:40.498)
Yeah, yeah, yeah.
Sam Erb (42:44.115)
It's both being jaded, but then also like, yeah, disassociating the, the bug bounty space with the money to some extent. Um, which like, I don't know, we're earning good money here. At the end of the day, it's like, you know, we're in, we're in a good position to be able to say no to certain things too, as well.
Justin Gardner (@rhynorater) (42:45.165)
desensitizing.
Justin Gardner (@rhynorater) (42:51.593)
Yeah, 100%.
Justin Gardner (@rhynorater) (42:57.218)
Yeah.
Joel Margolis (teknogeek) (42:57.805)
Yeah.
Justin Gardner (@rhynorater) (43:04.761)
Dude, that's so good, yeah. And I think that's a part of the guttural response of being successful in bookbounding in the beginning is like, ah.
I just got a bounty, like maybe I can do this more. And you're just like sort of constantly chasing that hit of the bounty, right? It's almost like a gambling thing, right? But once you learn to control that, once you, you know, and I talk about this a little bit as well, is like for me it was a part of an identity thing for me. Like, ah, I haven't fulfilled my.
Sam Erb (43:22.423)
Ha ha
Justin Gardner (@rhynorater) (43:40.061)
identity that I want to be, who I want to be if I'm not at the top of the leaderboard at every event, that sort of thing. Just redefining that away has helped me build some healthy boundaries. It sounds like for you, saying no to some opportunities, whether it be a private invite or maybe even a live hacking event that you just can't make it work right now, it's not going to be healthy. Saying no to those things have helped you build a more healthy relationship with Bug Bounty.
Sam Erb (44:08.711)
Yeah, yeah. And it's also, I mean, to be fair, one of the benefits of having a full-time job and kind of doing this like nights and weekends is like you do, you can just pause. Like there's, you still have health insurance. And I know you've talked about this as well. There are ways around this in the US. But like, you know, self-health insurance, you still have a paycheck coming in, like your bills are still paid. And it's nice to have that. Yeah.
Justin Gardner (@rhynorater) (44:14.476)
Hmm.
Justin Gardner (@rhynorater) (44:18.223)
Yeah.
Yeah.
Justin Gardner (@rhynorater) (44:28.342)
Yeah, that's good.
Justin Gardner (@rhynorater) (44:31.913)
Miss those days. I mean, to a degree I do, you know, and I love the day to day. But for me, you know, as a full time hunter, what that looks like is like realizing, okay, I don't actually have to be working all the time because the payoff, you know, the risk is greater and the reward is greater, you know? Um, and so, you know, if I need to take a couple of days here and there to like decompress and chill and, you know, not be working constantly, then I should be able to do that. But. You know,
you know, whenever I figure that out. All right, man, so that was a lot of the hacking methodology stuff. Let's jump back and look at the career a little bit because this podcast is primarily about bug bounty stuff and we'll definitely come back to more bug bounty stuff, but I also am the host of this podcast and I have some very, I'm very interested in career development and entrepreneurialism in particular, so I'd really like to talk to you about your career and then also the whole thing that you did
Sam Erb (45:03.159)
Thanks for watching!
Justin Gardner (@rhynorater) (45:31.597)
buffer over dot run and that whole experience. So start from wherever you'd like with the career thing.
Sam Erb (45:39.065)
Yeah, I was kind of going back a little bit, undergrad, I was a computer systems engineering, I think Joel was as well, right?
Justin Gardner (@rhynorater) (45:45.186)
Mm-hmm.
Joel Margolis (teknogeek) (45:47.041)
Well, I was, yeah, pretty much. Yeah, security engineering. I had a security engineering major when I went into college, so.
Sam Erb (45:49.535)
Yeah, okay.
Sam Erb (45:53.679)
Oh nice. Oh that's awesome. Um, yeah so I was like... okay.
Justin Gardner (@rhynorater) (45:54.873)
That's cool. Well, I'm just gonna stop you right there. No. No, no, no. Well, I'm gonna stop you right there. So, computer engineering versus computer science versus whatever the heck Joel did. What are your thoughts on that and what are the ways you think it's impacted your career?
Joel Margolis (teknogeek) (45:59.386)
I didn't know how to say that without making you sound old
Sam Erb (46:02.171)
Yeah, yeah. I didn't exist for a while, so yeah.
Sam Erb (46:18.067)
I actually dual majored with math and I thought I was going to be a cryptographer. And I found that the computer's engineering courses were pretty straightforward and I really struggled with the math courses. So I was like, well, maybe I should just go and do this computer thing. But honestly, I think really any of these will give you a good background. I think that that's like learning the fundamentals will...
Justin Gardner (@rhynorater) (46:22.902)
Yeah, I could see that.
Justin Gardner (@rhynorater) (46:31.266)
Mm-mm.
Yeah.
Sam Erb (46:45.259)
get you a lot of the way to being able to like really easily pivot into bug bounty, the bug bounty sites if you want to. Um, you know, for me, like starting bug bounty stuff was less about learning the security side of it and more about learning how burps we worked.
Justin Gardner (@rhynorater) (46:51.367)
Mm, mm.
Joel Margolis (teknogeek) (47:04.505)
Yeah, I totally see like so many people who are like technology adjacent, who are like, I don't know how to get started. And it's like, do you see something you don't know what it is? Just look it up and like start following every rabbit hole that you see. And that's the best way to get started. It's just like, take one step and just like slide down the hill.
Justin Gardner (@rhynorater) (47:05.7)
Damn.
Sam Erb (47:14.742)
Yeah.
Justin Gardner (@rhynorater) (47:19.957)
Yeah.
Sam Erb (47:20.208)
Yeah.
Justin Gardner (@rhynorater) (47:24.545)
Yeah, and also like you said, Sam, you know.
But it can also, you see a lot of people that aren't technical, just know how to use Burp Suite and know how to find iDoors and know, have a basic understanding, yeah, of how the browser works, right? And you'll find your XSSs, your iDoors, your CSURFs, your access control issues, and that's more than enough to pay the bills. If you know those four vuln types, iDoor, XSS, CSURF, access control, or client-side implementation of access controls, I mean, you can make a lot of money.
Sam Erb (47:33.395)
You can make good money that way, yeah.
Sam Erb (47:51.104)
Yeah.
Joel Margolis (teknogeek) (47:54.981)
And another thing that you can do is you can kind of target those as well. So you can like, if you say you work in some industry that's like maybe a little more obscure, but it's still technology adjacent, you probably have experience interfacing with some software that like most people don't use. And so you can probably find vulnerabilities in that way better than other people can because you're used to this software. You might even already kind of know a vulnerability and not realize it's a vulnerability and it's just like, oh yeah, this is kind of like works this way. And it's like, it's probably not supposed to work that way.
Justin Gardner (@rhynorater) (48:11.437)
Mm.
Yeah.
Justin Gardner (@rhynorater) (48:22.573)
Hmm. Yeah.
Joel Margolis (teknogeek) (48:25.928)
So I'd recommend if you have experience with that kind of stuff, try and see if they have a program or see if they have a security team and try and find vulnerabilities because that's also a good place to start.
Justin Gardner (@rhynorater) (48:35.713)
100%, yeah. Yeah, sorry Sam for the interruption. I just, you know, the computer science versus computer engineering thing is kind of close to my heart because I see, you know, people like you and Joel that have a little bit lower level understanding of some of the stuff that I'm missing and I've had to claw that back, you know, since I've graduated because my CS program was focused entirely on programming.
Sam Erb (48:38.127)
Yeah, that's so good. Oh no, it's all good.
Justin Gardner (@rhynorater) (49:00.045)
it was like 100% programming. And now I'm like, I don't really understand some of this lower level stuff, and I've had to regain that knowledge as a professional. So my advice, I guess, to people that wanna go to college, which is a choice in and of itself, is to go computer engineering or security engineering, degree to get that lower level aspect.
Sam Erb (49:23.703)
Yeah, and I've always kind of looked at it from like, you don't... what you learn there is almost, to me anyways, it was kind of irrelevant to what I actually ended up doing, but it was more of the exposure to all these like different technologies, different ways to do things, and then you can then apply that later. Like, you know, never in my life will I ever write MIFS assembly again, but I knew how to at some point.
Justin Gardner (@rhynorater) (49:31.537)
Mm-mm.
Justin Gardner (@rhynorater) (49:38.967)
Mm-hmm.
Justin Gardner (@rhynorater) (49:43.777)
Yeah, yeah, but I mean, that's the thing that doesn't make you scared of it though, right? Like if you talk to a web developer.
Somebody like me years ago, whenever I was just writing PHP and all that sort of thing, and talk about assembly, it's like, ah, that's that weird shit, where you're putting stuff in registers and stuff like that, right? It's intimidating, and I think exposure to it is one of the things that can help earlier on in your education. You don't need to know how to do.
everything for your job, right? Because you're going to have to learn a lot of it on the job. But just that base level exposure, I think is helpful.
Sam Erb (50:21.747)
Yeah, and I will say, one thing I wish I had taken more of were like, and I don't really even know if these existed back then, where it had to be a software dev type courses, because I think that would get you much further today than a lot of the college courses that I took.
Justin Gardner (@rhynorater) (50:30.543)
Mm-mm.
Justin Gardner (@rhynorater) (50:35.689)
Yeah, yeah man, yeah, if we could switch it and I'll trade you the how to via software dev stuff from my college and you can give me like, you know, OS 101 or something like that. All right, so college, computer engineering, from there, where'd you go?
Sam Erb (50:43.251)
Yeah. Yeah, exactly.
Sam Erb (50:52.871)
Yeah, so I went to work at Cisco and I was doing like working on VPN technology, the ASA, firewall, VPN aggregator. I actually wrote very minimal amount of code there. You know, a lot of the work I was doing were like integrating new platforms and stuff. But the folks there, I mean, they're brilliant at what they do, to be honest, trying to make all that work all in a single...
Justin Gardner (@rhynorater) (51:01.949)
Ah dude, you're the one that wrote all that Bucky code that's getting popped every- HAHAHA
Justin Gardner (@rhynorater) (51:13.028)
Yeah.
Justin Gardner (@rhynorater) (51:17.687)
Mm, mm, yeah.
Sam Erb (51:21.451)
X's like box is kind of nuts. But somebody there, it was like, hey, like you like security, you should go to Defcon. And so I guess actually kind of back it up a little bit even further, actually this is like dating myself, but like I started listening to the Security Now podcast, like right when it first launched. Yep, it spawned out of that. And like that to me actually got me interested in a lot of this.
Justin Gardner (@rhynorater) (51:29.317)
Mm-hmm. Classic.
Justin Gardner (@rhynorater) (51:40.721)
Yeah, dude. Was that, it's twit, right? Yeah.
Sam Erb (51:51.451)
into this whole world, honestly, because the way that they explained it, especially in the earlier episodes, was like very, very basic. So I went back to fundamentals very well there. And I don't know, I just have a huge amount of respect for them, especially because they're still going, which is just nuts.
Justin Gardner (@rhynorater) (51:58.119)
Mm, mm.
Justin Gardner (@rhynorater) (52:03.945)
Yeah, dude. Years later, man, it's your podcast OG, because that show's been around for forever. Wow.
Sam Erb (52:08.627)
It was like 2004, I wanna say. Yeah, I was listening to like, was it iPod mini or something? Yeah. Like downloading it over. But anyway, sorry, going back to, so I was at Cisco and I was like, hey, you should go to DEF CON. And so I went there and I was like, I have no idea what to do. Like, I don't really just wanna go to a talk. So I walked into the like the puzzle room and Brett was sitting right there. Yeah. How nuts is that? Sorry.
Justin Gardner (@rhynorater) (52:13.945)
Oh my gosh. It's amazing.
Justin Gardner (@rhynorater) (52:27.751)
Mm-hmm.
Justin Gardner (@rhynorater) (52:31.553)
Ah really? That's awesome. Zayit, yeah? Yeah, yeah.
Sam Erb (52:39.451)
And yeah, so we ended up competing for a few years. We won a few black badges, which was awesome. And then he was like, I think I helped him. I don't remember exactly the timeline here, but at some point, I helped him find a zero day RCE in Ruby, like a Ruby library. And he was like, hey, you should come to these live hacking events. And so I went to one of them and like.
Justin Gardner (@rhynorater) (52:44.962)
Yeah.
Justin Gardner (@rhynorater) (52:55.422)
Oh my gosh.
Justin Gardner (@rhynorater) (53:00.205)
Heck yeah you should.
Sam Erb (53:04.599)
I was like there, I was doing something with the curl commands, and I was watching everyone use burp, and I was like, what is this magic?
Justin Gardner (@rhynorater) (53:10.53)
He's using curl. He's hacking it with curl
Joel Margolis (teknogeek) (53:13.421)
Yeah, I remember a similar thing. When I started Bugman, I was using like Charles and everybody's using this thing called burp and I was like, burp, you guys should try Charles. And they were like, what the hell is Charles? Ha ha ha.
Sam Erb (53:14.096)
I knew exactly what they were...
Sam Erb (53:18.292)
Yeah!
Justin Gardner (@rhynorater) (53:23.298)
Oh my god, they're like, no.
Sam Erb (53:27.135)
Well it's like I knew exactly what everyone was doing but I just couldn't keep up with them, you know?
Justin Gardner (@rhynorater) (53:31.045)
Oh yeah, I mean if you're using curl and freaking inspector or DevTools versus burp, you're gonna be in trouble.
Sam Erb (53:38.419)
Yeah. Yeah, and so from there I was like, I think I actually said to Namsak, was at that event and I actually said to him, this is back when he worked at Hacker One, I said to him, I was like, I'm gonna get better at this, I wanna win one of these events one day. That was the event I won, yeah. Yeah, so I mean, that's pretty much it, to be honest. From there, it's like I went to work at Akamai, working on some TLS software and then...
Justin Gardner (@rhynorater) (53:45.479)
Mm.
Justin Gardner (@rhynorater) (53:48.651)
Mm.
Justin Gardner (@rhynorater) (53:54.453)
Lo and behold, he did. And then he did. That's great. Yeah.
Joel Margolis (teknogeek) (53:56.166)
That was your superhero arc.
Sam Erb (54:07.679)
I helped start a red team there kind of like part time with my job. Um, and that really pivoted me further into the security engineering world. Um, and then yeah, a year and a half ago, I joined Google. I was a security engineer.
Justin Gardner (@rhynorater) (54:17.165)
Hmm. Nice dude. Yeah. No, that that's a good, that's a good journey. And so the CTF thing, you know, you rolled into DEF CON, you meet, you meet Brett and then you guys just kind of hit it off and start like popping stuff on CTFs. That's, that's the, that's how you got into that space.
Sam Erb (54:33.927)
It wasn't as much CTF as it was like puzzles, which is like a different world, but it quickly pivoted into the bug bounty world.
Justin Gardner (@rhynorater) (54:37.728)
Okay.
Justin Gardner (@rhynorater) (54:42.037)
Yeah. And so talk to me about the black badge thing. Like that's pretty bad ass for one. And I think you have multiple or one or you have two. So how, how did, how? Like that, that's so amazing. And how did, how did this ETF come about and who's on your team? Give me the deets.
Sam Erb (54:49.135)
Yeah, so our team won two. Um, yep.
Sam Erb (54:54.575)
Um
Sam Erb (54:59.599)
It was, in retrospect, it doesn't feel like a real experience. But we got a team together of like, there were 10 to 13 of us, and these puzzles were put on by Lost1057, who's like the organizer at DEF CON, or was one of the organizers at DEF CON at the time, making the badges and everything. And so he would put together these puzzles as well, and our team would compete against other...
Justin Gardner (@rhynorater) (55:05.544)
Mm-mm.
Justin Gardner (@rhynorater) (55:10.447)
Mm-hmm.
Justin Gardner (@rhynorater) (55:16.009)
Mm-hmm.
Justin Gardner (@rhynorater) (55:21.386)
Mm-hmm.
Sam Erb (55:26.639)
similarly sized teams and we'd spend nothing, we'd do nothing else during DEF CON. Like that was our DEF CON. Yep.
Justin Gardner (@rhynorater) (55:32.525)
So is only DEF CON, like you show up DEF CON and you get the info and then you start hacking. Wow.
Sam Erb (55:37.619)
Yeah, I mean, we would go as far as like, we would actually, you know, and I think I've actually gave a talk on this. Part of my TLS recon was scanning the internet for, originates, origin story of the TLS recon, was scanning the internet for secrets before, yeah, before DEF CON launched, like the week before, yeah.
Justin Gardner (@rhynorater) (56:00.997)
Defcon launched? No way, you're trying to like find the servers in advance of the oh my gosh that's amazing.
Sam Erb (56:08.443)
Yeah, we were successful. Um, but, uh, it turns out somebody, so we were monitoring the entire dot codes TLT, like everything, like every single domain and somebody somehow, one of the competitor competing teams got wind of this and they put up a fake puzzle and we spent three days solving this and the last step was, you know, email, email lost and say you've won this and so we sent him an email. He's like, what the hell? Like this wasn't me.
Justin Gardner (@rhynorater) (56:16.245)
Yeah, yeah, yeah.
Justin Gardner (@rhynorater) (56:27.385)
Oh, no way! No!
Joel Margolis (teknogeek) (56:31.272)
Oh no.
Justin Gardner (@rhynorater) (56:38.761)
What the heck? Wow, that is some that is some grey hat shit right there, dude. That's nuts It's like freaking ARP spoofing people at a live hacking event or something. Jeez
Joel Margolis (teknogeek) (56:39.213)
Wow!
Sam Erb (56:41.02)
Yeah.
Sam Erb (56:46.775)
Yeah, I mean, it certainly like...
It was three days roughly of our time where we had nothing but solve these puzzles. Like to be fair, that was really well done, but like they were all faked.
Justin Gardner (@rhynorater) (56:57.401)
Dude.
If I was you, I would have been pissed. Yeah. Wow, dude, that's crazy, man. What an experience.
Sam Erb (57:02.887)
I mean we're just impressed by anything else, we're a little disappointed but...
Sam Erb (57:13.383)
Yeah, those were fun days. I think the puzzle's still around. I'm not quite sure if it's still all black badge, sure not. I've kind of drifted a little bit. DEF CON for me has become more of like a volunteering thing than a precipitation thing, but.
Justin Gardner (@rhynorater) (57:20.334)
Mm.
Justin Gardner (@rhynorater) (57:28.045)
Right, right. Now that makes sense. So we'll come back to some other career stuff, but you mentioned the TLS scanning thing. Talk to me a little bit about bufferover.run, both dns. and tls. And tell me a little bit about that project and how it ended up.
Sam Erb (57:46.631)
Yeah, so the DNS.inpoint was just, I would take the, these are all just like, honestly, these are all just engineering efforts, like, yeah. Yep.
Justin Gardner (@rhynorater) (57:53.161)
Okay, let me also just add a little bit. So, Sam ran a service, DNS.bufferover.run and TLS.bufferover.run. These were tools that you could use for reconnaissance. So, you're saying DNS. is an endpoint where, yeah.
Sam Erb (58:08.275)
Yes, they're both free. You know, the DNS endpoint, it... Sorry, I just got a... Something just dropped, who knows what that was. Sorry. So the DNS endpoint was one where I would take the rapid7 public data and I would just index it and make it searchable because the rapid7, it was project sonar data at the time was publicly available, but you would just get these like...
Justin Gardner (@rhynorater) (58:19.044)
You're good.
Justin Gardner (@rhynorater) (58:27.416)
Mm, yeah.
Justin Gardner (@rhynorater) (58:33.058)
Yeah.
Sam Erb (58:35.303)
You know, it was like seven gig downloads or something absurd. And they would be published like very often, like weekly. And it was just like what's on the internet essentially. And so I think that they had like hooks into a bunch of DNS infrastructure and then they just made it available, which was great. And then they realized that they could charge for it and they started charging for it. Yeah. Oh yeah. They inked it. I don't think it's free anymore.
Justin Gardner (@rhynorater) (58:45.074)
Mm.
Justin Gardner (@rhynorater) (58:50.642)
Awesome.
Justin Gardner (@rhynorater) (58:56.569)
Did they ink it? No!
Justin Gardner (@rhynorater) (59:03.121)
Oh my gosh, I haven't looked into that in so long.
Sam Erb (59:05.127)
It's been years, yeah, like two or three years now.
Joel Margolis (teknogeek) (59:08.409)
It says that you can still get data. You might have to register for it, but. Oh no, it's actually, yeah. 47, 74 terabytes of data. You still get all the, all the stuff. It was last updated, uh, as early as today.
Sam Erb (59:11.671)
I think it... yeah, I think it requires registration at least. I thought...
Justin Gardner (@rhynorater) (59:19.865)
Holy moly.
Sam Erb (59:26.339)
Oh, interesting. Okay. Sorry, I didn't.
Justin Gardner (@rhynorater) (59:27.593)
Existing open data users can sign in for access it says. Yeah, I think you're right Sam.
Joel Margolis (teknogeek) (59:34.305)
Yeah, yeah, you got to sign up for and like request access, but after that. They, they listed all at least.
Justin Gardner (@rhynorater) (59:39.758)
Okay.
Sam Erb (59:40.571)
Yeah, it was nowhere publicly available. And if I recall the time I felt it was a little bit sketchy, like re-hosting something that you had to get through a registration page for. Yeah.
Joel Margolis (teknogeek) (59:48.117)
Yeah, that's probably, probably part of why they did it.
Justin Gardner (@rhynorater) (59:48.557)
Oh yeah. Oh yeah, that makes sense. Yeah, they're like, hey, this guy's making it accessible.
Sam Erb (59:56.244)
Yeah.
Joel Margolis (teknogeek) (59:56.405)
Stingy enough someone is constantly pulling this down
Sam Erb (01:00:00.411)
Yeah, right. But I mean, to be fair, no, I was only doing like once a week. You know, I had a little like, I don't know how to call it. No, it was the smallest VPS I could possibly buy. I was one of those European VPS hosters that was like, you know, like two euros a month. Yeah, there was, I don't recall exactly which hosting provider it was.
Justin Gardner (@rhynorater) (01:00:05.246)
Mm-hmm.
Justin Gardner (@rhynorater) (01:00:08.825)
Cron job or whatever.
Justin Gardner (@rhynorater) (01:00:16.447)
Mm-mm.
Joel Margolis (teknogeek) (01:00:19.605)
Oh, that's awesome.
Sam Erb (01:00:26.387)
but they were European and yeah, for whatever reason, it was just enough power to run this web server. It was right at like 70% CPU, like 100% of the time, but it stayed up. So it essentially cost me nothing to run it. Yeah.
Justin Gardner (@rhynorater) (01:00:34.038)
Oh jeez.
Justin Gardner (@rhynorater) (01:00:38.865)
It stayed up, question mark. And then after you, so what you did with DNS prep was you took that whole data set, extracted it down a little bit, sorted it so it'd be really easy to jump through the indexes and then you made it, you know, exposed an HTTP API for that, right?
Sam Erb (01:00:46.195)
Yep. Yes.
Sam Erb (01:00:57.055)
Yep, exactly. Yeah. And you could download like good amount of data, like a hundred megabytes ish or so at a time if you wanted to from that end point. Um, and I actually ended up putting, uh, at the time, CloudFlare in front of it just because like I would get like 90, 95% offload cause everyone's searching for like yahoo.com. Like, you know, like everyone's searching for the same thing as a recon pipeline. So I was like, all right, I'll just put CloudFlare in front of it. And they offloaded like, like just an absurd amount of data for me. Um,
Justin Gardner (@rhynorater) (01:01:04.823)
Nice.
Justin Gardner (@rhynorater) (01:01:12.481)
Oh my gosh. Yeah, oh yeah, of course, yeah, yeah.
Justin Gardner (@rhynorater) (01:01:22.713)
That's amazing.
Joel Margolis (teknogeek) (01:01:23.413)
Was there a lot of people who were like hitting you as well, like hitting your APIs essentially and like done the same thing where they were like now taking all of your data and farming it into their own systems or okay.
Justin Gardner (@rhynorater) (01:01:32.589)
He... Oh yeah, 100%.
Sam Erb (01:01:32.615)
Oh, absolutely. Yeah. Well, it is. So I'll talk about that in a second with the TLS endpoint, because that one was much worse, honestly. So with the TLS endpoint, I, um, I implemented a system where I was like, without, I don't know how much detail I should get into here without like, I'll explain why in a second, I guess. Um, uh, but I was scanning the public internet for TLS search and I used like.
Justin Gardner (@rhynorater) (01:01:55.641)
Mm-hmm.
Sam Erb (01:01:58.16)
I have a lot of TLS backgrounds, so I use every trick I can think of to make it as fast as possible. I wrote my own modified and existing TLS library to make it work.
Justin Gardner (@rhynorater) (01:02:06.641)
Oh my gosh, Sam, what is wrong with you? Yeah.
Joel Margolis (teknogeek) (01:02:09.805)
You know, normal things that everybody does.
Sam Erb (01:02:11.976)
Well, that made it, honestly though, that made it affordable. That made it so that I could afford to run this, you know? So it was...
Joel Margolis (teknogeek) (01:02:15.27)
That's awesome
Justin Gardner (@rhynorater) (01:02:15.865)
Dude, the amount of time you spent rewriting that TLS library cost you more than that freaking compute power ever would.
Sam Erb (01:02:22.879)
Uh, I don't know. I was scanning the entire internet for like 20 bucks a month. Like, it was nothing and... yeah.
Joel Margolis (teknogeek) (01:02:26.235)
Hey man, have you seen the cost of AC too? I don't know.
Justin Gardner (@rhynorater) (01:02:28.718)
Freaking Sam-erb. Okay, so you rewrote a TLS. Did you write this whole thing in Go or C or what? It was all Go? Okay, gotcha.
Sam Erb (01:02:35.839)
That was all go, yeah. Yeah, it was all go. And so just as quick as I possibly could. And honestly, that made it affordable for me to run this. And then similar hosting structure where like a teaser with Cloudflare in front of it. And so I was a...
Sorry, lost my train of thought there.
Justin Gardner (@rhynorater) (01:02:59.089)
No, no, you're good. So you built up that extra, you know, fast TLS scanning. You're pumping that into a database and then serving that via the API so people can query against TLS certificates that are popping up over the public internet. Yeah.
Sam Erb (01:03:04.959)
Yep.
Sam Erb (01:03:12.531)
Yeah, exactly. Yeah, sorry. Yeah, and so for me anyways, this data was very valuable because there's exposed origins all over the place. There's a lot of endpoints without DNS routing, necessarily, or internally DNS routing that you'll find valid TLS search for. Especially, I wasn't validating any of the TLS search. So there were a lot of like.
Justin Gardner (@rhynorater) (01:03:19.735)
Yeah.
Justin Gardner (@rhynorater) (01:03:25.621)
Mm-hmm. Yeah.
Justin Gardner (@rhynorater) (01:03:32.167)
Mm.
Justin Gardner (@rhynorater) (01:03:35.801)
like self-signed stuff or whatever. Yeah.
Sam Erb (01:03:36.859)
So that's the right term, sorry. Yeah. Self-signed certs that exist on the internet for like dev instances of things. And, you know, finding all those is extremely valuable. And like as a bug bounty hunter and like that led to a lot of bugs, like significant number of my finds for years, honestly. And so I then realized as I was like running this, I eventually realized that folks were using this data and reselling it.
Justin Gardner (@rhynorater) (01:03:43.597)
Yeah, 100%.
Sam Erb (01:04:01.947)
Like I essentially put up like a, you know, a rough terms of service and every request response download that was like, please don't resell this. Essentially it's all sold. Like don't, don't do legal things with this and don't resell it. And I found a bunch of folks who were selling it, I think like three or four different companies and like, I put it up.
Justin Gardner (@rhynorater) (01:04:08.606)
Yeah.
Justin Gardner (@rhynorater) (01:04:17.127)
Oh man.
Joel Margolis (teknogeek) (01:04:18.233)
super interesting. How did you find that they were reselling it?
Sam Erb (01:04:22.299)
A couple of cases someone reached out to me and then a couple other cases like there was just like somebody posted like, oh, like look at all this data, like, and it listed their sources and it's like Buff Rover Run was one of their sources and I was like, yeah.
Joel Margolis (teknogeek) (01:04:32.557)
You should have just put like samurby is, is amazing dot example.com in all your data sets and then just start looking and seeing if any of them have it in there.
Justin Gardner (@rhynorater) (01:04:35.225)
Ha. Ha ha.
Justin Gardner (@rhynorater) (01:04:40.576)
Yeah.
Sam Erb (01:04:41.875)
Well, yeah, I mean, there's a lot of less ethical things I could have done with that, like embedding fake IP addresses in there, but it just didn't seem like the right thing to do. You can call somebody out, it's like a lot of these were like startups and stuff and they might not even know better. Or the other thing too is like, in a lot of cases, these are another country, so there's no legal recourse. So long story short, I was like, okay, I'm gonna go rev three, which is...
Joel Margolis (teknogeek) (01:04:49.246)
Yeah.
Justin Gardner (@rhynorater) (01:04:49.313)
Yeah.
Joel Margolis (teknogeek) (01:04:55.716)
Yeah.
Justin Gardner (@rhynorater) (01:05:01.59)
Mm-hmm. Sure.
Joel Margolis (teknogeek) (01:05:02.757)
Hmm.
Sam Erb (01:05:08.327)
well, I guess round four would shut this all down. Round three was turn this into a real product. And so I spent, you know, I finally spent a little bit of an AWS budget. I think I was spending like...
Justin Gardner (@rhynorater) (01:05:13.859)
Yeah.
Sam Erb (01:05:26.551)
probably like a few hundred bucks a month on AWS and like turned this into like a whole real pipeline. Like everything was much faster. It was getting the internet way faster. And then I used a rapid API to actually turn this into like a productionized API endpoint. And yeah, it turns out like a bunch of companies that immediately signed up for it. So like a data reselling companies, you know, immediately signed up for it at like an enterprise level. So there was actually interest in this. The enterprise tier was only like 250 a month.
Justin Gardner (@rhynorater) (01:05:29.826)
Mm, mm. Yeah.
Justin Gardner (@rhynorater) (01:05:51.065)
How much were you charging?
Sam Erb (01:05:56.483)
I... yeah. It's nothing. A little bit. A little bit.
Justin Gardner (@rhynorater) (01:05:57.953)
Underpriced that one a little bit, didn't you, Sam? Heh. Yeah.
Joel Margolis (teknogeek) (01:06:01.637)
But still like that's still like pretty good right like I think like again We're a little like Jay did abstract you with $250 a month per customers like pretty solid like
Justin Gardner (@rhynorater) (01:06:09.851)
Yeah. No, it is good, it is good. Not to diminish that at all, but like, you know, they're definitely... Mm-hmm.
Sam Erb (01:06:10.824)
Yeah.
Joel Margolis (teknogeek) (01:06:13.733)
But yeah, but again, like right on the scale of like enterprise applications, I mean people charge 250,000 a month, but it's like, you know, like, yeah.
Justin Gardner (@rhynorater) (01:06:19.433)
Yeah, yeah, I mean it's crazy. Yeah.
Sam Erb (01:06:20.347)
Yeah. Yeah, so from there, you know, I brought in a few customers, but like, I also deployed some much bigger scanning servers. There's a web host that exists that will sell you bare-metal servers that, they don't, it's not that they don't care what you do with them, it's that they, yeah.
Joel Margolis (teknogeek) (01:06:42.688)
That's a good way to preface it.
Justin Gardner (@rhynorater) (01:06:43.705)
Don't say it like that!
Sam Erb (01:06:46.807)
It's that they're very scanner friendly. They're kind of known as being a scanning friendly host. They're, yeah, they're, yeah, I'll just...
Justin Gardner (@rhynorater) (01:06:49.561)
Gotcha. Sure.
Joel Margolis (teknogeek) (01:06:50.277)
Okay. Nice. I'm gonna go.
Joel Margolis (teknogeek) (01:06:55.589)
I'm going to host like that too. It's they're located in Sweden.
Justin Gardner (@rhynorater) (01:06:58.462)
Yeah.
Sam Erb (01:06:58.559)
Well, so that's the thing is they're like a US, I don't fully understand how they operate to be blunt. Sorry, I won't name them, but like they're.
Justin Gardner (@rhynorater) (01:07:03.017)
Yeah. Well, hey man, you know, port scanning isn't legal. I don't understand why everyone freaks out, but like, port scanning is legal. So yeah.
Joel Margolis (teknogeek) (01:07:12.257)
Yeah, yeah, port-scaring is totally legal.
Sam Erb (01:07:12.731)
Yeah, yeah, I, you know, I just, yeah, I don't, maybe they just send their abuse email to, you know, to the void. Yeah, but it's, you know, whatever, whatever they do about it. You know, I was paying them like $500 a month for like a bare metal server to scan much faster. So I was getting results on the entire Internet in 24 hours and certain subsets of it in like an hour to 20 minutes, depending on.
Justin Gardner (@rhynorater) (01:07:19.769)
The garbage can. Dev null.
Joel Margolis (teknogeek) (01:07:20.194)
Yeah.
Justin Gardner (@rhynorater) (01:07:26.542)
Mm. Yeah.
Justin Gardner (@rhynorater) (01:07:40.734)
Ugh, that's great, man. That's fast.
Sam Erb (01:07:41.811)
the day. Yeah, so it was like almost instantaneous, like I would see hosts pop up. And yeah, I mean, that to me was extremely valuable. But then, yeah, the both getting this job, but then also the like support of all that infrastructure, became a little bit much. So I reached out over to a friend who was at the time over at Record Future. And they ended up purchasing the whole thing. You know, it was
Justin Gardner (@rhynorater) (01:07:45.122)
Yeah.
Justin Gardner (@rhynorater) (01:07:58.166)
Mm-hmm.
Justin Gardner (@rhynorater) (01:08:04.881)
Mm.
Justin Gardner (@rhynorater) (01:08:08.073)
Nice. Very cool.
Sam Erb (01:08:11.219)
It wasn't like amazing money, but it was still good money, so I was very happy to hand it off to them.
Justin Gardner (@rhynorater) (01:08:14.733)
Very good, man. Yeah, it's cool to, it's definitely on my bucket list to do some little project like that, turn it into the software as a service, and then eventually exit it. So that's, I love to hear people doing that.
Joel Margolis (teknogeek) (01:08:28.92)
Yeah. What year was it when you exited? Do you know? Oh, okay. Okay.
Sam Erb (01:08:31.635)
Oh, this is last year. Yeah, it was around the time I joined Google.
Justin Gardner (@rhynorater) (01:08:36.765)
Yeah, very cool. So the final little section we had here was just talking about Google and your experience there and the whole program that you guys have in place. So talk to me a little bit about your current role at Google and involvement with the Bug Bounty side of it.
Joel Margolis (teknogeek) (01:08:37.541)
Awesome. Yeah, that's really, really interesting.
Sam Erb (01:08:57.255)
Yeah, so my time at Google split between the Bug Bounty program and then what we refer to as kind of like product security reviews. So looking at signs or things about to launch for their security, making sure it's good to go, or answering any questions that the team seems to have. But then the Bug Bounty side, I am part of the so.
Justin Gardner (@rhynorater) (01:09:08.086)
Mm.
Sam Erb (01:09:24.735)
The way that our bug bounty program works is like our, at least on the Google bug bounty program, we have a rotation of bug hunters, or security engineers, sorry, who will accept reports from bug hunters. We'll treat them on a weekly basis, and then we essentially take turns as treasurer for the week. You know, I think that this is...
Justin Gardner (@rhynorater) (01:09:39.045)
Mm.
Justin Gardner (@rhynorater) (01:09:42.177)
Mm.
Justin Gardner (@rhynorater) (01:09:47.936)
Mm.
Sam Erb (01:09:52.135)
Sorry, to answer your question, I'm on that rotation. But then I also helped out on the help with the overall program, helped to try and make it better. Yeah.
Justin Gardner (@rhynorater) (01:10:03.673)
Sure. Yeah, that's gotta be interesting, man, being on the triage side, especially for, you know, an attack surface as big as Google. Like, that's gotta be nuts.
Joel Margolis (teknogeek) (01:10:12.717)
Yeah. Before you started like working at Google, or I guess maybe the same thing, but before you were in the rotation, before you started working there, whatever, had you hacked on Google at all? Like, but did you have any precursor knowledge about like what vulnerabilities looked like, what the infrastructure was like, anything like that?
Sam Erb (01:10:30.611)
Um, yes and no to some extent. Uh, like I had found, um, some bugs in Chrome actually. Um, I didn't.
Joel Margolis (teknogeek) (01:10:39.081)
Okay, again, no big deal.
Justin Gardner (@rhynorater) (01:10:39.845)
Okay, alright. Just off to the side, like, yeah, just a couple chromium bugs, you know.
Sam Erb (01:10:43.899)
Yeah. But I didn't actually, I don't think I had, I have to look through my email. I don't think I actually had any on Google proper because like the minimization actually really slowed me down. We actually just released a verb extension that will do proto decompiling for you and recompiling.
Justin Gardner (@rhynorater) (01:10:53.723)
Mm.
Justin Gardner (@rhynorater) (01:11:01.801)
Yeah, dude, I freaking love that. We mentioned it on the pod a couple weeks back when it first came out, and I was like, dang, there goes my moat, because Lupin and I sort of like built that exact same tool, except it's not as good as yours, a couple weeks back when we were hacking on Bard. And so, yeah, no, I was glad to see that drop though, because that really opens accessibility up a lot.
Joel Margolis (teknogeek) (01:11:04.474)
That's awesome.
Sam Erb (01:11:07.723)
Down this.
Sam Erb (01:11:28.199)
Yeah, that was really what was holding me back anyways when I was looking at Google originally. I think honestly being on the triage team, you really do see just how many different services we have. Because we get bugs from all over the place. Yeah.
Justin Gardner (@rhynorater) (01:11:33.152)
Mm-hmm.
Justin Gardner (@rhynorater) (01:11:43.393)
Yeah. Everywhere. Yeah. Can you talk a little bit? Sorry, real quick, Joel. Can you talk a little bit about that extension? And were you writing it? I know you're one of the collaborators on the, but were you the one hands on the keyboard writing it? And could you talk a little bit about the structure that Google uses in their request that makes it kind of tricky?
Joel Margolis (teknogeek) (01:11:47.673)
So now.
Sam Erb (01:12:06.599)
Yeah, so that extension was actually authored by an intern we had over the summer. So we're just getting around to releasing it now. Our process was a little bit slow, but yeah, yeah. I think it worked out really well. It turns out that somebody, or the Protobuf team, actually had released a tool called Protoscope, I believe it was called, that...
Justin Gardner (@rhynorater) (01:12:12.405)
Oh, great. That is awesome. What a cool project.
Justin Gardner (@rhynorater) (01:12:28.56)
Mm.
Sam Erb (01:12:30.947)
kind of did exactly what we needed. And so we kind of just glued the burp extension bits onto that. And it actually ended up working out really well. But the reason that it's so complicated, well, the reason that protobuffs are complicated is that you don't necessarily know the definition going over the wire. And so the
Justin Gardner (@rhynorater) (01:12:39.958)
Nice. Yeah, that's cool.
Justin Gardner (@rhynorater) (01:12:52.841)
Mm-hmm, exactly.
Sam Erb (01:12:55.327)
When the JavaScript gets compiled, it knows the definition, but to minify JavaScript, you're not given the definition of the protobuf. And that can make request and response manipulation very difficult in websites that use it.
Justin Gardner (@rhynorater) (01:13:12.057)
Hmm.
Joel Margolis (teknogeek) (01:13:12.685)
Yeah, and what I had noticed, there were maybe two websites that I really were like my go-tos that were able to do, essentially this, like the proto, taking just like a hex or a basic C4 blob of protobuf data and just saying like, here's what the type structure looks like, here's what the field indexes are, good luck. And it's awesome that this is now like an official tool and a CLI tool and
now also a burp extension so that you can sort of tie this into all your own tooling and all that, all that other good fun stuff and sort of lower that, that bar for, for hacking on Google, which is so super, super awesome to see.
Justin Gardner (@rhynorater) (01:13:52.965)
Mm.
Sam Erb (01:13:53.587)
Yeah, there's also a few others in there also. If there's something that the extension can't do, there are a few other options listed. We definitely looked at those as we were implementing this, to see how they worked, but then also to, we wanted to provide that list just to give folks a choice.
Justin Gardner (@rhynorater) (01:14:16.833)
Yeah, so with the extension that Lupin built and that we were using on Bard, essentially it would kind of do some deobfuscation and formatting and that sort of thing. But at the end of the day, you're still not getting parameter names and stuff like that. You're just getting like, okay, here's an array. We know that this is a string, that sort of thing.
Joel Margolis (teknogeek) (01:14:43.221)
And like for protobuf, like the parameter names are completely like decorative, like, like structurally wise, like all it cares about is what, what is the index, like the field ID number and what type is it? And beyond that, like you could name it, whatever you want. It's really just like, you know, extra.
Sam Erb (01:14:49.461)
Yeah.
Justin Gardner (@rhynorater) (01:15:00.489)
Okay, so for those of you with, because I hacked on Bard for like a week, so I've got a very rudimentary understanding of it, but the, part of the reversing process then becomes, okay, what the heck is this thing, right? Like, because there's no perimeter names. Yeah.
Joel Margolis (teknogeek) (01:15:14.881)
What is this? Yeah. This request now added field 19. What is field 19? Where does this come from? Yeah.
Sam Erb (01:15:23.019)
Yeah.
Justin Gardner (@rhynorater) (01:15:23.169)
Exactly. So you're much more blind. You're trying to sort of create deductions about how the request is actually being formatted and what numeric ID is being used where, what string is being used where. Any tips or thoughts for that? Or is it really just got to, you know, we're getting a little bit closer to...
a little bit more difficult hacking where you're just sitting there and just trying to make sense of even what you're seeing, let alone trying to figure out how the backend works.
Sam Erb (01:15:57.443)
Um, I mean, the, if you see, you know, the one tip I'll give actually is like, if you see missing fields, try putting something in there. Um, you know, especially missing the dexes. Well, you'll get stuff out of order, but you also get field numbering when you, um, when you decompile and recompile it. And so like, I don't know if you have like fields like one, two and five, like what are three and four?
Justin Gardner (@rhynorater) (01:16:07.172)
Yeah.
Joel Margolis (teknogeek) (01:16:08.237)
Yeah, so it's like stuff that's out of order.
Justin Gardner (@rhynorater) (01:16:16.865)
Mm.
Joel Margolis (teknogeek) (01:16:17.061)
Mm-hmm.
Joel Margolis (teknogeek) (01:16:22.945)
Yeah, three and four. Yeah.
Justin Gardner (@rhynorater) (01:16:23.721)
Yeah, yeah, yeah. Okay, that's interesting. Because, you know, also one of the problems I ran into was that it doesn't give verbose errors ever, right? So it's like, error five.
And I'm like, no, not error five again. And so it's good to hear that actually just trying to insert stuff at different indexes may actually do something. Because in my head, I was thinking that they probably have something very specific they're looking for. And if I do anything besides change the ID numbers or anything like that, it's going to heck everything up.
Sam Erb (01:17:00.563)
Yeah, I mean, it really depends on the application. Sorry. Yeah.
Justin Gardner (@rhynorater) (01:17:03.085)
Yeah. Dang it. I was, I wanted the, I wanted the silver bullet Sam, like dang it man. Yeah, Sam's like, yeah dude, I don't have to deal with this shit. I got the proto files, you know, like I got the, I got the definitions on this side.
Sam Erb (01:17:11.688)
I, yeah, become a Google employee and read the front of us. Yeah. You know, we have hired Bug Hunters before. I think Ezekiel, who's won a bunch of the Cloud Awards, we hired him.
Joel Margolis (teknogeek) (01:17:15.058)
Ha ha ha!
Justin Gardner (@rhynorater) (01:17:26.361)
Oh, very cool. Yeah, we'll have to chat about that after. So I guess, so you're on the triage team, you see a lot of bugs coming through there. Any crazy stories? Any crazy stories you wanna share? Or are we diving too deep into the unknown and the things that we're not allowed to probe about, Sam?
Joel Margolis (teknogeek) (01:17:28.237)
Nice. That's awesome.
Sam Erb (01:17:46.991)
Well, and so this is like, this is the interesting part is that I can't talk about these bugs, but as bug reporters, we don't hold folks to NDAs. So you're welcome to talk about your bugs.
Justin Gardner (@rhynorater) (01:17:51.175)
Mm.
Justin Gardner (@rhynorater) (01:17:56.53)
Mmm.
Like active bugs? Yeah, wow, that's crazy, man. That's kinda nuts. I've heard that before. Yeah. Okay.
Sam Erb (01:18:06.204)
If you say you're going to disclose, we'll send you a standardized disclosure notice. I'm not going to attempt to paraphrase it, but yeah, we're not going to hold you to NDAs.
Justin Gardner (@rhynorater) (01:18:15.605)
Wow. That's really cool. Yeah. That's one of the few programs out there that really, like, actually lets you do that pretty freely because almost every other big program is like, all right, if you want to get paid for it, then mom's the word. You know?
Joel Margolis (teknogeek) (01:18:16.677)
That's really cool.
Sam Erb (01:18:18.133)
So that's it.
Sam Erb (01:18:28.915)
Yeah. Well, yeah, well also, I mean, if you read blog posts about it, too, like, well, like, retweet it and stuff. And yeah, you know, we want more folks finding bugs. And yeah. But I will say, actually, one of the more interesting reports that came through, and so sorry, backing up, I guess. So the, I don't know if you've paid attention recently, but like the.
Justin Gardner (@rhynorater) (01:18:35.857)
Oh wow, that's Clutch, and they're even gonna promote it.
Justin Gardner (@rhynorater) (01:18:51.82)
Mm.
Sam Erb (01:18:58.139)
over the past year, I guess, we've launched a lot of, fair, we've launched a bunch of different bug bounty programs, is the best way to describe them within Google.
Justin Gardner (@rhynorater) (01:18:59.237)
Probably not. Ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha
Justin Gardner (@rhynorater) (01:19:10.497)
Yeah, I wanted to ask about that, because there's like, you know, the one where it's like, you get a bunch of, you know, any app above 100 million, you know, installs, that's in scope for Google, the Google Play, you know, thing, and there's the VRP, and then there's some AI stuff, so like, there's a bunch going on.
Sam Erb (01:19:20.157)
Mm-hmm.
Sam Erb (01:19:25.971)
Well, so one in particular that I want to call out is the abuse VRP. And this is something that I didn't, before joining, I didn't have a lot of insight into. And it's actually really interesting. So when we are rewarding payments through the Google VRP, we'll actually, at the same time, do the abuse VRP. Because often, there are bugs where it's applicable to both. And we'll just reward everyone's highest.
Justin Gardner (@rhynorater) (01:19:30.737)
Mmm.
Justin Gardner (@rhynorater) (01:19:34.724)
Mm.
Justin Gardner (@rhynorater) (01:19:46.767)
Hmph.
Justin Gardner (@rhynorater) (01:19:52.641)
What is the abuse of ERP? Is it like? Yeah.
Joel Margolis (teknogeek) (01:19:54.029)
Yeah, I was going to say like what falls under.
Sam Erb (01:19:54.275)
And so sorry, going back to it. So to use the example, recently somebody reported that they received a really convincing phishing email. And this is like, they actually wrote about this publicly, so I can talk about it. It became a problem with the BIMI protocol. And it's some sort of email. This is pushing the limits of what I know about this. But it's some sort of email user identification system.
Justin Gardner (@rhynorater) (01:20:17.723)
Mm.
Justin Gardner (@rhynorater) (01:20:23.87)
Oh, interesting. What is this called?
Joel Margolis (teknogeek) (01:20:24.121)
Yeah, I'm reading. I've never even heard of this brand indicators for message identification. It's it's kind of like SPF for DKIM or DMARC or one of those like I guess it's just an additional sort of security layer and their additional DNS BIMI brand brand. What was it brand indicators for message identification?
Sam Erb (01:20:24.831)
and
Justin Gardner (@rhynorater) (01:20:38.521)
Print indicators for what? Sorry?
Justin Gardner (@rhynorater) (01:20:48.897)
Wow, interesting.
Sam Erb (01:20:49.727)
And so somebody reported a problem that they received a really convincing phishing email through this. And phishing emails don't quite fit cleanly under the rules of our VRP necessarily, but under the abuse of our P, this is a very clear cut. This can be used to abuse our products. And so we accepted through that. But then I think that actually resulted in RC updates. They weren't...
Justin Gardner (@rhynorater) (01:21:02.214)
Mm-mm.
Justin Gardner (@rhynorater) (01:21:17.166)
Wow.
Sam Erb (01:21:18.923)
weren't aware that this could be an attack. I don't recall the exact details, but it did it did get some news back a few months ago.
Joel Margolis (teknogeek) (01:21:25.349)
So it sounds like kind of the distinction is like more like repurposing intended functionality or at least like featured functionality for like a bad purpose versus something that's like a security vulnerability. And maybe they kind of like overlap a little bit, but like one is more like classical than the other. Is that kind of the gist?
Sam Erb (01:21:44.999)
Exactly, yeah. I think that's the right way to look at it. There's things that are cleanly technical security vulnerabilities. Like an RC would be a clean security vulnerability or clean up use. You can, I don't know, steal money from somebody. Like.
Justin Gardner (@rhynorater) (01:21:57.341)
Yeah, let me read this really quickly. It says, OK, note in addition, significant abuse-related methodologies are also in scope for this program. If the reported attack scenario displays a design or implementation issue in Google products, they could lead to significant harm. An example of an abuse-related methodology would be a technique by which an attacker is able to manipulate the rating score of a listing on Google Maps by submitting a sufficiently large volume of fake reviews that go undetected by our abuse system. That's really cool.
Sam Erb (01:22:26.515)
Yeah, and so we get a lot of reports where, like, you know, we forward it to that team.
Joel Margolis (teknogeek) (01:22:33.069)
Yeah, so I wanted to ask, because Google is so huge, and you guys probably receive a lot of reports, how do you go sort of about drawing the line for this is stuff we don't care about? Because I can imagine that you can rework all your bugs in one way or another and make it so that technically it falls under some form of either abuse or technical security issue.
Justin Gardner (@rhynorater) (01:22:33.302)
I imagine so.
Joel Margolis (teknogeek) (01:23:02.305)
And how do you stop like that from overloading the internal teams with like lots of little nitpicky things that don't really like that, that maybe they should fix, but not doesn't need to happen tomorrow, right? Like, you know what I mean?
Justin Gardner (@rhynorater) (01:23:02.617)
Mm.
Sam Erb (01:23:16.943)
Yeah, I mean, we have a lot of automation on our side that helps us, especially with that first level of like, look at the report, like, do we even own this URL? Those are.
Joel Margolis (teknogeek) (01:23:27.621)
Mm-hmm.
Justin Gardner (@rhynorater) (01:23:29.269)
Yeah, actually, I got a, I got, when I submitted a bug, it's like, I got an email back saying, hey, by Google magic, we know, this is like within like two minutes of me submitting the bug, it's like, by Google magic, we know that this is a valid report. Or like something like that. And I'm like, oh wow, that's kind of crazy. Like, yeah, seriously.
Sam Erb (01:23:37.075)
Yeah, so our... Yep.
Sam Erb (01:23:46.639)
So we do have a... So I should preface this by saying, well, I should preface this by saying that security engineers will still look at every report. That really impacts the order which would look at things. But we do have like a first pass like machine learning system that will look at every report and knows what's previously been valid and invalid. I should also say just before we go any further that like...
Joel Margolis (teknogeek) (01:23:48.705)
I need some of that magic. Holy cow.
Justin Gardner (@rhynorater) (01:23:58.051)
Yeah.
Sam Erb (01:24:14.683)
Anything I say here at the rules pages take precedent, in case I make any mistakes. You know, if you go to the Bug Hunters website, yeah, yeah.
Justin Gardner (@rhynorater) (01:24:19.742)
Yeah, that's fair. That's fair Sam you said on the CTPV podcast that I could submit phishing emails
Joel Margolis (teknogeek) (01:24:27.573)
Ha!
Sam Erb (01:24:30.06)
But then, so then if you go to the other bug hunters, I'd say if you go under the invalid reports section, there are a few different categories there which actually often will catch folks off guard. The one big one especially is open redirects. Like there's a few cases where we have intentional open redirects that do have security properties, but yeah, those are intentional. And we often get reports around those.
Justin Gardner (@rhynorater) (01:24:35.288)
Yeah.
Justin Gardner (@rhynorater) (01:24:40.716)
Mmm.
Justin Gardner (@rhynorater) (01:24:45.251)
Yeah.
Justin Gardner (@rhynorater) (01:24:48.826)
Yep, slash amp.
Justin Gardner (@rhynorater) (01:24:56.612)
I saw...
a tweet a while back that was like, hey, I'm trying to get this OpenRedirect to work, but I can only redirect to the site in Google.com. And I was like, ah, you're in luck because slash amp, you hit slash amp with that OpenRedirect and it'll redirect you to whatever domain you want. And that's just a gadget that's been in place for a while. And this is just a comment on a Twitter response. And everyone was like, retweet. I spend so much time writing very thoroughly thought through.
tweet threads and like, yeah, very, very thorough responses. And then this one just like blows up because Google's got an open redirect they won't fix. It's like, come on.
Sam Erb (01:25:26.794)
Single line comment.
Sam Erb (01:25:35.647)
I would, once they won't fix, I would look on the about page to understand more of our logic behind it.
Justin Gardner (@rhynorater) (01:25:40.745)
No, yeah, let me be clear. I don't think it's a problem. I don't think it's a problem because Google, you know, well, I guess, let me challenge that a little bit. I think it is, I don't have all the context in your organization. I think it's a little bit irresponsible to.
Joel Margolis (teknogeek) (01:25:43.577)
Hehehe
Justin Gardner (@rhynorater) (01:25:56.641)
to do something like that because I use open redirecting chains like every day. And so ideally that would be fixed. But the thing is Google is a search engine and part of the business case is redirecting users to things. So I'm sure there's a perfectly valid use case word on your side. But I will say I've used that redirecting chains many times.
Sam Erb (01:26:24.183)
I will say also that there are, and once again, look at the invalid reports redirect page for a lot more technical information around this. I probably should've read it before I joined this. Ta-da. Okay, cool. There are a lot of cases where we do accept those. There is JavaScript redirects, obviously, but ones that aren't intentional might get rewarded.
Justin Gardner (@rhynorater) (01:26:28.777)
Mm-hmm.
Justin Gardner (@rhynorater) (01:26:34.502)
Heh.
Joel Margolis (teknogeek) (01:26:35.885)
No, I haven't opened it. We'll put a link down below.
Justin Gardner (@rhynorater) (01:26:38.134)
Yeah, yeah, we'll drop it.
Oh really? Interesting.
Justin Gardner (@rhynorater) (01:26:51.681)
Hmm, interesting. Okay, all right, yeah, we'll add it. Yeah, JavaScript redirects, client-side redirects, we've talked about this plenty of times on the pod, but there is...
Sam Erb (01:26:53.08)
And look at that page once again.
Justin Gardner (@rhynorater) (01:27:05.289)
a lot of use cases with that, especially with the onset of same-site cookies and just whether or not things are going to be sent cookies depending on where the request originated from. And when you do a client-side redirect via JavaScript, you're sending different cookies along if you had just done it from a third-party top-level nav. So...
Sam Erb (01:27:26.163)
Well, actually, I know, sorry, I made like a little bit more like, uh, open redirect across that scripting, but yeah. Yeah.
Justin Gardner (@rhynorater) (01:27:31.645)
Okay, gotcha open redirect to process scripting. Okay. Well, it's interesting, you know JavaScript level redirects I think that I'd be surprised if you guys accepted those but I also wouldn't be surprised in that there I think as Okay
Sam Erb (01:27:45.031)
Yeah, I actually, sorry, I don't know if we accept those. I really, I really didn't mean the, you know, the JavaScript URI. Like the.
Justin Gardner (@rhynorater) (01:27:50.565)
Oh, the redirect too, yeah sure. The client side redirect, and where you can snag it via process scripting. Yeah, no, that makes sense. Well, that's cool, man. There's a lot of different, there's a lot of, I'm on the page for, I guess, bughunters.google.com, and there's a ton of different programs here on the left-hand side. Android and Google Device Security Reward Program, all the way to Alphabet Program, to Open Source, to OSFuzz, or OSSFuzz, so like, there's a ton of stuff here.
Sam Erb (01:27:52.555)
Sorry, sorry, yeah.
Justin Gardner (@rhynorater) (01:28:20.75)
So that's a big job you got going on there Sam, because it's got to keep you busy. Yeah.
Sam Erb (01:28:24.435)
We have a large team, and we want to reward good vulnerabilities. I think that if you want to see some other cool ones, some of my favorites have been on the yearly, there in the past years have been cloud rewards for the best GCP vulnerabilities. Those are always super creative. There have been some great write-ups there as well.
Justin Gardner (@rhynorater) (01:28:40.608)
Ah, GCP stuff, nice.
Justin Gardner (@rhynorater) (01:28:48.225)
Very nice. You added something to the doc here, hacking Google. And it's one of the coolest things I've ever seen, to be honest. Dude, this was not on my radar at all. And we're going to link this down below. But apparently, it's like a multi-series, almost like a documentary of essentially hacks that have happened with Google and hacking related stuff at Google. Super well produced.
Sam Erb (01:28:57.056)
Ha ha.
Joel Margolis (teknogeek) (01:28:58.053)
I love seeing brands embrace hacking like this by the way.
Sam Erb (01:29:17.535)
Yeah, they went all out with this. It's...
Justin Gardner (@rhynorater) (01:29:18.983)
talk to me about that, I mean, and it's only a year old and there's like millions and millions and millions of views on it.
Sam Erb (01:29:23.943)
Yeah, yeah, this, I probably can best describe this as like, this is what I show my grandparents to them, but I do, you know? Like it tries to, it tries to put into like, you know, into something that's publicly consumable, like, you know, by everybody, like what we do as security engineers. You know, it talks about a bunch of different teams. The one that I linked to there is the one on our bug bounty team.
Justin Gardner (@rhynorater) (01:29:32.144)
Hahaha, that's great.
Joel Margolis (teknogeek) (01:29:33.704)
That's awesome.
Justin Gardner (@rhynorater) (01:29:44.567)
Yeah.
Justin Gardner (@rhynorater) (01:29:52.445)
Mm, yeah, is that the, let me see, episode four? Yeah, because I only got through some of them. Yeah, episode four, that was the one I clicked.
Sam Erb (01:29:56.488)
episode four here.
Sam Erb (01:30:00.823)
But yeah, the production quality on those is wild. Yeah, it's really well done. I really like that series.
Justin Gardner (@rhynorater) (01:30:06.053)
Yeah, the first one, operation.
Aurora I watched that one. I was like this is amazing and then I clicked right on the book hunters one and then I realized I gotta go record the podcast So no, that's great. I'm gonna go back and check that out. Also. They've got a project zero section as well, which is such a cool part of Google So definitely check out the hacking Google series Dude, I think that we're already at an hour 30 that kind of concludes what I had on the on the list here
Sam Erb (01:30:15.16)
Yeah.
Sam Erb (01:30:32.4)
Ah, cheers, yeah.
Justin Gardner (@rhynorater) (01:30:36.065)
Did you have anything else you wanted to shout out or talk about before we wrap the pod?
Sam Erb (01:30:40.871)
Now, thank you all for having me on here. This was a great experience.
Justin Gardner (@rhynorater) (01:30:43.733)
Yeah, of course man, you can find Sam on Twitter, urbysam, and I don't know, do you even use your LinkedIn? I've got it here, but no, he doesn't even use his LinkedIn. So just find him on Twitter, or X, urbysam, with two Vs. Sam, thanks for coming on and sharing your knowledge, man, and looking forward to our next collab where you give it 110%, and you know. All right, man, talk to you soon, peace.
Joel Margolis (teknogeek) (01:30:45.558)
Yeah, absolutely.
Sam Erb (01:30:52.819)
Not really, no. Ha ha. Yeah.
Sam Erb (01:31:04.275)
Yeah. Yeah, same.
Joel Margolis (teknogeek) (01:31:07.498)
Awesome.
Sam Erb (01:31:09.131)
Take care.
