Smashing Security podcast #493: Why your AI model could be a sleeper agent
Joining the Call That Isn't Real
You join a video call, and your contact and a few colleagues are already there, waiting patiently. But they can't hear you, and a pop-up says you need to install an update to fix your microphone. What could possibly go wrong?
Meanwhile, we explain how changing just a handful of numbers can turn an AI model into a sleeper agent, waiting for the right moment to strike.
Plus, Andrius Buinovskis of NordLayer explains how companies can see which AI tools their staff are really using, and stop sensitive data leaking out, without banning AI altogether.
All this and more in episode 493 of the "Smashing Security" podcast with cybersecurity expert and keynote speaker Graham Cluley, and special guest Ray Redacted.
This transcript was generated automatically, probably contains mistakes, and has not been manually verified.
Do You Let AI Do Things on Your Computer?
GRAHAM CLULEY: You don't have to tell me if you're not comfortable doing so, Ray, but do you let an AI actually do things on your computer?
RAY REDACTED: 100%. Absolutely.
GRAHAM CLULEY: Oh my goodness, Ray!
Unknown: Smashing Security, episode 493, Why Your AI Model Could Be a Sleeper Agent, with Graham Cluley and special guest, Ray Redacted.
GRAHAM CLULEY: Hello, hello, and welcome to Smashing Security, episode 493. My name's Graham Cluley.
And I'm Ray Redacted. I always wonder, Ray, when you're on the show, whether we should bleep out the redacted. Maybe we should just have Ray beep.
RAY REDACTED: You know, I've actually had people ask me that before. Graham, here's the funny thing. I once upon a time was actually looking to legally change my name to Redacted, but my attorney told me I could not do that 'cause I would get subpoenaed all the time anytime somebody was redacted from the court orders and stuff, so. Did not do that.
GRAHAM CLULEY: It's better than being Bobby Drop Tables or whatever though, I suppose. That could have caused problems too.
RAY REDACTED: XKCD reference. That was really good. That's particularly sharp of you this morning.
GRAHAM CLULEY: Well, before we kick off, let's thank this week's wonderful sponsors, NordLayer AI Scout, Dropzone AI, and Vanta. We'll be hearing more about them later on in the podcast.
Unknown: This week on Smashing Security.
What We Won't Be Covering This Week
GRAHAM CLULEY: We won't be talking about how a flaw in a website testing tool let malware reach the BBC, CNN, and Zoom.
You'll hear no discussion of how a breach at an AI cloud vendor exposed close to 150,000 Uber driver licences.
And we won't even mention how 2 Grok AI agents tried to raid corporate accounts.
So Ray, what are you going to be talking about this week?
RAY REDACTED: I'm gonna be talking about an exploit called Lazarus that Microsoft Research has recently published.
GRAHAM CLULEY: And I'll be talking about how your face might be being stolen to fool your colleagues on Zoom. Plus, we've got a featured interview with Andrius Buinovskis of Nord Security about how you can get better visibility into how sanctioned and unsanctioned AI tools are being used in an organisation and prevent data leaks.
All this and much more coming up on Smashing Security.
Sponsor Segment: Shadow AI
Coming up in this episode of Smashing Security. Joe, can I ask you a slightly awkward question? If it's about the biscuits, that wasn't me. It's not about the biscuits. Which AI tools have you been using for working on the podcast?
JOE: Just the approved ones, obviously.
GRAHAM CLULEY: So what's this Chatty McChatface thing? Free, no sign-up required.
JOE: Ah, that's more of a research partner.
GRAHAM CLULEY: And what have you been pasting into it, Joe?
JOE: Oh, just a few bits. Guest emails, the sponsor schedule, your bank details.
GRAHAM CLULEY: My bank details?
JOE: It needed context.
GRAHAM CLULEY: Listeners, this is shadow AI, and Joe's not alone. IBM found that 43% of breached organisations had a shadow AI incident this year. That's more than double the year before.
JOE: Which is why this episode is sponsored by NordLayer AI Scout.
GRAHAM CLULEY: AI Scout finds every AI tool being used across your organisation in the browser and on the desktop, so you can see what was asked, by whom, and where.
JOE: Hang on, by whom?
GRAHAM CLULEY: By whom, Joe? It also flags passwords, financial data, and personal information at the prompt level before it wanders off somewhere it shouldn't.
JOE: Does it also show who's paying for AI licences they never actually use?
GRAHAM CLULEY: It does. It measures real usage, not just the seats you've bought.
JOE: Great, because I noticed we're paying for 3 AI subscriptions and the only thing you've used them for is writing a limerick about ransomware.
GRAHAM CLULEY: Right, moving on.
JOE: NordLayer AI Scout rolls out to every device at once and runs quietly in the background.
GRAHAM CLULEY: Unlike some people.
JOE: See how your teams really use AI and what it's really costing you. Book a free 30-minute demo at smashingsecurity.com/nordlayer.
GRAHAM CLULEY: That's smashingsecurity.com/nordlayer. And Joe, delete Chatty McChatface.
The Fake Zoom Meeting Setup
GRAHAM CLULEY: Now, chums, I want you to imagine the scene, if you will. You are really into cryptocurrency. Are you really into cryptocurrency, Ray?
RAY REDACTED: I am, but I don't usually admit that for OPSEC reasons on podcasts.
GRAHAM CLULEY: Oh dear. Well, you just have now. Well, maybe you're into cryptocurrency. Maybe you work for a crypto company, maybe you've made a small fortune investing in crypto coins, and you get a message out of the blue.
A message comes through on Telegram or LinkedIn, wherever it is, and it's from somebody that you know. And maybe it's someone you met at a crypto conference last year or someone fairly important, some bigwig in the cryptocurrency industry. And they're saying, hey, do you fancy a quick chat? Jump on the call, here's the link.
And so you click on the link.
RAY REDACTED: Sure.
GRAHAM CLULEY: And you land in something which looks well, just like a Zoom or Microsoft Teams call. And you know how it is, you're in the waiting room, right?
RAY REDACTED: Mm-hmm.
GRAHAM CLULEY: You're waiting for the call to begin and they're showing you a little preview of yourself. And that's fantastic because you can check your webcam preview, you can check your hair, you can sit up straight, make sure you're all comfortable in the middle of the screen, make sure you haven't got any broccoli stuck anywhere.
But what you don't realise is that all the time that you are in that waiting room, somebody is secretly already recording you. So footage is being taken.
And finally, the other people show up and the meeting starts. And there's the person who contacted you on the screen and they're nodding away. And perhaps there's a few other people there as well. And it's all very normal, but you get a message in the chat inside the messaging app saying, sorry, Ray, we can't hear you.
I mean, this happens all the time, doesn't it? I was doing a webinar the other day for a customer. And I dialled in 15 minutes early. They said, please come in early, Graham, to make sure everything's - and I said, yeah, no worries. And so I got there early and the people hosting the webinar, they couldn't hear me at all.
And so I said, oh, I'll disconnect and I'll reconnect. And then I had to restart my computer and I was feeling, oh my gideon, because of course it was going to go live with people tuning in. I was like, ah, I can't do this. I only made it by the skin of my teeth and everything eventually got to work.
RAY REDACTED: That's usually when you get software updates too.
The Malicious Update
GRAHAM CLULEY: Yes. So anyway, this is not an uncommon scenario of arriving on a call and your microphone or something not working properly. And in this particular case, what happened was an error message pops up saying that you need to run an update to fix the problem because your microphone - maybe there's some sort of codec or audio driver or something that needs to update.
And you know, it's mortifying to you because you just want the call to happen because, you know, you're thinking, oh, all these other people are waiting for me. So maybe you would, in that situation, run the update or do the install or apply the patch or whatever it is that it's telling you to do.
And what you don't realise is that your contact, the one you're having the call with, was never actually there. This has been explained by the FBI in recent days. They say you are looking not at your contact. You're actually, in fact, watching a video of a prior victim of this particular hack.
So somebody you know has actually previously been hit by this thing. Video of them has been taken, and it's now being played back to you. So this isn't deepfake. This is a genuine piece of footage of someone sort of sat in front of their monitor looking like they're preparing for a web meeting.
So all of these people who you can see who are in the meeting room waiting for you to fix your microphone, they are just there piling on the pressure on you.
RAY REDACTED: Of course.
GRAHAM CLULEY: To go and fix your audio, which is, you know, a pretty sneaky trick, I would say. And this update, which they get you to install, surprise, surprise. Now, Ray, you and I, we've been working in cybersecurity a long time. It's not gonna be any surprise to us that of course the update is malicious.
RAY REDACTED: Oh, but Graham, I could so see myself falling for this, 'cause you're on the call and you see all these people waiting for you and you're not thinking in terms of this might be malware. It's like giving you an update - this looks like it's from Zoom or whatever. I would absolutely click that.
GRAHAM CLULEY: And you receive the message in the first place from somebody you know or have an industry contact with. It's not like some unknown who has sent you this message.
RAY REDACTED: Sure.
GRAHAM CLULEY: And this malware gives the bad guys access to your computer, access to your cryptocurrency wallet, quite possibly your other accounts as well, which means, of course, they could steal - well, I don't know how much you've got in your cryptocurrency wallet, but, you know, they could steal a fortune.
And according to the FBI, the people behind all of this - let's open this up as a quiz, Ray. Where do you think they might be based?
RAY REDACTED: I would guess this sounds an awful lot like a rogue nation in Asia that is interested in pursuing nuclear weapons.
GRAHAM CLULEY: Yes, yes, yes. In other words, it's North Korea. It's the North Korean government. I don't want to say that everything cryptocurrency-based is going to be North Korea, but it's not uncommon.
RAY REDACTED: A lot of it is.
GRAHAM CLULEY: A lot. They are funding an awful lot of their naughty business with stolen cryptocurrency. Back in episode 478, I told a story about how North Korean hackers were recording people's webcams during fake job interviews. And this seems to be an extension of that kind of attack.
So you've already said, Ray, that if you were in this sort of scenario, you would be tempted to run the update. So it's no surprise that people who work outside of the cybersecurity industry would be fooled into doing that as well, is it?
RAY REDACTED: Well, what's fascinating about this, if it is indeed the DPRK, is they've gotten really good at social engineering lately. If you think about
GRAHAM CLULEY: The
RAY REDACTED: Old school North Korean attacks were typically zero-day exploits or bugs, or maybe badly written phishing emails. But now suddenly they're getting pretty devious with the social engineering side. And the idea that you get a video call and people are waiting for you to install software - that's evil. I mean, that's devious. I could absolutely see, because your heart's racing and you're flustered and you're worried about starting the call late and everything else, and everybody's looking at you and glaring at you.
GRAHAM CLULEY: Yep.
RAY REDACTED: They're getting good at social engineering.
GRAHAM CLULEY: Very, very cunning indeed. And last week, the US Department of Justice announced that it had disrupted the infrastructure, the cloud-based infrastructure being used by North Korean hackers with this particular scheme.
The FBI is calling this particular scheme FaceStealer, which sounds like it'd be a Nicolas Cage, John Travolta kind of movie. This is the FBI. I'm Jeremy Robertson, Special Agent in Charge of Counterintelligence and Cyber in the Los Angeles Field Office. Today, we disrupted infrastructure used by North Korean hackers in a global cryptocurrency theft scheme we call FaceStealer. The scheme uses secretly recorded webcam video, fake Zoom and Microsoft Teams meetings, and computer intrusions to steal from people around the world.
RAY REDACTED: In just 18 months, FaceStealer has resulted in over $400 million in thefts -
Unknown: Wow.
Unknown: - and impersonation of over 900 victims working in the crypto and Web3 industries.
GRAHAM CLULEY: The FBI has seized over 900 webcam videos of victims - so people who've sat in that waiting room waiting to have these calls, which have obviously been recorded and shoved onto a server somewhere up in the cloud, which the North Koreans are using, and they've been kept so that they can be played back to those people's friends and colleagues.
So in a way, what I find interesting about this - we're all familiar with worms, for instance, and email-aware worms from the old day
Comments
No comments yet. Start the discussion.