I have some thoughts on Sander Schulhoff’s appearance on Lenny’s Podcast. The episode, entitled The coming AI security crisis is a deep dive into the state of AI security (concerning) and what application developers can do about it (less than you’d think).
In terms of threat modeling, the easiest way to think about it to think about the LLMs as a person, and the inherent threats being very similar to social engineering. With enough tenacity, an LLM can be convinced to say anything you want it to say, divulge any information it has access to, and perform any task it has the ability to perform. This was a known, relatively minor risk before agents took off. It was fairly obvious that you could trick LLMs into doing things that their creators didn’t want them to do.