AI agents are coming, is your security program ready?

LLM powered agents, the idea of using Large Language Models to automate tasks by having an LLM make decisions and take action via system integrations or by acting on behalf of users through standard tools requires a wholesale rethink to traditional approaches to both business and information security.

Today, agents are often derided as not reliable. They are autonomous in the way that my kid can “autonomously” get dressed when it’s time to leave - neither is likely to complete the task without a lot of hand holding. Give an agent a task with open-ended complexity and you’ll be amazed at the creative ways the process falls apart.

While it’s easy to criticize the current state of LLM automation, not taking agents seriously would be a huge mistake. Things have only accelerated since the release of ChatGPT and a massive amount of effort is being focused on improving and operationalizing models to solve problems that previously landed squarely in the domain of humans.

We’re not far away from a future where interactions between semi-autonomous LLM agents will be the glue that connects disparate systems to do work with humans (for now) holding an oversight role to ensure things don’t go flying off the rails and to help when they do.

Although new, “AI-first” services are already cropping up, going for a piece of legacy provider market share, for the foreseeable future, the world will continue to operate with copilot powered versions of entrenched providers like Salesforce, Netsuite and Workday. Custom automation between these systems is where things will get very interesting. Work will be offloaded to agents who interact with these systems, often in ways that aren't as straightforward to assess as when a human does it.

What does that mean for security professionals?

1. Identity Management: The distinction between user and agent identities will blur as agents both gain autonomy and act on behalf of individuals. Mechanisms must be defined to ensure that an agent's actions are tied to a verifiable identity, whether acting on its own or as a delegate for a human user. This requires rethinking the static authentication and authorization frameworks currently in use for service level integrations.
2. Containing the Blast Radius: Similar to the impact of leaked keys but with many more ways of occurring, the potential for compromised or misbehaving agents underscores the need to design systems with the principle of least privilege. Permissions need to be segmented based on the specific tasks and data an agent requires to limit the damage an errant agent can cause.
3. Navigating Non-Determinism: The inherent unpredictability of LLM agents, coupled with their susceptibility to manipulation, requires rethinking security monitoring and response. Traditional deterministic models may fall short in detecting unintended agent behavior, necessitating a shift towards adaptive, behavior-based security analytics. Incorporating human oversight at critical decision points can provide a safety net, ensuring that anomalous or potentially harmful actions are reviewed by a human.
4. Prioritizing Upskilling and Knowledge Sharing: Effective management of security risks associated with LLMs and autonomous agents necessitates an understanding of how the technology works. Without comprehension of how these systems operate, identifying risks, effectively mitigating them and recognizing potential value-add opportunities becomes significantly harder. Proactively upskilling teams via training and projects where they can get hands-on now will be critically important to ensuring AI competency. This shouldn’t be relegated to technical teams, but across the organization to ensure all stakeholders understand the risks and opportunities.
5. Ensuring Explainability and Accountability: As agents interact in ever more complex ways, tracing the lineage of decisions and actions becomes more challenging and increasingly critical. Maintaining transparency and accountability will be vital to adoption across the enterprise. This may require developing newer mechanisms to capture the nuanced and potentially complex context of agent decisions, taking into consideration that LLMs can be manipulated so an attacker could convince it to log something entirely different, something you don’t really worry about on today’s systems. This could be augmented by yet another layer of AI to interpret, summarize and alert on interactions for human reviewers.
6. Reassessing priorities: The shift towards AI-driven automation calls for a reassessment of priorities. Security teams must become more anticipatory and adaptive, prepared to evolve with the technologies they safeguard. Proactively engaging with AI development teams, participating in the design of these systems, and advocating for security-by-design principles from the outset are crucial steps. Fostering a culture of continuous learning and agility within will become even more important in a world where the pace of change is coming at an ever increasing rate.
7. Preemptive Action and Future-Proofing: To stay ahead, organizations should begin mapping their current and projected use of AI agents, identifying potential security issues via established techniques like threat modeling, and developing a phased plan to get ahead of them. Investing in learning, exploring AI-specific security tools, and becoming engaged with the broader security and AI research communities for insights and best practices will be key to navigating the future securely.

LLM agents will become a part of every individual’s and every business’s internet footprint, fundamentally altering the landscape. Given the rate of change and progress we’re seeing today, security will be playing catch up for quite some time. By understanding what’s coming and developing strategies to adapt to this new reality, security leaders can ensure they are ready to welcome (and oversee) our new agent driven lifestyles.

Today, agents are often derided as not reliable. They are autonomous in the way that my kid can “autonomously” get dressed when it’s time to leave - neither is likely to complete the task without a lot of hand holding. Give an agent a task with open-ended complexity and you’ll be amazed at the inventive ways the process falls apart.

Just as it’s easy to criticize the current state of LLM automation, it’s difficult to visualize the rate of progress since ChatGPT was released. Today, things have only accelerated and huge proportions of technical effort is focused on improving and operationalizing models to solve problems that previously landed squarely in the domain of humans.

We’re not far away from a future where interactions between semi-autonomous LLM agents will be the glue that connects disparate systems to get work done with humans holding an oversight role to ensure things don’t go flying off the rails and help when they do.

New, “AI-first” services built will crop up, going for a piece of legacy provider market share, but for the foreseeable future, the world will continue to operate with copilot powered versions of Salesforce, Netsuite and Workday.

Custom automation between these systems is where things will get interesting. Decisions and work will be handled by agents who interact with provider APIs or copilot interfaces.

What does that mean for security professionals?

1. Identity Management: The distinction between user and agent identities will blur as agents both gain autonomy and act on behalf of individuals. Mechanisms must be defined to ensure that an agent's actions are tied to a verifiable identity, whether acting on its own or as a delegate for a human user. This requires rethinking the static authentication and authorization frameworks currently in use for service level integrations.
2. Containing the Blast Radius: Similar to the impact of leaked keys but with many more ways of occurring, the potential for compromised or misbehaving agents underscores the need to design systems with the principle of least privilege. Permissions need to be segmented based on the specific tasks and data an agent requires to limit the damage an errant agent can cause.
3. Navigating Non-Determinism: The inherent unpredictability of LLM agents, coupled with their susceptibility to manipulation, requires rethinking security monitoring and response. Traditional deterministic models may fall short in detecting unintended agent behavior, necessitating a shift towards adaptive, behavior-based security analytics. Incorporating human oversight at critical decision points can provide a safety net, ensuring that anomalous or potentially harmful actions are reviewed by a human.
4. Prioritizing Upskilling and Knowledge Sharing: Effective management of security risks associated with LLMs and autonomous agents necessitates an understanding of how the technology works. Without comprehension of how these systems operate, identifying risks, effectively mitigating them and recognizing potential value-add opportunities becomes significantly harder. Proactively upskilling teams via training and projects where they can get hands-on now will be critically important to ensure AI competency. This shouldn’t be relegated to solely technical teams, but across the organization, ensuring all stakeholders understand the risks and opportunities.
5. Ensuring Explainability and Accountability: As agents interact in ever more complex ways, tracing the lineage of decisions and actions becomes more challenging and increasingly critical. Maintaining transparency and accountability will be vital to adoption across the enterprise. It may require developing new mechanisms to capture the nuanced and potentially complex context of agent decisions, taking into consideration that LLMs can be manipulated so an attacker could convince it to log something entirely different, something you don’t really worry about on today’s systems. This could be augmented by yet another layer of AI to interpret, summarize and alert on interactions for human reviewers.
6. Reassessing priorities: The shift towards AI-driven automation calls for a reassessment of priorities. Security teams must become more anticipatory and adaptive, prepared to evolve with the technologies they safeguard. Proactively engaging with AI development teams, participating in the design of these systems, and advocating for security-by-design principles from the outset are crucial steps. Fostering a culture of continuous learning and agility within will become even more important in a world where the pace of change is coming at an ever increasing rate.
7. Preemptive Action and Future-Proofing: To stay ahead, organizations should begin mapping their current and projected use of AI agents, identifying potential security issues via established techniques like threat modeling, and developing a phased plan to get ahead of them. Investing in learning, exploring AI-specific security tools, and becoming engaged with the broader security and AI research communities for insights and best practices will be key to navigating the future securely.

LLM agents will become a part of every individual’s and every business’s internet footprint, fundamentally altering the landscape. Given the rate of change and progress we’re seeing today, security will be playing catch up for quite some time. By understanding what’s coming and developing strategies to adapt to this new reality, security leaders can ensure they are ready to welcome (and oversee) our new agent driven lifestyles.