This article is available exclusively to Business Insider subscribers. If you're not yet a member, consider becoming one to access in-depth analyses and news updates now.

Generative artificial intelligence (AI) is a cutting-edge technology that has rapidly emerged in recent years, bringing with it a range of innovative possibilities as well as significant new security challenges. These challenges are particularly daunting for organizations that may not fully comprehend the vulnerabilities that generative AI can introduce into their systems.

At the forefront of these concerns are chatbots that leverage large language models (LLMs). These advanced systems are not only capable of generating human-like text but also susceptible to a variety of novel cybersecurity threats. For instance, prompt injections can manipulate the behavior of a model by using carefully crafted prompts, while data exfiltration involves issuing countless prompts to extract sensitive or valuable information hidden within the model's dataset.

Such vulnerabilities exploit the unpredictable behavior of LLMs, leading to considerable financial repercussions. Chuck Herrin, the field chief information security officer at F5, a firm specializing in multicloud application and security solutions, noted, "The largest security breach I'm aware of, in monetary terms, happened recently, and it was an attack against OpenAI." Herrin's statements highlight the severity of the risks associated with generative AI.

Herrin was referring to a significant incident involving DeepSeek, a language model developed by a Chinese company of the same name. On January 20, DeepSeek made headlines by releasing its reasoning model, DeepSeek-R1, which surprisingly performed just below the premier models from OpenAI in popular AI assessments. However, users of DeepSeek quickly began reporting unusual behaviors. For one, the model frequently mimicked the response style of OpenAI's ChatGPT and falsely identified itself as a product trained by OpenAI.

In the weeks following DeepSeek's launch, OpenAI disclosed to the Financial Times that it had gathered evidence suggesting that DeepSeek had employed a technique known as "distillation" to train its model by leveraging prompts directed at ChatGPT. Although OpenAI claimed to have this evidence, it has not been made public, and it remains uncertain whether the company will take further action against DeepSeek.

This revelation raised alarm bells within the cybersecurity community. Herrin stated, "DeepSeek was accused of distilling OpenAI's models down and stealing its intellectual property. When the news of that hit the media, it took a trillion dollars off the S&P." This incident underscores the potential threat that unregulated AI models pose not only to individual companies but also to the broader financial landscape.

The implications of AI vulnerabilities are well documented. LLMs are trained on massive datasets, which enable them to respond to a wide variety of user prompts. However, these models do not typically "memorize" the specific data they are trained on. Instead, they develop a generalized understanding of the relationships within the data, which can lead to issues of data leakage. While memorization can happenan important aspect in the ongoing copyright infringement lawsuit filed by The New York Times against OpenAIprompting a model several thousand times can provide insights into the model's training data through a process known as distillation.

This is precisely why Herrin emphasizes the need for comprehensive security measures: "You can't secure your AI without securing the application programming interface (API) used to access the model and the rest of the ecosystem." If APIs are accessible without sufficient safeguards, they become a potential target for exploitation.

Compounding these challenges is the inherent 'black box' nature of LLMs. When training an LLM, a neural network is created that allows for a generalized comprehension of the training data, but it does not provide clarity on which specific components of the network are responsible for particular outputs. This obscurity makes it impossible to restrict access to certain data within an LLM as one might do with a traditional database.

Sanjay Kalra, head of product management at Zscaler, elaborated on this issue by stating, "Traditionally, when you place data, you place it in a database somewhere. At some point, an organization could delete that data if it wanted to. But with LLM chatbots, there's no easy way to roll back information." This lack of control over data adds another layer of complexity to managing AI security.

To address these vulnerabilities, cybersecurity companies are exploring solutions from multiple angles, with two approaches standing out. The first is rooted in traditional cybersecurity protocols. Herrin emphasized, "We already control authentication and authorization and have for a long time." He pointed out that while the authentication process itself doesnt change much when dealing with LLMs compared to other services, it remains a critical aspect of security.

Kalra reiterated the necessity of implementing solid security fundamentals, such as access controls and logging user access. He remarked, "Maybe you want a copilot that's only available for engineering folks, but that shouldn't be available for marketing, or sales, or from a particular location." These measures can help mitigate some risks associated with AI systems.

However, the irony lies in the fact that the solution to AI vulnerabilities may also involve the use of more AI. The 'black box' nature of LLMs complicates security as it remains uncertain which prompts might circumvent existing safeguards or exfiltrate data. Fortunately, LLMs excel at analyzing text and data, and cybersecurity firms are capitalizing on this capability to create AI watchdogs.

These models serve as an additional layer between the LLM and its users, actively scrutinizing user prompts and model responses for indications that a user is attempting to extract information, bypass safeguards, or otherwise manipulate the model. Herrin aptly described the scenario by stating, "It takes a good-guy AI to fight a bad-guy AI. It's sort of this arms race. We're using an LLM that we purpose-built to detect these types of attacks." F5 offers services that allow clients to leverage this capability, whether they are deploying their own AI models on-premises or accessing AI models in the cloud.

Despite these advancements, challenges remain, particularly concerning cost. Deploying a security-tuned variant of a robust model like OpenAI's GPT-4.1 may seem like an ideal route to achieve optimal security, but these advanced models come with high price tags, making them impractical for many organizations.

Kalra highlighted this economic dilemma, stating, "The insurance can't be more expensive than the car. If I start using a large language model to protect other large language models, it's going to be cost-prohibitive." As a solution, there is growing interest in utilizing smaller language modelsthose with fewer parameterswhich require less computational power for training and deployment. Noteworthy examples include Meta's Llama 3-8B and Mistral's Ministral 3B. Zscaler is also developing its own internal models through its AI and machine learning team.

As artificial intelligence continues to advance, organizations find themselves navigating an unexpected security landscape: the very technology that possesses vulnerabilities is also becoming central to their defense strategies against those vulnerabilities. By adopting a multilayered approach that combines robust cybersecurity practices with security-optimized AI models, organizations can begin to bridge the security gaps present in LLMs and better protect themselves against potential threats.