AI Cybersecurity Risks: A Detailed Examination

Organisations and individuals face significant cybersecurity risks when using AI systems without proper safeguards. Understanding these risks in depth is essential for developing effective protection strategies.

This lesson is going to list all of the current and common bad parts of AI. We're intentionally listing everything that could and will possibly go wrong with a business' use of AI. It's like ripping a band-aide off. Diving into the bad, ugly and dirty world of AI -

Personal Identifiable Information and Public Generative AI

One of the most serious and common risks occurs when people use public AI services to process personal identifiable information (PII) without understanding the implications. PII includes names, addresses, phone numbers, email addresses, social security numbers, financial information, medical records, employee data, and any other information that can identify a specific individual.

When someone pastes PII into a public AI tool like the free version of ChatGPT, Gemini, or similar services, they're sending that information to external servers where it may be stored indefinitely, used to train future versions of the AI model, or potentially exposed through data breaches or security vulnerabilities. The person providing the information loses control over it completely.

This creates multiple problems. First, it may violate privacy regulations like Australian OIAC Data Breach legislation. The GDPR in Europe requires explicit consent and specific legal bases for processing personal data. The US HIPAA (in healthcare settings) mandates strict protections for patient information. Organisations can face substantial fines and legal consequences for these violations. Second, it breaches the trust of customers, employees, or clients whose information is being shared without their knowledge or consent. Third, once PII enters a public AI system, there's no reliable way to remove it or ensure it won't appear in responses to other users' queries.

It's worth noting here - AI has no concept of the security classification of the data you're sending it. It doesn't understand that the tax file number or SSN you just pasted into the chat window is extremely sensitive information. Therefore, it has no guardrails to prevent it regurgitating that information back to another user. It's up to the AI user to classify data as public, private, sensitive, or confidential (as examples) and limit what the AI system receives.

Common scenarios where this risk materialises include HR staff pasting employee performance reviews or salary information into AI to help draft communications, customer service representatives using AI to analyze support tickets containing customer details, healthcare workers seeking AI assistance with cases that include patient information, or developers testing code with real user databases instead of anonymised test data.

The problem is compounded because the conversational nature of AI interfaces makes people forget they're interacting with a public service. It feels private and confidential, like talking to a colleague, when it's actually more like posting information on a public bulletin board.

Exposure of Private Information

Beyond PII, organisations face risks from exposing various types of private and confidential information to AI systems. This encompasses proprietary business information, trade secrets, source code, security configurations, strategic plans, financial data, customer lists, competitive intelligence, unpublished research, and internal communications.

When employees use public AI tools to help with their work, they often share information without considering its sensitivity. A marketing manager might paste an entire unreleased product roadmap asking for help with messaging strategy. An engineer might share proprietary algorithms asking for optimisation suggestions. A finance team member might input detailed budget information seeking analysis. In each case, competitive advantages and confidential information flow out into the public domain.

The risk extends beyond direct data exposure to inference risks. Even if someone doesn't share complete information, AI systems that process fragments from multiple users across an organisation enable the reconstruction of sensitive details. An AI trained on enough snippets from a company's employees could potentially piece together information about unreleased products, strategic initiatives, or business challenges. This is an active Reconnaissance method that is used today on the Internet: we cover Reconnaissance in the Defence Against CyberAttackers module.

There's also the supply chain dimension. When vendors, contractors, or partners use public AI to work on your projects, they may expose your confidential information even though you haven't directly shared it. A consulting firm drafting strategy recommendations might use AI that incorporates your business details into its training data. A software development contractor might use AI code assistants that learn from your proprietary codebase.

The permanence of this exposure is particularly concerning. Unlike a document accidentally sent to the wrong email address, which can potentially be deleted or contained, information that enters an AI training pipeline may be impossible to fully remove. It becomes embedded in the model's learned patterns and could surface in unexpected ways for years.

Inaccurate AI Advice Used Without Filtering in Business-Critical Areas

AI systems can produce confident-sounding but incorrect, outdated, or contextually inappropriate advice, and when this advice is implemented in business-critical areas without proper verification, the consequences can be severe.

AI models are trained on historical data with specific cutoff dates, meaning their knowledge about regulations, best practices, security vulnerabilities, and technical standards may be outdated. An AI might recommend a security configuration that was best practice three years ago but is now known to be vulnerable. It might suggest compliance approaches based on old versions of regulations that have since changed. It might reference software libraries or dependencies that have been deprecated or found to contain security flaws.

Beyond outdated information, AI systems can simply be wrong. They predict what text should come next based on patterns, not on verified facts or genuine understanding. This leads to plausible-sounding but factually incorrect advice. An AI might recommend a way to solve a problem that seems logical but contains fundamental flaws. It might suggest legal language for contracts that doesn't actually protect your interests or might even create liabilities. It might provide medical or safety guidance that sounds authoritative but contradicts established protocols.

The risk intensifies in technical domains where errors might not be immediately obvious. AI-generated programs or procedures can contain security vulnerabilities, logic errors, or inefficiencies that only manifest under specific conditions or after the system has been in production for some time.

Context blindness compounds these problems. AI systems don't fully understand your specific business context, regulatory environment, risk tolerance, or technical infrastructure. Advice that might be appropriate for a startup might be disastrous for a highly regulated enterprise like a medical practice. Security recommendations suitable for a public-facing website could be inadequate for systems handling financial transactions. The AI doesn't know these nuances unless explicitly told, and even then may not properly account for them.

People tend to trust AI advice because it's presented with apparent confidence and expertise. The polished, authoritative tone can create a false sense of security. Someone without deep expertise in an area may lack the knowledge needed to critically evaluate whether AI recommendations are sound. This is particularly dangerous when junior staff use AI to complete tasks beyond their current skill level without having senior colleagues review the work.

The speed and convenience of AI can also create pressure to skip verification steps. When AI generates a solution in seconds, taking hours or days to properly validate it feels inefficient. Organisations may rush to implement AI-generated strategies, code, or processes to maintain velocity, inadvertently building serious problems into their operations.

Compounding Factors Across All Three Risks

Several factors make these risks more severe and harder to manage. The ease of AI access means anyone in an organisation can use these tools without going through IT or security approvals. There's often no visibility into what AI services employees are using or what information they're sharing. Training and policies haven't kept pace with adoption, leaving many users unaware they're creating security exposures.

The sophistication of modern AI also means these tools are genuinely useful, creating pressure to use them despite risks. Organisations that prohibit AI usage entirely may find employees using Public AI covertly through personal accounts, creating even less visibility and control. The solution isn't banning AI but rather implementing appropriate safeguards, education, and enterprise-grade tools with proper security controls.

Drawing the AI Pitfalls lesson to a close.

Understanding these risks in detail is the first step toward managing them effectively.

The call-out of this lesson is - We need need clear guidelines and policies about what information can be shared with AI systems, which AI services are approved for different types of work, what verification processes must be followed for AI-generated outputs, and how to handle incidents when sensitive information is inadvertently exposed.

As employees, we may need training to recognize sensitive information, understand the implications of sharing it, critically evaluate AI outputs, and know when to seek expert review rather than trusting AI advice directly.

Lesson 3 – AI Pitfalls