Intro to AI Security Part 5: AI Security and Cyber Security *(applying cyber security to AI)
When I first started my PhD no-one talked about the relationship between AI and cyber security. Now I’ve just returned from DEF CON and everyone is talking about it. However I still see a big divide between the methodologies that could be transferred between the two disciplines. Most data scientists I know are intimidated by the vast amount of infosec terms and required knowledge, and most cyber security professionals dabble in AI but don’t understand the mathematics behind machine learning that a data scientist should.
This blog will serve as an intro to cyber and information security that is aimed at anyone who wants a quick and dirty understanding of the field, but specifically targeted at data scientists. Towards the end I’ll include a section on how cyber and infosec principles can be translated to AI security, in addition what will need to be adapted due to intrinsic differences between the fields.
Definitions
Cyber security and information security are often used interchangeably, but they are not the same thing. Cyber security is the practice of protecting computer systems and networks from unauthorised access, use, disclosure, disruption, modification, or destruction. Information security is the practice of protecting the confidentiality, integrity, and availability of information.
In other words, cyber security is focused on protecting the physical systems that store and process information, while information security is focused on protecting the information itself.
The term “hacker” is often used to refer to someone who gains unauthorised access to computer systems and networks. However, the term has a broader meaning in the context of hacker culture, as anyone who really challenges the status quo. It’s the offensive side of cyber security, but these days they are just two sides of the same coin.
Cyber security 101
Cyber and information security are critical disciplines for safeguarding digital systems, data, and networks from unauthorised access, breaches, and malicious activities. As technology has become increasingly integrated into all aspects of modern life, the need to protect sensitive information and digital assets has grown as well.
In the early days of computing, security concerns were relatively limited due to the isolated nature of systems and the small user base. ‘Security by obscurity’ was often a good enough defence. However, as technology advanced, the proliferation of networks and the internet brought about new challenges. And these days, the amount of effort criminals and hackers will go to in order to penetrate a system is mind-boggling.
The Morris Worm, released in 1988, is considered one of the earliest instances of a widespread cyberattack. It was a self-replicating computer worm that was released onto the Internet in 1988. It was written by Robert Tappan Morris, a graduate student at Cornell University. The worm spread quickly and infected an estimated 6,000 computers, and caused a lot of damage.
In the 1990s and early 2000s, the focus shifted towards developing encryption techniques, firewalls, and intrusion detection systems to counter emerging and high profile threats. The ILOVEYOU worm was a computer worm that infected over ten million Windows personal computers on and after 5 May 2000. It started spreading as an email message with the subject line “ILOVEYOU” and the attachment “LOVE-LETTER-FOR-YOU.TXT.vbs.” At the time, Windows computers often hid the latter file extension (“VBS,” a type of interpreted file) by default because it is an extension for a file type that Windows knows, leading unwitting users to think it was a normal text file and open it, unwittingly letting it propagate.
Code Red and Nimda were two of the most destructive computer worms in history. They both exploited vulnerabilities in Microsoft’s Internet Information Server (IIS) web server software to spread rapidly across the internet. Code Red was first observed on July 15, 2001. It infected computers by exploiting a buffer overflow vulnerability in IIS. Once infected, Code Red would deface the victim’s website with a message that read “Hacked by Chinese!”. It could also be used to launch denial-of-service attacks. Nimda was first observed on September 18, 2001. It was even more destructive than Code Red, and it spread even faster. Nimda could exploit multiple vulnerabilities in IIS, as well as vulnerabilities in other software. It could also spread through email attachments, file shares, and even instant messaging.
You may notice a commonality here — Microsoft. Microsoft computers were by far the most widely used, and therefore fertile ground for hackers. They were therefore one of the first major technology companies to adopt a secure by design approach to software development, implementing a number of policies and procedures to improve the security of its products. These included things like security reviews (requiring all new software to undergo a security review before it was released), vulnerability disclosure (allowing security researchers to report vulnerabilities in Microsoft products, which they would then work to fix as quickly as possible), and security training for employees. There were of course other companies and thought leaders who were instrumental in the development of secure by design thinking.
These days, there are new security trends (AI, data breaches, dark web, cryptocurrencies, quantum computing, cloud security) that are impacting the cyber security environment. It’s an ongoing challenge.
Why is security a thing?
It takes a lot of code to write a computer program. Many millions of lines usually. Code (especially software code) requires you to be very explicit about what you want the computer to do. This means that it’s inevitable that sometimes there will be some sort of ‘bug’ in the code that exposes some aspect of that program or system to something it wasn’t intended. This is known as a vulnerability. However, just because a vulnerability exists doesn’t mean it will necessarily be discovered or used. If it is used, we refer to it as an exploit. In cyber security, a zero day is an exploit that has never been used before (ie. the first day it’s used is day zero). The market for zero days is massive, with zero day exploits costing millions of dollars to those who might want to buy them (who may be criminals to exploit them, or governments to prevent criminals from exploiting them).
This might sound like a lot of money, but the potential income from exploiting any kind of vulnerability in a system can lead to much more value — money, information, reputation. Therefore, when software makers identify vulnerabilities in systems they want to patch those systems and patch them fast (these are often in software updates or specific patches you might install on your computer). However ideally the vulnerability never existed in the first place. This is the nature of cyber and information security — the practice of identifying and patching vulnerabilities, with the caveat that secure software design principles should prevent as many software vulnerabilities as possible. Just as vulnerabilities and exploits exist in traditional computer systems, they can also exist in AI systems — this is my trade.
An analogy often used in cyber security is that of a house. If you’re looking to secure a house, you might start by analysing how many windows and doors (and chimneys?) the house has. These are all access vectors, and contribute to the attack surface. To secure these you would do things like put locks on the doors and windows, and hide the keys in very secure places. If you really care about your house’s security, you could go so far as to build an impenetrable wall around the house, or even hire a mover to relocate your house to the moon. However a house is also for living in, and you need to make a compromise between how efficiently you can get in your house, and how much you want to keep your nosy neighbours out. Such is the balance in software security as well.
Common cyber attack types
Data breach: A data breach is an incident in which sensitive, confidential, or otherwise protected data is exposed to an unauthorised individual or entity. This can happen through a variety of ways, such as hacking, phishing, or social engineering. In Australia, the Office of the Australian Information Commissioner (OAIC) is responsible for managing data breaches. The OAIC has a number of resources available to help organisations manage data breaches, including a data breach notification scheme and a guide to responding to data breaches. In the US, the Federal Trade Commission (FTC) is responsible for managing data breaches. In the European Union (EU), responsibility for data breaches is regulated by the General Data Protection Regulation (GDPR).
DDoS (Distributed Denial-of-Service) attack: A DDoS attack is an attempt to make a website or service unavailable to its intended users by overwhelming it with traffic from multiple sources. This can be done by using a botnet, which is a network of infected computers that are controlled by the attacker.
Malware: Malware is software that is designed to harm a computer system or network. Malware can take many forms, such as viruses, worms, trojan horses, and ransomware.
Virus: A virus is a type of malware that can replicate itself and spread from one computer to another. Viruses can be spread through email attachments, infected websites, or removable media such as USB drives.
Worm: A worm is a type of malware that can replicate itself and spread from one computer to another, but it does not need a host file to do so. Worms can spread through email attachments, infected websites, or network vulnerabilities.
BEC (Business Email Compromise): BEC is a type of fraud in which criminals impersonate a legitimate business in order to trick the victim into making a payment. BEC scams often involve sending emails that appear to be from a senior executive of the company, requesting a wire transfer or other financial transaction. AI technologies like video and audio deep fakes are making these kinds of scams (among others) harder to detect.
Ransomware: Ransomware is a type of malware that encrypts the victim’s files and demands a ransom payment in order to decrypt them. Ransomware attacks can be very costly, both in terms of the ransom payment and the damage to the victim’s data. You may not hear about this so much in the media, because organisations are embarrassed to say they’ve been impacted and/or paid a ransom, but they are shockingly common.
Security by Design
Security by design (SbD) is an approach to security that incorporates security principles into the design of systems and products from the outset. This means that security is not an afterthought, but is instead considered at every stage of the development process.
SbD is based on the principle that it is easier and more effective to design security into a system than to try to retrofit it later. This is because security principles can be embedded into the system’s architecture, design, and implementation, making it more difficult for attackers to exploit vulnerabilities.
There are a number of different security principles that can be applied to SbD, including:
Principle of Least Privilege: emphasises restricting users’ access rights to the minimum necessary for their specific roles. This reduces the potential impact of unauthorised actions and limits the damage caused by security breaches.
Defence-in-depth: involves layering multiple security measures throughout an organisation’s infrastructure to provide overlapping protection. This approach reduces the likelihood of a single point of failure compromising the entire system’s security.
Zero Trust: challenges traditional approaches by assuming that threats exist within the network and emphasising the need for continuous verification and strict access controls regardless of location.
Patching and Vulnerability Management: underscores the importance of keeping software and systems up to date to address known vulnerabilities and reduce the risk of exploitation by attackers.
Fail-safe defaults: This principle ensures that systems are configured in a secure state by default. This helps to reduce the risk of security vulnerabilities being exploited.
Continuous monitoring: This principle involves continuously monitoring systems for security threats and vulnerabilities. This helps to identify and mitigate threats early on.
What about Information Security?
Without wanting to drown you in terminology, the main takeaway from the field of information security is that of the CIA Triad (no, not that CIA).
Confidentiality: This principle ensures that sensitive information is accessible only to authorised individuals or entities. Confidentiality prevents unauthorised disclosure of data and aims to maintain privacy and protect sensitive data from falling into the wrong hands.
Integrity: Integrity focuses on maintaining the accuracy and reliability of data. It ensures that information remains unaltered during storage, processing, and transmission. Maintaining data integrity prevents unauthorised modifications, corruption, or tampering.
Availability: Availability ensures that authorised users have access to information and systems when needed. It aims to prevent disruptions, downtime, or denial of service attacks that could impact the availability of critical resources.
The Ecosystem
Perhaps most importantly, in my opinion at least, is that cyber and infosec has a massive ecosystem.
The cyber and information security ecosystem is a comprehensive framework encompassing technical innovations, governance structures, policy frameworks, human expertise, and legal considerations. This holistic approach is crucial for effectively managing cybersecurity and enhancing the resilience of digital systems. Each facet of the ecosystem contributes to the overall strength of the field.
Technical: This involves leveraging advanced technologies, tools, and solutions to secure digital assets. Firewalls, intrusion detection systems, encryption mechanisms, and vulnerability scanning tools form the technical defences that safeguard against cyber threats and attacks.
Governance: Effective cybersecurity governance establishes roles, responsibilities, and accountability across an organisation. It includes establishing security strategies, risk management protocols, and incident response plans. Governance ensures a coordinated approach to cybersecurity and aligns security efforts with business objectives.
Policy: Policies provide guidelines and rules for secure practices and compliance. Security policies address data protection, access controls, authentication protocols, and more. Well-defined policies promote consistent security practices, reducing vulnerabilities arising from ad-hoc decisions.
People: The human factor is paramount in cybersecurity. Educating and training individuals on security best practices and creating a culture of cybersecurity awareness are essential. Empowered and security-conscious employees play a pivotal role in identifying and mitigating threats.
Legal: Legal frameworks and regulations establish the boundaries and responsibilities of organisations in maintaining cybersecurity. Compliance with regulations such as GDPR, HIPAA, and others ensures that organisations prioritise data protection and privacy. Organisations can even buy cyber insurance.
What does this mean for AI Security?
These are all principles that can be applied to AI Security. In fact, some Standards organisations are starting to recommend ways we can do just that, which I’m very happy about (also because it keeps me in business.. and makes the world a safer place).
However there are some key differences that need to be considered from a security perspective.
- Cybersecurity systems are designed to protect against known threats, while AI systems are constantly learning and evolving. This means that cybersecurity systems may not be able to keep up with the latest threats posed by AI systems.
- Cybersecurity systems are typically rule-based, while AI systems are often probabilistic. This means that cybersecurity systems may not be able to handle the uncertainty and ambiguity that is often present in AI systems.
- Cybersecurity systems are typically reactive, while AI systems can be proactive. This means that cybersecurity systems may not be able to prevent attacks that are initiated by AI systems.
Many cyber and information security controls will not necessarily be effective against AI attacks, since the attack surfaces are completely different technologies. However, there are many ways cyber and information security controls can be adapted for use in AI Security.
And therefore we might need to consider:
- Data security: AI systems often require access to large amounts of data in order to learn and improve. This data must be protected from unauthorised access, modification, or destruction.
- Model security: AI models are often proprietary and valuable assets. They must be protected from theft, reverse engineering, and tampering.
- Model selection: the choice of model will impact which vulnerabilities are present and how they can be exploited.
- Inference security: AI systems can be used to make inferences about individuals or groups. These inferences must be protected from unauthorised disclosure.
- Human-AI teams: humans interact with AI in a totally different way to computer systems, especially when it comes to trust and personification of human-like AI systems. Human psychology is a crucial consideration.
- Real world interaction: AI systems may be make or contribute to decisions that have real world consequences. There must be enough context in the system to enable good predictions.
This is an area I would love to see more active development, and I highly encourage any readers to be creative and learn more about how cyber security could be applied to AI security.
Cyber security references and resources
MITRE ATT&CK is a resource for tactics, techniques and procedures (TTPs) in cyber — basically how adversaries do things. They now also have ATLAS for adversarial machine learning.
OWASP stands for Open Web Application Security Project. It is a non-profit organisation that works to improve the security of software. OWASP provides free and open resources, including documentation, tools, and training, to help developers and organisations secure their web applications.
I also love the Darknet Diaries podcast by Jack Rhysider for learning about real world events that I often can’t believe really happened.
I definitely need to write another blog on this…
I still haven’t really reached the tip of the iceberg when it comes to cyber security, and it makes me realise I must definitely dedicate another couple of blogs to cyber security and other possible intersections with AI. For example, physical penetration testing and social engineering are FASCINATING rabbit holes to go down and could both have their place in the AI Security world.
I highly encourage you to develop your hacking skills — but as a white hat only, of course ;)
In the next part we’re going to be diving into AI Governance.. and it’s more interesting than you might think.
