The development of AI weapons systems is a chilling prospect. The European Parliament has recently demanded the start of “international negotiations on a legally binding instrument prohibiting lethal autonomous weapons systems”. The UN Convention on Conventional Weapons (CCW) is assessing the issue and the mandate for 2019 will be decided at the CCW’s annual meeting on 23 November 2018.
Some form of international regulatory control is desirable but greater recognition is needed that AI is becoming ubiquitous, cheap and easily obtainable. AI will be used in weapons by terrorists and rogue states. We need to be prepared.
This article by Roger Bickerstaff from the SCL Journal, “Computers & Law” (set out below) summarises the current legal situation in the UK, the US and at the CCW.
The use of artificial intelligence in the defence sector raises strong emotions. The risk of ‘killer robots’ running wild is a common Hollywood theme. However, as AI becomes commonplace in the business environment, the defence sector is following quickly, and the potential risks from use of AI in weapon systems are becoming more real. There is a need for more widespread and informed debate on the need for and extent of regulation on the use of AI in the defence sector.
UK: House of Lords Select Committee on AI
The issue was considered in some depth recently by the House of Lords Select Committee on Artificial Intelligence (very ably chaired by Lord Clement-Jones – who recently chaired a panel discussion on AI at the SCL Annual Conference 2018). The Select Committee report warned that a ‘lack of semantic clarity could lead the UK towards an ill-considered drift into increasingly autonomous weaponry’. The Select Committee identified the MoD’s definition of AI weapons systems as being a significant concern, as it establishes a considerably higher standard for AI weapons as compared to the definition used by most countries.
The most recent MoD definition of AI weapons systems is set out in the MoD’s guidance on unmanned aircraft systems (September 2017) which provides that:
An autonomous system is capable of understanding higher-level intent and direction. From this understanding and its perception of its environment, such a system is able to take appropriate action to bring about a desired state. It is capable of deciding a course of action, from a number of alternatives, without depending on human oversight and control, although these may still be present.
In the hearings of the Select Committee, Professor Noel Sharkey commented that the requirement in the MoD’s definition for an autonomous weapon system to be capable of being ‘aware and show intention’ was to set the bar so high for autonomy that it was effectively meaningless. This allows the MoD to avoid the issue of whether or not it should be developing and potentially deploying AI weapons systems. The MoD can claim on the basis of this very demanding definition that ‘the UK does not possess fully autonomous weapon systems and has no intention of developing them. Such systems are not yet in existence and are not likely to be for many years, if at all’.
However, AI-based weapons systems are starting to be developed and deployed in practice. During the Select Committee oral hearings, Mike Stone (ex- MoD Chief Digital and Information Officer) commented that at the moment in the defence area AI is ‘most prevalent’ in the back office, rather than in the military arena but that there is some AI in cyber defence and a limited amount of AI in defensive ship-to-air missiles. In his view, ‘the genie is out of the bottle’ in the sense that the usage of AI is becoming so widespread in non-military usage that it is inevitable that it will cross-over into military usage, particularly as much military innovation and investments ‘coming from the civilian world and being brought into the military world’.
In the same oral hearing, Professor Noel Sharkey commented that, in his view, the major concerns around the use of AI in the defence areas relate primarily to the use of AI in two areas: autonomy of systems in target selection and autonomy in the application of violent force. More generally, and from the perspective of international humanitarian law, there are concerns that the use of AI in target selection and the application of violent force may not comply with the principles of distinction, proportionality and military necessity or precautions in attack. It is arguable that, if weapon systems can identify and prioritise targets, some form of meaningful human control is required in order for them to comply with international humanitarian law.
The MoD is able to avoid having to deal with this issue on the basis of its overly demanding definition of autonomous weapons systems. However, in its response to the Select Committee report, the UK government has said that it has no intention of changing its definition of autonomous weapons systems.
US Directive on AI Weapons
The USA introduced a specific legal framework for AI weapons systems in November 2012. The USA issued Directive 3000.09 (updated in 2017) (see http://www.esd.whs.mil/DD/), establishing policy for the ‘design, development, acquisition, testing, fielding, and … application of lethal or non-lethal, kinetic or non-kinetic, force by autonomous or semi-autonomous weapon systems.’
It was a first attempt at establishing policy prescriptions and demarcating lines of responsibility for the creation and use of semi-autonomous, ‘human supervised’ and fully autonomous weapons systems. In layman’s terms, it attempts to answer the who, what, when, where and how of autonomous systems in military combat.
The Directive sets out reasonably clear lines of responsibility for system development, testing and evaluation, equipment/weapons training, as well as developing doctrine, tactics, techniques and procedures. The explicit purpose of the Directive is to establish guidelines to ‘minimize the probability and consequences of failures in autonomous and semi-autonomous weapons systems that could lead to unintended engagements’. In other words, ‘the use of force resulting in damage to persons or objects that human operators did not intend to be the targets of US military operations’.
The Directive raises as many questions as it answers. One of the key worries was the extent to which the policy could be avoided in certain circumstances. Also, the Directive may erode the concept of ‘proper authority’ for the use of violent force and thus raise questions over compliance with international humanitarian law.
UN Convention on Conventional Weapons
The United Nations is actively considering the imposition of a ban on AI weapons systems. A process has been set up to consider whether AI systems should be restricted under the Convention on Certain Conventional Weapons, a disarmament treaty that has regulated or banned several other types of weapons, including incendiary weapons and blinding lasers.
Meetings of the Group of Governmental Experts to review this issue have been held since 2014. At the most recent meeting in September 2018, 26 countries supported an outright ban on fully autonomous weapons. A small number of countries, including Australia, Israel, Russia, South Korea, and the United States, oppose a new treaty, political declaration, or any other new measures dealing with the introduction of AI weapons systems.
The issues will continue to be debated at a meeting of the UN High Contracting Parties in November 2018. The options currently ‘on the table’ relate to proposals for:
- a legally-binding instrument stipulating prohibitions and regulations on lethal autonomous weapons systems, potentially including a requirement for human control over the critical functions in lethal autonomous weapons systems;
- a political declaration that would outline important principles such as the necessity of human control in the use of force and the importance of human accountability, and with elements of transparency and technology review;
- further discussions on the human-machine interface and the application of existing international legal obligations, including legal weapons reviews required by the Geneva Conventions, and the identification of practical measures, best practices and information sharing; and
- a view that, as international humanitarian law is fully applicable to potential lethal autonomous weapons systems, no further legal measures are needed.
These options are not necessarily mutually exclusive. It is not at all clear which option or options may be chosen – if any – given the power of the relatively small number of countries that oppose positive action. Campaigning organisation (such as the Arms Control Association) make the point that ‘the time for discussion is over and that the dangers of deploying lethal autonomous weapons have been sufficiently demonstrated to warrant the initiation of formal negotiations on meaningful control mechanisms’.
Concluding Thoughts
Whilst the risks of AI weapons systems merit the need for a serious assessment of their use in armed conflict, a ban on AI weaponry aimed at preventing AI weaponry getting into the ‘wrong hands’ is not likely to be effective. AI is becoming pervasive. AI is different to most chemical weapons and many other weapons of mass destruction which are complex, expensive and their manufacture can be observed.
As AI becomes more easily accessible, it is inevitable that AI solutions will be linked to weapons systems to create weapons systems with AI capabilities. Ironically, Elon Musk is one of the leading promoters of Open AI and he is also actively campaigning for legislative controls on AI weapons systems.
Terrorists and rogue states are probably already in the process of developing AI weaponry. An autonomous car with a bomb in the boot may well be an autonomous weapons system. Effective defensive capabilities are needed to counter AI weapons used by the ‘bad guys’. There is a need to ensure that well-meaning constraints on the use of AI weapons systems do not result in exposure to AI weaponry used by terrorists and rogue states.
Roger Bickerstaff
