In April this year Group Captain Jo Brick (an esteemed editor at The Central Blue) gave a compelling presentation to the Williams Foundation Seminar discussing Next Generation Autonomous Systems. Specifically, she addressed the history of automation and ‘human-machine’ teaming - but not in a plain or linear approach. Using three popular stories from popular culture, Jo was able to plot the history of automation and its consequences for the conduct of warfare. More importantly however, she was able to highlight the wider questions we face as a human society that has advanced technology to the degree that we can imbue machines with human characteristics.
Good morning ladies and gentlemen. It is a very great privilege to be a part of this Williams Seminar on the ‘Next Generation Autonomous Systems’. As you all know, the Royal Australian Air Force commemorated 100 years of service just a few weeks ago. It was an occasion to reflect on the evolution of the Air Force over the last century. I think that this seminar allows us to consider the concepts and capabilities that will impact, not just on air forces, but the joint force, into the next 100 years. I am honoured to be invited to speak and contribute to this discussion.
In preparing for this presentation, I discovered that trying to understand automation is impossible if you focus only on the technology and machines. An earnest approach to understanding automation inevitably involves a multidisciplinary approach that includes fields such as anthropology, philosophy and ethics, mathematics and engineering, law, history, economics, and sociology. This is because the history of automation and ‘human-machine’ teaming tells us more about ourselves and our humanity than it does about the machines. The story of automation is one of our desires and fears, and of our drive and determination to exceed the biological and cognitive limits of body and mind. Futurist and inventor, Ray Kurzweil states that ‘technology is the continuation of evolution by other means…’. [1] This idea is consistent with the philosopher Manuel De Landa’s thesis in a book titled War in the Age of Intelligent Machines, where he discusses the transference of human cognition to machines as part of the ongoing development of military capability.
My aim in providing this overview is to establish a foundation for the remainder of the seminar by discussing a few of the historical aspects, concepts, and challenges that underpin automation, particularly its impact on warfare. Rather than providing a plain, linear approach to the history of automation, I will take a different approach that uses three stories from popular culture to plot the history of automation and the consequences for the conduct of warfare:
Mary Shelley’s Frankenstein.
Iron Man from the Marvel Cinematic Universe.
Osamu Tezuka’s Astro Boy television series.
Frankenstein – warnings about technology and industrialisation
Mary Shelley’s Frankenstein conveys several themes. Victor Frankenstein, the creator and not the creature, has violated Nature by assembling various body parts into a new creature that he is disgusted by and rejects. The story is often cited as a warning against human hubris and its blinding focus on the creator’s ego at the expense of truly thinking about the consequences of what we create. It is also a story that incorporates the zeitgeist of the early 19th century. This society did not yet have the word ‘scientist’ but was experiencing discoveries by its ‘alchemists’ and ‘natural philosophers’. Shelley does not describe Victor as a ‘scientist’ as the word did not exist in 1818, when the book was first published. Victor would have been considered as a ‘natural philosopher’ – a person who tried to understand and describe nature. [2]
Shelley wrote the story at the time of the Industrial Revolution, characterised by the rapid rate of scientific development that led to the steam engine and the flourishing of the iron, coal and cotton industries. Interestingly, Shelley’s world in the early 19th century, was one of significant social upheaval caused by the pursuit of scientific understanding. During this time, science was performance – with public demonstrations of experiments by natural philosophers expressing their theories or displaying an invention or curio [3]. Automata were some of the items displayed – including mechanical dolls containing elaborate ‘clockwork’ mechanisms and a wood carved chess player called ‘The Turk’. Automation in the industrial era also meant the design of machines to replace human labour. Further, industrialisation meant the design of processes that resulted in the mechanistic behaviour of humans as part of a monotonous and repetitive routine [4]. The work of Frederick Winslow Taylor in creating a scientific and standardised approach to manufacturing effectively removed the individual artisan approach of workers with set procedures to be followed by skilled workers as cogs in a large industrial machine. This was a time of making humans the automatons for industry [5]. It was during this period that Karel Čapek coined the term ‘robot’ (based on the Czech verb robota ‘to work’) in a play, Rossum’s Universal Robots (RUR, 1920), which represented men as machines.
Industrialisation impacted warfare in the late 19th century and into the 20th century. Its impact was demonstrated by the horrific results of the collision between Napoleonic era military tactics and mechanised weaponry. Like the industrialisation and standardisation of the factories at the time, this war of the industrial era can be perceived as one of automatons (men) subjected to drill procedures and constant calls of ‘over the top’ at various locales in the French countryside – the Somme, Hamel, Villiers-Bretonneux – where men were pitted against mechanised fire – machine guns and artillery. The contest was determined by which side exhausted its stocks of men and armaments first. In the journey towards automation, the industrialisation of society – including warfare – meant the replacement of human muscle and effort with machines, and the fusion of humans into standardised and repetitive procedures.
Iron Man – extending human ability.
In the Marvel Cinematic Universe, Anthony Stark develops an armoured suit that allows him to ‘integrate’ with a natural language interface computer system called JARVIS (Just A Rather Very Intelligent System). The suit and its iterations (50 plus variants) feature various weapons, flight systems, and armour. A notable aspect of the Iron Man suits is that it binds the human with the machine, but the human maintains control and uses the computer interface (JARVIS) to assist in decision making. Iron Man represents a change in the relationship between humans and machines.
Humans interfaced with the basic machines of the industrial revolution – such as steam engines, furnaces, power looms, agricultural machinery – as part of industrialised processes. The advent of aircraft, motor vehicles (including tanks and other armoured vehicles) provide a further example of human-machine integration, as human operators controlled these machines of war to overcome inherent human limitations in a bid to gain an advantage over the adversary. The human and the machine form a symbiotic system – the human needs the aircraft to fly, and the aircraft needs the human to fly it [6]. The relationship between human and machine is encapsulated by the term ‘cybernetics’; developed by MIT mathematician Norbert Wiener, who also developed its first theory. His book, Cybernetics; or, Control and Communication in the Animal and Machine, was published in 1948 and advanced three ideas that supported his thoughts on human-machine interaction: control (control or shape the environment); feedback (the use of sensors to receive information about actual performance); and merging human and machine to form a single entity [7].
A rudimentary example of these cybernetic ideas in practice is the Sperry Gyroscope, founded in 1910 by American entrepreneur Elmer Ambrose Sperry. The company’s value in developing military products in the 1940s was to ‘extend the physical and mental powers of men in the armed forces enabling them to hit the enemy before and more often than the enemy can hit them’ [8]. The Sperry ball turret was developed for the B-17 Flying Fortress to allow the gunner to use a series of controls for the turret and the two Browning .50-calibre machine guns that gave the gunner a broad vertical and horizontal range of fire against aerial targets.
The fundamental theories of cybernetics articulated by Wiener are enhanced in practice through the increased processing power of computers. As the operation of JARVIS in Iron Man demonstrates, advanced computing power has enabled the enhancement or replacement of human decision-making in limited circumstances. The processing of data feeds by JARVIS means that Stark does not have to occupy his mind with parsing data, but can focus on important decisions about how to use Iron Man in a hostile environment. Similarly, systems in modern military aircraft have replaced human air crew (flight engineers, navigators) and are able to manage and process volumes of data that would have overwhelmed the human crew they replaced. The increase in computing power of these systems over time is a vital step towards semi and full autonomous systems.
Astro Boy – machines become ‘human’.
The 1980s version of the TV series ‘Astro Boy’ or ‘Mighty Atom’, created by Japanese manga artist Osamu Tezuka, is set in the 2030s. Dr Bonyton has been asked to create a robot with a ‘soul’. He ultimately replicates his son, Toby, who died in a car accident. Astro Boy is ‘raised’ by positive influences – Dr Elefun and Daddy Walrus. His treatment and growth as a character is often contrasted with his brother, Atlas, who is made from the same design template. Atlas has negative influences through a criminal, Skunk, who teaches Atlas to commit crimes for him. The interaction between Astro and Atlas throughout the series serves as a running commentary of the impact of human influence and human frailty in the development of artificial intelligence. Meanwhile, Astro Boy is reminiscent of Pinocchio – he just wants to be a real boy – but the world of humans around him struggle with whether he should be a hero or treated with derision like Frankenstein’s monster.
‘Astro Boy’ explores key themes about robotics and consciousness, and what a world of sentient robots would mean – how would they co-exist with humans and what kind of ‘quality of life’ and ‘rights’ would such sentient robots have?
Today we have autonomous and semi-autonomous systems, with humans in the loop that maintain the executive function of a system. A simple military example is the unmanned aerial or ground vehicles that incorporates artificial intelligence to enable autonomous operation in limited circumstances. The most recent example is the Boeing Loyal Wingman, which is part of an Airpower Teaming System that allows it to operate independently using artificial intelligence, or in partnership with a manned aircraft.
We have yet to realise the level of intelligence or consciousness that ‘Astro Boy’ represents. We face many obstacles in understanding ourselves before we can fulfil Kurzweil’s goal of using technology to further human evolution – or to realise the transference of human cognition to machines. The ability to learn and use data to make decisions is central to realising the level of artificial intelligence that enables the type of autonomy that is close to human decision-making.
The capacity of machines to compute and store memory has increased significantly in the last few decades. However, we are yet to understand how humans, and therefore machines can learn [9]. Further, according to Australian AI expert Ellen Broad, current AI systems sold commercially to make important decisions ‘are brittle, error-prone and poorly designed.’ [10] The central thesis of Broad’s work is that we leave our fingerprints on the systems and AI that we design and create. We can create a system of deep learning for AI and simulated neural networks, but these sophisticated systems cannot operate optimally if we build it on a poor foundation of mislabelled, biased, and poor quality data. In this context, it is difficult to countenance entrusting AI with the autonomous decision or whether or not to strike a target. When the current Chief of Defence Force, General Campbell was Chief of Army, he said: ‘I think not just the military but society … is going to go through a period of learning and understanding what is the right point of comfortably accepted machine analysis, integration, filtering, and machine decision… I don't think we have anywhere near the sophistication of understanding at this stage and for many years to be comfortable with human[s] not in the loop’.
Conclusion
I have given you only an exceedingly small and select overview into the history of automation. We can all do more to understand our relationship with the machines that we have created to assist us extend past our human limitations. As Kurzweil stated, we use technology to further our evolution beyond our inherent biological and cognitive limits. We used machines as muscle, and to help us with processing information and to assist in decision-making. We are now developing our technology to the point of creating machines with artificial intelligence and the ability to operate autonomously, without our intervention or input. We expect such superior technology to satisfy our military requirements to ‘be everywhere, know everything, and… to predict what happens next’ [11]. Yet we face many challenges with the use of such technology and many questions remain to be answered:
Do we trust the systems that we have created [12]? Are we expecting them to be perfect, or to accept that they are flawed just like us?
Do we understand autonomous systems enough to inform the creation of an effective system of accountability?
How would autonomous and intelligent systems make decisions, free from human intervention? Would they reflect the best of humanity or something less inspiring [13]?
Does the conduct of war by autonomous and intelligent systems dilute the sanctity of war as a societal function?
Who or what is permitted to fight wars and to take human life on behalf of the state?
What does the use of AI and autonomous systems in warfare mean for the profession of arms?
These issues are ones that should concern us as part of the military profession and defence industry. They are a subset of the wider questions we face as a human society that has advanced technology to the degree that we can imbue machines with human characteristics. Scientists, engineers, ethicists, futurists, and entrepreneurs such as Bill Gates and Elon Musk have urged caution in the development of AI. There are movements that are focused on keeping the development of AI beneficial to society (called AI-safety research) [14]. Strong activism at the United Nations by non-government organisations that advocate against the development of lethal autonomous weapons.
While it is easy to shut out such opposing voices, we should take the time to understand the issues they raise. Listening to opposing viewpoints as a means of testing our philosophies and assumptions is essential in avoiding ‘sleepwalking into a dangerous brave new world that is slowly slipping out of our control’ and to understand the risk we take in delegating and transferring more control and autonomy into the hands of more networked and intelligent machines [15]. I said at the start, the story of automation is the story of what it is to be human. As Ellen Broad states: ‘You don’t need to be a technical expert to appreciate the possibilities and pitfalls of using data. You just have to know what it’s like to be human’. After all, we are creating machines to be: Just. Like. Us.
FOOTNOTES
[1] Ray Kurzweil, The Age of Spiritual Machines (New York: Penguin, 1999), 16.
[2] The word ‘scientist’ was not coined until 1834 by William Whewell, a Professor of Moral Philosophy at Cambridge and associate of Charles Babbage – a pioneer of computational machines (the ‘Difference Engine’). National Public Radio. ‘How the word “scientist” came to be’. Talk of the Nation, NPR, 21 May 2010 <https://www.npr.org/templates/story/story.php?storyId=127037417> (accessed 04 April 2021).
[3] Burdon, ‘Mary Shelley’s Frankenstein’.
[4] Gaby Wood. Living Dolls – A Magical History of the Quest for Mechanical Life (London: Faber and Faber Limited, 2002), xviii
[5] See Stanley McChrystal. Team of Teams – New Rules of Engagement for a Complex World (Great Britain: Penguin, 2015), Chapter 2.
[6] See Thomas Rid. Rise of the Machines – The Lost History of Cybernetics (Melbourne: Scribe, 2016), 2.
[7] Rid, Rise of the Machines, 47-49.
[8] Quoted in Rid, Rise of the Machines, 13.
[9] Max Tegmark. Life 3.0 (Great Britain: Penguin Random House UK, 2017), 77-79.
[10] Ellen Broad. Made by Humans – The AI Condition (Melbourne: Melbourne University Press, 2018), xii.
[11] Quote from former Joint Special Operations Command Lieutenant-General Joseph Votel, April 2014. Quoted in Andrew Cockburn. Kill Chain – The Rise of the High-Tech Assassins (New York: Picador, 2015), 244.
[12] PW Singer provides a useful definition of ‘trust’ in this context: ‘Trust is having a proper sense of what the other is capable of, as well as being correct in your expectations of what the other will do’. Singer, Wired for War, 134.
[1] Question raised by Brad Smith. Tools and Weapons – The Promise and The Peril of the Digital Age (Great Britain: Hodder & Stoughton, 2019), 193.
[14] Tegmark, Life 3.0, 316-335.
[15] Rid, Rise of the Machines, 5.
REFERENCES
Broad, Ellen. Made by Humans – The AI Condition. Melbourne: Melbourne University Press, 2018.
Burdon, Suzanne. ‘Mary Shelley’s Frankenstein and the birth of modern science’, Ockham’s Razor with Tegan Taylor on ABC Radio National, 14 January 2015, < https://www.abc.net.au/radionational/programs/ockhamsrazor/mary-shelleys-frankenstein-and-the-birth-of-modern-science/6014790> (accessed 03 April 2021).
Cockburn, Andrew. Kill Chain – The Rise of the High-Tech Assassins. New York: Picador, 2015.
Davis, Malcolm. ‘Loyal Wingman leads the way to the RAAF of 2121’. ASPI The Strategist, 05 March 2021 https://www.aspistrategist.org.au/loyal-wingman-leads-the-way-to-the-raaf-of-2121/ (accessed 03 April 2021)
Jordan, John M. ‘The Czech Play that Gave Us the word “Robot”’. The MIT Press Reader, 29 July 2019 https://thereader.mitpress.mit.edu/origin-word-robot-rur/ (accessed 02 April 2021).
Knight, Will. ‘The Foundations of AI Are Riddled with Errors’ Wired Magazine online, 31 March 2021 https://www.wired.com/story/foundations-ai-riddled-errors/ (accessed 02 April 2021).
Kurzweil, Ray. The Age of Spiritual Machines. New York: Penguin, 1999.
McChrystal, Stanley. Team of Teams – New Rules of Engagement for a Complex World. Great Britain: Penguin, 2015.
National Public Radio. ‘How the word “scientist” came to be’. Talk of the Nation, NPR, 21 May 2010 < https://www.npr.org/templates/story/story.php?storyId=127037417> (accessed 04 April 2021).
Rid, Thomas. Rise of the Machines – The Lost History of Cybernetics. Melbourne: Scribe, 2016.
Singer, P.W. Wired for War – The Robotics Revolution and Conflict in the 21st Century. New York: Penguin, 2010.
Smith, Brad. Tools and Weapons – The Promise and The Peril of the Digital Age. Great Britain: Hodder & Stoughton, 2019.
Tegmark, Max. Life 3.0. Great Britain: Penguin Random House UK, 2017.
Wood, Gaby. Living Dolls – A Magical History of the Quest for Mechanical Life. London: Faber and Faber Limited, 2002.
Wroe, David. ‘Killer robots used by unethical enemies will threaten Australia, army chief warns’, Sydney Morning Herald, 28 January 2018 <https://www.smh.com.au/politics/federal/killer-robots-used-by-unethical-enemies-will-threaten-australia-army-chief-warns-20180128-h0pchl.html> (accessed 03 April 2021).