Super intelligent machines are a major step forward in industry. They are helpful, efficient and effective in enhancing the profits and the lives of all. That is, until they develop their own motivations which know no limits… Can they be stopped in time?
The four graduates from Trinity College sat outside the Anchor pub next to the River Cam in Cambridge on a beautful June day
in 2030. Several punts, propelled using long poles and loaded with undergraduates relaxing after exams, could be seen working their way slowly up and down the river, negotiating overhanging willows with varying degrees of success. The river levels had become more variable over the last few years due to climate change with wetter winters and drier summers but today everything seemed perfect. The four friends, Simon, Jane, Stephen and John had all graduated a year earlier and this was the first reunion, each having secured IT jobs in industry where their skills and academic achievements had been highly sought after.
Simon, always the extrovert, now working as an IT manager in Financial Services in a commercial bank in London, was quick to open the conversation. “Have you noticed that nearly all data acquisition and data processing work is done by machines these days with very little human intervention?” he commented.
“Probably less so in agriculture,” said Jane. “We still pride ourselves on muddy boots occassionally,” and continued, “but I agree once the crops are ready, the harvesting and delivery to the supermarkets is all highly automated so it arrives fresh next day!” Jane had involved herself in the college social scene during her three years as a Cambridge undergraduate and knew most of the resident academics and lecturers on first name terms. She had a soft spot for Simon since their brief affair three years earlier during their second year. He never formed long term relationships with women, preferring to move on and avoid any personal responsibility.
“John, what about telecoms?” Simon asked.
John was a thoughtful character, weighing up each word with care. “Of course, telecoms has almost always had a high degree of automation otherwise there would now be millions of telephone operators!” he commented.
“Yes, but what about today’s automation of data acquisition in telecoms?” persisted Simon.
“Its pretty much 100 % automated I would say,” responded John. “Monitoring, ordering, inventory control, you name it! Of course, people oversee the data analysis and set the direction,” he added.
Then Simon turned to Stephen. “How about power generation and distribution? Is all the data acquired and handled automatically?”
Stephen had always associated himself with green energy as a student and voted Green Party as soon as he was of age.
“Smartgrid generates masses of data in real time,” was his immediate response. “So almost everything at the ground level related to local and centralised power distribution is automated.”
He continued, “Today, in 2030, most households have some solar heating. New houses often have it built into the roof tiles and all excess power goes back into the local grid automatically.”
“What about the banks?” asked Jane, thinking it was time for Simon to come clean.
Simon thought for a few seconds and replied, “Data collection and data processing are already nearly 100 % automated but the new thing is artificial intelligence AI. This offers the ability to process and analyse vast amounts of finance data (often called Big Data) in more sophisticated ways to meet almost any requirement.”
Simon had always been fast to take on board new ideas, often ignoring clear risks until they were pointed out by his seniors.
The four friends agreed that almost 100 % of data acquisition and data processing across the industries they worked in had become automated. This had been essential to handle the growing volumes of data and also reduce response times for critical activities. People were no longer required or even capable of carrying out these activities manually on time to the required quality! But AI, the next obvious step, was still to make a real impact.
“Another thing,” said Simon, “has everyone noticed the numbers of robots around these days to carry out simple tasks?” From 2020, intelligent mobile machines (robots) had gradually been introduced into the marketplace in specific roles. But recently, simple generic robots had been available at relatively low cost and were able to learn most manual tasks quickly, often by simply observing a human for a day or so. Developments in natural language had even made meaningful man to machine conversations practical. This lead to increased use of sophisticated robots in the workplace and the home as prices plumeted through mass production.
Jane responded first. “In agriculture we have started to use robots as part of the farm labour force, working 24 hours a day and cheaper to operate than even overseas labour.”
“How do you find talking to them?” asked John.
“The farmers seem to like them,” replied Jane. “Keeps the language simple and it’s not like you can socialise with a tin man!”
“How do they perform compared with traditional labour?” contined John.
“Well, after a slow start, they seem to improve quite quickly as they learn the tricks,” replied Jane. “And they are physically much stronger than a normal farm labourer.”
“Do they go wrong much?” asked Stephen, thinking about his high reliability targets in the power industry.
“Only the occassional accident and we have one day replacement written into the supplier contract,” said Jane. “They just transfer the acquired learning to the replacement robot over WiFi.”
“And the Asimov three laws of robotics?” asked Simon, “about protecting humans and themselves?”
“Yes, they are in the software along with recently developed laws specific to agriculture,” replied Jane.
“So do they fuck the sheep as well?” asked John lightheartedly.
“The robots are missing the correct apendidges,” replied Jane, not easily shocked and used to John’s ‘off the cuff ’ comments.
“To be honest, we have all been surprised at how quickly the robots have been accepted into farm work,” she concluded. “Only thing they dont do is drink cider down the pub after work.”
This discussion made John think again about the possible use of robots in telecoms. How many jobs required predictable physical work suitable for robots in a managed environment such as a Data Centre? How about routine fieldwork such as connecting a customer from a street cabinet or repairing a line fault? Getting robots to the site should not be an issues using self-drive cars, now ubiquitous in 2030. The relative complexity of repairs had been the main issue to introducing field robots into telecommunications and of course, the unions where wary of potential job losses. But in principle, it could be done and needed be assessed.
Introducing robots with all the human skill requirements into telecoms would be a challenge! But if the use of skilled robots halved operational costs and improved quality, who would argue? Of course, direct customer facing roles and stakeholder interactions would remain with humans. John decided he would investigate the concept further when he returned to the office on Monday.
“Stephen, what about using robots in power generation and distribution?” continued Simon.
“Of course, much of the operations are already automated but are you talking about mobile autonomous units?” replied Stephen.
“Correct,” added Simon.
“Well, drones have been part of power plant inspections for some years. In many cases, they can eliminate the need to send workers to examine diff icult-to-access areas, such as boilers, stacks, and towers. Unmanned vehicles can enter piping, go underwater, survey power transmission lines from varying angles and analyse the results immediately. They also provide better coverage and more effective maintenance working 24 hours a day!” said Stephen.
“What about nuclear plants?” asked Simon.
“Today, robotics in power plants often replace workers, enabling operators to remotely access hazardous and inaccessible zones safely to minimize the exposure of workers to radiation,” replied Stephen. “And if a robot gets over exposed to radiation, we replace him and train up a new one.”
“Sounds effective,” said Simon, impressed.
“And in a desert, solar energy generation can fall by as much as 40 % when dust storms impact the panels so we use robots to carry out routine night time cleaning,” added Stephen.
“Simon, you mentioned artificial intelligence earlier?” said Jane.
“Yep, that is the new thing in town for my bank,” said Simon.
“It has come on a lot in the last year.”
“I thought AI was just about look up tables with responses to pre-defined questions?” said John. “A lot of work for not much benefit!”
“That used to be the case – a lot of hype then nothing for the last 30 years,” agreed Simon. “But now the approach to AI has changed so that it is more about self learning algorithms processing masses of real time data as you can see with the robots.”
John had to agree that robots which learned new tasks simply through observation were impressive. How they worked, fuck knows! Something else for him to look at on Monday back in the Telecom office.
This short conversation among the four friends had exposed the degree of automation across the four industries. More surprisingly, it indicated the accelerated use of robots in both specialist areas and the more general replacement of human workers simply on cost grounds. Even AI appeared to be moving forward! Were there significant dangers to this revolution? Were humans becoming too dependent on robots? Simon was reminded of the 1909 story ‘The Machine Stops’ by E.M. Forster where the ubiquitous machine controls every aspect of people’s lives until it eventually fails and society collapses! But that is just SFI.
“How about a punt on the river?” suggested Simon.
They spent the next two hours on a punt on the River Cam mostly going around in circles and eventually making it underneath the pedestrian bridges down to the next staging post. The River Cam was the main river flowing through Cambridge. It flowed north and east into the Great Ouse to the south of Ely.
“Great day,” said Simon. “Let’s meet up next year.”
They all agreed it would be great to meet up again. These were exiting times!
Simon had always been comfortable in his skin; a talker and natural leader. From age 15, he was into video games big time and an ability to engage with him on a games console to 4am in the morning would usually establish the player as a member of his social group. He had attended public school in Guildford but missed out on Saturday sports, instead participating in junior music classes at The Royal College of Music (RCM) in London where he met his first girlfriend at age 15. Strong in maths and computing, Simon enrolled to read IT at Cambridge in 2026, forming close friendships with John, Jane and Stephen all at Trinity. He graduated in 2029 and joined a major commercial bank as an IT expert working in London and now rented a flat in Southgate.
He worked in the IT development area sharing an office in a data centre within the City Finance district. Simon’s boss, George, was in his early forties and a little bewildered by all the new technologies being introduced by the suppliers so Simon saw his chance for early promotion if he became recognized as an expert in some of the newer areas. The disciplines of data collection and data processing were already nearly 100 % automated as in most industries. But in the newer areas grouped under artificial intelligence, AI offered the ability to process and analyse vast amounts of finance data intelligently to achieve defined objectives without human intervention. This new capability was underpinned by machine learning, allowing the machines to improve their ability to address problems and goals through accumulated experience. As the AI system received an increasing amount of similar sets of data its ability to rationalize increased, allowing it to better ‘predict’ on a range of outcomes. Simon understood this from the academic perspective and looked forward to putting AI into practice in IT finance. Other areas of AI he thought he could lead on included Virtual Agents (to assist customers real time), trend analysis and especially investment decision making by the bank on behalf of its customers.
So Simon worked with his team at the Commercial Bank and its suppliers to generate strategic business cases for each of the AI related developments. For a wide range of assumptions about the real world, these business cases surprisingly turned out to be either good or very good financially! This encouraged Simon and the team to continue although the risk assessment part of the business case included some items which were not particularly well understood. How exactly did the machine learning work and how were the self generated control algorithms produced? How reliable were they? How robust? Nevertheless his manager, George, put out a few feelers to the major stakeholders to assess the mood for introducing more powerful AI into the bank and on receiving a good response decided to present a provisional business case to the investment board.
On the day of the presentation, Simon and George made their way to the 60th f loor of the London office, reflecting that ivory towers in banks really do exist. There were two slides per item assessing benefits, costs and risks for each proposed investment in advanced AI. This was suff icient given only an hour was scheduled and most of that would be discussion. The purpose of the investment board was to decide which items to progress for further detailed work prior to full approval to proceed. Detailed implementation plans would then be produced by the project manager.
They were greeted by the chairperson and invited to give their presentation. George gave a preamble but the presentation was left to Simon who understood the detail. George had tried hard to ‘get his head around’ the concepts but in reality only understood the bottom line. So Simon, a new recruit of one year’s service, found himself in the ‘hot seat’. Seeing the numbers, the eyes of the board lit up! It appeared that the new enhancements around machine learning would significantly improve decision making whilst reducing manpower costs! The risks appeared managable as far as could be understood. So that day, the investment board gave approval for further analysis to produce a detailed investment case capable of approval and scheduled a follow-up meeting in six weeks.
After Simon and George had left the meeting, the chairperson asked if any of the board members had concerns? Everyone was excited so concerns were a bit scarce. Unexpectedly, the legal expert spoke up. “I just wonder as these machines become more intelligent through fast learning how much are we stepping into the unknown?”
“Can you give us an example?” said the chairman.
But the legal guy could go no further; it was just a feeling he had.
“OK, I will raise it as an issue next time,” said the chairman and made a note.
The Human Resources person commented next. “Is the young age of the presenter, Simon, a possible risk with regards to hands-on experience?”
“Well, his boss, George, is experienced,” said the chairman, “and it was a joint presentation, wasn’t it?”
The comment was noted with the action for an independent review of the updated business cases when available as part of the standard assurance procedure.
When Simon and George appeared six weeks later, the investment board had grown to twice the size with casual members drawn in, such was the interest. George summarized the additional work done by the team and Simon presented a more detailed business case which already had provisional sign off by finance. The presentation by Simon focused on the case to introduce machine learning (apparently a software upgrade at minimal cost) resulting in better use of human resources and improved forecasting and decision making. He also presented a high level implementation plan including a trial in selected branches followed by a general rollout if successful.
“Any questions?” asked the chairman.
The assurance director said, “In terms of unexpected risk, do we have detailed understanding of how the learning algorithms work?”
“Not exactly,” said Simon. “The machines generate their own algorithms based on the continual learning process.”
“So isn’t that a serious risk?” replied the director.
“The plan is to set internal limits to the amount of change permited on the key parameters as a precaution,” said Simon, “which we will, of course, control and monitor on the field testbed in the next few months.”
“And these checks will become a permanent feature in operational use, I assume?” asked the assurance officer, feeling they were stepping a little too boldly into the unknown.
“Yes, of course,” said Simon, gesturing to the chief engineer who added his confirmation.
“That sounds fine to me,” concluded the chairman who was approaching retirement and keen to move this development forward as part of his legacy.
“And we expect the overall performance of the machines to continually improve as the learning algorithms kick in,” added Simon.
The assurance director squirmed again in his seat at the phrase ‘learning algorithm’ but remained silent this time, unable to take his argument further.
The lawyer spoke up again. “Is there a chance that these machines will become more intelligent than humans and get out of control?”
“Do you have an example?” asked the chairman again.
“Well, suppose the super intelligent computer stops working due to a data overload or decides to switch itself off in the worst case scenario?” added the lawer.
“Is that possible, Simon,” asked the chairman.