Computers, humans, and automation: A dangerous mix when corners are cut
Steve Naidamast
Frank Drake, in the development of his famous equation in the early 1960s for the search for extra-terrestrial life, included what is known as the “L” variable. This variable stipulates, in addition to the viability of a civilization’s life-span, the concept that if an earth-like planet is found with no life upon it, this lack of life could be the result of the misused development of nuclear technology (i.e., the atomic bomb).
As a senior software engineer professional, I must expand this variable to represent not just nuclear technology but technology in general. Drake and his initial research team, though quite cognizant of the dangers of nuclear technology, were too singular in focusing on one technology, a lapse for which he can be forgiven since studies on technology’s effects on the daily lives of people were still in their infancy.
Today, most people rarely give a thought to the current hi-tech computers and gadgets they use every day, yet all of these technologies have a Dark Side since any technology can be used for good or evil… or both. Where and how one applies any technology also determines its long term success, to include whether its benefits will outweigh its drawbacks and whether its ill effects can be absorbed by society and the surrounding natural environments. A classic example of this was the invention of the cotton-gin, which had a ripple effect throughout American Southern society prior to the Civil War; slaves may have been able to produce more cleaned cotton but this only encouraged further support of the Southern slave culture. The economic consequences furthered the growing conflict between the Southern aristocratic/slave-labor economy and the Northern capitalistic/paid-labor one. This conflict would become an underlying motivation toward the “War Between The States.”
Thus, technologies affecting major institutions and people’s daily lives will not only have profound effects in a society but such effects, unfortunately, often seek the lowest common human denominator. This will always be true as a result of human penchants for individual power.
This power dynamic is especially dangerous when technology speeds up a faulty and dangerous process, as expressed by this axiom:
If a technology is applied to an erroneous process and that process is not corrected, the results will remain erroneous but be produced at a far faster pace. Humans are, by their very nature, highly flawed in their personal perceptions of their realities. Applying technology to such warped perceptions can and does cause terrifying results.
A classic example of this axiomatic collision of human perception and faulty automation was found at a hospital where I worked in their Information Technology organization. It involved a medical analysis and tracking system to authorize and reference blood–sample results. In this instance, hospitals always took a primary and a final sample of blood from a patient to ensure consistency in blood chemistry prior to a procedure being performed. If there was a difference, further testing would be required to verify if the procedure could go ahead as planned, be halted completely, or be modified toward another procedure.
When doing this analysis manually, mistakes were made but they were easy for hospital personnel to detect since the paperwork could be quickly analyzed for validity. Still, a mistake could filter through the process for a variety of reasons if there were no further checks on the validity of such tests other than the assigned nurse or doctor in charge of the patient.
When a new, automated system was implemented, it had a fatal flaw: no validation checks were made between the two blood samples since the system merely updated the final information with the primary information every time data was entered, whether it was for a primary or a final blood sample. Medical personnel simply could not see the inner workings of this system, which meant that they had no idea that the system was producing erroneous results.
For six months patients with serious conditions were being admitted to scheduled procedures without medical personnel having any knowledge of blood test consistency until my colleague, who had been performing analysis on the system, uncovered this flaw. The system was quite complex so it was not something anyone would have found unless they suspected something was amiss.
This kind of error in technology goes on all the time. People unknowingly continue to use data produced by flawed processes. For example, how many professional truckers have crashed their trucks as a result of GPS systems sending them down incorrect routes? It happens far too often.
Why is this allowed to happen if the technical profession is cognizant of such matters? The answer lies in the deeper sociological traits of society itself. Technological innovation is applied both to the individual as well as the society an individual is a member of. However, in this case we have to consider the entirety of humanity. And humanity, being the flawed entity it is, has never learned the critical lessons from its own history. One such criticality was what Roman Cynics were trying to teach people in their day (or the message of Christians who followed Christ): have compassion for and understanding of your neighbor. The avoidance of such a message has subsequently continued to encourage the most primeval of all human characteristics: its penchant for abject and exploitative violence.
The long-term result has been a seriously flawed sociological process toward modernity that has brought humanity to the brink of immediate extinction more than once. If you think that the 1962 Cuban Missile Crisis was the worst case of this in our history you would be sadly mistaken. Few people have read the documentation in the 1980s of a U.S. Air Force SAC (Strategic Air Command) bomber squadron that was headed to Moscow for a nuclear bombing run and passed its two fail-safe points because of system software “glitches” that were unnoticed until a lowly airmen caught them 20 seconds prior to the bombers passing their third and final fail-safe point. Had he not found this situation in time everything we know would have simply disappeared in the catastrophic nuclear response.
There is documented evidence that such events have occurred roughly 300 times in US military history since the continued development of nuclear technology during the Cold War and the development of the similarly complex technologies of the software programs used to monitor nuclear-based activities and endeavors.
There is a very simple reason for such failures: though the Information Technology field has been quite cognizant of the proper processes and procedures required to create high-quality software for many years, technical managers have consistently refused to adhere to such standards. And with few exceptions, this lack of adherence has been a consistent issue across all forms of software development, even when the underlying requirements entail life and death.
This trait of the profession has then consistently produced a pattern of software project failure across all project types that stands at 70%, which is roughly the same rate of failure that was observed over 35 years ago when the tracking of such statistics was first begun.
This is not a technical failure by any means but a human one in which the pursuit of profits, selfish agendas, and self-aggrandizement are found to be far more important than the production of quality. And this cuts across all professions, though unlike software development, many similar technical endeavors have seen a maturation in standards and processes that software development has quite intentionally refused to follow.
Today, software literally runs the large majority of transactions in daily life, but the expectation that it is reliable and fool-proof is not mirrored in reality. Quality and reliability cost time and money, and there are simply too many technical managers who are willing to side-step the critical foundations of quality production, at the potential expense of human life.
The result is that modern life today relies on a foundation that is akin to a very wobbly house of cards.
Steve Naidamast is a senior software engineer by profession and a military historian by avocation.
The Contrary Perspective 
The late Carl Sagan, a voice for Reason that is sorely missed in a world of increasingly irrational individuals–and whole societies!–was the most prominent scientist to argue that the odds are astronomically slim that planet Earth is THE sole harborer of life, and more specifically intelligent life, in the unfathomably vast Universe. An intelligent species (why the devil DO we use a plural for a single zoological entity? Hmmm…) does NOT have to condemn itself, and its fellow life forms sharing its environment, to extinction when it acquires technology. Nor does it have to construct a societal organizational model based on the pyramid of the very few privileged at the top exploiting the vast majority constituting the base. The faults lie not in the stars, but within ourselves. To be perfectly blunt, I see Humanity having a very, very dim future. But I claim title to Ultimate Optimist in my belief that elsewhere in the far reaches of space there are highly intelligent, enlightened beings who have found a better way of existence.
I agree, Greg. Survival does not require dominance. Surely there are other beings in the galaxy that have found other models to live by. And surely life is both richer and stranger than we can imagine it. Most of the Sci-Fi movies I’ve seen replicate humanity (humanoid species; driven by war and heavily armed spaceships; and so on). Very few imagine other possibilities. “Starman” is pretty good with Jeff Bridges. But to me the best movie to capture the “otherness” of ETs is the original “Solaris” from 1972.
When a man makes a mistake, he makes a mistake. When a machine makes a mistake … makes a mistake … makes a mistake … makes a …
LIFO = Last In, First Out
FIFO = First In, First Out
GIGO = Garbage In, Garbage Out
Whatever can go wrong, will go wrong. (Murphy’s Law)
START: Go To START. (World’s simplest computer virus — or U.S. Military Counter-[Whatever] Doctrine)
The problem is many people pass out – get qualification – without their fundamentals being clear. I will cite a very simple example.
I had prepared a spreadsheet with some very complex formulas and lookup tables to calculate cost of certain trade transactions. BTW I am an engineer, not an accountant. My so called commerce and accounting graduate accountant came up with some calculation he had done on a calculator and claimed that my formula was wrong. Now the question was what was wrong ? Was the calculator defective or was my formula wrong. I suggested a very simple solution to this. Do the calculations long hand with paper and pencil. There were a number of 6 and 7 digit values involved. Surprise surprise ! He did not know how to do long hand calculations !.
Oh boy, that sure doesn’t surprise me in the least! Do you think the 19-year-old cashier at your local convenience store could make proper change for you if the computer-run cash register broke down?? Hah! No way! Doing math in one’s head is a skill that will die with my generation (I’m, uh, let’s just say “over 60”), with exception of the occasional “idiot savant” perhaps. Rumor has it many schools are dropping teaching “handwriting,” too. “O brave new world, with such people in’t”–Shakespeare, ‘The Tempest’
I was fortunate to take math before calculators became common. So I learned to do long division, learned to calculate square roots and percentages, learned what “interest” meant, before reliance on calculators obscured the meaning. Many students nowadays are just taught to get “the answer” by hitting the right combination of keys on a calculator. Not good for numeracy.
And if you don’t learn to write, Greg, there goes a vital piece of literacy as well.
Innumerate and illiterate = easily led and easily fooled
Let us also not forget the two iron laws of bureaucracy: namely, Parkinson’s Law and The Peter Principle. The first states that “Work will expand to fill the time allotted for its completion.” The second says that “In any hierarchy, people tend to rise to their level of incompetence.” In practice, this means that any “Long War” conceived and executed by the hierarchical U.S. Military Government will occupy as long a time as human bureaucratic incompetence can possibly assure. Thus, to George Orwell’s famous Three Slogans of the Oligarchical Collective: namely, “Ignorance is Strength, “Freedom is Slavery,” and “War is Peace,” we must add a fourth: “Defeat is Victory.” Nothing can possibly work when the entire design of government, society, and the economy exists to see that it doesn’t.
So I guess diehard Dems must be all puffed-up with pride to have achieved parity with GOP in incompetent war-making! Please allow me a somewhat-related aside: I approve the awarding of the Nobel Peace Prize to Ms. Malala, the young Pakistani activist for rights of young women. But oh how I wish the Committee had decided to REPOSSESS the Prize they so foolishly awarded Obama a few years back! (Yeah, I know, that would be unprecedented…but a fella can fantasize occasionally, can’t he?)
Yes. A repossession of Bomber Obama’s Nobel Peace Prize would do much to redeem the prestige that award formerly enjoyed. Still, the one Henry Kissinger received probably sullied that prize for all time. As the folk-singer/comedian Tom Lehrer explained when he quit doing stand-up for a return to teaching university mathematics: “All comedy died the day Henry Kissinger won the Nobel Peace Prize.” So, I guess we can say that if Kissinger’s award killed comedy, then farce bit the dust the day Obama won his. One can’t even satirize this stuff any more, it has become its own parody.
Yes. Obama “won” the Nobel Peace Prize on the merit (ha!) of not being Bush, and perhaps also of speaking in complete sentences and occasionally mentioning the word “peace.” And with Nobel in hand, Obama has waged war for the last six years. Meanwhile, the Republicans still accuse him of being weak on defense, a dove in a world of hawks.
A slight correction to the ornithological metaphor, Professor. As the late Gore Vidal would say, President Obama ranks as neither a hawk nor a dove, but a capon. Worst of all, he castrated himself because the Republicans threatened to call him a “wimp” if he didn’t — and so he certainly showed them, didn’t he?
Pingback: The Sorrow of Magic Lost | The Contrary Perspective
Pingback: Comfort, risk and coal-fired furnances | Phil Ebersole's Blog