10 Horrifying Technologies That Should Never Be Allowed To Exist
Gregory Alan Bolcer stashed this in Dangerous Decisions
2. Conscious Machines
Futurist Louie Helm:
One of the best things about computers is that you can make them sum a million columns in a spreadsheet without them getting resentful or bored. Since we plan to use artificial intelligence in place of human intellectual labor, I think it would be immoral to purposely program it to be conscious. Trapping a conscious being inside a machine and forcing it to do work for you is isomorphic to slavery. Additionally, consciousness is probably really fragile. In humans, a few miscoded genes can cause Down Syndrome, schizophrenia, autism, or epilepsy. So how terrible would it feel to be a slightly misprogrammed form of consciousness? For instance, several well-funded AI developers want to recreate human intelligence in machines by simulating the biological structure of human brains. I sort of hope and expect that these near-term attempts at cortical simulations will be too coarse to really work. But to the extent that they do work, the first "success" will likely create cripplingly unpleasant or otherwise deranged states of subjective experience. So as a programmer, I'm generally against self-aware artificial intelligence. Not because it wouldn't be cool. But because I'm just morally opposed to slavery, torture, and unnecessary code.
There is the potential for a lot of things to go wrong.