Have you read Ted Chiang's Silicon Valley Is Turning Into Its Own Worst Fear ?
It's beyond remarkable that this essay is more than six years old at this point, it's still astonishingly accurate and insightful.
Elon Musk spoke to the National Governors Association and told them that “AI is a fundamental risk to the existence of human civilization.”
...
... in its pursuit of a seemingly innocuous goal, an AI could bring about the extinction of humanity purely as an unintended side effect.
But as Chiang observes, this really isn't a discussion about runaway superintelligence, it's a discussion about modern capitalism.
Consider: Who pursues their goals with monomaniacal focus, oblivious to the possibility of negative consequences? Who adopts a scorched-earth approach to increasing market share? This hypothetical strawberry-picking AI does what every tech startup wishes it could do — grows at an exponential rate and destroys its competitors until it’s achieved an absolute monopoly. The idea of superintelligence is such a poorly defined notion that one could envision it taking almost any form with equal justification: a benevolent genie that solves all the world’s problems, or a mathematician that spends all its time proving theorems so abstract that humans can’t even understand them. But when Silicon Valley tries to imagine superintelligence, what it comes up with is no-holds-barred capitalism.
Chiang's essay is extremely important, even more important now than when he wrote it more than six years ago.
I hope more people will go back and rediscover this under-acknowledged gem, and read it all the way through, and re-read it some more, and go and tell everyone they know about it.
