What do we want? Reflections on AI and morality
Essentially, these AIs solve all kinds of "how" problems for us. We now know how to play a good game of chess or go, how to reply to any text prompt, and how to create an image from just a text description. This leaves us with the "what" problem. What do we want? What do we value? This is a hard question. Can an AI answer it for us? No, I don't think so. At least not any of the AIs as we have them today. Remember, the fundamental function of all the AIs we've discussed is to take in data, and give us what we said we wanted. So it seems that there's some kind of bootstrapping step missing, in that we have to give the AI "what we want" as an input, but expect "what we REALLY want" as an output.