Different packaging, same product
In the world of the philosophy of consciousness, most formal theories rely on what's referred to as an “extra factor.” This extra factor is used to describe why humans have free will, while everything around us seems to be determined. There needs to be something to separate us from the rocks that we don't assign agency to.
One interesting modern interpretation of this problem is the idea from Daniel Dennett who theorized that if consciousness is defined as “there has be to something that it's like to be you,” then an explanation of the extra factor could be that we narrate ourselves into existence.
From an evolutionary perspective, this idea is genius because it enables long term thinking. If I didn't think I'm going to exist in the next second because I have no sense of self, I won't make plans that don't satisfy an immediate urge but accomplished a more abstract long term goal.
AI is such an exciting field because we get to dive into how intelligence works. If something works in a machine, we will likely be able to use it to augment our own capabilities. Intelligence is intelligence, regardless of the source so I have to suspect that if our rules of logic and math are sound, ai will inevitably end up similar to humans as it's human tasks they are being trained on and given human tasks to accomplish.
Source: Researchers Discover AI Language Models Are Mirroring the Human Brain’s Understanding of Speech - The Debrief https://share.google/SAdRN7TxSsG2Rk3tR