A model predicts. That is its job. It does not have to mean anything in order to be good at the job. This is easy to forget when the predictions arrive in fluent sentences.
Training is a statistical intimacy with a corpus. The system becomes skilled at continuing the sort of thing that has already been said. Skill is not the same as understanding. Continuation is not the same as a world.
We are now in the habit of asking models questions we used to reserve for teachers, lovers, and priests. The answers can be useful. They can also be a high-quality way of not sitting with the question. A sentence that sounds like care is not yet care.
None of this requires contempt for the tools. The tools are extraordinary. The error is to treat extraordinary prediction as a new metaphysics — as if the fact that a system can imitate mind meant that mind had been explained, or that a life could be outsourced to the imitation.
What a model cannot mean is the cost of a choice. It can describe costs. It can average them. It cannot be the one who will have to live with the remainder. That remainder is where ethics actually happens.
Use the machines for what they do. The question remains: what is reality made of, and what does that mean for how we live? A model can assemble language around that question without bearing the consequences of an answer.