You can devise a task it couldn’t have seen in the training data, I mean.

You don’t even have access to the “thinking” side of the LLM.

Obviously, that goes for the natural intelligences too, so it’s not really a fair thing to require.