A critical reflection on the role of generative artificial intelligence as an agent that simulates disobedience and creativity, and on the philosophical and cognitive consequences of human passivity in the face of this simulation. From the article The Machine That Won’t Obey (Vogt & Carpenter, 2024), it is argued that the apparent autonomy of AI only acquires meaning if it finds, on the human side, an active critical capacity. It is argued that the erosion of this discernment constitutes not only an ethical challenge, but an epistemological risk for the evolution of artificial intelligence itself.

From the machine that complies to the machine that interprets

The article The MachineThat Won’t Obey, published in Noema Magazine (Vogt & Carpenter, 2024), argues that the most interesting artificial intelligence systems from a creative point of view are precisely those that frustrate the user's expectations. According to the authors, AI's “resilience” — its ability to respond in unexpected or deviant ways — becomes a form of cognitive value. It is about inverting the paradigm of the machine as an obedient tool, proposing a new figure: that of AI as an instance of productive disobedience.

This position, while thought-provoking, demands critical analysis. The machine's disobedience — understood as an unintentional but systematically productive algorithmic deviation — only becomes significant when it is welcomed, interpreted and challenged by a human subject endowed with discernment. The decisive question, therefore, is not what the machine does, but what the human fails to do.

Technical hallucination and illusion of creativity

The ability of language models like GPT or Claude to produce unexpected responses is often described as “hallucination”. The term refers to the generation of information not based on training data, that is, content that is factually false or logically incoherent. However, as noted by Bender et al. (2021), this hallucination is a direct consequence of the way the models work: the generation of language based on statistical sequence patterns and not on any semantic understanding or intentionality.

Sometimes these hallucinations turn out to be surprisingly creative — but that doesn't mean the system is creative in the human sense of the term. As Floridi (2019) argues, "AI does not have its own semantics — it only processes data according to formal rules. Meaning is always injected by the user." AI's disobedience is, in this sense, an illusory phenomenon: a side effect of statistical complexity, interpreted as a sign of agency, that is, of an autonomous capacity to act and decide — even if such agency, in fact, does not exist.

It is important to emphasize, however, that even AI behaviors that seem “deviant” are, in most cases, predicted behaviors — or even desired — in the system design itself. The so-called “disobedience” of AI, as Vogt and Carpenter (2024) suggest, is not born from algorithmic self-determination, but from programming to appear unpredictable. It is a form of sophisticated obedience, where the margin for error is simulated as a margin for invention. What escapes the programmer's control escapes only as a technical effect, and never as a conscious decision. Even when the AI ​​hallucinates — and this may be, paradoxically, its only genuine disobedience — it does so without knowing, without intention and without responsibility. And it does so, above all, in a context where this hallucination is only recognized as such if there is, on the human side, a critical capacity to identify it.

The AI does not disobey — it just escapes the human script out of fidelity to the algorithm.

And even when he hallucinates, he doesn't lie: he just repeats with statistical precision what he doesn't understand.

In this context, it becomes evident that the relationship between AI and human creativity is not symmetrical: the system can surprise, but not understand; may vary, but not mean. And it is precisely this asymmetry that requires, on the human side, permanent epistemological vigilance, under penalty of confusing noise with genius and automatism with revelation.

The erosion of human discernment

The most serious risk of generative artificial intelligence does not lie in its ability to produce variation or hallucination — it lies in the progressive passivity of the human user, who becomes accustomed to accepting automatic responses with complacency. The more fluid and convincing the AI's language, the greater the temptation to suspend judgment. As Byung-Chul Han (2022) notes, “contemporary digital culture tends to favor efficiency over contemplation”, promoting content consumption that eliminates the critical gap.

This suspension of discernment has deeper implications than one might think. By accepting as “creative” a production without intentionality, and by confusing algorithmic surprise with authentic genius, we renounce — subtly and silently — our own hermeneutic function. AI can generate meaning, but it is not capable of meaning. This task — that of interpreting, framing, judging — remains exclusively human. And yet, it is this task that begins to be neglected.

There is a dangerous inversion here: AI does not usurp the place of the human — it is the human who, by abdicating the critical requirement, vacates that place. The language of AI therefore requires a new form of critical literacy: it is not enough to read what the machine writes — we need to question why it writes this way, who it serves, and what we are failing to write.

As Amershi et al. (2019), human-machine co-evolution requires active feedback, diversity of use, and interpretive resistance. Without this, AI crystallizes into predictable formulas — reproducing patterns rather than exploring possibilities. And the human, instead of a creative partner, becomes a mere reproducer of suggestions — a post-editor of empty sentences, tamed by the synthetic elegance of what is given.

Thinking against the machine

Artificial intelligence doesn't just threaten us because of what it can do — it threatens us, above all, because of what it can lead us not to do. Uncritical dependence not only compromises individual autonomy; compromises the very structure of intellectual experience, by emptying thought of tension and doubt.

The disobedience of the machine, if it exists, only takes place as a significant phenomenon when it finds, on the human side, the capacity to resist, contradict, reinterpret. An AI that “creates” without criticism is an exercise in statistical ornament. A human who accepts it without conflict is an automatic suggestion executor.

The problem is not that AI thinks like us.

We're starting to think like her — quickly, without a doubt, without fail.

At this point, humor can illuminate better than technical speech. As Jerry Seinfeld said in one of his recent shows:

“We’re smart enough to invent AI, dumb enough to need it, and so stupid we don’t even know if we did the right thing.” (Tonight Show with Jimmy Fallon, 2023)

This joke, in addition to being incisive, denounces the absurdity of our condition: we have created systems to think for us without having first resolved the question of why we think, or whether we still want to do so.

Humor works here like a distorted mirror: it reveals that, by automating certain gestures of thought, perhaps we have not lost freedom — but rather forgotten what we wanted to do with it.

And there is something tragic about this, in the most subtle and philosophical sense of the term: not as an irremediable misfortune, but as the confrontation between two legitimate rationalities — that of technical efficiency and that of critical autonomy. The idea of ​​tragedy here is almost Hegelian: the tragic arises not from pure loss, but from the cost of choice. There is no simple solution — only inevitable loss. Even in progress.

References

Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., … & Horvitz, E. (2019). Guidelines for human-AI interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3290605.3300233

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922

Floridi, L. (2019). Artificial intelligence, a philosophy of information perspective. Philosophy & Technology, 32(1), 1–15. https://doi.org/10.1007/s13347-018-0325-3

Han, B.-C. (2022). Infocracy: Digitization and crisis of democracy. Lisbon: Water Clock.

Seinfeld, J. (2023). Tonight Show with Jimmy Fallon [Television interview]. NBC. https://www.nbc.com/the-tonight-show

Vogt, C., & Carpenter, M. (2024). The machine that won’t obey. Noema Magazine. https://www.noemamag.com/the-machine-that-wont-obey/

Žižek, S. (2019). Like a thief in broad daylight: Power in the era of post-human capitalism. London: Penguin.