As AI-generated text continues to evolve, distinguishing it from human-authored content has become increasingly difficult. This study examined whether non-expert readers could reliably differentiate between AI-generated poems and those written by well-known human poets. We conducted two experiments with non-expert poetry readers and found that participants performed below chance levels in identifying AI-generated poems (46.6% accuracy, χ2(1, N = 16,340) = 75.13, p < 0.0001). Notably, participants were more likely to judge AI-generated poems as human-authored than actual human-authored poems (χ2(2, N = 16,340) = 247.04, p < 0.0001). We found that AI-generated poems were rated more favorably in qualities such as rhythm and beauty, and that this contributed to their mistaken identification as human-authored. Our findings suggest that participants employed shared yet flawed heuristics to differentiate AI from human poetry: the simplicity of AI-generated poems may be easier for non-experts to understand, leading them to prefer AI-generated poetry and misinterpret the complexity of human poems as incoherence generated by AI.
LLMs can’t use some literary devices and techniques, and I will illustrate with the following example of a poetry I wrote:
It seems like a “normal” (although mysterious) poetry until you isolate each initial letter from every word, finding out a hidden phrase:
It doesn’t stop here: if you isolate each initial letter again, you get a hidden word, “Shadows”.
Currently, no single LLM is capable of that. They can try to make up poetry with acrostics (the aforementioned technique) but they aren’t good at that. Consequently, they can’t write multilayered acrostics (an acrostic inside another acrostic). It’s not easy for a human to do that (especially if the said human isn’t a native English speaker), but it can be done by humans with enough time, patience and resources (a dictionary big enough to find fitting words).
They’re excellent for stream-of-consciousness and surrealist poetry, tho. They hallucinate, and hallucinated imagination is required in order to write such genres.