Two physicists may have found a hidden structure in the English language1. One can guess the syntactic or grammatical character of a word without knowing its meaning, they say, because different types of word are scattered differently through written texts, rather as certain wild flowers grow in certain parts of a meadow.

The new analysis might help to unravel how language arose, and could aid in the deciphering of coded messages, say the duo: Marcelo Montemurro of the National University of Cordoba, Argentina, and Damian Zanette of the Centre for Atomic Science in Bariloche, also in Argentina.

In the 1940s, US sociologist George Kingsley Zipf observed that words in English texts occur in an apparently universal pattern. Zipf's patient students counted all the words in Shakespeare's Hamlet and ranked them in order of decreasing frequency. The most common word, with 1,087 appearances, was 'the', followed by 'and'.

Zipf found that the distribution obeyed a mathematical relationship called a power law. In other words, a plot of the logarithm of a word's rank against the logarithm of its frequency is a straight line - for all the words in the play.

This curious result seemed to hold true for other texts. But it doesn't say much about what each word signifies. Word number 45, for example, is 'or'; word 47 is 'Hamlet'. All Shakespeare's plays probably contain 'or', but only one of them contains 'Hamlet'. So some words are highly context-specific, others are more general. 'Hamlet' looks like a relatively common word if you analyse Hamlet, but not Twelfth Night.

Montemurro and Zanette wanted to link this kind of statistical analysis to the meanings of words. They assigned to each word a 'Shannon entropy', named after another 1940s mathematician, Claude Shannon, who worked on information flow in communications systems.

The Shannon entropy is a measure of how context-specific a word is. A word confined to only one part of a text has low entropy, even if it is very common in that section. The entropy of a word that is evenly distributed throughout the text is high.

The researchers calculated the frequency and Shannon entropy of all 885,535 words in Shakespeare's 36 plays (23,150 different words). Unsurprisingly, the entropy of a word increased as its frequency increased - common words tended to be evenly distributed.

But there were big fluctuations in this relationship - some words are common but have very low entropy, perhaps appearing in only a few plays, or just one.

Then, reversing the 'monkeys with typewriters' scenario, Montemurro and Zanette produced 36 nonsensical 'Shakespearean' plays by scrambling all the words at random. This absurd oeuvre had a similar frequency and entropy relationship, but without the fluctuations.

The differences between a word's entropy in the real plays and in the randomized plays depends on that word's grammatical and even its narrative nature. For example, names, status nouns - such as 'duke' and 'king', which are common in Shakespeare - and verbs cluster into different regions of a plot of entropy difference versus frequency.

In other words, the distribution of a word through the text can indicate whether it is a noun or a verb, for example.

The researchers observed the same effect in the works of Charles Dickens and Robert Louis Stevenson. It remains to be seen how the findings translate to other languages.