`I' vs `the'

In the Sept 3-9th edition of New Scientist, James Pennebaker discusses the individual variations in frequency with which we use pronouns and other small words, and he considers what this metric might say about our personalities and relationships. The paper version (p. 45) has a graph entitled "The real word count" with the caption "The 20 most frequently used words in the English language, across both spoken and written texts." The graph shows that I is the most common word, followed closely by the.

This prompted the following query by Mike Scott to the Corpora mailing list:

"I wrote to the author, James Pennemaker of the U of Texas, about this, expressing my surprise at the pronoun I having greater frequency than THE, as even in the spoken-only section of the BNC (10m words) we find I occurring only just over half as often as THE. His data contains a mix of spoken and written with a large amount of blog data. He reports that with all his studies in the USA and Mexico, "people always use more I more than THE. It's never close." Can anyone help here, clearing up the position? Someone with access to a really top quality corpus, more up to date and representative than the BNC? "

An interesting discussion followed, which you can see in the list archives.

Adam Kilgarriff, remarked that it all depends on text type, which is certainly true and belies the universal nature of Pennebaker's claim and that of the article graph. He goes on to say that "Asking for a more representative corpus won't help because we all have different ideas about what it should be representative of."

Thereafter commenters presented counts from a variety of corpora, which led Marc Brysbaert to observe:

Maybe we can turn the question around and use the “the/I” ratio as an index of how socially vs. description oriented a corpus is? Here is a summary of the data I have at hand.

Source	the	I	ratio
COCA (academic)	5549547	204916	0.04
COCA (newspapers)	4648992	506030	0.11
Google (books)	22914473646	2744649681	0.12
COCA (magazines)	4878925	648344	0.13
American blogs	4200000	1300000	0.31
COCA (fiction)	4534433	1576303	0.35
COCA (television programs)	4190341	1623705	0.39
Shakespearean plays	182400	239200	1.31
SUBTLEX (film subtitles)	1501908	2038529	1.36

Ken Litkowski added the following:

This discussion has focused on only one aspect of James Pennebaker's work, the 'I' frequency, and perhaps not as much on his many contributions to content analysis, which may have even more relevance to discussions on this list.

Kyle Dent of Xerox has recently performed an analysis of 2400 tweets, with the aim of classifying them into "Questions" and "Not Questions". He developed an elaborate NLP system to deal with these tweets. He kindly provided me with these data, so that I could examine them with my content analysis program to see how well they could be analyzed without all the NLP superstructure. I happened to run a first analysis at the time of this thread. It simply compares the two sets as a whole.

The corpus size is 31,000 words (hardly the stature of BNC, COCA, or OEC). But, curiously, both "i" and "the" hold the top two frequency positions in both:

Set                "the"    "I"
Questions            400    327
Not Questions        437    575

Wow! Could this be a classification signature? Although this is not likely, various other statistics in various combinations generated in the program may very well be. So, here we have a micro-genre analysis that confirms the other comments on this thread, much like the Known Similarity Corpora of Adam Kilgarriff (15 years ago!).

Sentiment analysis is an emerging field, but is currently dominated by heavy NLP techniques. I would suggest that techniques from content analysis might provide a nice complement.


As I said, you can read the whole discussion in the list archives.