Paper published: Rank Dynamics of Word Usage at Multiple Scales

The recent dramatic increase in online data availability has allowed researchers to explore human culture with unprecedented detail, such as the growth and diversification of language. In particular, it provides statistical tools to explore whether word use is similar across languages, and if so, whether these generic features appear at different scales of language structure. Here we use the Google Books N-grams dataset to analyze the temporal evolution of word usage in several languages. We apply measures proposed recently to study rank dynamics, such as the diversity of N-grams in a given rank, the probability that an N-gram changes rank between successive time intervals, the rank entropy, and the rank complexity. Using different methods, results show that there are generic properties for different languages at different scales, such as a core of words necessary to minimally understand a language. We also propose a null model to explore the relevance of linguistic structure across multiple scales, concluding that N-gram statistics cannot be reduced to word statistics. We expect our results to be useful in improving text prediction algorithms, as well as in shedding light on the large-scale features of language use, beyond linguistic and cultural differences across human populations.

Morales, J. A., Colman, E., Sánchez, S., Sánchez-Puig, F., Pineda, C., Iñiguez, G., Cocho, G., Flores, J., and Gershenson, C. (2018). Rank dynamics of word usage at multiple scales. Frontiers in Physics, 6:45.

Part of the Research Topic Culturomics: Interdisciplinary Path Towards Quantitative Study of Human Culture


Comments

Post a Comment

Popular posts from this blog

The Past, Present and Future of Cybernetics and Systems Research

Summary of "The Past, Present and Future of Cybernetics and Systems Research" Symposium M at  European Meetings on Cybernetics and Systems Research (EMCSR), Vienna, Austria, April 12, 2012. Organizer: Carlos Gershenson This guided reflection on the challenges and opportunities of cybernetics and systems research (CSR) included initial interventions by panelists Peter Erdi, Helena Knyazeva, Stefan Thurner, Peter Csermely, and Alexander Lazlo. Afterwards, the floor was opened to interventions from the general public and further interventions by panelists. Commentaries were made from a broad variety of perspectives, but several general ideas can be distilled from the discussion. CSR have strongly influenced all scientific disciplines. As an example, the term "system" is used commonly in daily language. One of the breakthroughs of CSR lies in the search of commonalities across disciplines. Even when this was achieved to a certain degree, there is still a lack of...
Read more

New Book: Complexity: 5 Questions

Image
This volume consists of short, interview-style contributions by leading figures in the field of complexity, based on five questions. The answers trace their personal experience and expose their views on the definition, aspects, problems and future of complexity. The aim of the book is to bring together the opinions of researchers with different backgrounds on the emerging study of complex systems. In this way, we will see similarities and differences, agreements and debates among the approaches of different schools. Contributors: Peter M. Allen, Philip W. Anderson, W. Brian Arthur, Yaneer Bar-Yam, Eric Bonabeau, Paul Cilliers, Jim Crutchfield, Bruce Edmonds, Nigel Gilbert, Hermann Haken, Francis Heylighen, Bernardo A. Huberman, Stuart A. Kauffman, Seth Lloyd, Gottfried Mayer-Kress, Melanie Mitchell, Edgar Morin, Mark Newman, Grégoire Nicolis, Jordan B. Pollack, Peter Schuster, Ricard V. Solé, Tamás Vicsek, Stephen Wolfram. Get it at Amazon.com Check out more books from the 5 Questions s...
Read more

Sites that revolutionized the Web

I compiled a list of the websites that have changed the way information is produced, handled, or distributed over the Internet. Many of these sites have something in common: they enable users (human or software) to judge the relevance of their content, to provide "interesting" information to others. This has been called distributed cognition or collective intelligence: like diligent ants, the small contributions of many people or sites lead to great achievements and sources of information. Google . The most effective search engine for several years now. It judges the relevance of a webpage (its PageRank) by the PageRank of webpages that link to it. Thus, if many webpages link to a certain site, this should be interesting. And even more if those webpages are interesting themselves. Google has become an empire of free web-based utilities (in some cases buying other companies): GMail . One of the most flexible free web-based mail providers , it gives users almost 3Gb of disk sp...
Read more