New paper: Rank Diversity of Languages: Generic Behavior in Computational Linguistics

Statistical studies of languages have focused on the rank-frequency distribution of words. Instead, we introduce here a measure of how word ranks change in time and call this distribution rank diversity. We calculate this diversity for books published in six European languages since 1800, and find that it follows a universal lognormal distribution. Based on the mean and standard deviation associated with the lognormal distribution, we define three different word regimes of languages: “heads” consist of words which almost do not change their rank in time, “bodies” are words of general use, while “tails” are comprised by context-specific words and vary their rank considerably in time. The heads and bodies reflect the size of language cores identified by linguists for basic communication. We propose a Gaussian random walk model which reproduces the rank variation of words in time and thus the diversity. Rank diversity of words can be understood as the result of random variations in rank, where the size of the variation depends on the rank itself. We find that the core size is similar for all languages studied.

Cocho G, Flores J, Gershenson C, Pineda C, Sánchez S (2015) Rank Diversity of Languages: Generic Behavior in Computational Linguistics. PLoS ONE 10(4): e0121898. doi:10.1371/journal.pone.0121898

This is the first publication of an ongoing collaboration with colleagues from the Physics Institute at UNAM, there is more in the works... It has been a pleasure working with them.
Location: National Autonomous University of Mexico, Coyoacán, Mexico City, Federal District, Mexico

Comments

Post a Comment

Popular posts from this blog

The Past, Present and Future of Cybernetics and Systems Research

Summary of "The Past, Present and Future of Cybernetics and Systems Research" Symposium M at  European Meetings on Cybernetics and Systems Research (EMCSR), Vienna, Austria, April 12, 2012. Organizer: Carlos Gershenson This guided reflection on the challenges and opportunities of cybernetics and systems research (CSR) included initial interventions by panelists Peter Erdi, Helena Knyazeva, Stefan Thurner, Peter Csermely, and Alexander Lazlo. Afterwards, the floor was opened to interventions from the general public and further interventions by panelists. Commentaries were made from a broad variety of perspectives, but several general ideas can be distilled from the discussion. CSR have strongly influenced all scientific disciplines. As an example, the term "system" is used commonly in daily language. One of the breakthroughs of CSR lies in the search of commonalities across disciplines. Even when this was achieved to a certain degree, there is still a lack of...
Read more

New Book: Complexity: 5 Questions

Image
This volume consists of short, interview-style contributions by leading figures in the field of complexity, based on five questions. The answers trace their personal experience and expose their views on the definition, aspects, problems and future of complexity. The aim of the book is to bring together the opinions of researchers with different backgrounds on the emerging study of complex systems. In this way, we will see similarities and differences, agreements and debates among the approaches of different schools. Contributors: Peter M. Allen, Philip W. Anderson, W. Brian Arthur, Yaneer Bar-Yam, Eric Bonabeau, Paul Cilliers, Jim Crutchfield, Bruce Edmonds, Nigel Gilbert, Hermann Haken, Francis Heylighen, Bernardo A. Huberman, Stuart A. Kauffman, Seth Lloyd, Gottfried Mayer-Kress, Melanie Mitchell, Edgar Morin, Mark Newman, Grégoire Nicolis, Jordan B. Pollack, Peter Schuster, Ricard V. Solé, Tamás Vicsek, Stephen Wolfram. Get it at Amazon.com Check out more books from the 5 Questions s...
Read more

Sites that revolutionized the Web

I compiled a list of the websites that have changed the way information is produced, handled, or distributed over the Internet. Many of these sites have something in common: they enable users (human or software) to judge the relevance of their content, to provide "interesting" information to others. This has been called distributed cognition or collective intelligence: like diligent ants, the small contributions of many people or sites lead to great achievements and sources of information. Google . The most effective search engine for several years now. It judges the relevance of a webpage (its PageRank) by the PageRank of webpages that link to it. Thus, if many webpages link to a certain site, this should be interesting. And even more if those webpages are interesting themselves. Google has become an empire of free web-based utilities (in some cases buying other companies): GMail . One of the most flexible free web-based mail providers , it gives users almost 3Gb of disk sp...
Read more