Bangalore: So, what makes a website popular—a headstart, bells and whistles, or talent and novelty?
The evolving Internet has been traditionally conservative in spotting talent, showing a preference for experience, which has led to the rich-gets-richer syndrome among websites, but a new study shows that upstart new websites regularly oust existing leaders by providing enhanced novelty and utility.
In the first-ever empirical measurement of activities of the Web, researchers from the University of California, Los Angeles, the US, and University of Regina, Canada, report in Tuesday’s issue of the Proceedings of the National Academy of Sciences that the rules of a “meritocratic” society govern the Internet. “For the first time, we empirically tested how talent, or fitness is distributed on the Web and found that most pages tend to have low fitness; only a very small number of pages have high fitness,” said Vwani P. Roychowdhury, lead author, department of electrical engineering, UCLA.
The few high fitness pages start with only a handful of in-links, but accumulate links at a much higher rate than most of the established pages, and then over time, surpass them. At the same time, all alive pages continue to accumulate links with time, but the rate depends on their fitness, said Roychowdhury.
Top 1,000 Home Page Growth (Graphic)
In one experiment, Roychowdhury and co-authors Nima Sarshar and Joseph S. Kong, found that 48% of the websites that reached a threshold of 1,000 incoming links over a given period used “talent” to displace more experienced pages.
That’s not true for all types of websites, argues Navneet Kaushal, chief executive of PageTraffic Web-Tech Pvt. Ltd, a search engine optimization firm in New Delhi, which offers link building services through its affiliate Link Popularity India.
Citing the example of Google, as a search engine, he said there’ve been some promising new search engines, but none has been able to make any mark.
However, in the case of Google, one of the main reasons it became a hit way back in 1999 was its simplicity and speed in search, said Kaushal. “But if we consider some other examples like Myyearbook.com, it became instant hit just because of its novelty and offerings, which were unique; same goes for Twitter (a social networking and microblogging site).”
For all the increasing complexity of the Web, no one has really studied what dying pages do to it. This study for the first time shows that the “death rate” is pretty high. “For every new page created, 0.77 pages get deleted and with such high rates, if pages did not have different talent, or fitness, then the structure of the Web would be very different,” said Roychowdhury.
Researchers studied 22 million Web pages over a year. The most important discovery, they said, is that there’s a “fine balance” in the birth and death of pages and their accumulation of links as despite high churn, the overall structure of the Web remains seemingly static. Only 6.5% of websites had any perceptible growth in one year.
So, how does this new finding help the Internet? All major search engines already make some use of the structure of the Web—pages at the top in a search query are those with high in-links. And no points for guessing, it was Google that first utilized the structure of the Web and implemented the so-called page rank system.
Researchers argue that each page can be given a constant fitness, which is simple to calculate and determines the rate at which a website will accumulate links.
“It works in the real world as all one needs are regular crawls of the Web,” said Roychowdhury, who has founded a start-up, NetSeer Inc., in Los Angeles, which uses this knowledge to match online advertisements to user patterns.