The Vernacular Search Evolution and Challenges

The online search industry while being dominated by a few large generic players and one single player in different languages/verticals has seen a huge transformation. This has taken place both technologically and also on the usability front, innovative yet solely driven by the market. On one side is the consistent pressure of the ever growing online content and connected users and on the other the power of being the biggest revenue generating aspects in the online eco-system. With great power comes greater responsibilities and the online search industry has the responsibility to honestly lead users to the most relevant information in the shortest possible time.

This has led search to be embedded in the browser or on the device. Even before the user has typed in the word, search engines are already on the job. Processing chunks of data trying to predict what the user is looking for.

Giving suggestions, even showing a set of results which are constantly updated while the user is still typing. All this is enabled by innovatively processing continuously streams of terabytes of data. This is an outcome of years of effort on the part of various search technology evangelists. This effort has also led to the emergence of a host of new technologies for handling and processing of data.

In addition to all of this there are the challenges of the new screen sizes of various devices-mobiles, phablets, tablets, and televisions. Each screen has its own demand of a new level of relevance from search results. The importance of relevance customized for each user has thus become the holy grail of search engines.

VERTICAL SEARCH ENGINES

The pressure to meet these user expectations led to the evolution of vertical search engines. These are focused search engines restricting their domain to a particular vertical (Weather.com, Yelp, Kayak, Pricegrabber, etc) or language (Baidu, Naver, Yandex, Raftaar, etc). Better focus can lead to more relevant results. And the increase in the market share of these sites only confirms this. Even the generic search engines were forced to try out the vertical route and so photos/image search has a dedicated section in most search engines. Other than that Google did try out a recipe search, a music search, a news search and many others.

The vernacular search industry while catering to a smaller audience, still needs to respond to the technology led development by generic search engines. Initially, the vernacular search engines created a platform that could handle non-English search better. While the vertical search engines created a platform to handle specific domain data better. Their focused attempt at only doing a single language/vertical let them take the lead in relevance. However, many of them did not have a large user base and were forced to add multiple languages/verticals. A move that spelt blunder for them and led to lot of them closing down or selling out. The generic search engines are also facing the pressure to close the many multiple verticals with a few already been removed from their main sites.

The few vernacular and vertical search engines that have survived realized that they need to increase their usability and not the languages or verticals. These search engines therefore started to relook at the vanilla clean search look that had become synonymous to search engines. These evolved from being just a search bar to a more content aggregator, search and explore portal. Each ensuring that the user is being given the power to explore for more content within the vertical/language. Top trending terms, categories and relevant data are all available on a click. Search did not always necessarily be an outcome of a query being typed on a search bar, it could also be from a click.

An aspect that will further evolve as the access from smaller devices continues (mobiles and also TV remotes) to increase. This thought either consciously or through hit and trial is the biggest differentiator for the search engines especially the vertical focussed.

CHALLENGES

Vernacular search has its own challenges, while it is limited to a language it is still generic search within the language. Each language has its own specific challenges, as each language has evolved in a very different manner and within a different cultural setting. Even where the language is same the cultural settings give a new twist to the interpretation of language. Search based on pure text retrieval has long been abandoned. The vernacular search is still halfway through this transition. The development in and around the English language search has been very fast paced and has guided the efforts in other languages. Better natural language processing (NLP) tools are being evolved for various languages that can greatly improve the relevance of language search. But still there is a long way to go and the vernacular search needs to catch up.

The internet also has the ability to track users and their behaviour. This aspect has been used by search engines to improve their relevance by customizing their content. Most users move across various language sites, the tracking however, is pre-dominantly happening on Englishlanguage sites.

To add to it the current availability of content is also limited in the non-English space. The limited content that is there is optimized to drive traffic through the larger English search engines. Thus, the most relevant of all aspects of a webpage that can help search engine rank content better tend to be in English. User generated content from participation in social and professional networking sites in most languages is still missing. Whatever little is happening is predominantly in the English language and in the roman script.Making the job of a vernacular search engine all the more difficult and forcing them to innovate.

CONCLUSION

Despite this obstacle, understanding search behavior is very important for a non-English search engine because it revolves around the following unequivocal fact for countries who do not have English as their first language; the fact that users may learn English but do not think in English. Vernacular Search platform thus has a very crucial role to play in these languages and countries. They have the ability act as the catalyst that can encourage content development as they connect the right user to the relevant content. By routing traffic they channelize the demand that can lead publishers to create content.

Once this content generation is catalysed, publishers especially for vernacular content have to move from just putting content and optimizing for search to optimizing their presence on the web. For example, a company offering a product would want to ensure that its product page is in the concerned language, the online order store is in the language, the brand ambassador is communicating in the same language and the user interaction are in the same language. What now is being referred to as WPO (Web Presence Optimization) will be the future that drives content creation on the net and vernacular search engines would need to address this and account for this.