IIT Kharagpur researchers make Sanskrit accessible with Artificial Intelligence

IIT Kharagpur has announced that its researchers have developed a digital infrastructure for the efficient processing of Sanskrit texts, by effectively combining machine learning techniques and traditional linguistic knowledge from Sanskrit. This artificial intelligence based solution has been developed under the supervision of Dr. Pawan Goyal, and the work has been carried by research scholar Dr. Amrith Krishna, currently a postdoc at the University of Cambridge.

This work has been accepted for publication in the Computational Linguistics journal published by the MIT Press. The paper currently addresses the tasks of word segmentation, morphological parsing, dependency parsing and poetry to prose conversion of Sanskrit text. This solution has been developed as there has been a renewed interest in Sanskrit since the announcement of NEP 2020, said a statement from the institute. Various academic institutions both at school education as well as higher education are adopting various approaches for improving the reach of the language through training programs, research and outreach initiatives.

While various digital resources have improved the accessibility and use of world languages and even regional languages, Sanskrit presents unique challenges in automated computational processing. In addition to the sheer volume and diversity, both stylistic and chronological, found in these texts, the linguistic peculiarities expressed by the language, pose several challenges in making these works accessible to the world.

The proposed framework is based on Energy-based models and it enables the encoding of relevant linguistic information as constraints. “Processing of Sanskrit texts poses several challenges owing to the high lexical productivity of the words, free word order in poetry, euphonic assimilation of sounds at the word boundaries and phonemic orthography followed in writing. Keeping these in mind, we proposed a generic graph-based framework that takes advantage of the free word order nature of the language. Further, we make use of linguistic insights from the traditional Sanskrit grammar for learning the feature function and applying the relevant constraints.” explained Dr. Goyal.

He further adds: “Our proposed framework substantially reduces the training data requirements to as low as 10%, as compared to that of the neural state-of-the-art models. In all the Sanskrit-related tasks discussed in the work, we either achieve state-of-the-art results or ours is the only data-driven solution for those tasks,”

The team is now actively collaborating with several external research groups to extend the application of the proposed system for automatic speech recognition and question-answering in Sanskrit. The proposed artificial intelligence based system, used in conjunction with interactive tools such as the Sanskrit Heritage reader, can aid the users in the easier analysis of these manuscripts with word-by-word analysis and translation, the relation between words, poetry to prose conversion, search and question answering, etc.