Challenges in building an app to interpret Indian sign language

Jayasudan Munsamy hit a mid-career crisis after 21 years in the IT industry with Wipro, Infosys, Persistent Systems and Amazon’s app store certification platform. He quit two years ago to dabble in new-age technologies like machine learning and artificial intelligence. He wanted to focus on tech solutions that create a positive impact on society.

The first project he took up was LetsTalkSign, an app that would convert sign language to speech and text and vice versa, using natural language processing and computer vision. But he soon ran into a problem that made him realize it would entail a far deeper involvement than he had thought.

Also Read | Inside the rumble in India’s coding jungle

“The big challenge with sign language is that it’s not universal,” says Munsamy. It varies from country to country and even in different regions of a large country. American Sign Language (ASL) has gestures associated with words based on how these have evolved over time in the country. In Brazil, the gestures would be different for the same communication.

India doesn’t have a standard sign language. This has many ramifications, from lack of access to education for hearing impaired people to barriers in their employment. “If you go to schools for the deaf in India, you will find them mostly using ASL which is single-handed, whereas most variations of Indian sign language are derived from British sign language that uses two hands,” points out Munsamy.

Localization push

After a long campaign by the deaf community, the government set up the Indian Sign Language Research and Training Centre, which has so far agreed on standard Indian signs for around 10,000 words. Two months ago, it signed a memorandum of understanding with the National Council of Educational Research and Training to make textbooks and other educational material available to deaf children in sign language. This year the National Education Policy declared that Indian sign language will be standardized, even as local sign languages are respected and taught as well.

These developments augur well for Munsamy, who committed himself to building LetsTalkSign by forming a startup called DeepVisionTech.ai last year in Bengaluru. Automated two-way communication between sign language and text or speech is an unsolved problem globally.

Brazil’s Hand Talk, which won a $750,000 grant last year as one of the winners of Google’s AI Impact Challenge, automatically translates Portuguese text and speech to Brazilian sign language (Libras) and ASL. But this is one-way for now, which limits scope to carry on a conversation between a hearing-impaired person and others who don’t know the local sign language.

“It’s relatively easier to convert speech to sign language. Using computer vision to understand sign language and convert that into speech is harder,” says Munsamy. LetsTalkSign thus aims to break new ground by automating two-way communication. But it’s early days yet, with a minimum viable product (MVP) expected to be ready next year.

The use cases for this are many. “I know deaf people who spend ₹200-300 per hour to take an interpreter along with them to a doctor. A digital solution will make their lives easier. Inside a classroom, the app can be installed on a device so that an avatar of the teacher can be displayed on a monitor. Deaf students can follow what the teacher is saying from the avatar’s hand signs,” he says.

The Indian startup hopes that its head start with Indian sign language will place it in a defensible business position locally because of a high entry barrier for apps developed for sign languages in other cultures. Covid made it hard for the bootstrapped startup to raise funds, hire people and even connect with deaf communities or access CSR grants. But Munsamy is focusing on the MVP for now.

DeepVisionTech has received tech support from Nvidia Inception, the MNC’s deep-learning and AI startup incubator in Bengaluru. It has also been incubated by Oracle’s startup programme.

Critical factor

When it comes to deploying the app in the market, standardization of Indian sign language will be critical. This year’s education policy has taken a step in that direction, but there’s still a long way to go in implementing this.

“They have said schools have to teach in Indian sign language going forward. That’s not easy to do because first they have to train the teachers in ISL. And then they have to teach the students,” says Munsamy. “And if you look at the deaf students here, they are not taught a formal sign language at home from an early age. Family members use whatever basic signs are intuitive to them and that’s what the children learn. So, when they go to school at the age of 6 or 7, they don’t understand the sign language being used by the teachers and other students. That’s a big challenge to overcome.”

But at least a start has been made with the policy direction to standardize Indian sign language, which in turn will help create innovative digital tools for the hearing impaired.