Interpreting Sign Language Using Artificial Intelligence
Table of contents
While discussing cultural stereotypes is borderline illegal in ‘Murica these days, we have no problem opening those doors. That’s because cultural stereotypes largely have some truth to them which makes them amusing – unless you’re German in which case you probably don’t have a sense of humor. Corporations today are becoming increasingly globalized and it’s important to be aware of the idiosyncratic differences between cultures that you’ll encounter while training your lower-cost replacement in an Emerging Market Center (EMC) like Mumbai. What you’ll discover when outsourcing across continents is that some EMCs demonstrate higher levels of competence than others. One EMC where you will find above-average levels of competency – consistently – is Hungary.
According to the U.S. Department of Education and the National Institute of Literacy, about 32 million Americans can’t read (just under 10% of the population). In Hungary, that number falls to just 58,788 people (about 0.06% of the population). Maybe it’s because the children are told if they don’t learn how to read then Rézfaszú bagoly – the owl with a brass penis – will come terrorize them. Whatever the reason, the end result is a country that has its share of brainpower having invented the digital computer, the ballpoint pen, the hydrogen bomb, Vitamin C, the Rubik’s Cube, the helicopter, and even holograms. We recently airdropped one of our MBAs into the center of Hungary to spend the week experimenting with hard drugs at Ozora scouring the country’s capital in search of interesting startups to write about. One of the startups we found is SignAll.
SignAll – Interpreting Sign Language Using AI
Founded in 2016, Hungarian startup SignAll has taken in $3.6 million in funding and was recently featured in our article on the Top-10 AI Startups in Central and Eastern Europe. In short, the company is using computer vision to translate American sign language into written language which promises to improve the quality of life for the one million Americans who are functionally deaf. Said a VC who invested in SignAll, “In comparison to similar projects, SignAll is the only one that recognizes complete sentences, not just single signs. Its technology unites two skyrocketing sciences, computer vision and natural language processing.” In 2017, SignAll forged a partnership with Gallaudet University (GU), the world’s leading university for the Deaf and Hard of Hearing (located in Washington D.C.). In order to better understand the problems being solved here, let’s first learn a bit more about sign language.
About Sign Languages
We weren’t surprised to learn that there are 70 million deaf people around the globe, but we were surprised to learn that they all communicate using different sign languages – more than 300 of them. In American Sign Language (ASL), the alphabet can be demonstrated using just one hand. For other sign languages like British Sign Language, two hands are used, and in Italy, you actually need to use four hands. (We might have made that last part up.) Deaf people use name signs instead of names, and facial expressions, hand movements and position, and body language are all used in addition to just signing letters, words, and sentences. Similar to how sports teams will give players nicknames, it’s usually the deaf community that assigns your ASL name which can also contain some description which is unique to you. Says an article by Verywell Health:
If you love a specific animal, like cats, your name sign may be using the first letter of your birth name to then sign “cat’s whiskers” on your cheek. If you enjoy birds, your name sign could be the first letter of your birth name combined with the sign for bird.
By now you’re beginning to realize that the world of sign language is far more complex than most people would think. It’s not that simple to try and use technology to understand what a deaf person is saying, and that’s what makes what SignAll is doing so remarkable. What’s even more remarkable is that SignAll – for the first time in history – hopes to enable real-time translation between different sign languages, something which will broaden its usability significantly.
SignAll’s Technology
When venture capitalists invest in startups, they’re oftentimes investing in the founders as much as in the idea. The founders of SignAll, János Rovnyai and Zsolt Robotka, have pedigrees that help explain why SignAll has become one of the most successful startups in Hungary. (If you’re wondering why all Hungarians have Hungarian sounding names, that’s because Hungarian parents are required by law to choose their children’s names from an official list of several thousand names. Now you know.) Both men hail from a company called Dolphio Technologies which is the only Hungarian firm to make Deloitte’s “50 fastest growing technology companies in Central and Eastern Europe” list – for three consecutive years.
Dolphio Technologies provides consulting services and incubates ideas, SignAll being one of them. The company’s talent pool leans heavily towards mathematics, and it’s no surprise that they’ve managed to secure exclusive rights to one of the most important Hungarian inventions of the 21st century – the Gomboc – a self-righting geometrical shape that was solved in 2006 by two Hungarian mathematicians. It’s the first known convex, homogeneous object to have just one stable and one unstable equilibrium point.
We sat down to talk with Co-Founder and CEO of SignAll, Zsolt Robotka, about how the idea for SignAll came about when he hired a deaf employee and didn’t have the ability to communicate without an interpreter. The supply of sign language interpreters does not meet the demand, and consequently, rates can be very high. Mr. Robotka talked about how communicating with the Deaf using written language can be difficult since that’s actually their second language. That’s when he decided to apply technology to solve the problem.
As we mentioned before, translating sign language is not just about capturing hand gestures. We also need to take into account body language, facial expressions, and of course the hand/finger shapes.
SignAll is the only device in the world that allows spontaneous communication between a deaf person and someone who can hear via automated ASL translation technology. The deaf person wears a pair of colorful gloves and signs to the camera (the colors on the gloves help the technology to more accurately differentiate signs). A total of three cameras capture additional information including facial expressions and gestures. The hearing person simply speaks normally and their speech is translated to the deaf person’s screen using natural language processing.
SignAll found that off-the-shelf technologies didn’t work so well given how unique their use case was, so most of their technology has been developed in house. One competitive advantage the company has is their exclusive access to a big data set that’s used to train their algorithms. Working closely with Gallaudet University (GU), SignAll assembled a database of 250,000 manually annotated images that are used to train their algorithms. Companies that participate in pilots all agree to share the data they generate to further improve the algorithms. Over time, the Deaf who use the system may not even need to use gloves because the system will have learned to read sign language so well, and it’s a solution that’s currently under development. SignAll has successfully implemented pilot projects in Washington, D.C. at Gallaudet University, and more recently in Houston and Salt Lake City though the company wasn’t at liberty to expand on the use cases in these last two cities. What they could tell us is that they recently unveiled a teaching tool that will allow hearing users to learn and practice sign language.
ASL as a Second Language
More than 50% of our global audience is made up of Americans, and we like to wind them up every chance we get – mainly because it’s some really low hanging fruit. People from other countries are often surprised to learn that most Americans don’t speak a second language, and that might help explain why ASL is the third most learned second language in the United States. We came up with an interesting theory on why that is.
Apparently, many colleges in the U.S. consider ASL to meet the second language requirements for certain degrees and those darn millennials will choose ASL because it’s perceived to be easier than studying something like Mandarin – a language that can be used to communicate with 1 in 5 people on this planet. That’s great news for SignAll which unveiled their technology last month at the American Sign Language Teachers Association Conference in San Diego where they collected about 50 applications for the pilot installment of systems in educational organizations. At least three universities will install the system this month and start using it next semester. Many ASL teachers have expressed their concerns regarding a lack of students’ motivation to practice after class, and SignAll’s technology provides a fun way to get the students to get on with adulting and do their homework.
Conclusion
One thing we noted during our time spent researching this topic is that there appear to be no formidable competitors in this space. The only other firm working on this that we know of – MotionSavvy – went kaput. A lack of competition is something that company leaders don’t find comforting since it means that others will notice and try to enter the space. Without an exclusive big data set like the one SignAll built, it promises to be an uphill battle.
While many universities out there have projects going to try and tackle the problem, an understanding of the Deaf community is equally as important as having data. SignAll is using technology to better the lives of deaf people and it looks like they’ll make some money for their investors while doing so. If you’re an organization with deaf employees or customers, just contact SignAll if you’re interested in participating in your own pilot.
Share
can i use ai on my website to translate to sign language or on any platform?
You would need to contact the company directly.