Vassilis Athitsos               VLM Lab

ASL Dictionary Search Project

This project has been funded by NSF grants IIS-0705749, IIS-1055062 and CNS-1059235, and was conducted in collaboration with Professors Stan Sclaroff and Carol Neidle. The goal is to develop a system that lets users search dictionaries of American Sign Language (ASL) so as to look up the meaning of unknown signs. In the proposed system, the user submits as query a video of the sign of interest, or simply performs the sign in front of a camera. The system then searches a large database of sign videos in order to find the best matches for the query video, and presents the top results to the user. The user can then visually inspect the top results to verify which (if any) of them best matches the query sign.

A key part of this project is extensive data collection of video examples of signs. We have created the ASL Lexicon Video Dataset, described in this publication.

prototype

Figure 1: This image shows a sample result of an early prototype system that allows users to search for the meaning of a sign. The user provides as input a video sequence of a sign. Then, the computer searches through the videos of signs contained in the system database, and presents to the user the most similar matches, and English translations for those matches. The process is considered successful if those top matches include the sign that the user was searching for, since in that case the user can see the translation for that sign, and can also click on that sign to obtain more information. The top matches are determined automatically, using computer vision and machine learning techniques. The user can visually inspect each of the top matches to determine which (if any) of them is the correct match.


Designing techniques that produce accurate results as often as possible is a key challenge in this research project. Another key challenge is designing indexing methods that allow for efficient database search despite the vast amount of video data stored in the database. Over the years we have proposed several novel methods for these problems, as described in our publications. We are still actively working on these problems, as there is still significant room for improvement in the performance obtained by current methods.

References

Vassilis Athitsos               VLM Lab