Greetings from the Principal Investigator

I have been studying Japanese Sign Language and gestures in social interaction. I first encountered Japanese Sign Language when I was a university student. The religious department of the university organized a sign language study group, which I joined. There I met a Deaf teacher and had the experience of learning sign language without using spoken Japanese. After that, I attended a local sign language circle and became completely fascinated with sign language.

While attending the local sign language circle, I also researched language communication at a graduate school. At that time, there were few references for research on sign language, and I wondered if my research topic would be good enough to write a doctoral thesis. Because of that, my graduate research was not on sign language but on gestures and body movements in spoken Japanese communication, and I received my Ph.D. degree.

More than 20 years have passed since then, and the research environment surrounding sign language and gestures in language studies have changed dramatically. Deaf people, who use sign language as their everyday language, have begun to study it by themselves. Research and development of artificial intelligence technology to recognize and translate sign languages have gradually started. What about research on sign language and gestures? The study of gestures in spoken languages has a history of about 40 or 50 years. In contrast, there was a time when sign languages were called "gestural languages" and were not considered as languages. There have been few studies that treat sign languages and gestures in parallel or that treat gestures as a significant part of signed language communication.

However, what do you think? As with spoken language communication, it may be possible to study not only the vocabulary and grammar used in sign language communication, but also "how it is signed (e.g. intonation indicating signers' feelings and attitudes)" and "depicting elements (gestures describing visual images in detail) that cannot be captured in the vocabulary".By studying such elements, we believe that spoken language communication research and signed language communication can be placed on an equal footing, and that richer discussions can be made.

The purpose of this field of research is to develop a "theory of multimodal semiotics" on signs and gestures in language interactions, which has been discussed in the research fields of linguistics and philosophy of language, with the aim of studying phenomena that are not yet clearly characterized as signs, such as gestures and sign language in conversation. As a whole, we aim to deal with speech, gestures and sign languages methodologically and engineeringly across modalities. As a result, we would like to organize interaction data annotated with multimodal semiotics into a corpus and make it widely available to the related research community to clarify how to apply our theory to the development of multimodal dialogue translation technology. We call the above research activities "Embodied Semiotics" and attempt to create an academic revolutionary field that integrates the humanities and sciences.