Grants-in-Aid for Scientific Research-JSPS KAKENHI- for FY2022(Grant-in-Aid for Transformative Research Areas(B))

mainimage

News

2024/03/08 JASS48 WS Session
DATE:March 8, 2024
VENUE:FUKUOKA WOMEN’S UNIVERSITY,Kyoto,Japan
WS Session3 相互行為中の身体動作を対象としたマルチモーダル連鎖分析から身体記号学へ
See link :JASS48 Program

2024/03/07 A02 Group Meeting

2024/03/06 Embodied Semiotics Seminar (4th)
Introduction to Sign Language Processing
Presenter:Victor SKOBOV (A02)

2024/03/01 A03 Group Meeting

2024/02/24 身体記号学科研スピンオフ研究会「身体記号学と動物コミュニケーションの関連について議論する」
DATE: February 24, 2024
VENUE:KYOTO UNIVRSITY INAMORI CENTER,Kyoto,Japan

Project Outline

The purpose of this field of research is to develop a "theory of multimodal semiotics" on signs and gestures in language interactions. This has been discussed in the research fields of linguistics and philosophy of language with the aim of studying phenomena that are not yet clearly characterized as signs, such as gestures and sign language in conversation. As a whole, we aim to deal with speech, gestures and sign languages methodically across modalities. As a result, we would like to organize interaction data, annotated with multimodal semiotics, into a corpus and make it widely available to the related research community. This would clarify application of our theory to the development of multimodal dialogue translation technology. We have termed the above research activities "Embodied Semiotics" and are attempting to create an academic revolutionary field that integrates the humanities and sciences.

Principal Investigator

Mayumi Bono
National Institute of Informatics Associate Professor

Greetings from the Principal Investigator

I have been studying Japanese Sign Language and gestures in social interaction. I first encountered Japanese Sign Language when I was a university student. The religious department of the university organized a sign language study group, which I joined. There I met a Deaf teacher and had the experience of learning sign language without using spoken Japanese. After that, I attended a local sign language circle and became completely fascinated with sign language.

While attending the local sign language circle, I also researched language communication at a graduate school. At that time, there were few references for research on sign language, and I wondered if my research topic would be good enough to write a doctoral thesis. Because of that, my graduate research was not on sign language but on gestures and body movements in spoken Japanese communication, and I received my Ph.D. degree.

More than 20 years have passed since then, and the research environment surrounding sign language and gestures in language studies have changed dramatically. Deaf people, who use sign language as their everyday language, have begun to study it by themselves. Research and development of artificial intelligence technology to recognize and translate sign languages have gradually started. What about research on sign language and gestures? The study of gestures in spoken languages has a history of about 40 or 50 years. In contrast, there was a time when sign languages were called "gestural languages" and were not considered as languages. There have been few studies that treat sign languages and gestures in parallel or that treat gestures as a significant part of signed language communication.

However, what do you think? As with spoken language communication, it may be possible to study not only the vocabulary and grammar used in sign language communication, but also "how it is signed (e.g. intonation indicating signers' feelings and attitudes)" and "depicting elements (gestures describing visual images in detail) that cannot be captured in the vocabulary".By studying such elements, we believe that spoken language communication research and signed language communication can be placed on an equal footing, and that richer discussions can be made.

The purpose of this field of research is to develop a "theory of multimodal semiotics" on signs and gestures in language interactions, which has been discussed in the research fields of linguistics and philosophy of language, with the aim of studying phenomena that are not yet clearly characterized as signs, such as gestures and sign language in conversation. As a whole, we aim to deal with speech, gestures and sign languages methodologically and engineeringly across modalities. As a result, we would like to organize interaction data annotated with multimodal semiotics into a corpus and make it widely available to the related research community to clarify how to apply our theory to the development of multimodal dialogue translation technology. We call the above research activities "Embodied Semiotics" and attempt to create an academic revolutionary field that integrates the humanities and sciences.