Back
 ICA  Vol.10 No.3 , August 2019
The Optimization of Humanoid Robot’s Dialog in Improving Communication between Humanoid Robot and Older Adults
Abstract: Older adults with dementia are increasing in Japan. Because of this, “communication robots” are being introduced into nursing settings to substitute for the shortage of nurses and other care workers. Our research group is currently developing a humanoid nursing robot with caring function (HNR), specifically for the functional and practical use of older adults with dementia. The purpose of this study was to clarify improvement points of the current function of humanoid robot’s (Pepper) (SoftBank Robotics) dialog pattern for improving optimal communication between humanoid robot and older adults. Dialog programs were installed in the humanoid robot Pepper, including the application program Care Prevention Gymnastics Exercises for Pepper (Pepper-CPGE) that was made by Xing Company, Japan. Dialogues between older adults and Pepper-CPGE were recorded by video camera. Data from transcriptions of the conversations captured from video and from field notes were analyzed focusing on human-robot interaction. From the recorded scenes and conversations, the following were points to be improved: 1) Intonation of the words vocalized by Pepper-CPGE was different from that expected by the older adult resulted in inappropriate responses by the older adult; 2) The timing between Pepper’s questions and the responses of the older adult were not timely and did not match (differences in the question-response time), which lead to confusion among the older adults; and 3) Other surroundings older adults were interested in the dialogue with Pepper-CPGE. However, Pepper-CPGE cannot communicate with multiple older adults. ImprovingPepper-CPGE’s ability to communicate with multiple older adults as an older adult’s dialogue with Pepper-CPGE can cause other older adults to also interact not only with the older adult that Pepper-CPGE is interacting with but also with Pepper-CPGE. This study shows that transactive relations among humanoid robots and older adults can be facilitated. Improving communication between humanoid robots and the older adults by optimizing a structured dialogue is needed to enhance appropriate engagement and participation. It is necessary to create a “Caring dialogue Database” for HNRs in order to know the patient/client and to share the aesthetic experiences of human-robot interactions. Also, it is important to develop a dialog pattern that enables humanoid robots to sympathize with older adults.

1. Introduction

As the aging society with low birth rate progresses [1] , elderly populations with dementia are increasing in Japan [2] [3] , as well as the increasing burden of care for the elderly with dementia [4] . This phenomenon is also observed in many developed countries all over the world [5] . Solitary aged-person increased [6] and many people experience loneliness and depression in old age, either as a result of living alone or due to lack of close family ties and reduced connections with their culture of origin, which results in an inability to actively participate in community activities [7] .

Less opportunities for elderly dialogue cause risk for developing dementia [8] . Therefore, supportive communication is essential to promote good quality dementia care. Also, person-centered approach focusing on supporting an older adult is critical to use and retain their skills and abilities [9] [10] . Nurses and other care workers are decreasing or are in short supply [11] [12] ; for this reason, “communication robots” have been introduced into nursing settings as an alternative for the shortage of nurses and other care workers to support caring for elderly [13] .

For examples, there are small animal type robots that can respond to voice and stroking [14] [15] , and mobile phones with sensors that can detect sound and touch, and can answer back [16] . These technologies have limitations that do not support the dialogue of older adults. Even if there is a development of the elderly communication support system such as remote communication using telenoid etc., and the autonomous interactive robot [17] [18] , this current speech recognition technology is not well-developed and could be challenging for some researchers [19] [20] due to its imperfect two-way dialogue.

Our research group is currently developing a humanoid nursing robot with caring functions (HNR) [21] [22] specifically for the functional and practical use of older adults with dementia.

2. Purpose

The purpose of this study was to clarify improvement points of the current function of a humanoid robot’s dialog pattern for improving optimal communication between humanoid robot and older adults.

3. Methods of This Study

3.1. Method

This descriptive study utilized the observation method. The older adults in a hospital and a Long-Term Care Health facility were the participants of this study. The researchers employed the dialogical programs which were installed in the humanoid robot, Pepper (SoftBank Robotics), including the application program Care Prevention Gymnastics Exercises for Pepper (Pepper-CPGE) that was made by Xing Company, Japan. Dialogues between older adult participants and Pepper-CPGE were recorded by video camera. In addition, the facial expressions and behaviors of older adults with dementia during dialogues with Pepper-CPGE were observed, and field notes were done. The collected data were the descriptions of the interactions, transcriptions of verbalizations, facial expressions, and behaviors. In the analysis, transcripts and video recordings were analyzed from the interaction scenarios.

3.2. The Theoretical Framework

The Transactive Relationship Theory of Nursing (TRETON) was used as a theoretical framework to observe communication between/among humanoid robot and older adults [23] [24] (Figure 1). The transactions between human beings and HNRs are perceived as a means of promoting interaction which helps to improve patitnts’ experience. In essence, the integration of HNR-to-human relationships and the science of human caring have been carefully considered within

Figure 1. Conceptual diagram of optimal communication between/among humanoid robot and older adults [23] [24] .

Tanioka’s TRETON. This theory envisages the mutual engagement and the technological engagement that transpire between that HNRs and the human being. This process involved in establishing interactions among older adults with dementia, the nurse as a mediator, and the HNRs.

This theory describes that HNRs need to have functions that correspond according to older adult’s mood, feeling, and emotion (e.g. anxiety, depression or suffering). This HNRs’ functions require these following elements such as verbal and nonverbal communication; facial expression; eye contact; knowing patient; listening empathically, and sharing patient’s lived experiences. These functions will be able to provide a caring as “healing process” to older adults.

3.3. Data Collection Period

Data collection period was in May, 2018.

3.4. Ethical Consideration

The data collection procedure was approved by the Ethics Committee of the University of Tokushima Hospital (application number 3046). In addition, approval was also obtained from Mifune Hospital Clinical Research Ethics Review Committee (application number 201180502) where the data collection and analysis were conducted. The purpose of the study and the methods used in data generation were explained to all participants and their guardians. Participants confirmed that they fully understood that their personal information were protected, and that reports were compiled and used only for research purposes. Informed consent to participate in the study and to be audio-visually recorded were obtained from every participant before the start of the study.

4. Findings

From the recorded scenes and conversations (Tables 1-3), the following were points to be further developed: 1) Intonation of the words vocalized by Pepper-CPGE was different what was expected by the older adult, so there was no appropriate response from the older adult; 2) The timing between Pepper-CPGE’s questions and the responses of the older adult were not timely and did not match (differences in the question-response time), therefore, the older adult exhibited confusion; and 3) Improving Pepper-CPGE’s ability to communicate with multiple older adults as an older adult’s dialogue with Pepper-CPGE can cause other older adults to also interact not only with the older adult that Pepper-CPGE is interacting with but also with Pepper-CPGE.

5. Discussion

5.1. Problem of the Intonation of the Word Spoken by Pepper-CPGE

Prior to starting a dialogue, the older adult said her name was “Mak-chan”. When the dialogue started, Pepper-CPGE called her “Mak-Chan-San”, however,

Table 1. Conversation contents (1).

Table 2. Conversation contents (2).

Table 3. Conversation contents (3).

the older adult could not understand the name and did not respond. For smooth communication, it is important to correctly recognize the other person, and it is essential to call the other person’s name correctly. The older adult was expecting Pepper-CPGE to call her “Mak-chan ⤵”, however, Pepper-CPGE called her respectfully with the name “Mak-chan-san”. Pepper’s vocalization sounded flat and mechanical, without intonation. In addition, “Mr./Ms. is like ‘San’ in Japanese” was automatically said after the name. It was concluded that the use of these words may pose problems and obstacles for older adults because they may fail to recognize that her name was called.

In this first scene, it was considered that the difference in intonation between the Pepper-CPGE and the older adult and the honorific name given to her, deviated from the expectation of the older adult, thereby causing delayed recognition. In communicating with older adults with dementia, it is important to successfully include and integrate verbal communication and non-verbal communication functions with declining cognitive function, along with the decline of cognitive function [25] . Therefore, it seemed effective to call older adults with dementia with the expression as desired by them. It was considered necessary to have a mechanism for Pepper-CPGE to repeat the name of the other person appropriately.

Also, from the perspective of occupational ethics, it is not appropriate to call the older adult “XX chan” or “grandma” [26] . Even if the older adult wishes to be called “XX chan”, in order to make function as a nursing caring robot, Pepper-CPGE needs to use appropriate words that emphasize the dignity of older adults. Therefore, it was considered that Pepper-CPGE needs a function to select (judge) whether or not to call a person using the proper (courtesy) title according to the level of cognitive function of the older adults.

5.2. Scenes Where the Timing between Pepper-CPGE’s Question and the Response of the Older Adult Did Not Match

When the older adult tried to speak or while talking, Pepper-CPGE spoke out of topic or the topic changed suddenly. The timing between Pepper-CPGE’s question and the response from the older adult did not match. Therefore, the older adult was confused. In Pepper-CPGE’s dialogue program, timeliness was deficient, causing inadequate interactive conversation.

In the previous study, the average time of “pause” that a human takes for a robot was about 2000 ms. The time required for a reply in the case of a question that needs to be considered was 1600 ms for male and 2200 ms for female [27] . The older adult with dementia often takes time to understand the other person’s words and reaching accurate and appropriate verbal responses [28] . Koike [29] pointed out that when care workers talk to patients with dementia, their timing should be slow as it is the characteristic way of nursing care staff when talking to a patient with dementia. In conversation with an elderly with dementia, a slow and polite way of speaking to attunement with the elderly is important [30] . For that purpose, it is necessary to have a particular function in a Pepper-CPGE machine that allows facial expressions recognition of the elderly and selective dialogue content is necessary. In addition, there is a need to look at adjusting the speed of conversation according to the pace of elderly. Moreover, when Pepper-CPGE changes the content of the story, it is important that continuity of communication with the older adult is ensured, such as confirming the content with the older adult like “May I move on to the next topic?”

Also, it is necessary to make Pepper-CPGE’s dialogue pattern from the current question interactive type to the discourse type speech pattern. In dealing with the characteristics of the older adults, it is important to encourage elderly patients to speak and observe their non-verbal cues to ensure that they are also listening. Therefore, instead of Pepper-CPGE asking people one after another, a mechanism is required that accepts elderly people, such as “nodding with a smile laughing” or “Oh, I see”, “Was that so” and encouraging speech. Furthermore, it is needed a dialogue program with a diagnostic function of dementia in natural conversation, such as “I am getting cold, what month is it today?” A dialogue program with the function of accurately diagnosing the cognitive function of elderly people in natural conversation was considered to lead to the effective evaluation of dialogue rehabilitation.

5.3. Scenes Where the Older Adult Who Was Around Joined the Dialogue with Pepper-CPGE

This scene exhibits the attraction of Pepper-CPGE to other persons, especially other older adults. When Pepper-CPGE asked, “Have you ever experienced strawberry picking?” The older adult A who was watching the dialogue with Pepper-CPGE responded to the word “strawberry picking” and loudly exclaimed, “Strawberries are very delicious.” Furthermore, the older adult B who was interested in the dialogue with Pepper-CPGE had a reaction to this occurrence that he was disturbed by A and was upset by person A’s behavior.

In this scene, it was affirmed that interaction and/or transaction occurred not only between Pepper-CPGE and the older adult with dementia, but also with other older adults who were attracted to the conversation with Pepper-CPGE in the surrounding area [23] . These conversations showed a bigger response than the usual dialogue may exist between patients and between nurses and patient. This seemed to be caused by having a strong interest in the dialogue between the older adults and the robot. In addition, it can be said that the older adult A who was watching the dialogue between Pepper-CPGE and her, recognized Pepper-CPGE as a conversation partner. As a result, the older adult A joined the dialogue. Unfortunately, Pepper-CPGE can only have one-on-one conversation with the target who did face recognition. If Pepper-CPGE has the ability to talk with multiple people, Pepper-CPGE can respond to surrounding reactions, spreading and deepening further conversation becomes possible, and it becomes possible to expand opportunities of speech of the older adults. It is important to develop a dialog pattern of humanoid robots that sympathize with older adults.

5.4. Limitations of This Study

In the first implementation of the Pepper-CPGE, it does not only assist older adults to increase communication, but also to be able to improve the whole person producing good outcomes related to health and well-being among the elderly group. However, the observation was only conducted on one occasion. Further observation and evaluation should be made in the intermediate phase and extended phase.

6. Conclusion

This study analyzed the interaction/transaction dialogues in the recorded scenes between older adults with dementia and Pepper-CPGE. It became clear that dialogues with Pepper-CPGE enhance transactive relations between humanoid robots and older adults. Furthermore, structured dialogue with the Pepper-CPGE promotes interaction not only with the older adult but also with surrounding older adults. This shows the development of transactive relations between humanoid robots and older adults. For a humanoid robot to communicate optimally with the older adults, it is necessary to create caring dialogue databases focused on knowing patients/clients. In essence, the structured dialogues from a humanoid robot provide not only interaction but also an aesthetic experience for older adults. Moving forward, it is essential to develop a dialog pattern that humanoid robots who express emotions (e.g. sympathy) that can be used with older adults.

Acknowledgements

This work was supported in part by JSPS KAKENHI Grant Number JP 17H01609.

Cite this paper: Miyagawa, M. , Yasuhara, Y. , Tanioka, T. , Locsin, R. , Kongsuwan, W. , Catangui, E. and Matsumoto, K. (2019) The Optimization of Humanoid Robot’s Dialog in Improving Communication between Humanoid Robot and Older Adults. Intelligent Control and Automation, 10, 118-127. doi: 10.4236/ica.2019.103008.
References

[1]   Nomura, K. and Koizumi, A. (2016) Strategy against Aging Society with Declining Birthrate in Japan. Industrial Health, 54, 477-479.

[2]   Cabinet Office, Government of Japan (2017) Health and Welfare of Elderly People. 2017 Edition of the Annual Book Aging Society White Paper (Summary).
http://www.cao.go.jp/kourei/whitepaper/w-2017/html/gaiyou/s1_2_3.html

[3]   Ohara, T., Hata, J., Yoshida, D., Mukai, N., Nagata, M., Iwaki, T., et al. (2017) Trends in Dementia Prevalence, Incidence, and Survival Rate in a Japanese Community. Neurology, 88, 1925-1932. https://doi.org/10.1212/WNL.0000000000003932

[4]   Cheng, S.T. (2017) Dementia Caregiver Burden: a Research Update and Critical Analysis. Current Psychiatry Reports, 19, 64. https://doi.org/10.1007/s11920-017-0818-2

[5]   Yu, R., Chau, P.H., McGhee, S.M., Cheung, W.L., Chan, K.C., Cheung, S.H., et al. (2012) Trends in Prevalence and Mortality of Dementia in Elderly Hong Kong Population: Projections, Disease Burden, and Implications for Long-Term Care, International Journal of Alzheimer’s Disease, 2012, Article ID: 406852. https://doi.org/10.1155/2012/406852

[6]   Cabinet Office, Government of Japan (2017) Family and Household of the Elderly. 2017 Edition Elderly White Paper (the Entire Edition).
https://www.cao.go.jp/kourei/whitepaper/w-2017/zenbun/29pdf-index.html

[7]   Singh, A. and Misra, N. (2009) Loneliness, Depression and Sociability in Old Age. Industrial Psychiatry Journal, 18, 51-55. https://doi.org/10.4103/0972-6748.57861

[8]   Fratiglioni, L., Wang, H.X., Reicsson, K., Mwytan, M. and Winblad, B. (2000) Influence of Social Network on Occurrence of Dementia: A Community-Based Longitudinal Study. The Lancet, 355, 1315-1319. https://doi.org/10.1016/S0140-6736(00)02113-9

[9]   Kaneda, C. (2005) Research on Action Analysis of the Aged with Dementia with the High Degree of Physical Independence. Journal of Medical Welfare, 1, 57-65. (in Japanese)

[10]   Downs, M. and Collins, L. (2015) Person-Centred Communication in Dementia Care. Nursing Standard, 30, 37-41. https://doi.org/10.7748/ns.30.11.37.s45

[11]   Ministry of Health, Labor and Welfare (2014) About Securing Care Workers. (in Japanese) https://www.mhlw.go.jp/file/05-Shingikai-10801000-Iseikyoku-Soumuka/0000072895.pdf

[12]   Ministry of Health, Labor and Welfare (2014) Basic Materials on Supply and Demand of Nursing Staff. (in Japanese)
https://www.mhlw.go.jp/file/05-Shingikai-12201000-Shakaiengokyokushougaihokenfukushibu-Kikakuka/0000047617.pdf

[13]   Ministry of Internal Affairs and Communications (2015) Ministry of Internal Affairs and Communications Edition of the Annual Book Information Communication White Paper/Partner Robot Needs and Challenges. (in Japanese)
http://www.soumu.go.jp/johotsusintokei/whitepaper/ja/h27/html/nc241350.html

[14]   Shibata, T. and Wada, K. (2011) Introduction of Field Test on Robot Therapy by Seal Robot, PARO. Journal of the Robotics Society of Japan, 29, 246-249. (in Japanese)

[15]   Yamaoka, H., Imai, T., Watanabe, I., Nakajima, T. and Koga, Y. (2010) Robot-Assisted Activities for Elderly People with Dementia Using a Human Friendly Robot. The 24th Annual Conference of the Japanese Society for Artificial Intelligence, Nagasaki, 9-11 June 2010, 1-2. (in Japanese)

[16]   Melson, G.F., Kahn Jr., P.H., Beck, A. and Friedmen B. (2009) Robotic Pets in Human Lives: Implications for the Human-Animal Bond and for Human Relationships with Personified Technologies. Journal of Social Issues, 65, 545-567.
https://doi.org/10.1111/j.1540-4560.2009.01613.x

[17]   Yamazaki, R., Nishio, S., Ogawa, K. and Ishiguro, H. (2012) Teleoperated Android as an Embodied Communication Medium: A Case Study with Demented Elderlies in a Care Facility. 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, Paris, 9-13 September 2012, 1066-1071.
https://doi.org/10.1109/ROMAN.2012.6343890

[18]   Kanda, T., Kamashima, M., Imai, M., Ono,T., Sakamoto, D., Ishiguro, H., et al. (2005) Embodied Cooperative Behavior for a Humanoid Robot that Communicate with Humans. Journal of the Robotics Society of Japan, 23, 898-909. (in Japanese)

[19]   Iribe, Y. and Kitaoka, N. (2017) Speech Corpus Development toward Speech Recognition for OLDEST-OLD Japanese. The Journal of the Acoustical Society of Japan, 73, 303-310. (in Japanese)

[20]   Schermerhorn, P.W., Kramer, J.F., Middendorff, C. and Scheutz, M. (2006) DIARC: A Testbed for Natural Robot Interaction. The Twenty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, Boston, MA, 16-20 July 2006, 1972-1973.

[21]   Tanioka, T., Osaka, K., Locsin, R., Yasuhara, Y. and Ito, H. (2017) Recommended Design and Direction of Development for Humanoid Nursing Robots Perspective from Nursing Researchers. Intelligent Control and Automation, 8, 96-110. https://doi.org/10.4236/ica.2017.82008

[22]   Osaka, K., Sugimoto, H., Tanioka, T., Yasuhara, Y., Locsin, R., Zhao, Y., et al. (2017) Characteristics of a Transactive Phenomenon in Relationships among Older Adults with Dementia, Nurses as Intermediaries, and Communication Robot. Intelligent Control and Automation, 8, 111-125. https://doi.org/10.4236/ica.2017.82009

[23]   Tanioka, T. (2017) The Development of the Transactive Relationship Theory of Nursing (TRETON): A Nursing Engagement Model for Persons and Humanoid Nursing Robots. International Journal of Nursing Practice, 4, 223.

[24]   Tanioka, R., Sugimoto, H., Yasuhara, Y., Ito, H., Osaka, K., Zhao, Y., et al. (2019) Characteristics of Transactive Relationship Phenomena among Older Adults, Care workers as Intermediaries, and the Pepper Robot with Care Prevention Gymnastics Exercises. The Journal of Medical Investigation, 66, 46-49. https://doi.org/10.2152/jmi.66.46

[25]   Havey, E.A. (2018) How to Communicate with Someone with Dementia.
https://eldersense.com/communicate-someone-dementia/

[26]   The Japan Nursing Ethics Association (2015) Guidelines for Protecting the Dignity of Elderly People Who Receive Medical Care and Nursing. (in Japanese)

[27]   Oto, K., Hyo, K., Koyama, D. and Imai, M. (2017) The Design of the Silence in the Conversation between Human and Robot. THE 79th NATIONAL CONVENTION OF IPSJ. (in Japanese)

[28]   Ikeda, M. (2015) Communication Difficulties in Patients with Neurodegenerative Dementia. Higher Brain Function Research, 35, 30-34. (in Japanese)

[29]   Koike, T. (2012) Initiating Conversations with People Suffering from Dementia: Conversation Analysis on Initiating Conversations in Video Recordings. The Japanese Society of Health and Medical Sociology, 23, 96-105. (in Japanese)

[30]   Harwood, J. (2013) Communicating with Patients Who Have Dementia. Arizona Center on Aging.

 
 
Top