Prospective memory (PM), as an ability to remember to take actions in the future, is frequently used in our daily lives. Prospective memory is also important in maintaining everyday lives and social communications, for instance, taking medicine after lunch or meeting with some friends at a certain time.
For testing prospective memory performance in laboratory settings, slide-based PM task―as a standard psychological paradigm, had been developed (Maylor, 1993) . Basic slide-based task is conducted by giving participants instructions to perform a specific action when a target condition is met. As the ongoing component of the task, participants are also required to perform another action, the ongoing action, such that they are receiving constant background stimulus. The ongoing action should be trivial, such as comparing the length, size, or colors of an item, and must be different from the action required by the PM component, which is the target condition. A typical laboratory setup for slide-based task should follow five points (Einstein & McDaniel, 2005) : 1) a slide-based task must have instructions and practice trials for the participants; 2) the PM instructions should be presented to the participants; 3) a delay is introduced during which participants perform other activities; 4) reintroduce the slide-based task without reminding participants of the PM component; 5) the PM target condition occurs several times in the slide-based task, and PM performance is measured by the participants’ response time (RT) when the target condition occurs.
With the widespread use of slide-based task, many researchers came to realize that this basic laboratory setup cannot reflect all the PM processes under real-world conditions. There are two main reasons why laboratory setup (such as slide-based task) provides limited predictive information. Firstly, everyday real-life PM demands are well learned and determined by individual’s life experience, so they are different from the PM demands in laboratory tasks (Dismukes et al. 2008) . Secondly, laboratory experiments have traditionally focused on quantifying brain damage, rather than appraising PM functional performance in real-life condition (Titov & Knight, 2001) . While laboratory experiments have been useful in understanding basic processes related to encoding, storage, and retrieval of PM (Kliegel et al., 2008) , PM tasks derived from laboratory settings, however, may not simulate the PM demands placed on the memory in the real world (Titov & Knight, 2001) . In order to measure the PM performance of real world conditions in laboratory settings, the task requires real world-based design and real world- like environment that can be controlled. To overcome this obstacle, (Titov & Knight, 2001) designed and constructed a standardized PM testing task with video-based technology, called the Prospective Remembering Video Procedure (PRVP) to assess PM performance by stimulating everyday memory-based activities with video recordings. The procedure required participants to complete everyday memory-based actions in response to the observed cues while watching a video recording. The video presents a First-person perspective of walking through a shopping district to create the feeling of a shopping trip. Based on the result of the study, Knight considered the video recordings of everyday activities can be used as experimental measures of PM with high ecological validity. Moreover, the study showed that participants’ task performance of PRVP sensitively varies with environment in the task. In his later work, Knight also attempted to assess the impact of old age and noisy environment (Knight et al., 2008) using a similar setup. He designed a computer-based platform to simulate a shopping street, and linked a set of 1200 photographs of street scenes that allowed participants to walk along “the virtual street”. As for the PM task in this study, participants were required to respond to different cues, such as “if you hear a dog barking, tell me the name of the shop in front of you”. The result showed a significant contrast between young and elderly people. Elderly people had difficulty remembering future intentions, even with naturalistic stimuli. In addition, the noisy environments have a particular impact to older adults’ PM performance. This study suggested that the use of computer-based environments to measure functional cognition deficits should be encouraged.
With the advancement of 3D graphics technology, some studies used 3D modeling to design virtual experiment environment. For testing cognitive function performance of brain-damaged subjects, Okahashi et al. (2013) developed a virtual shopping test (VST) with 3D modeling technology. This VST task has Japanese style shopping street environment with shops on both sides of road. All the shops have a signboard written in Japanese. Participants are required to buy items and perform some actions in the VST. As a result, it is able to evaluate attention ability and everyday memory performance in patients with brain damage, and it has shown that the completion time of the VST increased by age.
While VR technology has been used in assessing PM performance, it has also been applied on the training front. Mitrovic et al. (2014) developed a 3D virtual model-based task to try to improve patient’s PM performance. In this study, the task environment is a house with common household objects and garden. Participants execute tasks and interact with objects by performing various actions, such as turning on a TV set or using the cooking stove. As a training study, Debarnot et al. (2015) used the theta burst stimulation (TBS) technique to stimulate the left frontopolar cortex (BA10), which has been shown to enhance PM performance in aged subjects. The behavioral effects were evaluated by a 3D model-based task. In this task, Debarnot designed a virtual city environment which has one road and 12 connecting scenes. It also included common city objects such as buildings and stores. After the TBS, participants entered the learning stage to learn how to detect PM cues. After the training, participants were asked to navigate in the virtual city. They were asked to stop at the appropriate time or place and tell researchers what action they were supposed to do. Debarnot successfully found the evidence that intermittent TBS might counteract the normal aging of PM performance, which was confirmed by the VR PM task.
These studies show the significant potential in using VR technology to assess real-life PM function. Although some studies (Okahashi et al., 2013; Mitrovic et al., 2014; Debarnot et al., 2015) claimed that they used VR technology to build the task environment, such experiments were only using computer monitor to present the task, which may not be technically regarded as true “VR” experiment environment. In order to simulate realistic PM function, using head-mounted display (immersive VR) to create a life-like experience is necessary. We consider the use of immersive VR device will enable us to create an environment that can test PM performance in real-life conditions more naturally and accurately. Moreover, there are very few studies that compare the validity of VR task with typical laboratory experiments. Therefore, in our research, we developed a VR system with immersive VR device to evaluate PM function under simulated daily life condition. This system is designed for daily life routine, such that the task tries to mimic a typical shopping experience in a shopping area (Titov & Knight, 2001) . The VR task also contains ongoing actions and PM actions. In order to examine its validity, we also created a standard slide-based task to compare with the VR task.
2. Materials & Methods
2.1. Participants and Procedure
10 healthy male participants were recruited from Kobe University, and the mean age was 25.4 years old. All participants performed both slide-based task and VR task. The whole experiment took 45-minute for each participant (15-minute for slide-based task and 35-minute for VR task), and was finished in one day (Figure 1). Before slide-based task and VR task, participants were required to perform a training session to ensure that the participants fully understand the instructions and can perform the task with minimal interference.
2.2. Slide-Based PM Task
The experiment was conducted in a quiet and well-lit room. Each participant was seated in front of a computer monitor and were instructed to use the keyboard to respond to the on-screen stimuli. Stimuli were displayed on the monitor with resolution at 1024 × 768 (EIZO, Color Edge CX240). All feedbacks were recorded in real-time on the computer. The task was created with Unity 5.0.
The task is based on Burgess et al. (2001) ’s number slide task. As shown in Figure 2, a word “question” was displayed at the center of the black background. The font of figure was in Times New Roman, and font size was 256 pixels. The viewing distance was 57 cm to monitor. Each block of test had 30 trials. The rate of stimulus (one question) was 17 s, with 2 s as the inter-stimulus interval, 3 s for the stimulus and 12 s for a resting period (Figure 2 and Figure 3). Participants were required to make responses by pressing keys on the computer keyboard whenever a stimulus was shown.
In the stimulus section, numbers were present to participants as stimulus between the center point, with a “.” mark in the center of background as in Figure 2. Whenever a pair of numbers appeared, participants had to press the “left” or “right” key. Press the “left” key when the number on the left side is larger, and press the “right” key when the number on the right side is larger. And when the numbers were both even numbers, press the “up” key instead. The block was designed such that each participant would press the “left” or “right” keys 75% of the time, and the “up” key 25% of the time. Participants would not need to press “left” or “right” keys simultaneously when both numbers are even numbers. This kind of judging number problem is attributed to
Figure 1. Time line of experiment.
Figure 2. Inter-stimulus screen and stimulus screen.
Figure 3. Resting screen.
event-based PM (Burgess et al. 2001) . The test task has 30 trials. Before the actual test task, an additional 6 demo trials were presented for practice demo.
2.3. VR PM Task
For the immersive VR experiment, the lights in the room were turned off in order to avoid vision interference and the laboratory room was kept quiet. The system includes a computer with a desktop monitor and immersive VR device (Oculus Rift Development Kit 2, with 960 × 1080 per eye resolution). The VR environment was created with Unity 5.0. Desktop monitor is used for the researchers to observe the participants’ movement in the VR task. Participants press the forward and backward buttons on a controller to move forward or backward. Turning around in VR task required natural body rotation on a rotatable office chair. The environment consists of a shopping street with 12 shops as shown in Figure 4, two special action points and a bus station as the exit. By using the controller and making body rotation, participants moved in the VR shopping street, buying items and taking actions.
In the VR tasks, participants were asked to buy three items and take two special actions in the VR environment for PM task component. Shopping is regard as event- based PM, and the three items are “banana, coke and steak” as shown in Figure 5. Each item has both image and text to help participants to memorize it. The two special actions are event-based: “put a letter in the post box” and “throw a piece of trash into a dustbin” (Figure 5). They were presented in text form. As the ongoing task component, participants are required to press a button and then read aloud the number under the shop signboard as shown in Figure 6. When a participant reads out the number on the signboard, we expect that he can also read the name of the store and determine what is available inside it. It is up to the participant to choose whether to enter the store to view
Figure 4. The view of VR shopping street.
Figure 5. Shopping list and action list.
Figure 6. Example of a shopping menu in a coffee shop.
the shopping menu. Before the formal VR task begins, participants need to complete demo training, which have the same basic components as the formal task. However, the demo only has two shops and one special action point in the same VR environment.
As soon as the participant enters a shop, a shopping menu is shown. All items are presented as images. A yellow dot works as a selector and is controlled by the head movement of the participant. Item is purchased by aiming at the item with the selector and pressing a button on the controller. When an item is purchased, a text “bought!” displays under the item as a feedback message. When participants take special actions, they need to move to a special action point such as a mailbox or a dustbin to take the action. The process was the same as buying item, and a feedback message as text “accomplished!” will display under the action icon.
The information of the participants’ path, camera angle, buttons and timing during VR task were recorded automatically and outputted as CSV file form after finishing the test. To ensure the correctness of the ongoing task, their voices were also recorded. When experiment has finished, we asked the participants for permission to save the voice recording data.
In slide-based task, the score was determined whether a participant was able to press the correct key (incorrect or missed key is considered as an incorrect response). The RT of slide-based task was referred to the time period between stimuli had been shown on screen and the key had been pressed by participants.
In VR task, if a participant forgot to do the ongoing cue (read out signboard number) or do the PM actions, it would be marked as an incorrect response. For example, participant who clearly saw the number under the coffee shop signboard, but forgot to press button or read out the numbers is also considered as an incorrect response. The RT of VR task referred to the time period between seeing the menu and buying an item.
SPSS was used to analyze the experimental data. Speaman’s correlations were used to determine the association between slide-based task and VR task. Speaman’s correlations were performed not only between task performance (correctness, as task score) of slide-based task and VR task, but also between the RT of slide-based task and VR task. However, the RT of slide-based task has two types in data analysis section. 1) The whole block slide-based task RT that include both ongoing component (judge which number is greater) and PM component (judge whether numbers are even numbers); 2) the slide-based task RT only includes PM component. Comparison between the two tasks was performed with Two-Tailed Test, and if p < 0.05, statistical significance differences were reported as significant.
3.1. Slide-Based PM Task
By comparing the slide-based PM task score and RT, we found that there was no significant correlation between slide-based task score and whole block RT (r = −0.32, p = 0.37). Also the slide-based task score and PM component RT had no significant correlation (r = −0.29, p = 0.41). This result shows that performance has no relation with RT in slide-based task.
3.2. VR PM Task
With regard to VR PM task, the VR PM task scores was significantly negatively correlated with the VR PM task RT (r = −0.66, p < 0.05). This result indicates that the better the performance in VR task, the shorter the time will be cost in VR PM task RT.
3.3. Slide-Based PM Task vs. VR PM Task
Comparing the slide-based PM task and VR PM task scores, we found that there was no significant correlation between the scores of slide-based task and VR task (r = 0.22, p = 0.53). The experimental data shows that the slide-based task performance was not related to VR task performance as shown in Table 1.
To examine the relationship between slide-based PM task RT and VR PM task performance, the analysis shows that there was significant negative correlation between slide-based task whole block RT and VR task score (r = −0.81, p < 0.01). Comparing the PM component RT with VR task score shows significantly negative correlation(r = −0.79, p < 0.01). This result suggests that the shorter the RT cost in slide-based task, the higher the score in VR task.
With regard to RT, there was significant positive correlation between slide-based PM RT and VR PM task RT (r = 0.72, p < 0.05). Also, there was significant positive correlation between PM component RT and VR PM task RT (r = 0.67, p < 0.05). The result indicates that the slide-based task PM RT is correlated to the short time cost in VR PM task RT.
In PM studies, slide-based PM task was often used in evaluating PM in laboratory environment. Our study developed a system that can simulate the shopping experience in a VR shopping street and evaluate PM performance in daily life condition in a VR environment. Furthermore, thanks to the use of immersive VR device, the VR task environment is closer to real-life experience than previous studies.
For ongoing task control condition, participant was asked to read out the numbers when they can clearly see them under the shop signboard. This setup leads the participant to search for the perspective memory cue (name of shop), while he is performing the ongoing task. With this setup, we purpose that participant’s searching behavior can be shown by the VR task.
Our study compares the slide-based task and the VR task with the aim to investigate whether traditional laboratory PM task produces a similar result as PM task for daily life condition in VR environment, and the potential relevance between the two tasks. Based on our result, PM score performance in slide-based task is not relevant to the PM score performance in VR task. Previous studies have shown that slide-based task, as a
Table 1. The correlations between slide-based task PM and VR PM task.
*p < 0.05, **p < 0.01.
lab-based task, cannot fully reflect all the PM processes in real world condition (Titov & Knight, 2001; Dismukes et al, 2008 ). Although our immersive VR task is also lab-based, it provides a more realistic environment and may reflect more PM processes than the slide- based task. As a result, its score may not significantly correlate to the pure lab-based slide-based task. However, in our study, there is a strong negative correlation between the RT of slide-based task and the score performance of VR task. These results show that although slide-based task score performance cannot correlate with VR task performance, the RT of slide-based task is able to relate to the performance part of VR task.
By noticing-search model (Einstein & Mcdaniel, 1996) , noticing is an automatic process, but searching process is requiring attention-demanding control and processing efficiency is decided by search ability. RT of slide-based task may be considered as the time cost in searching process. In other words, participant who shows lower RT in slide-based task has a higher “target” search ability, and the one who shows higher RT in slide-based task has a lower “target” search ability. According to the cognitive resource theory (Kahneman, 1973) , the total of cognitive resource is limited. If the cognitive load cost increased exceeded the general degree of cognitive resource, than the cognitive resource that can be allocate is lesser, so the information process would be affected. In our study, since slide-based task (laboratory environment) requires less cognitive resource than VR task (VR real-world environment), the search ability is not as apparently reflected in task score performance. It could also explain why there was no significantly correlation between slide-based task score and RT. In VR task, we developed a system to simulate and test PM in daily life condition of VR environment. In this VR environment, the cognitive load of the participants was increased. It also consumed more cognitive resource than in laboratory environment. Therefore, in VR task, participants have less cognitive resource to allocate, and search ability differences are able to influence the VR task performance. In other words, by our VR task, we can observe how PM “target” search ability influences task performance. Compared with slide-based task, VR task can accurately assess PM in daily life. VR task RT, which refers to the time it takes to buy an item after the item list had been shown, can reflect PM “target” search ability. That is, the faster the VR task RT is, which means shorter “target” search time cost, correspond to better “target” search ability. In addition, the correlation between VR task score and RT also supported this finding.
This study used immersive virtual reality technology to create realistic shopping street experience in a controlled virtual environment. To test the task performance, we used a slide-based task to compare with the VR task.
Different from previous studies, our VR task not only highly simulated the real-life environment, but also considered the composition of the PM task. In particular, the VR task contains both ongoing component (reading out the numbers) and PM component (shopping and taking action). By comparing with the result of the slide-based task, it suggests that both VR task score and RT are related to the PM “target” search ability, and this has not been discussed in previous VR studies. Therefore, comparing to previous studies, our VR task is able to more reliably assess real-life PM ability in controlled laboratory setting.
In conclusion, these results suggest that comparison between PM laboratory psychological model and VR model is necessary. VR task can provide more detailed contrast with PM in real-life. Future study should apply neuroscience technology that may discover deeper potential relevance about laboratory, VR and real-world environment.