The effect of the Phonological Store material on an Auditory Control task Ruhi Dang, Department of Biological Sciences and Bioengineering, IIT Kanpur Abstract -- The human working memory system consists of 2 parts, a visual-spatial 
sketchpad and a verbal loop, often called the phonological loop. The phonological loop is further known to be constituted of a phonological store that maintains temporary traces of auditory material and an articulatory rehearsal component that aids in maintaining these traces. The present study aims at understanding the influence of the material stored in the Phonological Store in the form of Phonological codes on articulatory tasks. The results show that the phonological store material does influence auditory tasks (in this case, an auditory search task). It also reveals the automatic diversion of overt attention towards objects that are stored in the phonological store and its influence on the search task. The current study will help enhance our understanding of the articulatory working memory. This understanding is, in turn, crucial for duplicating systems for computational purposes and formulating theories about the auditory working memory. Keywords: Phonological Store, Phonological loop, working memory, articulatory task, audio memory INTRODUCTION Baddeley proposed a model of the working memory consisting of a central executive and 3 slave systems: the phonological loop, the episodic Buffer and the Visuo-Spatial Sketchpad dedicated to articulatory, episodic and visual memory and related processing respectively [1] . According to Baddeley(1986,1992) and Sperling(1967), the phonological loop is formed of 2 components: the Phonological and the Articulatory Rehearsal Component. The functional independence of these 2 components is established from neurological data and memory experiments [4;7] . For example: the phonological similarity effect* and the word length effect** have been shown not to interfere with each other. This proves the functional independence of the components as the phonological similarity effect is based on the Phonological Store whereas the word length similarity effect is based on articulatory rehearsal. Given the evidence that the two components of the Phonological Loop are functionally independent, a task involving one should not be influenced by the contents of the other. However, any articulatory task will require storage of the audio input in the Phonological Store and further rehearsal and processing requires the articulatory control component. Thus, it is hypothesized that the Phonological Store material will influence an Auditory Control task. To test the hypothesis, an experiment was designed that included an auditory memory task and an auditory search task. The findings of the experiment were analyzed to form a model of the auditory working memory. PREVIOUS WORK Several effects that have been shown to be important with respect to the Phonological Loop are [10] : *Phonological similarity effect: Similar sounding words are difficult to recall as compared to different sounding words owing to less number of distinctive features during storage in the Phonological Store for similar sounding words and hence, leading to difficult retrieval. **Word Length Effect: The longer the word length, the more difficult is their storage and retrieval. This is because the rehearsal in the the Articulatory Rehearsal Component will take a longer time and hence poorer performance. Phonological similarity effect: Similar sounding words are difficult to recall as compared to different sounding words owing to less number of distinctive features during storage in the Phonological Store for similar sounding words and hence, leading to difficult retrieval. Word Length Effect: The longer the word length, the more difficult is their storage and retrieval. This is because the rehearsal in the the Articulatory Rehearsal Component will take a longer time and hence poorer performance. Articulatory Suppression: Uttering of irrelevant words leads to disruption of the rehearsal process. Several experiments and neuropsychological data have verified the functional independence of the components of the Phonological Loop [4;7] . Previous work for the visual working memory has shown that changes can be detected by means of an unlimited-capacity comparison process, which can be used to direct covert and overt attention but that manual responses depend on a limited-capacity process. Finally, it has shown that the unlimited-capacity comparison process can be limited to specific feature dimensions [9] . Similar aspects pertaining to the auditory working memory are explored in the present work. METHODS AND MATERIALS The following experiment was designed with Php. The code and the experiment can be viewed on the following links, respectively: http://home.iitk.ac.in/~ruhidang/se367/project/code.htm http://home.iitk.ac.in/~ruhidang/se367/experiment/. The experiment consists of a practice test followed by 15 test trials. The practice test is added to get the subject acquainted to the test and to the accent involved in the test. There are 3 categories in the test each further being divided into 5 different kinds of test trials. The test consists of clip1 which contains words or sounds to be remembered, followed by a beep, this being followed by clip2 which is a search sequence and then a beep followed by a single word or sound. The subject has to identify whether the last word/sound in the entire audio was contained in clip2 i.e. the search sequence or not by pressing a yes or a no button correspondingly. Finally, the subject is asked to recall the words/sounds of clip1 that were to be remembered. Figure1. The sequence of the audio clip presented to the subject The 3 categories and their sub parts are as follows: Set A: Clip1 includes sequence of meaningful English words like play, book, eat etc. Clip2 includes meaningful sentences like she went to the park, he likes red flowers etc. Clip3 is a meaningful English word like sleep, table etc. Set B: Clip1 includes sequence of nonsensical words like rheoni, parishink etc. Clip2 again includes sequence of nonsensical words like straik, direngi etc. Clip3 is a nonsensical word. Set C: Clip1 includes sequence of meaningful English words like play, book, eat etc. Clip2 includes sequence of meaningful words like play, book, eat etc. Clip3 is a meaningful English word like sleep, table etc. The purpose of having these three categories is to check the effect of semantics on storage of auditory data and its processing and retrieval. Set A involves high semantic content, Set C involves slight semantic content and Set B involves no semantic content. Hence, this gradation will aid in understand the extent to which semantics influence the working memory processing. Also, each set has 5 sub parts as follows: 1. Clip2 contains a word that is similar in meaning(Set A)/sound(Set B,C) to one of the words of Clip1 and is also the target word 2. Clip2 contains words/sounds that are distinct from the ones in Clip1 3. Clip2 contains a word that exactly same as one of the words of Clip1 but is not the target word 4. Clip2 contains a word that exactly same as one of the words of Clip1 and is also the target word 5. Clip2 contains words/sounds that are distinct from the ones in Clip1 but the target is similar sounding to one of the words of Clip2 The purpose of having these 5 sub parts to each set is to check the effects of automatic diversion of attention and the relationship between the two components of the auditory working memory i.e. the store and the rehearsal component. Parts 3,4 check the attention aspect, Part 1,5 check the storage of material in the form of phonological codes and its consequences and Part 2 is a control for the experiment. Figure2. Plot of response time in seconds versus the test trial number averaged for all the 12 subjects The three groups are marked as follows: 20 subjects were made to sit for the experiment with a pair of earphones. The 15 test trials were given to the subject in random order to avoid any biasing. Their response time with the accuracy up to micro seconds was noted using the same tool developed (the php code). The results were filtered based on correctness of response, coherence of response times and completion of the experiment. This narrowed down the results to 12 subjects. The results were organized as an Excel database and further analyzed. RESULTS AND DISCUSSION The plot of time v/s test trial number along with the standard deviation is given in Fig2. 1-5 stand for set A, 6-10 for set B and 11-15 for set C in the same order as given under the methods and materials section. Figure3. Plot of response time in seconds versus the test trial number averaged for all the 12 subjects showing the three sets of questions differentiated by vertical lines The plot very clearly shows the striking difference in response times in different tests within each group. The response times of different trials are clearly either in the above range or the lower range. 4 memory. Storage and hence processing of this extra stored information i.e. semantics takes a longer time hence, a larger response time. It also indicates that verifying meaning is much more difficult than verifying sounds Figure4. Plot of response time in seconds versus the test trial number averaged for all the 12 subjects showing the significant difference in response times across various test trials A striking fact to note is the coherent trend observed which is marked as follows. Figure5. Plot of response time in seconds versus the test trial number averaged for all the 12 subjects showing striking trend across the test trials Discussion: Questions 1,6,11 have response time higher than 2 nd question in set B, C and very high in set A. These questions correspond to Clip2 containing a word that is similar in meaning (Set A)/sound (Set B, C) to one of the words of Clip1 and is also the target word. For set B and C, it can be explained from the assumption that similar sounding words lead to less confidence in the response and hence, requires verification before the motor responses needed. This arises from the theory that auditory input is stored in the form of phonological codes and similar codes interfere with each other. Set A shows meaning is stored along with the phonological codes even when processing is occurring for a very short span i.e. even at the level of working Questions 2,7,12 have the least response time in the respective sets. These questions correspond to Clip2 containing words/sounds that are distinct from the ones in Clip1. This shows that lack of interference in phonological codes leads to faster response hence, implying ease of search task. Questions 5,10,15 have very low response times in sets B (Q10) and C (Q15) but comparatively higher in set A (Q5). These questions correspond to Clip2 containing words/sounds that are distinct from the ones in Clip1 but the target is similar sounding to one of the words of Clip2. Intuitively, these questions should have taken greater response time due to confusion arising from similar phonological cues. We note here that the accuracy (shown later) is very low in Q10 and Q15 so this observation is discarded. Question 5 has high accuracy and hence this result is valid. Q5 has a greater response time owing to verification of the similar sounding word. Also note that this time is less that of Q1. This shows more time is required for verifying semantics rather than sounds. This implies that storage occurs as both, semantics and phonological codes even in the working memory. However, an interesting point to note here is that if semantics storage is hindering the process of search task (evident from the greater time), why then does this occur. One reason could be the importance of semantic storage in other contexts such as reading where you have to interpret what you are reading at the same time not store it in the long term memory but in the working memory itself. Also, this aspect can be deliberately masked. This can be seen from a simple task: Ask a friend to find a certain word in a paragraph as quickly as possible. After this, ask the person about the contents of the paragraph. In more than 90% of the cases, the person will not be able to describe the contents as he/she did not store them while searching the paragraph Questions 3, 4; 8, 9; 13, 14 correspond to Clip2 containing a word that exactly same as one of the words of Clip1 but is not the target word and is the target respectively. Set A: Questions 4 have is extremely low response time. This suggests that the storage of word diverts automatic attention when it is heard again and hence a lesser response time. Response time of Q3 is high because unnecessary time is wasted on the repeated word because of diversion of attention towards it whereas it is not the search word. Set B and C: Questions 9, 14 take more time than Q8, 13 respectively and all 4 have comparatively greater response times. This is because more time is spent when the word is repeated due to diversion of attention. Also, sounds seem to be a lesser reliable source of verification leading to lesser confidence and more verification time requirement as opposed to that in the case of semantics. The plot of accuracy v/s test number is given as follows; the higher the number, the greater the accuracy. where the clip2 contains one of the words in clip1 leading to a rehearsal of the stored word and hence, a better recall. Rest of the questions show average accuracy as expected in all groups out of which 3 rd question of each group is more accurate than 4 th . This is explained by the assumption that more processing is required in case of repeated word being target and hence is a little less accurate as compared to 3 rd question. CONCLUSIONS The present study has explored several new aspects of the phonological working memory and the experiment done has revealed a number of aspects of this memory. The experiment conducted utilizes the simultaneous storage of material in the phonological store along with an articulatory search task to explore the articulatory working memory. The aspects of the auditory working memory as shown by this experiment are: Figure6. Plot of score as a measure of correctness of response versus the test trial number averaged for all the 12 subjects Discussion: Questions 1; 6, 10; 11, 15 have comparatively lower accuracies. This indicates a replacement of previously stored phonological cues when other similar cues are processed. As explained earlier, this is due to the interference of phonological cues. Except question 5 that shows high accuracy. This could possibly arise from the ease of detection in sentences which are semantically associated. Questions 2, 12 have high accuracies as expected except for Q7 which may arise because of difficult to understand accent of nonsensical words. Questions 8, 9 and 13, 14 show relatively higher accuracy. These questions correspond to the trials ? Auditory input is stored in the form of phonological codes and similar codes interfere with each other. ? In case of semantic or sound similarity of two words, an additional verification step is involved in order to be sure of a response. ? Verifying meaning is much more difficult than verifying sounds. ? Storage in the working memory also includes a dimension of semantics despite the fact that it hinders the search tasks. However, this aspect can be deliberately masked. ? People tend to associate meanings even with nonsensical words in order to remember them. This was evident from the responses in set B. ? The storage of word diverts automatic attention when it is heard again. REFERENCES 1. A.D. Baddeley and G.J. L Hitch. Working Memory. The psychology of learning and motivation: advances in research and theory New York: Academic Press, 8:47–89, 1974. 2. A.M. Longoni, J. T, E. Richardson and A. Aiello. Articulatory rehearsal and phonological storage in working memory. Memory & Cognition, 21( 1):11-22 1993. 3. Edward Awh, John Jonides, Edward E. Smith, Eric H. Schumacher, Robert A. Koeppe and Stewart Katz. Dissociation of Storage and Rehearsal in Verbal Working Memory. American Psychological Society, 7:25-31, 1996. 4. Cherie L. Marvel & John E. Desmond Functional Topography of the Cerebellum in Verbal Working Memory. Neuropsychol Rev, 20: 271-279, 2010. 5. Pierre Salami, Strasbourg and Alan Baddeley. Disruption of Short-Term Memory by Unattended Speech: Implications for the Structure of Working Memory. Journal of verbal learning and verbal behavior, 21:150- 164, 1982. 6. Pia Rama. Domain-dependent activation during spatial and nonspatial auditory working memory. Cogn Process, 9:29–34, 2008. 7. Michael Petrides, Bessie Alivisatos, Ernst Meyer and Alan C. Evans.Functional activation of the human frontal cortex during the performance of verbal working memory tasks. Neurobiology, 90:878-882 ,1993. 8. Daniel J. Acheson and Maryellen C. MacDonald.Verbal Working Memory and Language Production: Common Approaches to the Serial Ordering of Verbal Information. American Psychological Association, 135:50-68, 2009. 9. Joo-seok Hyun, Geoffrey F. Woodman, Edward K. Vogel, Andrew Hollingworth and Steven J. Luck. The Comparison of Visual Working Memory Representations with Perceptual Inputs. Journal of Experimental Psychology, 35:1140-1160, 2009. 10. A .D. Baddeley, N. Thomson and M. Buchanan. Word length and the structure of short-term memory. Journal of Verbal Learning and Verbal Behavior, 14: 575–589, 1975. 1 2 3 5 6
