The Joystick Task

Video: Skinner answers some questions

2011-03-22T21:06:00.000Z

Where: youtube.com/user/sciinfo18
What: in front of an audience of psychologists and psychiatrists, Burrhus Skinner answers questions on a range of issues including: verbal behaviour, consciousness, the conflict between contingencies and 'reasons', gambling, rumours about his daughters, the placebo effect, accusations of fascism, why his work is largely atheoretical, why primary reinforcers can be maladaptive, the accusation that the concept of a reinforcer is circular and the difference between 'dehumanising' and 'dehomunculising'.
Thoughts: I haven't found many videos of Skinner on the Web and certainly none as long as this one. What is particularly interesting about this video is that Skinner responds to some of the criticisms that have been levelled against him, something he rarely did in writing (Palmer, 2006). It is also nice to get an insight into the personal side of someone who was often, unfairly, portrayed as cold. For example, he is very funny when answering the rumours that one of his daughters was raised in a Skinner box, became psychotic, tried to sue him and killed herself (suffice to say, none of these things happened). There is also a lovely anecdote which describes the folly of trying to experiment on one's own children; Skinner's children were too clever and too close to him to submit to silly psychological tricks. As he says, with a smile: "they turned out well for, perhaps, the wrong reasons".

References

Palmer, D. C. (2006). On Chomsky’s Appraisal of Skinner’s Verbal Behavior : A Half Century of Misunderstanding. The Behavior Analyst, 29(2), 253-267.

Announcement: thinking about a PhD?

2011-02-16T10:41:00.009Z

This is just a quickie to spread Tom Stafford's announcement to potential PhD students. In his words:

"If you want to do a PhD in Sheffield supervised by me, now is the time to let yourself be known!"

Tom is currently supervising my PhD and I highly recommend him to anyone wishing to do research in experimental psychology. If you're interested, don't be shy and make sure you get in touch using the details on his academic web page: http://www.shef.ac.uk/psychology/staff/academic/tom-stafford.html

Audio: Randy Gallistel

2010-09-01T19:02:00.000+01:00

Where: brainsciencepodcast.libsyn.com
What: an interview in which Ginger Campbell and Randy Gallistel discuss, in Campbell’s words, “why read/write memory is an essential element of computation with an emphasis on the animal experiments that support the claim that brains must possess read/write memory." Gallistel challenges the theory that changes in the connections between synapses in the brain are the primary way in which animals store information from experience in order to inform subsequent behaviour. He explains that some species of insect rely on dead reckoning to calculate their location relative to a specific starting position and that such calculations are much easier to perform with a read-write memory as opposed to a neural network. Gallistel claims that there is an “enormous price that you pay when you attempt to construct a computing machine without read-write memory."
Thoughts: in this interview, Gallistel gives us a flavour of a sophisticated argument about the necessary components of biological memory systems and suggests that we might benefit by adopting a new theoretical framework for conducting memory research. From the point of view of my own work, two things particularly stand out. On the one hand, we have developed an appreciation in the brain sciences for the idea that incoming sensory information is most useful to an animal when that animal has some kind of expectation about the information. This expectation, sometimes called a world model or schema, allows for large steps in performance that can be bewildering from the perspective of a researcher. On the other hand, we often explain behaviour within an associationist framework, despite the fact that this approach is particularly unsuitable for answering questions about the large amount of animal learning that occurs after little or no repetition of a particular behaviour. Indeed, as I mentioned in the previous post, one of the things that occupies of lot of my development time is trying to come up with tasks that can only be learned through repeated practice; the more I struggle with this problem, the more I believe that a new theoretical framework might be the very thing we need. Perhaps I should read Gallistel's book on the subject.

Paradigm: the serial reaction time task

2010-08-27T23:05:00.000+01:00

For research purposes, psychologists split human learning into many distinct processes, which share in common the effect of modulating future behaviour, but differ in terms of the conditions that are necessary for the learning to take place and the brain structures that are involved. Dividing learning into categories is compatible with some basic insights we might have from personal experience. For example, most people would agree that the experience of acquiring the ability to ride a bike is rather different from honing one’s verbal presentation skills through practice at public speaking. One of the challenges I face in my research into action learning, is the problem of designing learning environments in which action discovery employs only those learning processes or brain structures that are relevant to the theory underpinning my research, whilst not abstracting so far from normal behaviour as to make the research irrelevant. In particular, I struggle with trying to design tasks which stop my human participants from circumventing the learning processes of interest by using their high-level understanding of how the task works to complete experimental trials.

My own struggles have given me a great appreciation for the methodological triumphs achieved by other researchers. One such triumph, in my opinion, is the serial reaction time task (SRTT) (Nissen & Bullemer, 1987), an experimental paradigm designed to measure learning at different levels of attention. In a typical set-up, participants press one of four buttons on a keypad in response to visual signals that occur at one of four corresponding positions on a computer monitor. As the buttons are pressed, the time between the signal appearing on the monitor and the associated button being pressed is recorded. For most trials in a testing session the sequence of button presses is randomly determined so the participants can’t anticipate which button should be pressed next and therefore the speed of response is a genuine measure of their ability to react to the visual stimulus provided. However, in addition to the random trials, experimenters also include repeating sequences without telling the participants. It turns out that these non-random sequences can be learned (indicated by shorter reaction times) even when participants aren’t aware of the repetition, allowing for the investigation of learning in the absence of the high-level understanding which is such a problem in my research.

So there you have it, the wonderful serial reaction time task! The articles referenced below are recommended for anyone wanting to know more; in particular, Robertson (2007) provides a succinct introduction to the paradigm.

References

Nissen, M., & Bullemer, P. (1987). Attentional requirements of learning: Evidence from performance measures* 1. Cognitive psychology, 19(1), 1-32.
Robertson, E. M. (2007). The serial reaction time task: implicit motor skill learning? The Journal of neuroscience, 27(38), 10073-5.
Willingham, D. B., Salidis, J., & Gabrieli, J. D. (2002). Direct comparison of neural systems mediating conscious and unconscious skill learning. Journal of neurophysiology, 88(3), 1451-60.

Definition: action discovery

2010-07-08T22:41:00.004+01:00

A reduction of behavioural variance towards a stable sequence of movements as a result of prior reinforcement of similar sequences of movements.

Definition: action

2010-07-08T22:10:00.001+01:00

A stable sequence of movements that reliably produces specific changes in the environment or conveys situation relevant information.

Article: evidence of action sequence chunking in goal-directed instrumental conditioning and its dependence on the dorsomedial prefrontal cortex

2010-06-18T13:09:00.002+01:00

Ostlund, Winterbauer and Balleine (2009) tackle the tricky subject of action sequence chunking with a study designed to test whether, in rats, an action sequence can be treated as a single reinforceable entity (chunk) or whether each action component must be dealt with in isolation from the sequence as a whole.

The study involved two groups of rats: one with surgical destruction of the medial agranular cortex and another that underwent a sham surgical procedure. The rats were tested in two experiments. Experiment one involved no sequences and instead simply tested the ability of the rats to make decisions about which of two levers to press following devaluation of the outcome for one of the levers (by prefeeding with the food that the lever was contingent with). They found that both groups of rats had apparently learnt the consequences of pressing each lever to the extent that they showed a preference during an extinction session for whichever lever hadn't been devalued. In other words, damage to the medial agranular cortex appears not to interfere greatly with this type of goal directed behaviour. (For more on the devaluation paradigm and goal-directed learning have a look here).

The second experiment involved testing the rats on two simple sequences. Sequence one was to press the left-hand lever followed immediately by the right-hand lever in order to receive sucrose pellets. Sequence two was to do the opposite in order to receive polycose solution. The rats were then tested when one of the outcomes had been devalued (again, by prefeeding). The results showed that lesioned rats attempted to reduce the number of devalued outcomes by avoiding whichever lever was situated closest in time to the delivery of the food outcome; however, they did not avoid the whole sequence that was contingent with the devalued outcome. The healthy rats, by contrast, tended to avoid the devalued sequence as a whole. Lesioned rats were also insensitive to contingency degradation. That is to say, when one of the food outcomes was delivered non-contingently with respect to its previously associated sequence, the lesioned group failed to change their response preferences at the sequence level. The sham group, on the other hand, showed a reduced preference for the degraded sequence.

Thoughts

The concept of chunking has been applied to the human motor domain on many occasions and few would doubt that chunking of actions is an important aspect of animal behaviour. However this study goes further, showing that, to paraphrase Adams and Dickinson (1981, p.109), animals know about the consequences of action sequences, not merely the components within those sequences.

References

Adams, C. D., & Dickinson, A. (1981). Instrumental responding following reinforcer devaluation. The Quarterly Journal of Experimental Psychology, 33B(2), 109–121.
Ostlund, S. B., Winterbauer, N. E., & Balleine, B. W. (2009). Evidence of action sequence chunking in goal-directed instrumental conditioning and its dependence on the dorsomedial prefrontal cortex. The Journal of neuroscience : the official journal of the Society for Neuroscience, 29(25), 8280-7.

The power of variable-ratio reinforcement

2010-04-13T16:25:00.005+01:00

The ability to manipulate behaviour simply by changing schedules of reinforcement, has always amazed me. This quotation from Skinner (1961, pp. 106-107) gives some idea of just how powerful variable-ratio reinforcement can be...

We are all familiar with this schedule because it is the heart of all gambling devices and systems. The confirmed or pathological gambler exemplifies the result: a very high rate of activity is generated by a relatively slight net reinforcement. Where the “cost” of a response can be estimated (in terms, say, of the food required to supply the energy needed, or of the money required to play the gambling device), it may be demonstrated that organisms will operate at a net loss.

When the food magazine is disconnected after intermittent reinforcement, many responses continue to occur in greater number and for a longer time than after continuous reinforcement ... The potential responding built up by reinforcement may last a long time. We have obtained extinction curves six years after prolonged reinforcement on a variable-ratio schedule. Ratio schedules characteristically produce large numbers of responses in extinction. After prolonged exposure to a ratio of 900:1 (Figure 4) the bird was put in the apparatus with the magazine disconnected. During the first 4½ hours it emitted 73,000 responses.

References

Skinner, B. F. (1961). Cumulative Record. (Enlarged ed.) New York: Appleton-Century-Crofts. (Reprinted from Skinner, B. F. (1957). The experimental analysis of behavior. American Scientist, 45, 343-371).

Article: instrumental responding following reinforcer devaluation

2009-12-22T14:24:00.006Z

Adams and Dickinson get right to the heart of action-outcome learning with two experiments investigating whether information gained by an animal through exposure to a reinforcer in one context can influence behaviour related to that same reinforcer in a different context without the animal having to experience the reinforcer again; or, in their words, "whether or not animals know about the consequences of their actions" (1981, p.109).

Rats were trained to press a lever in return for a food reinforcer. They were also fed on a different kind of food which was not paired with lever pressing. Following this training phase, half of the rats had the contingent food devalued whilst the other half had the non-contingent food devalued; this was done by injecting them with lithium chloride (a toxic salt that causes sickness) following consumption of the relevant food. It was found that the number of lever presses performed during subsequent extinction trials was lower in the group of rats for whom the contingent food had been devalued. In other words, the rats' experience of the contingent food outside of the normal lever-pressing context affected their behaviour when they returned to the lever pressing situation even though they didn't experience the reinforcer again; apparently, this difference in behaviour depended crucially on the rats' knowledge of what lever pressing does.

Thoughts

The method sections are highly recommended for clear discussion of the need to minimise the effects of classical conditioning processes.

On a separate issue, writing this post has given me a taste for how much meaning is packed into the word 'know'. It is clear that Adams and Dickinson's explanations in terms of 'knowledge about actions' are far more concise than my attempt to frame the experiment in terms of the reinforcer.

References

Adams, C. D., & Dickinson, A. (1981). Instrumental responding following reinforcer devaluation. The Quarterly Journal of Experimental Psychology, 33B(2), 109–121.

What's so special about a Skinner box?

2009-12-04T22:42:00.009Z

An operant chamber (or ‘Skinner box’) is an apparatus used by psychologists to investigate animal learning, with a particular emphasis on schedules of reinforcement. Until a couple of days ago, I thought the benefits of the Skinner box paradigm were all fairly obvious: it reduces exposure to irrelevant stimuli, keeps the animal near the operandum (response lever or key), allows some behavioural freedom (the animal isn't restrained) and enables the researcher to automatically control all important parameters within the experiment. However, good though these reasons are, it turns out that there is a more sophisticated theoretical rationale. Skinner (1969) explains this by contrasting operant chamber paradigms with the learning experiments conducted by Thorndike (1911). In one such experiment, a cat learns over successive trials to escape from a box by pressing a foot pedal linked to a door release mechanism. Thorndike describes the learning process as one where all responses that are not relevant to escaping from the box are gradually "stamped out" whilst all actions resulting in pressing the pedal are "stamped in" (p.36). Skinner explains that operant conditioning paradigms are a refined version of Thorndike's experiments and that they allow for a more focused investigation of the relevant behaviour by stamping out irrelevant behaviours before the main part of the experiment begins:

By thoroughly adapting the rat to the box before the lever is made available, most of the competing behaviour can be “stamped out” before the response to be learned is ever emitted. Thorndike’s learning curve, showing the gradual disappearance of unsuccessful behaviour, then vanishes. In its place we are left with a conspicuous change in the successful response itself: an immediate, often quite abrupt, increase in rate (1969, pp.6-7).

References

Skinner, B. F. (1969). Contingencies of reinforcement. New Jersey: Prentice-Hall.
Thorndike, E. L. (1911). Animal intelligence: experimental studies. New York: The Macmillan Company.

Definition: law of effect

2009-11-30T20:20:00.003Z

A rule employed to explain the process of action learning in behavioural terms. It equates to the hypothesis that actions which result in pleasant sensations are more likely to be repeated and actions that result in unpleasant sensations are less likely to be repeated.

On this occasion we can pin the term on a single person: the psychologist Edward Lee Thorndike. His explanation of the principle is as follows:

The Law of Effect is that: Of several responses made to the same situation, those which are accompanied or closely followed by satisfaction to the animal will, other things being equal, be more firmly connected with the situation, so that, when it recurs, they will be more likely to recur; those which are accompanied or closely followed by discomfort to the animal will, other things being equal, have their connections with that situation weakened, so that, when it recurs, they will be less likely to occur. The greater the satisfaction or discomfort, the greater the strengthening or weakening of the bond (1911, p.244).

References

Thorndike, E. L. (1911). Animal intelligence: experimental studies. New York: The Macmillan Company.

Volitional control of actions and decisions

2009-11-13T16:54:00.003Z

When studying action-outcome learning, it is not uncommon to stumble on the issue of free will. The scientific literature gives us little reason to believe in free will but in spite of this most sane people do. The philosopher John Searle vividly captures how counter-intuitive it is to view consciousness as having no power to cause actions:

Every time I decide to raise my arm, it goes up. And it is not a random or statistical phenomenon. I do not say, "Well, that's the thing about the old arm. Some days she goes up and some days she doesn't." (2004, p.30)

And in this scenario, featuring a person choosing between pork and veal in a restaurant, he shows how difficult it is to imagine making a decision without assuming free will:

Look, I am a determinist—que sera sera, I’ll just wait and see what I order (2002, p.494).

References

Searle, J. R. (2002). Free will as a problem in neurobiology. Philosophy, 76, 491-514.
Searle, J. R. (2004). Mind: A Brief Introduction (Fundamentals of Philosophy). Oxford University Press.

Article: perspectives and problems in motor learning

2009-11-13T16:18:00.004Z

Wolpert, Ghahramani and Flanagan give an introduction to the subject of motor learning. They cover internal models, dynamic systems theory, types of motor learning (supervised, reinforcement and unsupervised), neural correlates, the challenges of motor control and some of the learning algorithms that have been used in computational models of motor learning. They also provide a nice reminder of how important the topic is:

Movement is the only way we have of interacting with the world. All communication, including speech, sign language, gestures and writing, is mediated via the motor system. All sensory and cognitive processes may be viewed as inputs that determine future motor outputs (Wolpert, Ghahramani & Flanagan, 2001, p.487).

Thoughts

The article is highly recommended as an introduction to motor learning; it clearly and succinctly covers lots of important issues. I particularly enjoyed the section contrasting reinforcement learning with supervised learning. They make the theoretical dissociation clear and their examples show just how difficult it is to come up with scenarios that irrefutably invoke one kind of learning but not the other.

References

- Wolpert, D. M., Ghahramani, Z., & Flanagan, J. R. (2001). Perspectives and problems in motor learning. Trends in Cognitive Sciences, 5, 487-494.

Definition: reward

2009-11-02T22:08:00.002Z

Something that elicits approach behaviour in a non-stressed, non-satiated animal irrespective of context.

Article: roles for nigrostriatal - not just mesocorticolimbic - dopamine in reward and addiction

2009-11-02T15:55:00.001Z

Roy Wise argues that the tradition of treating dopamine systems involving the ventral tegmental area and the substantia nigra as being functionally distinct with respect to reward is not supported. He points to experiments involving the direct electrical and chemical stimulation of these areas and the resulting effects on reward and reinforcement related behaviour in animals. Of particular interest to the topic of action-outcome learning is the section on reward prediction; Wise gives the following, satisfyingly confident, conclusion: "SN and VTA dopamine neurons share the same qualifying characteristics for whatever role dopamine neurons play in reward function" (p.521).

Thoughts

This is a useful reference point when looking for articles concerning the function of dopamine pathways. It is clear on the the separation of reward and reinforcement and provides measured interpretations of experimenal findings. For me, the article highlighted the difficulty in assigning specific functions to areas of the midbrain. For example, Wise indicates that there have been cocaine self-administration experiments involving animals with lesions to the ventral tegmental area but not the substantia nigra. He suggests that this might be because complete lesions of the latter "render animals aphagic [unable to eat], adipsic [insensitive to dehydration] and akinetic [unable to initiate movement]" (p.520). In other words, there are profound functional differences between these two midbrain areas and yet their dopamine neurons appear to behave in very similar ways.

References

- Wise, R. A. (2009). Roles for nigrostriatal - not just mesocorticolimbic - dopamine in reward and addiction. Trends in Neurosciences, 32, 517-524.

Video: Joshua Klein

2009-11-01T00:11:00.002Z

Where: crowboxunleashed.com
What: in this TED talk Joshua Klein discusses crow intelligence, his crow vending machine and how we might live more harmoniously with animals currently treated as vermin.
Thoughts: there are some nice examples of operant conditioning, insight learning and cultural learning combining to produce sophisticated patterns of behaviour. I'd love to see Klein implement his idea of using animals such as crows and rats in ways that would benefit both man and beast.

Dreams of a brave new world

2009-10-31T23:35:00.003Z

Aldous Huxley anticipating the strength of conviction held by behaviourists such as Burrhus Skinner:

As if anyone believed anything by instinct! One believes things because one has been conditioned to believe them (p.207).

References

- Huxley, A. (2007). Brave new world. London: Vintage. (Original work published 1932)

Article: search in external and internal spaces: evidence for generalized cognitive search processes

2009-10-30T10:26:00.003Z

Hills, Todd and Goldstone (2008) reasoned that if cognitive and spatial foraging both rely on the same underlying neural architecture then it should be possible to find behavioural similarities between the two. For example, if an individual has a foraging style that is peculiar to them, such as the tendency to perseverate, we would expect this to be the same whether they are doing cognitive or spatial foraging. Furthermore, because animals often adapt their behaviour to whatever resource distribution they are faced with, we might expect that engaging in one kind of foraging would result in a behavioural after-effect when engaging in the other kind of foraging.

Their experiment required participants to complete a computer based spatial search for resources that were either patchily or evenly distributed. This acted as a priming condition for the subsequent cognitive foraging task in which participants were required to make as many words as possible from a set of letters in a scrabble-like game. Each set of letters represented a patch and a between-patch delay was imposed when requesting a fresh set of letters, analogous to the between-patch travel time in food foraging. They found that those who had searched in the patchy spatial environment persisted with letter sets for longer. These people also showed longer giving-up times; in other words, they waited longer following their last correct word submission before requesting a new set. In addition to this, they found that the foraging habits of individuals were consistent between the tasks so, irrespective of whether they had experienced the patchy or even-spread spatial task, an individual's tendency to explore more in spatial search was mirrored by a tendency to explore more in the scrabble task.

Thoughts

This experiment is a bit of a departure from the learning literature I normally cover. It is relevant because it deals with simple goal-directed behaviour and behaviour that is controlled, to some degree, by dopamine. However, I am interested in this kind of study because it highlights a difference between search behaviour and action-outcome learning. When the former is studied in the laboratory, the experimental task is directly comparable to the natural challenges a wild animal would face. By contrast, when scientists study the latter, the task demands often bear no resemblance to those in real survival situations. Whilst I can accept that a laboratory study must create a caricature of normal behaviour, it appears that some learning research has gone too far because there is no attempt to interpret laboratory findings in terms of the natural behaviour animal brains were selected for.

References

- Hills, T., Todd, P.M., Goldstone, R.L. (2008). Search in external and internal spaces: evidence for generalized cognitive search processes. Psychological Science, 19, 676-682.

Article: reward or reinforcement: what's the difference?

2009-10-20T16:20:00.001+01:00

In this review, Norman White attempts to draw an anatomical distinction between reinforcement and reward. In the abstract he refers to rewards as stimuli which "have the property of eliciting approach responses" (p.181) and reinforcement as "the tendency of certain stimuli to strengthen learned stimulus-response tendencies" (p.181). There is an attempt to draw a behavioural distinction by referring to the 'spread of effect' phenomenon (the finding that rewarding a response has the effect of reinforcing other temporally contiguous but non-contingent responses). In other words, if events occur close together in time but are otherwise unrelated to a stimulus-response association, some of the memorable nature of the stimulus-response association will still rub off on them. White suggests that this could mean the action of reward is dissociable from the association-strengthening aspect of learning because the temporally contiguous events aren't themselves rewarded. The section dealing with the effect of reinforcement on physiology is convincing in suggesting that not only is dopamine involved in improving memory during post-training periods but its effects are markedly different depending on which area of the striatum is targeted. For example, by targeting specific areas, experimenters can improve memory for visual associations but not olfactory and vice versa. The related section dealing with the physiological basis of the effects of rewards suggests that we can't dissociate reinforcement and reward in terms of the involvement of dopamine but that they appear to differ in terms of the area of the striatum that is most important. The final section covers the anatomical correlates of reinforcement and reward with White suggesting that the striatal matrix and neostriatal patch system are implicated respectively.

Thoughts

White's discussion of the spread of effect was surprising as such a phenomenon would seem to tell us more about the problems we have with deciding what counts as an event than it does about the differences between reward and reinforcement. The discussion of the role of dopamine in reinforcement learning and the marked differences between different areas of the striatum in terms of how they mediate learning is very interesting. Whilst I was excited by the idea that reinforcement and reward might be dissociable anatomically, White's treatment of this issue in the final section was, to me, almost unintelligible; I'm sure this is due to my limited understanding of the relevant neuroanatomy because I found his use of technical vocabulary to be quite merciless.

References

- White, N. M. (1989). Reward or reinforcement: what's the difference? Neuroscience and Behavioural Reviews, 13, 181-186.

Article: animal foraging and the evolution of goal-directed cognition

2009-10-14T22:40:00.010+01:00

Thomas Hills gives an account of the evolution of goal-directed cognition from its behavioural precursors. The argument moves from area restricted search in invertebrates to more abstract resource acquisition learning such as operant conditioning and finally to the kind of highly abstracted internal search that humans engage in as we search our memories for concepts and ideas. He further suggests that dopamine could be a common driving force behind these behaviours. Because he is attempting to tell an evolutionary story, it is extremely difficult to give a short yet satisfactory summary of the paper. However, from the perspective of action-outcome learning, perhaps the most important aspect is the discussion of the changing role of dopamine over evolutionary time:

The evolutionary theory described here is therefore completely consistent with the reward theory of dopamine, but adds the evolutionary hypothesis that the initial rewards represented by the release of dopamine were food. Only later was this system co-opted to represent the expectation of a reward, which allows for goal-directed cognition (p.16).

Thoughts
Hills is an original thinker and this is a refreshing new approach to the topics of dopamine and learning, areas that have seen increasingly repetitive contributions over the past few years. His discussion of the connection between dopamine and food is extremely well argued and has left me wondering whether any form of associative learning can be truly divorced from the provision of rewards... outside the special case of juvenile animals, this seems unlikely.

References

- Hills, T. (2006). Animal foraging and the evolution of goal-directed cognition.Cognitive Science, 30, 3-41.

Article: contiguity and contingency in action-effect learning

2009-10-11T21:33:00.004+01:00

Elsner and Hommel investigated action-outcome learning in the form of two experiments within which acquisition of an association and production of an action were separated by the use of learning and testing phases. Participants implicitly learned to associate key presses with particular tones: they performed a simple task and were exposed to the tones but were told that the sounds were not relevant. Following this, they were tested on the ability to press a key in response to the tones. The testing phase was split into two conditions: a consistent condition required that the buttons were pressed in a manner consistent with the learning phase; in the inconsistent condition the keys were effectively swapped such that the participants were required to press the relevant key with the opposite hand to that used in the learning phase. In the first experiment feedback delay was manipulated such that the sound of the tone following a correct key press was delayed by 50ms, 1000ms or 2000ms. The results from the testing phase show differences in mean reaction time between the inconsistent and consistent groups at 50ms and 1000ms delay but not at 2000ms delay. The 50ms and 1000ms groups did not differ significantly from one another in terms of the level of the contrast found. The second experiment involved the manipulation of the probability of receiving or not receiving the relevant tone depending on whether or not a key was pressed (in a go/no-go task). It was found that it is important for a particular tone to occur as little as possible when its related action is not being performed.

Thoughts

The findings for the feedback delay experiment are notable because it didn't rely on the rate of responding or other similar performance measure, instead the emphasis was on creating a contrast during the test phase. Interestingly, it wasn't necessary to learn the association during the acquisition phase in order to perform the task during the testing phase but responses during the testing phase were affected by the associations developed during the acquisition phase. The experiment is, therefore a relatively pure measure of association. However, I am not sure whether this is good or not in terms of interpreting the findings with respect to feedback delay because using the technique appears to have resulted in a relatively small effect. If we accept it as a good measure of the effect of feedback delay on action-outcome learning, this places the critical delay length at somewhere between one and two seconds. It would be nice to see if the same experiment would yield a significant difference between the 50ms and 1000ms conditions with larger group sizes (i.e. more than eight people per group... I should note that the authors also acknowledge this as a potential limitation).

References

- Elsner, B. & Hommel, B. (2004). Contiguity and contingency in action-effect learning. Psychological Research, 68, 138-154.

Article: reinforcement delay of one second severely impairs acquisition of brain self-stimulation

2009-10-07T21:56:00.005+01:00

Black, Belluzzi and Stein recognised two problems with investigations into delayed reinforcement. Firstly, food is normally used as a reinforcer and this takes time to be delivered and consumed thus affecting the length of delay being imposed. Secondly, all sorts of things can become associated with the arrival of reinforcement over the course of training such as the sound of food delivery; as they put it, these stimuli "can bridge the delay period by providing interpolating secondary reinforcement" (p.113). They attempted to avoid these issues by providing rats with reinforcement through brain self-stimulation, provided at various delays following a lever press. As the title of the article suggests, they found that delaying reinforcement by one second has a large and significant impact on the number of lever presses.

Thoughts
With such a direct delivery of reinforcement we can be confident about the accuracy of the delay being imposed. With hindsight, however, it is disappointing that reinforcement delays between zero and one second weren't investigated. From the data provided, it seems likely that delays of considerably less than one second would have produced a large reduction in the acquisition of the behavioural response. Nonetheless, this finding gives us good reason to believe that, in the absence of bridging signals, reinforcement is required within one second of producing the relevant behaviour for normal learning to occur.

References

- Black, J., Belluzzi, J. D. & Stein, L. (1985). Reinforcement delay of one second severely impairs acquisition of brain self-stimulation. Brain Research, 359, 113-119.

Article: effects of delayed visual information on the rate and amount of prism adaptation in the human

2009-10-06T22:37:00.008+01:00

It is generally accepted that some movements must rely on feedforward control because they occur at latencies that are too short to allow for control based on sensory feedback. That is to say, the inherent delay due to transmission and processing of sensory information in the body is too great for that information to provide immediate feedback during the course of our fastest movements. In the hope of gaining an understanding of the underlying neural mechanisms involved in motor adaptation, Kitazawa, Kohno and Uka (1995) asked: given that we can cope with an intrinsic physiological feedback delay, how much more delay can we cope with? They used a paradigm involving rapid reaching movements and measured adaptation of these movements when visual feedback was distorted using prisms. They found that feedback delays as short as 50ms have a detrimental effect on prism adaptation and, additionally, that conditions involving substantially longer delays, up to 5000ms, do not differ appreciably from the 50 ms trials. They conclude that adaptation must rely on feedback occurring within 50ms of making a movement. Furthermore, they note that since adaptation occurred in the longest delay conditions, there are probably two brain processes dealing with adaptation.

Thoughts

This is a clever experiment with an interesting finding. That people should be sensitive to delays as short as 50ms is very surprising, given that visual saccades don't occur until around 100ms (Hikosaka & Wurtz, 1983). The question is: what kind of theory does such a result inform? At first it seems that this experiment doesn't test action-outcome learning (after all, it wasn't intended to) because the participants are not required to discover a particular action; they already know how to perform the correct reaching movement. Instead they are required to improve the performance of a familiar action which has been deliberately displaced. However, adaptation in this kind of task could be compared to the gradual learning of new actions we see in operant conditioning or more basic forms of action-outcome learning. It is surely important, therefore, to be able explain where the difference lies. That is to say, why do we view one task (such as the present experiment) as 'motor control', probably involving the cerebellum, whereas we view another as 'motor learning', probably involving the basal ganglia?

References

- Hikosaka, O. & Wurtz, R. H. (1983). Visual and oculomotor function of monkey substantia nigra pars reticulata. I. Relation of visual and auditory responses to saccades. Journal of Neurophysiology, 49, 1230–1253.

- Kitazawa, S., Kohno, T. & Uka, T. (1995). Effects of delayed visual information on the rate and amount of prism adaptation in the human. The Journal of Neuroscience, 75, 7644-7652.

Article: dialogues on prediction errors

2009-10-05T23:10:00.005+01:00

This is basically a really nicely polished email transcript composed of ten questions and the corresponding answers regarding prediction errors in animal learning. Lots of emphasis is placed on contrasting the Rescorla-Wagner model of classical conditioning with temporal difference models of learning. Having said that, the article includes the other main issues such as the search for the neural correlates of prediction error detection using single cell recordings and brain imaging.

Thoughts

The short answers make this an excellent article for quick reference regarding prediction errors. In particular, the description of the models and their relative merits is easy to understand and concise. Perhaps the most satisfying aspect is the even-handed coverage of the competing theories regarding the activity of dopamine neurons in the midbrain (question six); in their opinion, this issue is still up for grabs.

References

- Niv, Y. & Schoenbaum, G. (2008). Dialogues on prediction errors. Trends in cognitive sciences, 12, 265-272.

Video: Paul Glimcher

2009-10-04T13:24:00.004+01:00

Where: cns.nyu.edu/~glimcher/GlimcherSokol_web.mov (requires QuickTime player)
What: lecture giving an overview of the work in Glimcher's lab which centres on their attempts to use findings from neuroscience to inform our understanding of economics. The experiments covered all probe decision making in monkeys and humans. Of most relevance to reinforcement learning is Glimcher's description of his work with Hannah Bayer. They have extended the findings of Wolfram Schultz, and others, concerning neurons in the ventral tegmental area and the substantia nigra pars compacta and the role of these neurons in the detection of reward prediction errors.
Thoughts: a useful overview of some of the work from Glimcher's lab. The emphasis of the lecture is on how the laboratory studies can ultimately contribute to our understanding of human decision making in complex real-world scenarios. However, I was left unconvinced that we are close to bridging the gap between measurements of neuron activity in tasks that are iterated hundreds of times and the kind of decision scenarios that economists are normally interested in. For me, the value of the work is much easier to appreciate when it is framed in terms of our understanding of basic learning and valuation processes with less emphasis on potential behavioural applications for humans.