- SharpBrains - http://sharpbrains.com -

BBC “Brain Training” Experiment: the Good, the Bad, the Ugly

Posted By Alvaro Fernandez On April 20, 2010 @ 10:50 In Education & Lifelong Learning,Health & Wellness | Comments Disabled

You may already have read the hun­dreds of media arti­cles today titled “brain train­ing doesn’t work” and sim­i­lar, based on the BBC “Brain Test Britain” experiment.

Once more, claims seem to go beyond the sci­ence back­ing them up … except that in this case it is the researchers, not the devel­op­ers, who are responsible.

Let’s recap what we learned today.

The Good Sci­ence

The study showed that putting together a250px-ClintEastwood [1] vari­ety of brain games in one web­site and ask­ing peo­ple who hap­pen to show up to play around for a grand total of 3–4 hours over 6 weeks (10 min­utes 3 times a week for 6 weeks) didn’t result in mean­ing­ful improve­ments in cog­ni­tive func­tion­ing. This is use­ful infor­ma­tion for con­sumers to know, because in fact there are web­sites and com­pa­nies mak­ing claims based on sim­i­lar approaches with­out sup­port­ing evi­dence. And this is pre­cisely the rea­son Sharp­Brains exists, to help both con­sumers (through our book [2]) and orga­ni­za­tions (through our report [3]) to make informed deci­sions. The paper only included peo­ple under 60, which is sur­pris­ing, but, still, this is use­ful infor­ma­tion to know.

A TIME arti­cle sum­ma­rizes [4] the lack of trans­fer well:

But the improve­ment had noth­ing to do with the interim brain-training, says study co-author Jes­sica Grahn of the Cog­ni­tion and Brain Sci­ences Unit in Cam­bridge. Grahn says the results con­firm what she and other neu­ro­sci­en­tists have long sus­pected: peo­ple who prac­tice a cer­tain men­tal task — for instance, remem­ber­ing a series of num­bers in sequence, a pop­u­lar brain-teaser used by many video games — improve dra­mat­i­cally on that task, but the improve­ment does not carry over to cog­ni­tive func­tion in general.”

The Bad Sci­ence

The study, which was not a gold stan­dard clin­i­cal trial, angeleyescleef1.thumbnail [5]con­tained obvi­ous flaws both in method­ol­ogy and in inter­pre­ta­tion, as some neu­ro­sci­en­tists have started to point out. Back to the TIME article:

Kling­berg (note: Torkel Kling­berg is a cog­ni­tive neu­ro­sci­en­tist who has pub­lished mul­ti­ple sci­en­tific stud­ies on the ben­e­fits of brain train­ing, and founded a com­pany on the basis of that pub­lished work)…criticizes the design of the study and points to two fac­tors that may have skewed the results.

On aver­age the study vol­un­teers com­pleted 24 train­ing ses­sions, each about 10 min­utes long — for a total of three hours spent on dif­fer­ent tasks over six weeks. “The amount of train­ing was low,” says Kling­berg. “Ours and oth­ers’ research sug­gests that 8 to 12 hours of train­ing on one spe­cific test is needed to get a [gen­eral improve­ment in cognition].”

Sec­ond, he notes that the par­tic­i­pants were asked to com­plete their train­ing by log­ging onto the BBC Lab UK web­site from home. “There was no qual­ity con­trol. Ask­ing sub­jects to sit at home and do tests online, per­haps with the TV on or other dis­trac­tions around, is likely to result in bad qual­ity of the train­ing and unre­li­able out­come mea­sures. Noisy data often gives neg­a­tive find­ings,” Kling­berg says.”

More remark­able, a critic of brain train­ing pro­grams had the fol­low­ing to say in this Nature arti­cle [6]:

I really worry about this study — I think it’s flawed,” says Peter Sny­der, a neu­rol­o­gist who stud­ies age­ing at Brown University’s Alpert Med­ical School in Prov­i­dence, Rhode Island.

…But he says that most com­mer­cial pro­grams are aimed at adults well over 60 who fear that their mem­ory and men­tal sharp­ness are slip­ping. “You have to com­pare apples to apples,” says Sny­der. An older test group, he adds, would have a lower mean start­ing score and more vari­abil­ity in per­for­mance, leav­ing more room for train­ing to cause mean­ing­ful improve­ment. “You may have more of an abil­ity to see an effect if you’re not try­ing to cre­ate a super­nor­mal effect in a healthy per­son,” he says.

Sec­ond, the “dosage” was small, Sny­der said. The par­tic­i­pants were asked to train for at least 10 min­utes a day, three times a week for at least six weeks. That adds up to only four hours over the study period, which seemed mod­est to Snyder.

Update (04/26): just found this com­ment by Michael Valen­zuela, respond­ing to Nature article:

In our meta-analysis of cog­ni­tive brain train­ing RCTs in healthy elderly*, doses of active train­ing ranged from 10hours to 45 hours, with an aver­age dosage of 33 hours. Over­all, the effect was sig­nif­i­cant and robust.

The min­i­mum cited total dose in the BBC study was 3 hours (10mins three times a week for 6 weeks), and an aver­age num­ber of ses­sions is given as 23.86 and 28.39 for the two exper­i­men­tal groups. What was the aver­age dura­tion of each ses­sion? This infor­ma­tion is not pro­vided, nor con­trolled for, so let us assume 20minutes per ses­sion, lead­ing to an aver­age total active train­ing dose of 9.5hours.

The BBC study there­fore did not trial a suf­fi­cient dose of brain train­ing, leav­ing aside the issue of the qual­ity of training.

This study was seri­ously flawed and its con­clu­sions are invalid.

*Valen­zuela M., Sachdev P. Can cog­ni­tive exer­cise pre­vent the onset of demen­tia? A sys­tem­atic review of clin­i­cal tri­als with lon­gi­tu­di­nal fol­low up. Amer­i­can Jour­nal of Geri­atric Psy­chi­a­try 2009 17:179–187.”

The Ugly Logic

Let’s ana­lyze by anal­ogy. Aren’t thetucogbu1.thumbnail [7] BBC-sponsored researchers bas­ing their extremely broad claims on this type of faulty logic?

  1. We have decided to design and man­u­fac­ture our first car ever
  2. Oops, our car doesn’t work
  3. There­fore, cars DON’T work, CAN’T work, and WON’T work
  4. There­fore, ALL car man­u­fac­tur­ers are steal­ing your money.
  5. Case closed, let’s all con­tinue rid­ing horses.

Kling­berg points out this too, stress­ing to TIME that the study “draws a large con­clu­sion from a sin­gle neg­a­tive find­ing” and that it is “incor­rect to gen­er­al­ize from one spe­cific train­ing study to cog­ni­tive train­ing in general.”

Posit Sci­ence dares to debunk the debunker [8] (I have been crit­i­cal of sev­eral Posit Sci­ence’ mar­ket­ing claims in the past, but in this case agree with what they are saying):

This is a sur­pris­ing study method­ol­ogy,” said Dr. Henry Mah­ncke, VP Research at Posit Sci­ence. “It would be like con­clud­ing that there are no com­pounds to fight bac­te­ria because the com­pound you tested was sugar and not penicillin.”

We do need seri­ous sci­ence and analy­sis on the value and lim­i­ta­tions of scal­able approaches to cog­ni­tive assess­ment, train­ing and retrain­ing. There are very promis­ing pub­lished exam­ples of method­olo­gies that seem to work (which the BBC study design not only ignored but some­how man­aged to directly con­tra­dict), mixed with many claims not sup­ported by evi­dence. What con­cerns me is that this study may not only man­age to con­fuse the pub­lic even more, but to sti­fle much needed inno­va­tion to ensure we are bet­ter equipped over the next 5–10 years than we are today to meet the demands of an aging soci­ety in a rapidly chang­ing world.


Pre­vi­ous Sharp­Brains articles:

Be Socia­ble, Share!

    Article printed from SharpBrains: http://sharpbrains.com

    URL to article: http://sharpbrains.com/blog/2010/04/20/bbc-brain-training-experiment-the-good-the-bad-the-ugly/

    URLs in this post:

    [1] Image: http://www.sharpbrains.com/blog/2010/04/20/bbc-brain-training-experiment-the-good-the-bad-the-ugly/250px-clinteastwood/

    [2] book: http://www.amazon.com/dp/0982362900

    [3] report: http://www.sharpbrains.com/market-report/

    [4] TIME arti­cle sum­ma­rizes: http://www.time.com/time/health/article/0,8599,1983306,00.html#ixzz0lhe3Oiri

    [5] Image: http://www.sharpbrains.com/blog/2010/04/20/bbc-brain-training-experiment-the-good-the-bad-the-ugly/angeleyescleef1-thumbnail/

    [6] Nature arti­cle: http://www.nature.com/news/2010/100420/full/4641111a.html?s=news_rss

    [7] Image: http://www.sharpbrains.com/blog/2010/04/20/bbc-brain-training-experiment-the-good-the-bad-the-ugly/tucogbu1-thumbnail/

    [8] debunk the debunker: http://www.prnewswire.com/news-releases/posit-science-disputes-results-of-the-bbc-brain-training-study-91634379.html

    [9] Here: http://www.nature.com/nature/journal/vnfv/ncurrent/pdf/nature09042.pdf

    [10] Here: http://www.time.com/time/health/article/0,8599,1983306,00.html#ixzz0lh0ZMO77

    [11] Here: http://online.wsj.com/article/SB10001424052748703763904575196400381395576.html?mod=WSJ_latestheadlines

    [12] Here: http://www.sharpbrains.com/blog/2010/04/19/brain-training-bbc-nature-fact-hope-hype/

    [13] Sharp­Brains Sum­mit on Tech­nol­ogy for Cog­ni­tive Health and Per­for­mance: http://www.sharpbrains.com/z/jan2010-sharpbrains-summit/

    [14] Here: http://www.sharpbrains.com/resources/

    [15] Does Cog­ni­tive Train­ing Work? For whom? for what?: http://www.sharpbrains.com/blog/2009/02/25/does-cognitive-training-work-for-whom-for-what/

    [16] Are videogames good for you? It depends on who YOU are: http://www.sharpbrains.com/blog/2008/12/28/are-videogames-good-for-you-arthur-kramer-posits/


  • : http://www.facebook.com/share.php?u=http%3A%2F%2Fsharpbrains.com%2Fblog%2F2010%2F04%2F20%2Fbbc-brain-training-experiment-the-good-the-bad-the-ugly%2F&t=BBC%20%E2%80%9CBrain%20Training%E2%80%9D%20Experiment%3A%20the%20Good%2C%20the%20Bad%2C%20the%C2%A0Ugly


  • : http://twitter.com/intent/tweet?text=BBC%20%E2%80%9CBrain%20Training%E2%80%9D%20Experiment%3A%20the%20Good%2C%20the%20Bad%2C%20the%C2%A0Ugly%20-%20http%3A%2F%2Fsharpbrains.com%2Fblog%2F2010%2F04%2F20%2Fbbc-brain-training-experiment-the-good-the-bad-the-ugly%2F%20 SHARETAG


  • : http://delicious.com/post?url=http%3A%2F%2Fsharpbrains.com%2Fblog%2F2010%2F04%2F20%2Fbbc-brain-training-experiment-the-good-the-bad-the-ugly%2F&title=BBC%20%E2%80%9CBrain%20Training%E2%80%9D%20Experiment%3A%20the%20Good%2C%20the%20Bad%2C%20the%C2%A0Ugly&notes=You%20may%20already%20have%20read%20the%20hundreds%20of%20media%20articles%20today%20titled%20%22brain%20training%20doesn%27t%20work%22%20and%20similar%2C%20based%20on%20the%20BBC%20%22Brain%20Test%20Britain%22%20experiment.%0D%0A%0D%0AOnce%20more%2C%20claims%20seem%20to%20go%20beyond%20the%20science%20backing%20them%20up%20...%20except%20that%20in%20t


  • : http://reddit.com/submit?url=http%3A%2F%2Fsharpbrains.com%2Fblog%2F2010%2F04%2F20%2Fbbc-brain-training-experiment-the-good-the-bad-the-ugly%2F&title=BBC%20%E2%80%9CBrain%20Training%E2%80%9D%20Experiment%3A%20the%20Good%2C%20the%20Bad%2C%20the%C2%A0Ugly

    [21] Image: http://www.google.com/bookmarks/mark?op=edit&bkmk=http%3A%2F%2Fsharpbrains.com%2Fblog%2F2010%2F04%2F20%2Fbbc-brain-training-experiment-the-good-the-bad-the-ugly%2F&title=BBC%20%E2%80%9CBrain%20Training%E2%80%9D%20Experiment%3A%20the%20Good%2C%20the%20Bad%2C%20the%C2%A0Ugly&annotation=You%20may%20already%20have%20read%20the%20hundreds%20of%20media%20articles%20today%20titled%20%22brain%20training%20doesn%27t%20work%22%20and%20similar%2C%20based%20on%20the%20BBC%20%22Brain%20Test%20Britain%22%20experiment.%0D%0A%0D%0AOnce%20more%2C%20claims%20seem%20to%20go%20beyond%20the%20science%20backing%20them%20up%20...%20except%20that%20in%20t


  • : http://news.ycombinator.com/submitlink?u=http%3A%2F%2Fsharpbrains.com%2Fblog%2F2010%2F04%2F20%2Fbbc-brain-training-experiment-the-good-the-bad-the-ugly%2F&t=BBC%20%E2%80%9CBrain%20Training%E2%80%9D%20Experiment%3A%20the%20Good%2C%20the%20Bad%2C%20the%C2%A0Ugly


  • : http://www.blinklist.com/index.php?Action=Blink/addblink.php&Url=http%3A%2F%2Fsharpbrains.com%2Fblog%2F2010%2F04%2F20%2Fbbc-brain-training-experiment-the-good-the-bad-the-ugly%2F&Title=BBC%20%E2%80%9CBrain%20Training%E2%80%9D%20Experiment%3A%20the%20Good%2C%20the%20Bad%2C%20the%C2%A0Ugly


  • : http://posterous.com/share?linkto=http%3A%2F%2Fsharpbrains.com%2Fblog%2F2010%2F04%2F20%2Fbbc-brain-training-experiment-the-good-the-bad-the-ugly%2F&title=BBC%20%E2%80%9CBrain%20Training%E2%80%9D%20Experiment%3A%20the%20Good%2C%20the%20Bad%2C%20the%C2%A0Ugly&selection=You%20may%20already%20have%20read%20the%20hundreds%20of%20media%20articles%20today%20titled%20%22brain%20training%20doesn%27t%20work%22%20and%20similar%2C%20based%20on%20the%20BBC%20%22Brain%20Test%20Britain%22%20experiment.%0D%0A%0D%0AOnce%20more%2C%20claims%20seem%20to%20go%20beyond%20the%20science%20backing%20them%20up%20...%20except%20that%20in%20t


  • : https://mail.google.com/mail/?view=cm&fs=1&to&su=BBC%20%E2%80%9CBrain%20Training%E2%80%9D%20Experiment%3A%20the%20Good%2C%20the%20Bad%2C%20the%C2%A0Ugly&body=http%3A%2F%2Fsharpbrains.com%2Fblog%2F2010%2F04%2F20%2Fbbc-brain-training-experiment-the-good-the-bad-the-ugly%2F&ui=2&tf=1&shva=1

    [26] Tweet: https://twitter.com/share

  • Copyright © 2009 SharpBrains. All rights reserved.