BBC “Brain Training” Experiment: the Good, the Bad, the Ugly

You may already have read the hun­dreds of media arti­cles today titled “brain train­ing does­n’t work” and sim­i­lar, based on the BBC “Brain Test Britain” experiment.

Once more, claims seem to go beyond the sci­ence back­ing them up … except that in this case it is the researchers, not the devel­op­ers, who are responsible.

Let’s recap what we learned today.

The Good Sci­ence

The study showed that putting togeth­er a250px-ClintEastwood vari­ety of brain games in one web­site and ask­ing peo­ple who hap­pen to show up to play around for a grand total of 3–4 hours over 6 weeks (10 min­utes 3 times a week for 6 weeks) did­n’t result in mean­ing­ful improve­ments in cog­ni­tive func­tion­ing. This is use­ful infor­ma­tion for con­sumers to know, because in fact there are web­sites and com­pa­nies mak­ing claims based on sim­i­lar approach­es with­out sup­port­ing evi­dence. And this is pre­cise­ly the rea­son Sharp­Brains exists, to help both con­sumers (through our book) and orga­ni­za­tions (through our report) to make informed deci­sions. The paper only includ­ed peo­ple under 60, which is sur­pris­ing, but, still, this is use­ful infor­ma­tion to know.

A TIME arti­cle sum­ma­rizes the lack of trans­fer well:

But the improve­ment had noth­ing to do with the inter­im brain-train­ing, says study co-author Jes­si­ca Grahn of the Cog­ni­tion and Brain Sci­ences Unit in Cam­bridge. Grahn says the results con­firm what she and oth­er neu­ro­sci­en­tists have long sus­pect­ed: peo­ple who prac­tice a cer­tain men­tal task — for instance, remem­ber­ing a series of num­bers in sequence, a pop­u­lar brain-teas­er used by many video games — improve dra­mat­i­cal­ly on that task, but the improve­ment does not car­ry over to cog­ni­tive func­tion in general.”

The Bad Sci­ence

The study, which was not a gold stan­dard clin­i­cal tri­al, angeleyescleef1.thumbnailcon­tained obvi­ous flaws both in method­ol­o­gy and in inter­pre­ta­tion, as some neu­ro­sci­en­tists have start­ed to point out. Back to the TIME article:

Kling­berg (note: Torkel Kling­berg is a cog­ni­tive neu­ro­sci­en­tist who has pub­lished mul­ti­ple sci­en­tif­ic stud­ies on the ben­e­fits of brain train­ing, and found­ed a com­pa­ny on the basis of that pub­lished work)…criticizes the design of the study and points to two fac­tors that may have skewed the results.

On aver­age the study vol­un­teers com­plet­ed 24 train­ing ses­sions, each about 10 min­utes long — for a total of three hours spent on dif­fer­ent tasks over six weeks. “The amount of train­ing was low,” says Kling­berg. “Ours and oth­ers’ research sug­gests that 8 to 12 hours of train­ing on one spe­cif­ic test is need­ed to get a [gen­er­al improve­ment in cognition].”

Sec­ond, he notes that the par­tic­i­pants were asked to com­plete their train­ing by log­ging onto the BBC Lab UK web­site from home. “There was no qual­i­ty con­trol. Ask­ing sub­jects to sit at home and do tests online, per­haps with the TV on or oth­er dis­trac­tions around, is like­ly to result in bad qual­i­ty of the train­ing and unre­li­able out­come mea­sures. Noisy data often gives neg­a­tive find­ings,” Kling­berg says.”

More remark­able, a crit­ic of brain train­ing pro­grams had the fol­low­ing to say in this Nature arti­cle:

I real­ly wor­ry about this study — I think it’s flawed,” says Peter Sny­der, a neu­rol­o­gist who stud­ies age­ing at Brown Uni­ver­si­ty’s Alpert Med­ical School in Prov­i­dence, Rhode Island.

…But he says that most com­mer­cial pro­grams are aimed at adults well over 60 who fear that their mem­o­ry and men­tal sharp­ness are slip­ping. “You have to com­pare apples to apples,” says Sny­der. An old­er test group, he adds, would have a low­er mean start­ing score and more vari­abil­i­ty in per­for­mance, leav­ing more room for train­ing to cause mean­ing­ful improve­ment. “You may have more of an abil­i­ty to see an effect if you’re not try­ing to cre­ate a super­nor­mal effect in a healthy per­son,” he says.

Sec­ond, the “dosage” was small, Sny­der said. The par­tic­i­pants were asked to train for at least 10 min­utes a day, three times a week for at least six weeks. That adds up to only four hours over the study peri­od, which seemed mod­est to Snyder.

Update (04/26): just found this com­ment by Michael Valen­zuela, respond­ing to Nature article:

In our meta-analy­sis of cog­ni­tive brain train­ing RCTs in healthy elder­ly*, dos­es of active train­ing ranged from 10hours to 45 hours, with an aver­age dosage of 33 hours. Over­all, the effect was sig­nif­i­cant and robust.

The min­i­mum cit­ed total dose in the BBC study was 3 hours (10mins three times a week for 6 weeks), and an aver­age num­ber of ses­sions is giv­en as 23.86 and 28.39 for the two exper­i­men­tal groups. What was the aver­age dura­tion of each ses­sion? This infor­ma­tion is not pro­vid­ed, nor con­trolled for, so let us assume 20minutes per ses­sion, lead­ing to an aver­age total active train­ing dose of 9.5hours.

The BBC study there­fore did not tri­al a suf­fi­cient dose of brain train­ing, leav­ing aside the issue of the qual­i­ty of training.

This study was seri­ous­ly flawed and its con­clu­sions are invalid.

*Valen­zuela M., Sachdev P. Can cog­ni­tive exer­cise pre­vent the onset of demen­tia? A sys­tem­at­ic review of clin­i­cal tri­als with lon­gi­tu­di­nal fol­low up. Amer­i­can Jour­nal of Geri­atric Psy­chi­a­try 2009 17:179–187.”

The Ugly Log­ic

Let’s ana­lyze by anal­o­gy. Aren’t thetucogbu1.thumbnail BBC-spon­sored researchers bas­ing their extreme­ly broad claims on this type of faulty logic?

  1. We have decid­ed to design and man­u­fac­ture our first car ever
  2. Oops, our car does­n’t work
  3. There­fore, cars DON’T work, CAN’T work, and WON’T work
  4. There­fore, ALL car man­u­fac­tur­ers are steal­ing your money.
  5. Case closed, let’s all con­tin­ue rid­ing horses.

Kling­berg points out this too, stress­ing to TIME that the study “draws a large con­clu­sion from a sin­gle neg­a­tive find­ing” and that it is “incor­rect to gen­er­al­ize from one spe­cif­ic train­ing study to cog­ni­tive train­ing in general.”

Posit Sci­ence aimed to debunk the debunker (I have been crit­i­cal of sev­er­al Posit Sci­ence’ mar­ket­ing claims in the past, but in this case agree with what they are saying):

This is a sur­pris­ing study method­ol­o­gy,” said Dr. Hen­ry Mah­ncke, VP Research at Posit Sci­ence. “It would be like con­clud­ing that there are no com­pounds to fight bac­te­ria because the com­pound you test­ed was sug­ar and not penicillin.”

We do need seri­ous sci­ence and analy­sis on the val­ue and lim­i­ta­tions of scal­able approach­es to cog­ni­tive assess­ment, train­ing and retrain­ing. There are very promis­ing pub­lished exam­ples of method­olo­gies that seem to work (which the BBC study design not only ignored but some­how man­aged to direct­ly con­tra­dict), mixed with many claims not sup­port­ed by evi­dence. What con­cerns me is that this study may not only man­age to con­fuse the pub­lic even more, but to sti­fle much need­ed inno­va­tion to ensure we are bet­ter equipped over the next 5–10 years than we are today to meet the demands of an aging soci­ety in a rapid­ly chang­ing world.

Resources:

Pre­vi­ous Sharp­Brains articles:

10 Comments

  1. Ed Batista on April 20, 2010 at 11:34

    Thanks, Alvaro–very help­ful clarification.



  2. Laura Fay on April 21, 2010 at 1:08

    Alvaro, You raise some very valid points regard­ing the BBC exper­i­ment. Bernard Croisile, Chief Sci­ence Office of Sci­en­tif­ic Brain Train­ing / HAP­PYneu­ron, makes addi­tion­al points on the con­struc­tion and inter­pre­ta­tion of the exper­i­ment. They can be read at http://www.brainfitnessforlife.com. — Laura



  3. Robyn on April 21, 2010 at 1:42

    thanks, very inter­est­ing. I’m inter­est­ed in design­ing apps and games that can have a trans­fer­able effect (www.timestableclock.com, for exam­ple). While I appre­ci­ate the attempts to bring sci­ence to a wider audi­ence, I wish they would­n’t make it sound like they were doing a valid exper­i­ment, and instead explained the lim­i­ta­tions and that some­thing like this can only be a demonstration.



  4. Kathy O'Brien on April 22, 2010 at 10:47

    I went to a non-com­put­er, cog­ni­tive train­ing work­shop about 5 years ago and have nev­er for­got­ten what the psy­chol­o­gist (Feuern­stein) said, “If we don’t believe we can change brain behav­ior, then why do we teach?” 

    My expe­ri­ence with a com­put­er-based, inten­sive, audi­to­ry train­ing pro­gram was that it enhanced the audi­to­ry sys­tem momen­tar­i­ly, but the pos­i­tive effects did­n’t last over time. There’s def­i­nite­ly room for advance­ment in the ‘brain train­ing’ world.



  5. Alvaro Fernandez on April 26, 2010 at 9:42

    Hel­lo every­one, thank you for your comments.

    Let’s hope some­thing good comes out all of this, such as more clar­i­ty into what “brain train­ing” is and isn’t. 

    We have decid­ed to pub­lish online close to the full con­tent of our book and con­sumer guide The Sharp­Brains Guide to Brain Fitness:
    https://sharpbrains.com/resources/



  6. Billiam James on April 26, 2010 at 7:59

    I agree. The show’s log­ic is ugly. And stu­pid. It’s like say­ing that since most diets don’t work, that all the research on healthy eat­ing is just a bunch of baloney. But then again, it’s a TV show. And on TV, stu­pid log­ic works. 

    So what do we need to do? Dumb down brain-fit­ness? I don’t think so… But if you are going to sell Brain Fit­ness to TV audi­ence, you need to use visu­al and emo­tion­al hooks to deliv­er the message.



  7. Luc P. Beaudoin on April 28, 2010 at 11:49

    Test­ing brain/cognitive train­ing prod­ucts is impor­tant. How­ev­er, in sci­ence, one can’t con­clude that an effect (e.g., cog­ni­tive improve­ment through soft­ware use) is impos­si­ble or unlike­ly based on obser­va­tion (even if the sam­ple size is very large). Owen and col­leagues could refer to the wikipedia arti­cle on the “null hypoth­e­sis” which most researchers under­stand. The range of pos­si­ble cog­ni­tive mech­a­nisms, strate­gies and skills that one can poten­tial­ly train through soft­ware, and the range of man­ners in which one could train them, are so broad as to call for much more the­o­riz­ing, soft­ware devel­op­ment and empir­i­cal test­ing of the effects of such soft­ware. (And of course many cog­ni­tive train­ing effects have already been documented.)



  8. Richard on April 29, 2010 at 5:09

    I’ve tried brain train­ing on-line and I’ve tried it on a Nin­ten­do DS and notice con­sid­er­able improve­ments. Try med­i­tat­ing. That’s REAL brain training.



  9. Mark A Smith on May 10, 2010 at 11:09

    Excel­lent points. But why did­n’t the study draw from the lit­er­a­ture and use brain train­ing pro­grams that have been sci­en­tif­i­cal­ly demon­strat­ed to have a broad trans­fer effect — such as the dual n‑back (e.g. http://www.highiqpro.com/high-iq-pro/scientific-basis-of-software)? It’s not just a dose prob­lem but a task prob­lem. Sin­gle n‑back train­ing gains have NOT been found to trans­fer to flu­id intel­li­gence gains, while the dual n‑back has. This kind of speci­fici­ty is important.



  10. Alvaro Fernandez on May 14, 2010 at 3:53

    Yes, BBC basi­cal­ly mud­dled the waters for con­sumers (even more, yes) by nam­ing “brain train­ing” some­thing that has noth­ing to do with the cog­ni­tive train­ing that has enabled trans­fer in pre­vi­ous stud­ies. If they want­ed to debunk Nin­ten­do, well, then, test Nintendo.

    Cog­ni­tive train­ing has been iden­ti­fied in the recent NIH inde­pen­dent pan­el report as the only pro­tec­tive fac­tor against cog­ni­tive decline based on the high­est degree of evi­dence. Cog­ni­tive engage­ment over­all and phys­i­cal activ­i­ties are also pro­tec­tive, based on low­er qual­i­ty evidence.

    It would seem as if the BBC may be inad­ver­tent con­tribut­ing to the cog­ni­tive decline of its viewers/ read­ers who trust its news and pro­grams with­out crit­i­cal judgment.



About SharpBrains

SHARPBRAINS is an independent think-tank and consulting firm providing services at the frontier of applied neuroscience, health, leadership and innovation.
SHARPBRAINS es un think-tank y consultoría independiente proporcionando servicios para la neurociencia aplicada, salud, liderazgo e innovación.

Top Articles on Brain Health and Neuroplasticity

Top 10 Brain Teasers and Illusions

Newsletter

Subscribe to our e-newsletter

* indicates required

Got the book?