- SharpBrains - http://sharpbrains.com -

BBC “Brain Training” Experiment: the Good, the Bad, the Ugly

You may already have read the hun­dreds of media arti­cles today titled “brain train­ing doesn’t work” and sim­i­lar, based on the BBC “Brain Test Britain” experiment.

Once more, claims seem to go beyond the sci­ence back­ing them up … except that in this case it is the researchers, not the devel­op­ers, who are responsible.

Let’s recap what we learned today.

The Good Sci­ence

The study showed that putting together a250px-ClintEastwood [1] vari­ety of brain games in one web­site and ask­ing peo­ple who hap­pen to show up to play around for a grand total of 3–4 hours over 6 weeks (10 min­utes 3 times a week for 6 weeks) didn’t result in mean­ing­ful improve­ments in cog­ni­tive func­tion­ing. This is use­ful infor­ma­tion for con­sumers to know, because in fact there are web­sites and com­pa­nies mak­ing claims based on sim­i­lar approaches with­out sup­port­ing evi­dence. And this is pre­cisely the rea­son Sharp­Brains exists, to help both con­sumers (through our book [2]) and orga­ni­za­tions (through our report [3]) to make informed deci­sions. The paper only included peo­ple under 60, which is sur­pris­ing, but, still, this is use­ful infor­ma­tion to know.

A TIME arti­cle sum­ma­rizes [4] the lack of trans­fer well:

But the improve­ment had noth­ing to do with the interim brain-training, says study co-author Jes­sica Grahn of the Cog­ni­tion and Brain Sci­ences Unit in Cam­bridge. Grahn says the results con­firm what she and other neu­ro­sci­en­tists have long sus­pected: peo­ple who prac­tice a cer­tain men­tal task — for instance, remem­ber­ing a series of num­bers in sequence, a pop­u­lar brain-teaser used by many video games — improve dra­mat­i­cally on that task, but the improve­ment does not carry over to cog­ni­tive func­tion in general.”

The Bad Sci­ence

The study, which was not a gold stan­dard clin­i­cal trial, angeleyescleef1.thumbnail [5]con­tained obvi­ous flaws both in method­ol­ogy and in inter­pre­ta­tion, as some neu­ro­sci­en­tists have started to point out. Back to the TIME article:

Kling­berg (note: Torkel Kling­berg is a cog­ni­tive neu­ro­sci­en­tist who has pub­lished mul­ti­ple sci­en­tific stud­ies on the ben­e­fits of brain train­ing, and founded a com­pany on the basis of that pub­lished work)…criticizes the design of the study and points to two fac­tors that may have skewed the results.

On aver­age the study vol­un­teers com­pleted 24 train­ing ses­sions, each about 10 min­utes long — for a total of three hours spent on dif­fer­ent tasks over six weeks. “The amount of train­ing was low,” says Kling­berg. “Ours and oth­ers’ research sug­gests that 8 to 12 hours of train­ing on one spe­cific test is needed to get a [gen­eral improve­ment in cognition].”

Sec­ond, he notes that the par­tic­i­pants were asked to com­plete their train­ing by log­ging onto the BBC Lab UK web­site from home. “There was no qual­ity con­trol. Ask­ing sub­jects to sit at home and do tests online, per­haps with the TV on or other dis­trac­tions around, is likely to result in bad qual­ity of the train­ing and unre­li­able out­come mea­sures. Noisy data often gives neg­a­tive find­ings,” Kling­berg says.”

More remark­able, a critic of brain train­ing pro­grams had the fol­low­ing to say in this Nature arti­cle [6]:

I really worry about this study — I think it’s flawed,” says Peter Sny­der, a neu­rol­o­gist who stud­ies age­ing at Brown University’s Alpert Med­ical School in Prov­i­dence, Rhode Island.

…But he says that most com­mer­cial pro­grams are aimed at adults well over 60 who fear that their mem­ory and men­tal sharp­ness are slip­ping. “You have to com­pare apples to apples,” says Sny­der. An older test group, he adds, would have a lower mean start­ing score and more vari­abil­ity in per­for­mance, leav­ing more room for train­ing to cause mean­ing­ful improve­ment. “You may have more of an abil­ity to see an effect if you’re not try­ing to cre­ate a super­nor­mal effect in a healthy per­son,” he says.

Sec­ond, the “dosage” was small, Sny­der said. The par­tic­i­pants were asked to train for at least 10 min­utes a day, three times a week for at least six weeks. That adds up to only four hours over the study period, which seemed mod­est to Snyder.

Update (04/26): just found this com­ment by Michael Valen­zuela, respond­ing to Nature article:

In our meta-analysis of cog­ni­tive brain train­ing RCTs in healthy elderly*, doses of active train­ing ranged from 10hours to 45 hours, with an aver­age dosage of 33 hours. Over­all, the effect was sig­nif­i­cant and robust.

The min­i­mum cited total dose in the BBC study was 3 hours (10mins three times a week for 6 weeks), and an aver­age num­ber of ses­sions is given as 23.86 and 28.39 for the two exper­i­men­tal groups. What was the aver­age dura­tion of each ses­sion? This infor­ma­tion is not pro­vided, nor con­trolled for, so let us assume 20minutes per ses­sion, lead­ing to an aver­age total active train­ing dose of 9.5hours.

The BBC study there­fore did not trial a suf­fi­cient dose of brain train­ing, leav­ing aside the issue of the qual­ity of training.

This study was seri­ously flawed and its con­clu­sions are invalid.

*Valen­zuela M., Sachdev P. Can cog­ni­tive exer­cise pre­vent the onset of demen­tia? A sys­tem­atic review of clin­i­cal tri­als with lon­gi­tu­di­nal fol­low up. Amer­i­can Jour­nal of Geri­atric Psy­chi­a­try 2009 17:179–187.”

The Ugly Logic

Let’s ana­lyze by anal­ogy. Aren’t thetucogbu1.thumbnail [7] BBC-sponsored researchers bas­ing their extremely broad claims on this type of faulty logic?

  1. We have decided to design and man­u­fac­ture our first car ever
  2. Oops, our car doesn’t work
  3. There­fore, cars DON’T work, CAN’T work, and WON’T work
  4. There­fore, ALL car man­u­fac­tur­ers are steal­ing your money.
  5. Case closed, let’s all con­tinue rid­ing horses.

Kling­berg points out this too, stress­ing to TIME that the study “draws a large con­clu­sion from a sin­gle neg­a­tive find­ing” and that it is “incor­rect to gen­er­al­ize from one spe­cific train­ing study to cog­ni­tive train­ing in general.”

Posit Sci­ence dares to debunk the debunker [8] (I have been crit­i­cal of sev­eral Posit Sci­ence’ mar­ket­ing claims in the past, but in this case agree with what they are saying):

This is a sur­pris­ing study method­ol­ogy,” said Dr. Henry Mah­ncke, VP Research at Posit Sci­ence. “It would be like con­clud­ing that there are no com­pounds to fight bac­te­ria because the com­pound you tested was sugar and not penicillin.”

We do need seri­ous sci­ence and analy­sis on the value and lim­i­ta­tions of scal­able approaches to cog­ni­tive assess­ment, train­ing and retrain­ing. There are very promis­ing pub­lished exam­ples of method­olo­gies that seem to work (which the BBC study design not only ignored but some­how man­aged to directly con­tra­dict), mixed with many claims not sup­ported by evi­dence. What con­cerns me is that this study may not only man­age to con­fuse the pub­lic even more, but to sti­fle much needed inno­va­tion to ensure we are bet­ter equipped over the next 5–10 years than we are today to meet the demands of an aging soci­ety in a rapidly chang­ing world.


Pre­vi­ous Sharp­Brains articles:

Be Socia­ble, Share!