132| Reliable Change – A Conversation With Dr. Kevin Duff

In this episode, we discuss reliable change with Dr. Kevin Duff. Specific topics covered include the purpose of serial assessment, classical test theory, test retest reliability, an introduction to practice effects, factors that increase or decrease practice effects, the reliable change index, standardized regression-based equations, and clinical factors impacting the interpretation of reliable change data.

If you’d like to receive APA-approved CE credit for listening to this episode, click here.

About Kevin

Dr. Kevin Duff has specialized in neuropsychology for over 20 years. He obtained his Ph.D. in Clinical Psychology from the State University of New York in Albany. He completed his neuropsychology internship at the Southern Arizona Healthcare System in Tucson, AZ, and his post-doctoral fellowship at the University of Oklahoma Health Sciences Center in Oklahoma City. He joined the Psychiatry Department at the University of Iowa in 2003, where had clinical and research responsibilities working with patients with dementia, Huntington’s disease, and a variety of other neuropsychiatric conditions. In 2009, he joined the University of Utah as Associate Professor of Neurology and neuropsychologist for Alzheimer’s Care, Imaging and Research. In 2022, he moved (hopefully for the last time) to Oregon Health & Science University in Portland, where he is the senior neuropsychologist at the Layton Aging & Alzheimer’s Disease Research Center and Professor in their Department of Neurology.

Dr. Duff’s research has focused primarily on the early identification of cognitive decline in neuropsychiatric illnesses. Across multiple studies, Dr. Duff has used short-term practice effects as a marker of brain plasticity in patients with Mild Cognitive Impairment to examine if short-term changes in cognition can predict the diagnosis and prognosis of dementia, as well as its brain pathology and response to interventions. He has published over 180 peer-reviewed papers in scientific journals and he has lectured nationally and internationally on his areas of expertise. His research on practice effects in Mild Cognitive Impairment has been continually funded by the National Institutes of Health since 2005.

Resources

Predicting 1 yr scores with baseline and practice effects based on ACN paper

Change score calculator revised

RBANS subtest and index predictor using manual norms

References

Bartels, C., Wegrzyn, M., Wiedl, A., Ackermann, V., & Ehrenreich, H. (2010). Practice effects in healthy adults: A longitudinal study on frequent repetitive cognitive testing. BMC Neuroscience, 11(1). https://doi.org/10.1186/1471-2202-11-118

Beglinger, L. J., Gaydos, B., Tangphao-Daniels, O., Duff, K., Kareken, D. A., Crawford, J., Fastenau, P. S., & Siemers, E. R. (2005). Practice effects and the use of alternate forms in serial neuropsychological testing. Archives of Clinical Neuropsychology, 20(4), 517–529. https://doi.org/10.1016/j.acn.2004.12.003

Calamia, M., Markon, K., & Tranel, D. (2012). Scoring higher the second time around: Meta-analyses of practice effects in neuropsychological assessment. The Clinical Neuropsychologist, 26(4), 543–570. https://doi.org/10.1080/13854046.2012.680913

Chelune, G. J., Naugle, R. I., Lüders, H., Sedlak, J., & Awad, I. A. (1993). Individual change after epilepsy surgery: Practice effects and base-rate information. Neuropsychology, 7(1), 41–52. https://doi.org/10.1037/0894-4105.7.1.41

Cysique, L. A., Franklin, D., Abramson, I., Ellis, R. J., Letendre, S., Collier, A., Clifford, D., Gelman, B., McArthur, J., Morgello, S., Simpson, D., McCutchan, J. A., Grant, I., & Heaton, R. K. (2011). Normative data and validation of a regression based summary score for assessing meaningful neuropsychological change. Journal of Clinical and Experimental Neuropsychology, 33(5), 505–522. https://doi.org/10.1080/13803395.2010.535504

Duff, K. (2012). Evidence-based indicators of neuropsychological change in the individual patient: Relevant concepts and methods. Archives of Clinical Neuropsychology, 27(3), 248–261. https://doi.org/10.1093/arclin/acr120

Duff, K., Beglinger, L. J., Moser, D. J., Paulsen, J. S., Schultz, S. K., & Arndt, S. (2010). Predicting cognitive change in older adults: The relative contribution of practice effects. Archives of Clinical Neuropsychology, 25(2), 81–88. https://doi.org/10.1093/arclin/acp105

Durant, J., Duff, K., & Miller, J. B. (2019). Regression-based formulas for predicting change in memory test scores in healthy older adults: Comparing use of raw versus standardized scores. Journal of Clinical and Experimental Neuropsychology, 41(5), 460–468. https://doi.org/10.1080/13803395.2019.1571169

Duff, Kevin, Dorociak, K. E., & Yamada, T. H. (2022). Serial assessment in the older patient. A Handbook of Geriatric Neuropsychology, 361–378. https://doi.org/10.4324/9781003100058-23

Duff, K., & Hammers, D. B. (2020). Practice effects in mild cognitive impairment: A validation of Calamia et al. (2012). The Clinical Neuropsychologist, 36(3), 571–583. https://doi.org/10.1080/13854046.2020.1781933

Duff, Kevin, Beglinger, L. J., Schoenberg, M. R., Patton, D. E., Mold, J., Scott, J. G., & Adams, R. L. (2005). Test-retest stability and practice effects of the RBANS in a community dwelling elderly sample. Journal of Clinical and Experimental Neuropsychology, 27(5), 565–575. https://doi.org/10.1080/13803390490918363

Duff, Kevin. (2014). One-week practice effects in older adults: Tools for assessing cognitive change. The Clinical Neuropsychologist, 28(5), 714–725. https://doi.org/10.1080/13854046.2014.920923

Duff, Kevin, Suhrie, K. R., Dalley, B. C. A., Anderson, J. S., & Hoffman, J. M. (2018). External validation of change formulae in neuropsychology with neuroimaging biomarkers: A methodological recommendation and preliminary clinical data. The Clinical Neuropsychologist, 33(3), 478–489. https://doi.org/10.1080/13854046.2018.1484518

Harvey, P. D. (2012). Clinical applications of neuropsychological assessment. Dialogues in Clinical Neuroscience, 14(1), 91–99. https://doi.org/10.31887/dcns.2012.14.1/pharvey

Hageman, W. J. J. M., & Arrindell, W. A. (1993). A further refinement of the reliable change (RC) index by improving the pre-post difference score: Introducing RCID. Behaviour Research and Therapy, 31(7), 693–700. https://doi.org/10.1016/0005-7967(93)90122-b

Hageman, W. J. J. M., & Arrindell, W. A. (1999). Establishing clinically signi®cant change: increment of precision and the distinction between individual and group level of analysis. Behaviour Research and Therapy, 37(12), 1169–1193. https://doi.org/10.1016/s0005-7967(99)00032-7

Hammers, D. B., & Duff, K. (2019). Application of different standard error estimates in reliable change methods. Archives of Clinical Neuropsychology, 36(3), 339–346. https://doi.org/10.1093/arclin/acz054

Hammers, D. B., Porter, S., Dixon, A., Suhrie, K. R., & Duff, K. (2020). Validating 1-year reliable change methods. Archives of Clinical Neuropsychology, 36(1), 87–98. https://doi.org/10.1093/arclin/acaa055

Hammers, D. B., Suhrie, K. R., Dixon, A., Porter, S., & Duff, K. (2020). Reliable change in cognition over 1 week in community-dwelling older adults: A validation and extension study. Archives of Clinical Neuropsychology, 36(3), 347–358. https://doi.org/10.1093/arclin/acz076

Hammers, D. B., Suhrie, K. R., Porter, S. M., Dixon, A. M., & Duff, K. (2020). Validation of one-year reliable change in the RBANS for community-dwelling older adults with amnestic mild cognitive impairment. The Clinical Neuropsychologist, 36(6), 1304–1327. https://doi.org/10.1080/13854046.2020.1807058

Heaton, R. K., Temkin, N., Dikmen, S., Avitable, N., Taylor, M. J., Marcotte, T. D., & Grant, I. (2001). Detecting change: A comparison of three neuropsychological methods, using normal and clinical samples. Archives of clinical neuropsychology : the official journal of the National Academy of Neuropsychologists16(1), 75–91.

Heilbronner, R. L., Sweet, J. J., Attix, D. K., Krull, K. R., Henry, G. K., & Hart, R. P. (2010). Official position of the American Academy of Clinical Neuropsychology on serial neuropsychological assessments: The utility and challenges of repeat test administrations in clinical and forensic contexts. The Clinical Neuropsychologist, 24(8), 1267–1278. https://doi.org/10.1080/13854046.2010.526785

Hermann, B. P., Wyler, A. R., Vanderzwagg, R., LeBailly, R. K., Whitman, S., Somes, G., & Ward, J. (1991). Predictors of neuropsychological change following anterior temporal lobectomy: Role of regression toward the mean. Journal of Epilepsy, 4(3), 139–148. https://doi.org/10.1016/s0896-6974(05)80039-8

Hsu L. M. (1999). A comparison of three methods of identifying reliable and clinically significant client changes: commentary on Hageman and Arrindell. Behaviour research and therapy37(12), 1195–1233. https://doi.org/10.1016/s0005-7967(99)00033-9

Iverson, G. L., & Schatz, P. (2014). Advanced topics in neuropsychological assessment following sport-related concussion. Brain Injury, 29(2), 263–275. https://doi.org/10.3109/02699052.2014.965214

Iverson, G.L. (2012). Interpreting change on repeated neuropsychological assessments of children. In E. Sherman and B. Brooks (eds.), Pediatric Forensic Neuropsychology, pp. 89-112. New York: Oxford University Press.

Iverson, G. L. (2018). Reliable change index. Encyclopedia of Clinical Neuropsychology, 1–4. https://doi.org/10.1007/978-3-319-56782-2_1242-3

Jacobson, N. S., & Truax, P. (1991). Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. Journal of consulting and clinical psychology59(1), 12–19. https://doi.org/10.1037//0022-006x.59.1.12

Knight, R. G., McMahonn, J., Skeaff, C. M., & Green, T. J. (2007). Reliable change index scores for persons over the age of 65 tested on alternate forms of the Rey Avlt. Archives of Clinical Neuropsychology, 22(4), 513–518. https://doi.org/10.1016/j.acn.2007.03.005

Rinehardt, E. (Ed.). (2014). National academy of neuropsychology bulletin. https://www.e-digitaleditions.com/i/401821-nan-fall-bulletin/0?

Pagnacco, G., Carrick, F. R., Wright, C. H. G., & Oggero, E. (2015). Between-subjects differences of within-subject variability in repeated balance measures: Consequences on the minimum detectable change. Gait & Posture, 41(1), 136–140. https://doi.org/10.1016/j.gaitpost.2014.09.016

Port, J. (Ed.). (2010, October). PsyPag quarterly. https://explore.bps.org.uk/content/bpspag

Reed, C., Calamia, M., Sanderson-Cimino, M., DeVito, A., Toups, R., & Keller, J. (2023). Four Year practice effects on the RBANS in a longitudinal study of older adults. Applied Neuropsychology: Adult, 1–7. https://doi.org/10.1080/23279095.2023.2180361

Reid-Arndt, S. A., Hsieh, C., & Perry, M. C. (2010). Neuropsychological functioning and quality of life during the first year after completing chemotherapy for breast cancer. Psycho-Oncology, 19(5), 535–544. https://doi.org/10.1002/pon.1581

Rinehardt, E., Duff, K., Schoenberg, M., Mattingly, M., Bharucha, K., & Scott, J. (2010). Cognitive change on the repeatable battery of neuropsychological status (RBANS) in parkinson’s disease with and without bilateral subthalamic nucleus deep brain stimulation surgery. The Clinical Neuropsychologist, 24(8), 1339–1354. https://doi.org/10.1080/13854046.2010.521770

Sawrie, S. M., Marson, D. C., Boothe, A. L., & Harrell, L. E. (1999). A method for assessing clinically relevant individual cognitive change in older adult populations. The journals of gerontology. Series B, Psychological sciences and social sciences54(2), P116–P124. https://doi.org/10.1093/geronb/54b.2.p116

Schmitt, J. S., & Di Fabio, R. P. (2004). Reliable change and minimum important difference (MID) proportions facilitated group responsiveness comparisons using individual threshold criteria. Journal of Clinical Epidemiology, 57(10), 1008–1018. https://doi.org/10.1016/j.jclinepi.2004.02.007

Schoenberg, M. R., Rinehardt, E., Duff, K., Mattingly, M., Bharucha, K. J., & Scott, J. G. (2012). Assessing reliable change using the repeatable battery for the assessment of neuropsychological status (RBANS) for patients with parkinson’s disease undergoing deep brain stimulation (DBS) surgery. The Clinical Neuropsychologist, 26(2), 255–270. https://doi.org/10.1080/13854046.2011.653587

Sherman, E. M., Wiebe, S., Fay-McClymont, T. B., Tellez-Zenteno, J., Metcalfe, A., Hernandez-Ronquillo, L., Hader, W. J., & Jetté, N. (2011). Neuropsychological outcomes after epilepsy surgery: Systematic Review and pooled estimates. Epilepsia, 52(5), 857–869. https://doi.org/10.1111/j.1528-1167.2011.03022.x

Stein, J., Luppa, M., Brähler, E., König, H.-H., & Riedel-Heller, S. G. (2010). The assessment of changes in cognitive functioning: Reliable change indices for neuropsychological instruments in the elderly – a systematic review. Dementia and Geriatric Cognitive Disorders, 29(3), 275–286. https://doi.org/10.1159/000289779

Stratford, P. W., Binkley, J., Solomon, P., Finch, E., Gill, C., & Moreland, J. (1996). Defining the minimum level of detectable change for the Roland-Morris questionnaire. Physical therapy76(4), 359–368. https://doi.org/10.1093/ptj/76.4.359

Temkin, N. R., Heaton, R. K., Grant, I., & Dikmen, S. S. (1999). Detecting significant change in neuropsychological test performance: A comparison of four models. Journal of the International Neuropsychological Society, 5(4), 357–369. https://doi.org/10.1017/s1355617799544068

Weir J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of strength and conditioning research19(1), 231–240. https://doi.org/10.1519/15184.1

 

Episodes