In this episode, we discuss reliable change with Dr. Kevin Duff. Specific topics covered include the purpose of serial assessment, classical test theory, test retest reliability, an introduction to practice effects, factors that increase or decrease practice effects, the reliable change index, standardized regression-based equations, and clinical factors impacting the interpretation of reliable change data.
If you’d like to receive CE credit for listening to this episode, click here.
Guest disclosures: None.
About Kevin
Dr. Kevin Duff has specialized in neuropsychology for over 20 years. He obtained his Ph.D. in Clinical Psychology from the State University of New York in Albany. He completed his neuropsychology internship at the Southern Arizona Healthcare System in Tucson, AZ, and his post-doctoral fellowship at the University of Oklahoma Health Sciences Center in Oklahoma City. He joined the Psychiatry Department at the University of Iowa in 2003, where had clinical and research responsibilities working with patients with dementia, Huntington’s disease, and a variety of other neuropsychiatric conditions. In 2009, he joined the University of Utah as Associate Professor of Neurology and neuropsychologist for Alzheimer’s Care, Imaging and Research. In 2022, he moved (hopefully for the last time) to Oregon Health & Science University in Portland, where he is the senior neuropsychologist at the Layton Aging & Alzheimer’s Disease Research Center and Professor in their Department of Neurology.
Dr. Duff’s research has focused primarily on the early identification of cognitive decline in neuropsychiatric illnesses. Across multiple studies, Dr. Duff has used short-term practice effects as a marker of brain plasticity in patients with Mild Cognitive Impairment to examine if short-term changes in cognition can predict the diagnosis and prognosis of dementia, as well as its brain pathology and response to interventions. He has published over 180 peer-reviewed papers in scientific journals and he has lectured nationally and internationally on his areas of expertise. His research on practice effects in Mild Cognitive Impairment has been continually funded by the National Institutes of Health since 2005.
Resources
Predicting 1 yr scores with baseline and practice effects based on ACN paper
Change score calculator revised
RBANS subtest and index predictor using manual norms
References
Bartels, C., Wegrzyn, M., Wiedl, A., Ackermann, V., & Ehrenreich, H. (2010). Practice effects in healthy adults: A longitudinal study on frequent repetitive cognitive testing. BMC Neuroscience, 11(1). https://doi.org/10.1186/1471-2202-11-118
Beglinger, L. J., Gaydos, B., Tangphao-Daniels, O., Duff, K., Kareken, D. A., Crawford, J., Fastenau, P. S., & Siemers, E. R. (2005). Practice effects and the use of alternate forms in serial neuropsychological testing. Archives of Clinical Neuropsychology, 20(4), 517–529. https://doi.org/10.1016/j.acn.2004.12.003
Calamia, M., Markon, K., & Tranel, D. (2012). Scoring higher the second time around: Meta-analyses of practice effects in neuropsychological assessment. The Clinical Neuropsychologist, 26(4), 543–570. https://doi.org/10.1080/13854046.2012.680913
Chelune, G. J., Naugle, R. I., Lüders, H., Sedlak, J., & Awad, I. A. (1993). Individual change after epilepsy surgery: Practice effects and base-rate information. Neuropsychology, 7(1), 41–52. https://doi.org/10.1037/0894-4105.7.1.41
Cysique, L. A., Franklin, D., Abramson, I., Ellis, R. J., Letendre, S., Collier, A., Clifford, D., Gelman, B., McArthur, J., Morgello, S., Simpson, D., McCutchan, J. A., Grant, I., & Heaton, R. K. (2011). Normative data and validation of a regression based summary score for assessing meaningful neuropsychological change. Journal of Clinical and Experimental Neuropsychology, 33(5), 505–522. https://doi.org/10.1080/13803395.2010.535504
Duff, K. (2012). Evidence-based indicators of neuropsychological change in the individual patient: Relevant concepts and methods. Archives of Clinical Neuropsychology, 27(3), 248–261. https://doi.org/10.1093/arclin/acr120
Duff, K., Beglinger, L. J., Moser, D. J., Paulsen, J. S., Schultz, S. K., & Arndt, S. (2010). Predicting cognitive change in older adults: The relative contribution of practice effects. Archives of Clinical Neuropsychology, 25(2), 81–88. https://doi.org/10.1093/arclin/acp105
Durant, J., Duff, K., & Miller, J. B. (2019). Regression-based formulas for predicting change in memory test scores in healthy older adults: Comparing use of raw versus standardized scores. Journal of Clinical and Experimental Neuropsychology, 41(5), 460–468. https://doi.org/10.1080/13803395.2019.1571169
Duff, Kevin, Dorociak, K. E., & Yamada, T. H. (2022). Serial assessment in the older patient. A Handbook of Geriatric Neuropsychology, 361–378. https://doi.org/10.4324/9781003100058-23
Duff, K., & Hammers, D. B. (2020). Practice effects in mild cognitive impairment: A validation of Calamia et al. (2012). The Clinical Neuropsychologist, 36(3), 571–583. https://doi.org/10.1080/13854046.2020.1781933
Duff, Kevin, Beglinger, L. J., Schoenberg, M. R., Patton, D. E., Mold, J., Scott, J. G., & Adams, R. L. (2005). Test-retest stability and practice effects of the RBANS in a community dwelling elderly sample. Journal of Clinical and Experimental Neuropsychology, 27(5), 565–575. https://doi.org/10.1080/13803390490918363
Duff, Kevin. (2014). One-week practice effects in older adults: Tools for assessing cognitive change. The Clinical Neuropsychologist, 28(5), 714–725. https://doi.org/10.1080/13854046.2014.920923
Duff, Kevin, Suhrie, K. R., Dalley, B. C. A., Anderson, J. S., & Hoffman, J. M. (2018). External validation of change formulae in neuropsychology with neuroimaging biomarkers: A methodological recommendation and preliminary clinical data. The Clinical Neuropsychologist, 33(3), 478–489. https://doi.org/10.1080/13854046.2018.1484518
Harvey, P. D. (2012). Clinical applications of neuropsychological assessment. Dialogues in Clinical Neuroscience, 14(1), 91–99. https://doi.org/10.31887/dcns.2012.14.1/pharvey
Hageman, W. J. J. M., & Arrindell, W. A. (1993). A further refinement of the reliable change (RC) index by improving the pre-post difference score: Introducing RCID. Behaviour Research and Therapy, 31(7), 693–700. https://doi.org/10.1016/0005-7967(93)90122-b
Hageman, W. J. J. M., & Arrindell, W. A. (1999). Establishing clinically signi®cant change: increment of precision and the distinction between individual and group level of analysis. Behaviour Research and Therapy, 37(12), 1169–1193. https://doi.org/10.1016/s0005-7967(99)00032-7
Hammers, D. B., & Duff, K. (2019). Application of different standard error estimates in reliable change methods. Archives of Clinical Neuropsychology, 36(3), 339–346. https://doi.org/10.1093/arclin/acz054
Hammers, D. B., Porter, S., Dixon, A., Suhrie, K. R., & Duff, K. (2020). Validating 1-year reliable change methods. Archives of Clinical Neuropsychology, 36(1), 87–98. https://doi.org/10.1093/arclin/acaa055
Hammers, D. B., Suhrie, K. R., Dixon, A., Porter, S., & Duff, K. (2020). Reliable change in cognition over 1 week in community-dwelling older adults: A validation and extension study. Archives of Clinical Neuropsychology, 36(3), 347–358. https://doi.org/10.1093/arclin/acz076
Hammers, D. B., Suhrie, K. R., Porter, S. M., Dixon, A. M., & Duff, K. (2020). Validation of one-year reliable change in the RBANS for community-dwelling older adults with amnestic mild cognitive impairment. The Clinical Neuropsychologist, 36(6), 1304–1327. https://doi.org/10.1080/13854046.2020.1807058
Heaton, R. K., Temkin, N., Dikmen, S., Avitable, N., Taylor, M. J., Marcotte, T. D., & Grant, I. (2001). Detecting change: A comparison of three neuropsychological methods, using normal and clinical samples. Archives of clinical neuropsychology : the official journal of the National Academy of Neuropsychologists, 16(1), 75–91.
Heilbronner, R. L., Sweet, J. J., Attix, D. K., Krull, K. R., Henry, G. K., & Hart, R. P. (2010). Official position of the American Academy of Clinical Neuropsychology on serial neuropsychological assessments: The utility and challenges of repeat test administrations in clinical and forensic contexts. The Clinical Neuropsychologist, 24(8), 1267–1278. https://doi.org/10.1080/13854046.2010.526785
Hermann, B. P., Wyler, A. R., Vanderzwagg, R., LeBailly, R. K., Whitman, S., Somes, G., & Ward, J. (1991). Predictors of neuropsychological change following anterior temporal lobectomy: Role of regression toward the mean. Journal of Epilepsy, 4(3), 139–148. https://doi.org/10.1016/s0896-6974(05)80039-8
Hsu L. M. (1999). A comparison of three methods of identifying reliable and clinically significant client changes: commentary on Hageman and Arrindell. Behaviour research and therapy, 37(12), 1195–1233. https://doi.org/10.1016/s0005-7967(99)00033-9
Iverson, G. L., & Schatz, P. (2014). Advanced topics in neuropsychological assessment following sport-related concussion. Brain Injury, 29(2), 263–275. https://doi.org/10.3109/02699052.2014.965214
Iverson, G.L. (2012). Interpreting change on repeated neuropsychological assessments of children. In E. Sherman and B. Brooks (eds.), Pediatric Forensic Neuropsychology, pp. 89-112. New York: Oxford University Press.
Iverson, G. L. (2018). Reliable change index. Encyclopedia of Clinical Neuropsychology, 1–4. https://doi.org/10.1007/978-3-319-56782-2_1242-3
Jacobson, N. S., & Truax, P. (1991). Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. Journal of consulting and clinical psychology, 59(1), 12–19. https://doi.org/10.1037//0022-006x.59.1.12
Knight, R. G., McMahonn, J., Skeaff, C. M., & Green, T. J. (2007). Reliable change index scores for persons over the age of 65 tested on alternate forms of the Rey Avlt. Archives of Clinical Neuropsychology, 22(4), 513–518. https://doi.org/10.1016/j.acn.2007.03.005
Rinehardt, E. (Ed.). (2014). National academy of neuropsychology bulletin. https://www.e-digitaleditions.com/i/401821-nan-fall-bulletin/0?
Pagnacco, G., Carrick, F. R., Wright, C. H. G., & Oggero, E. (2015). Between-subjects differences of within-subject variability in repeated balance measures: Consequences on the minimum detectable change. Gait & Posture, 41(1), 136–140. https://doi.org/10.1016/j.gaitpost.2014.09.016
Port, J. (Ed.). (2010, October). PsyPag quarterly. https://explore.bps.org.uk/content/bpspag
Reed, C., Calamia, M., Sanderson-Cimino, M., DeVito, A., Toups, R., & Keller, J. (2023). Four Year practice effects on the RBANS in a longitudinal study of older adults. Applied Neuropsychology: Adult, 1–7. https://doi.org/10.1080/23279095.2023.2180361
Reid-Arndt, S. A., Hsieh, C., & Perry, M. C. (2010). Neuropsychological functioning and quality of life during the first year after completing chemotherapy for breast cancer. Psycho-Oncology, 19(5), 535–544. https://doi.org/10.1002/pon.1581
Rinehardt, E., Duff, K., Schoenberg, M., Mattingly, M., Bharucha, K., & Scott, J. (2010). Cognitive change on the repeatable battery of neuropsychological status (RBANS) in parkinson’s disease with and without bilateral subthalamic nucleus deep brain stimulation surgery. The Clinical Neuropsychologist, 24(8), 1339–1354. https://doi.org/10.1080/13854046.2010.521770
Sawrie, S. M., Marson, D. C., Boothe, A. L., & Harrell, L. E. (1999). A method for assessing clinically relevant individual cognitive change in older adult populations. The journals of gerontology. Series B, Psychological sciences and social sciences, 54(2), P116–P124. https://doi.org/10.1093/geronb/54b.2.p116
Schmitt, J. S., & Di Fabio, R. P. (2004). Reliable change and minimum important difference (MID) proportions facilitated group responsiveness comparisons using individual threshold criteria. Journal of Clinical Epidemiology, 57(10), 1008–1018. https://doi.org/10.1016/j.jclinepi.2004.02.007
Schoenberg, M. R., Rinehardt, E., Duff, K., Mattingly, M., Bharucha, K. J., & Scott, J. G. (2012). Assessing reliable change using the repeatable battery for the assessment of neuropsychological status (RBANS) for patients with parkinson’s disease undergoing deep brain stimulation (DBS) surgery. The Clinical Neuropsychologist, 26(2), 255–270. https://doi.org/10.1080/13854046.2011.653587
Sherman, E. M., Wiebe, S., Fay-McClymont, T. B., Tellez-Zenteno, J., Metcalfe, A., Hernandez-Ronquillo, L., Hader, W. J., & Jetté, N. (2011). Neuropsychological outcomes after epilepsy surgery: Systematic Review and pooled estimates. Epilepsia, 52(5), 857–869. https://doi.org/10.1111/j.1528-1167.2011.03022.x
Stein, J., Luppa, M., Brähler, E., König, H.-H., & Riedel-Heller, S. G. (2010). The assessment of changes in cognitive functioning: Reliable change indices for neuropsychological instruments in the elderly – a systematic review. Dementia and Geriatric Cognitive Disorders, 29(3), 275–286. https://doi.org/10.1159/000289779
Stratford, P. W., Binkley, J., Solomon, P., Finch, E., Gill, C., & Moreland, J. (1996). Defining the minimum level of detectable change for the Roland-Morris questionnaire. Physical therapy, 76(4), 359–368. https://doi.org/10.1093/ptj/76.4.359
Temkin, N. R., Heaton, R. K., Grant, I., & Dikmen, S. S. (1999). Detecting significant change in neuropsychological test performance: A comparison of four models. Journal of the International Neuropsychological Society, 5(4), 357–369. https://doi.org/10.1017/s1355617799544068
Weir J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of strength and conditioning research, 19(1), 231–240. https://doi.org/10.1519/15184.1
