Confounds of insensitivity and blind luck: Statistical conclusion validity in stroke rehabilitation clinical trials

Thomas A. Matyas, Kenneth Ottenbacher

Research output: Contribution to journalArticle

13 Citations (Scopus)

Abstract

We present theoretical analyses of some infrequently considered consequences of low power and quantitatively review power in 36 stroke rehabilitation clinical trials with multiple outcome variables. We demonstrate that low power affects the proportion of false alarms (type 1 errors) in the literature and probability of replication, not only the ability to detect true intervention effects; and that these consequences of low power are modulated by the probability that effective interventions have been preselected for study via theory and prior evidence. Rehabilitation trials showed low power to detect small, medium, or large treatment effects at α = 0.05 (0.09, 0.33, and 0.69, respectively). Effect sizes were generally small with adverse consequences on power. As expected, sample size was an important determinant of power. Power was lower for motor/reflex measures due to differences in sample sizes rather than effect sizes. These results and the conceptual analyses predict and address conflicting conclusions from stroke rehabilitation clinical trials.

Original languageEnglish (US)
Pages (from-to)559-565
Number of pages7
JournalArchives of Physical Medicine and Rehabilitation
Volume74
Issue number6
DOIs
StatePublished - 1993
Externally publishedYes

Fingerprint

Clinical Trials
Sample Size
Aptitude
Power (Psychology)
Stroke Rehabilitation
Reflex
Rehabilitation

Keywords

  • Replication
  • Research design
  • Statistical

ASJC Scopus subject areas

  • Rehabilitation

Cite this

@article{f7c4b36dce0a474192ad3c75d4801ed8,
title = "Confounds of insensitivity and blind luck: Statistical conclusion validity in stroke rehabilitation clinical trials",
abstract = "We present theoretical analyses of some infrequently considered consequences of low power and quantitatively review power in 36 stroke rehabilitation clinical trials with multiple outcome variables. We demonstrate that low power affects the proportion of false alarms (type 1 errors) in the literature and probability of replication, not only the ability to detect true intervention effects; and that these consequences of low power are modulated by the probability that effective interventions have been preselected for study via theory and prior evidence. Rehabilitation trials showed low power to detect small, medium, or large treatment effects at α = 0.05 (0.09, 0.33, and 0.69, respectively). Effect sizes were generally small with adverse consequences on power. As expected, sample size was an important determinant of power. Power was lower for motor/reflex measures due to differences in sample sizes rather than effect sizes. These results and the conceptual analyses predict and address conflicting conclusions from stroke rehabilitation clinical trials.",
keywords = "Replication, Research design, Statistical",
author = "Matyas, {Thomas A.} and Kenneth Ottenbacher",
year = "1993",
doi = "10.1016/0003-9993(93)90152-Z",
language = "English (US)",
volume = "74",
pages = "559--565",
journal = "Archives of Physical Medicine and Rehabilitation",
issn = "0003-9993",
publisher = "W.B. Saunders Ltd",
number = "6",

}

TY - JOUR

T1 - Confounds of insensitivity and blind luck

T2 - Statistical conclusion validity in stroke rehabilitation clinical trials

AU - Matyas, Thomas A.

AU - Ottenbacher, Kenneth

PY - 1993

Y1 - 1993

N2 - We present theoretical analyses of some infrequently considered consequences of low power and quantitatively review power in 36 stroke rehabilitation clinical trials with multiple outcome variables. We demonstrate that low power affects the proportion of false alarms (type 1 errors) in the literature and probability of replication, not only the ability to detect true intervention effects; and that these consequences of low power are modulated by the probability that effective interventions have been preselected for study via theory and prior evidence. Rehabilitation trials showed low power to detect small, medium, or large treatment effects at α = 0.05 (0.09, 0.33, and 0.69, respectively). Effect sizes were generally small with adverse consequences on power. As expected, sample size was an important determinant of power. Power was lower for motor/reflex measures due to differences in sample sizes rather than effect sizes. These results and the conceptual analyses predict and address conflicting conclusions from stroke rehabilitation clinical trials.

AB - We present theoretical analyses of some infrequently considered consequences of low power and quantitatively review power in 36 stroke rehabilitation clinical trials with multiple outcome variables. We demonstrate that low power affects the proportion of false alarms (type 1 errors) in the literature and probability of replication, not only the ability to detect true intervention effects; and that these consequences of low power are modulated by the probability that effective interventions have been preselected for study via theory and prior evidence. Rehabilitation trials showed low power to detect small, medium, or large treatment effects at α = 0.05 (0.09, 0.33, and 0.69, respectively). Effect sizes were generally small with adverse consequences on power. As expected, sample size was an important determinant of power. Power was lower for motor/reflex measures due to differences in sample sizes rather than effect sizes. These results and the conceptual analyses predict and address conflicting conclusions from stroke rehabilitation clinical trials.

KW - Replication

KW - Research design

KW - Statistical

UR - http://www.scopus.com/inward/record.url?scp=0027200303&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0027200303&partnerID=8YFLogxK

U2 - 10.1016/0003-9993(93)90152-Z

DO - 10.1016/0003-9993(93)90152-Z

M3 - Article

C2 - 8503744

AN - SCOPUS:0027200303

VL - 74

SP - 559

EP - 565

JO - Archives of Physical Medicine and Rehabilitation

JF - Archives of Physical Medicine and Rehabilitation

SN - 0003-9993

IS - 6

ER -