Skip to main content
    Sample Size CalculatorBeta

    Method document

    Sample size to show that an intraclass correlation exceeds a benchmark

    How many participants, each rated k times, are needed to show that test–retest or inter-rater reliability exceeds a benchmark, by the exact F test of the one-way ICC. Covers dropout.

    Engine version 1.0.0 · Reviewed 9 October 2026

    Open in calculator

    See also: Sample size to estimate an intraclass correlation

    https://statwisely.com/methods/measurement-icc/

    When to use it

    • A reliability study that must show the ICC exceeds a standard such as 0.75, or that compares designs with different numbers of raters.

    Not for

    • Reporting the ICC with a confidence interval, with no benchmark to meet. Use the intraclass correlation under “Estimate a value”.

    Notation

    nn
    Number of participants
    kk
    Ratings of each participant
    ρ0, ρ1\rho_0,\ \rho_1
    Benchmark ICC, and the ICC expected
    θ\theta
    ρ/(1 − ρ), the ratio of between- to within-participant variance
    Fa,bF_{a,b}
    F distribution on a and b degrees of freedom
    α\alpha
    Significance level: the chance of a false positive
    1−β1-\beta
    Power: the chance of detecting the effect if it is real
    τ\tau
    Number of tails: 2 for a two-sided test, 1 for one-sided
    qq
    Expected dropout

    Formulas

    Exact F test of the one-way ICC

    1−β=P ⁣(Fn−1, n(k−1)>1+kθ01+kθ1 Fn−1, n(k−1); 1−α)1-\beta = P\!\left(F_{n-1,\,n(k-1)} > \frac{1+k\theta_0}{1+k\theta_1}\, F_{n-1,\,n(k-1);\,1-\alpha}\right)

    The mean-square ratio MSB/MSW is (1 + kθ) times an F variable, so power is exact for every n; the smallest n reaching it is found by search [1].

    Dropout

    nrecruit=⌈n1−q⌉n_{\text{recruit}} = \left\lceil \frac{n}{1 - q} \right\rceil

    Applied last, to the participants actually recruited.

    Parameters

    Expected ICCρ1\rho_1

    Values: Between 0 and 1

    From a pilot or an earlier reliability study.

    Benchmark ICCρ0\rho_0

    Values: 0 or more, below 1

    0.75 is often read as good reliability and 0.50 as moderate.

    Ratings per participantkk

    Values: At least 2

    More ratings per participant need fewer participants, with diminishing returns beyond three or four.

    Significance levelα\alpha

    Values: Between 0 and 1; usually 0.05

    The false-positive rate you accept. A smaller α, for example to allow for several primary outcomes, needs a larger sample.

    Power1−β1-\beta

    Values: Between 0 and 1; usually 0.80 or 0.90

    The chance of a significant result if the assumed effect is real. Funders and ethics committees commonly expect at least 0.80.

    One- or two-sidedτ\tau

    Values: 2 (two-sided) or 1 (one-sided)

    Two-sided is the default. A one-sided test is justified only when an effect in the other direction would be treated exactly as no effect, and should be stated in advance.

    Expected dropoutqqOptional

    Values: 0% up to, but not including, 100%

    The share of participants expected to provide no outcome, from comparable studies. Recruiting more restores the numbers, not the comparability of the groups: if dropout is related to the outcome, the estimate can still be biased.

    Assumptions

    • A one-way random-effects model with normal true scores and errors.
    • Benchmark tests are one-sided by default: the property must be shown to exceed the benchmark, not merely to differ from it.
    • Dropout is unrelated to the outcome; recruiting more participants restores the numbers but cannot remove bias.

    Worked examples

    ICC 0.8 against 0.6, test–retest

    ICC 0.8 against 0.6 with 2 ratings: n = 39, as in Walter, Eliasziw and Donner’s table.

    Sample size: 39

    Protocol statement

    To show that the ICC exceeds 0.6 when it is 0.8 with 80% power at a one-sided significance level of 0.05, using an exact F test of the one-way intraclass correlation from 2 ratings per participant, the required sample size is 39 participants. Calculated with StatWisely (statwisely.com) using: N: exact F test of the one-way ICC, MSB/MSW ~ (1+kθ)F(n-1, n(k-1)) (Walter-Eliasziw-Donner). These are planning estimates; confirm the final design with a qualified statistician.

    Open this example in the calculator

    Three raters

    With 3 ratings per participant: n = 26.

    Sample size: 26

    Open this example in the calculator

    90% power

    At power 0.9: n = 54.

    Sample size: 54

    Open this example in the calculator

    Validation

    • Every worked example on this page is recomputed by the StatWisely engine whenever the software is built. If a number or formula here stopped matching the calculator, the release would be stopped until this document was corrected.
    • An independent implementation in R recomputes this test every night and before each production release, and the results are compared with the engine’s.
    • Simulated one-way designs at the planned size gave power of 0.799 against a planned 0.801; the sizes reproduce the published table of Walter, Eliasziw and Donner.

    How to cite

    StatWisely. Sample size to show that an intraclass correlation exceeds a benchmark. Method document, engine version 1.0.0, reviewed 2026-10-09. https://statwisely.com/methods/measurement-icc/

    Cite the original references for the formulas your design uses as well.

    References

    1. 1.Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Statistics in Medicine. 1998;17(1):101–110.