Skip to main content
    Sample Size CalculatorBeta

    Method document

    Sample size to show that Cohen’s kappa exceeds a benchmark

    How many participants two raters must both rate, yes or no, to show that their agreement exceeds a benchmark, by Donner and Eliasziw’s goodness-of-fit test. Covers dropout.

    Engine version 1.0.0 · Reviewed 9 October 2026

    Open in calculator

    See also: Sample size to estimate Cohen’s kappa

    https://statwisely.com/methods/measurement-kappa/

    When to use it

    • An inter-rater study of a binary judgement that must reach, say, moderate agreement (kappa above 0.40).

    Not for

    • A rating of several categories; this test is for a yes/no rating.

    Notation

    nn
    Number of participants rated by both raters
    κ0, κ1\kappa_0,\ \kappa_1
    Benchmark kappa, and the kappa expected
    pp
    Share of ratings that are “yes”
    PjiP_{ji}
    Probability of agreement cell i (both yes, one yes, both no) under κ_j
    z1−α, z1−βz_{1-\alpha},\ z_{1-\beta}
    Standard normal quantiles for the significance level and power
    α\alpha
    Significance level: the chance of a false positive
    1−β1-\beta
    Power: the chance of detecting the effect if it is real
    τ\tau
    Number of tails: 2 for a two-sided test, 1 for one-sided
    qq
    Expected dropout

    Formulas

    The goodness-of-fit test of kappa

    n=⌈(z1−α+z1−β)2∑i=13(P1i−P0i)2/P0i⌉n = \left\lceil \frac{(z_{1-\alpha} + z_{1-\beta})^{2}}{\sum_{i=1}^{3} (P_{1i} - P_{0i})^{2} / P_{0i}} \right\rceil

    Under kappa κ the three cells have probabilities p² + p(1 − p)κ, 2p(1 − p)(1 − κ) and (1 − p)² + p(1 − p)κ; the χ² comparing the observed cells with those under κ₀ gives the sample size [1]. The test’s α is halved for a two-sided test.

    Dropout

    nrecruit=⌈n1−q⌉n_{\text{recruit}} = \left\lceil \frac{n}{1 - q} \right\rceil

    Applied last, to the participants actually recruited.

    Parameters

    Expected kappaκ1\kappa_1

    Values: Between 0 and 1

    From a pilot or published agreement studies.

    Benchmark kappaκ0\kappa_0

    Values: Below the expected kappa

    0.40 is often read as moderate agreement and 0.60 as substantial.

    Share rated “yes”pp

    Values: Between 0 and 1

    Agreement is hardest to show when “yes” is rare.

    Significance levelα\alpha

    Values: Between 0 and 1; usually 0.05

    The false-positive rate you accept. A smaller α, for example to allow for several primary outcomes, needs a larger sample.

    Power1−β1-\beta

    Values: Between 0 and 1; usually 0.80 or 0.90

    The chance of a significant result if the assumed effect is real. Funders and ethics committees commonly expect at least 0.80.

    One- or two-sidedτ\tau

    Values: 2 (two-sided) or 1 (one-sided)

    Two-sided is the default. A one-sided test is justified only when an effect in the other direction would be treated exactly as no effect, and should be stated in advance.

    Expected dropoutqqOptional

    Values: 0% up to, but not including, 100%

    The share of participants expected to provide no outcome, from comparable studies. Recruiting more restores the numbers, not the comparability of the groups: if dropout is related to the outcome, the estimate can still be biased.

    Assumptions

    • Each participant is rated once by each of two raters, who say “yes” equally often.
    • Benchmark tests are one-sided by default: the property must be shown to exceed the benchmark, not merely to differ from it.
    • Dropout is unrelated to the outcome; recruiting more participants restores the numbers but cannot remove bias.

    Worked examples

    Kappa 0.6 against 0.4

    With 0.3 rated “yes”, the cells are 0.216, 0.168 and 0.616 against 0.174, 0.252 and 0.574, so Σ = 0.04121110176619007 and n = (1.645 + 0.842)² / 0.0412 = 150.0, rounded up to 151.

    Sample size: 151

    Protocol statement

    To show that kappa exceeds 0.4 when it is 0.6 with 80% power at a one-sided significance level of 0.05, using a goodness-of-fit test of Cohen’s kappa between two raters on a yes/no rating (Donner and Eliasziw), and assuming 30% of ratings "yes" from each rater, the required sample size is 151 participants. Calculated with StatWisely (statwisely.com) using: N = (z_(1-α)+z_(1-β))²/Σ(P₁ᵢ-P₀ᵢ)²/P₀ᵢ over the three agreement cells (Donner-Eliasziw). These are planning estimates; confirm the final design with a qualified statistician.

    Open this example in the calculator

    An even split

    With 0.5 rated “yes”: n = 130.

    Sample size: 130

    Open this example in the calculator

    Substantial agreement

    Kappa 0.8 against 0.6: n = 117.

    Sample size: 117

    Open this example in the calculator

    Validation

    • Every worked example on this page is recomputed by the StatWisely engine whenever the software is built. If a number or formula here stopped matching the calculator, the release would be stopped until this document was corrected.
    • An independent implementation in R recomputes this test every night and before each production release, and the results are compared with the engine’s.
    • Simulated two-rater tables at the planned size, tested by the goodness-of-fit χ² with the prevalence estimated, gave power of 0.82 against a target of 0.80: the approximation errs slightly on the safe side.

    How to cite

    StatWisely. Sample size to show that Cohen’s kappa exceeds a benchmark. Method document, engine version 1.0.0, reviewed 2026-10-09. https://statwisely.com/methods/measurement-kappa/

    Cite the original references for the formulas your design uses as well.

    References

    1. 1.Donner A, Eliasziw M. A goodness-of-fit approach to inference procedures for the kappa statistic: confidence interval construction, significance-testing and sample size estimation. Statistics in Medicine. 1992;11(11):1511–1519.