Skip to main content
    Sample Size CalculatorBeta

    Method document

    Sample size to estimate Cohen’s kappa

    How many participants two raters must both rate, yes or no, to estimate their agreement beyond chance, Cohen’s kappa, within a chosen margin. Covers non-response.

    Engine version 1.0.0 · Reviewed 9 October 2026

    Open in calculator

    See also: Sample size to show that Cohen’s kappa exceeds a benchmark

    https://statwisely.com/methods/precision-kappa/

    When to use it

    • Inter-rater agreement on a binary judgement, such as the presence of a sign on examination, reported with a confidence interval.

    Not for

    • A rating of several categories, or ordered ones; this planner is for a yes/no rating.

    Notation

    nn
    Number of participants rated by both raters
    κ\kappa
    Expected kappa
    pp
    Share of ratings that are “yes”, the same for both raters
    VV
    Variance of kappa per participant
    EE
    Margin of error
    z1−α/2z_{1-\alpha/2}
    Standard normal quantile for confidence 1 − α (1.960 for 95%)
    qq
    Non-response rate

    Formulas

    Participants from the variance of kappa

    n=⌈z1−α/22 VE2⌉,V=(1−κ)[(1−κ)(1−2κ)+κ(2−κ)2p(1−p)]n = \left\lceil \frac{z_{1-\alpha/2}^{2}\, V}{E^{2}} \right\rceil, \qquad V = (1-\kappa)\left[(1-\kappa)(1-2\kappa) + \frac{\kappa(2-\kappa)}{2p(1-p)}\right]

    V is the large-sample variance of kappa when both raters say “yes” with probability p [1]. It grows as “yes” becomes rare or nearly universal, which is why kappa is hard to estimate for uncommon findings.

    Non-response

    napproach=⌈n1−q⌉n_{\text{approach}} = \left\lceil \frac{n}{1 - q} \right\rceil

    The number to approach so that n respond. Applied last, after any clustering.

    Parameters

    Confidence level1−α1-\alpha

    Values: 90%, 95% or 99%

    95% is the convention in health research. A higher level widens the interval, so the same margin needs a larger sample.

    Margin of errorEE

    Values: Above 0 and below 1

    Half the interval’s width on the coefficient’s scale: 0.1 gives an interval from 0.5 to 0.7 around a kappa of 0.6.

    Expected kappaκ\kappa

    Values: Between 0 and 1

    From a pilot or published agreement studies of similar raters.

    Share rated “yes”pp

    Values: Between 0 and 1

    How common the finding is among those to be rated.

    Expected non-response rateqqOptional

    Values: 0% up to, but not including, 100%

    The share of people approached who will not take part; use the response rate of a comparable study. Approaching more people restores the number of respondents, not their representativeness: if non-responders differ, the estimate can still be biased.

    Assumptions

    • Each participant is rated once by each of two raters, who say “yes” equally often; participants are independent.
    • Non-response is unrelated to the outcome.

    Worked examples

    A finding in three in ten

    Kappa 0.6 within ±0.1 with 0.3 rated “yes”: V = 0.4 × (−0.08 + 0.84/0.42) = 0.768, so n = 1.96² × 0.768 / 0.1² = 295.0, rounded up to 296.

    Sample size: 296

    Open this example in the calculator

    A rare finding

    With 0.1 rated “yes”: n = 705.

    Sample size: 705

    Open this example in the calculator

    An even split

    With 0.5 rated “yes”, the most informative case: n = 246.

    Sample size: 246

    Open this example in the calculator

    Validation

    • Every worked example on this page is recomputed by the StatWisely engine whenever the software is built. If a number or formula here stopped matching the calculator, the release would be stopped until this document was corrected.
    • Kappa from simulated two-rater tables at the planned size varied with a standard deviation that gives a 95% half-width of 0.100 against a target of 0.10. An independent implementation in R recomputes these sizes and is compared with the engine’s.

    How to cite

    StatWisely. Sample size to estimate Cohen’s kappa. Method document, engine version 1.0.0, reviewed 2026-10-09. https://statwisely.com/methods/precision-kappa/

    Cite the original references for the formulas your design uses as well.

    References

    1. 1.Bloch DA, Kraemer HC. 2 × 2 kappa coefficients: measures of agreement or association. Biometrics. 1989;45(1):269–287.