Systematic and Random Method Effects. Estimating Method Bias and Method Variance

Germà Coenders and Willem E. Saris

Two lines of research have been followed to assess the effect of the method of measurement on data quality. Split-ballot experiments allowed researchers to assess the effect of the measurement method on the marginal distribution of the responses and to make assessments of relative bias. Structural equation models allowed researchers to assess the effect of the method on reliability and validity.

This paper develops and illustrates a strategy based on fitting mean-and-covariance structure models to multitrait-multimethod data, which allows researchers to assess relative bias, reliability and validity simultaneously. Two major advantages of this approach over split-ballot designs are that relative bias is assessed after partialling out the effects of measurement errors and that alternative definitions of relative bias are possible. A complete sequence of statistical tests of relative unbiasedness of methods is provided and applied.