Authors: Hasnain M.¹; Onishi H.²; Elstein A.S.²

Source: Medical Education, Volume 38, Number 6, June 2004 , pp. 609-616(8)

Publisher: Wiley-Blackwell

Abstract:

Context and objective

Inter-rater agreement is essential in rating clinical performance of doctors and other health professionals. The purpose of this study was to establish inter-rater agreement in categorising errors in the diagnostic process made by clinicians using computerised decision support systems. Methods

Eight possible error categories were developed for coding errors in diagnostic hypotheses and plans for next steps in the work-up. Two independent doctor judges rated 54 work-ups (representing 2 cases, each worked-up by 27 doctors). Inter-rater agreement between the 2 raters was computed using the kappa coefficient. Results

High inter-rater agreement was achieved in all categories except where the manual was not sufficiently specific and raters had to use their judgement. As is typical of the kappa coefficient, however, agreement corrected for chance fell markedly into the ‘poor’ range when the percentages of expected and observed agreement were about the same. Conclusion

Raters can achieve good agreement in categorising errors provided they are given explicit scoring rules and do not rely solely upon clinical judgement. The kappa coefficient has limitations in cases where the expected agreement between judges is high and variability is low. The use of 2 indices to assess agreement, analogous to test sensitivity and specificity, is recommended.

Keywords: clinical competence/*standards; diagnostic errors/*standards; health personnel/*standards; *decision making; computers/methods; educational measurement; reproducibility of results

Document Type: Research article

DOI: http://dx.doi.org/10.1046/j.1365-2923.2004.01789.x

Affiliations: 1: Department of Family Medicine, College of Medicine, University of Illinois at Chicago, USA 2: Department of Medical Education, College of Medicine, University of Illinois at Chicago, USA

Publication date: 2004-06-01

Related content

• In this: publication
• By this: publisher
• In this Subject: Medicine (General)
• By this author: Hasnain M. ;  Onishi H. ;  Elstein A.S.

Website © Publishing Technology. Article copyright remains with the publisher, society or author(s) as specified within the article.

• Terms and conditions
• Privacy policy
• Information for advertisers