Authors: Gazendam, Luit¹; Wartena, Christian¹; Malaisé, Véronique²; Schreiber, Guus²; de Jong, Annemieke³; Brugman, Hennie⁴

Source: Interdisciplinary Science Reviews, Volume 34, Numbers 2-3, September 2009 , pp. 172-188(17)

Publisher: Maney Publishing

Abstract:

In the context of large and ever growing archives, generating annotation suggestions automatically from textual resources related to the documents to be archived is an interesting option in theory. It could save a lot of work in the time consuming and expensive task of manual annotation and it could help cataloguers attain a higher inter-annotator agreement. However, some questions arise in practice: what is the quality of the automatically produced annotations? How do they compare with manual annotations and with the requirements for annotation that were defined in the archive? If different from the manual annotations, are the automatic annotations wrong? In the CHOICE project, partially hosted at the Netherlands Institute for Sound and Vision, the Dutch public archive for audiovisual broadcasts, we automatically generate annotation suggestions for cataloguers. In this paper, we define three types of evaluation of these annotation suggestions: (1) a classic and strict evaluation measure expressing the overlap between automatically generated keywords and the manual annotations, (2) a loosened evaluation measure for which semantically very similar annotations are also considered as relevant matches, and (3) an in-use evaluation of the usefulness of manual versus automatic annotations in the context of serendipitous browsing. During serendipitous browsing, the annotations (manual or automatic) are used to retrieve and visualize semantically related documents.

Keywords: SEMANTIC BROWSING; EVALUATION; EXTRACTION; SEMANTIC EVALUATION; AUDIOVISUAL ARCHIVES; AUTOMATIC ANNOTATION

Document Type: Research Article

DOI: http://dx.doi.org/10.1179/174327909X441090

Affiliations: 1: Novay, Enschede, The Netherlands 2: Department of Computer Science, Vrije Universiteit Amsterdam, The Netherlands 3: Netherlands Institute for Sound and Vision, Hilversum, The Netherlands 4: Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands

Publication date: 2009-09-01

More about this publication?

• Editorial Board
• Information for Authors
• Subscribe to this Title
• Membership Information
• Information for Advertisers
• Terms & Conditions
• Read the journal blog
• Top articles
• ingentaconnect is not responsible for the content or availability of external websites

Related content

• In this: publication
• By this: publisher
• In this Subject: Arts (General) ,  Social Science (General)
• By this author: Gazendam, Luit ;  Wartena, Christian ;  Malaisé, Véronique ;  Schreiber, Guus ;  de Jong, Annemieke ;  Brugman, Hennie

Website © Publishing Technology. Article copyright remains with the publisher, society or author(s) as specified within the article.

• Terms and conditions
• Privacy policy
• Information for advertisers