Linked Hypernyms Dataset

v1.3.8, v2.3.8_draft

Evaluation Guidelines - Overall accuracy

This document describes the guidelines for evaluating hypernyms and linked hypernyms in the linked hypernyms dataset and its subset.

INPUT FOR EVALUATION

The input for evaluation is a random sample containing records, each containing the following fields:

ARTICLE: the full Wikipedia article URL, there is an unanimous mapping to a DBpedia resource by replacing en.wikipedia.org/wiki by dbpedia.org/resource in the URL
PLAIN TEXT HYPERNYM: the hypernym discovered from the article text
DISAMBIGUATED HYPERNYM: disambiguation of the hypernym to a Wikipedia page (again, unanimously mappable to a DBpedia resource) or DBpedia ontology concept.

Examples

ARTICLE	PLAIN TEXT HYPERNYM	DISAMBIGUATED HYPERNYM
http://en.wikipedia.org/wiki/Olympus_E-1	camera	http://wikipedia.org/wiki/Camera
http://en.wikipedia.org/wiki/Legal_case	dispute	http://wikipedia.org/wiki/Dispute
http://en.wikipedia.org/wiki/Midnight_Magic_(Atari_2600)	game	http://dbpedia.org/ontology/Game

FIELDS FILLED BY THE RATER

There are the following fields:

PLAIN TEXT HYPERNYM CORRECT
DISAMBIGUATED HYPERNYM CORRECT
DISAMBIGUATED HYPERNYM CORRECT
DIFFERENT SENSE
OVERLAPPING SENSE
REDIRECT SENSE SHIFT
ENTITY NOT INSTANCE
DISCOVERED HYPERNYM NOT FIRST
NO HYPERNYM IN ARTICLE

PLAIN TEXT HYPERNYM CORRECT

Empty: the article does not contain any hypernym. Also see the NO HYPERNYM IN ARTICLE field.

value	condition
1	At least one of the senses of the PLAIN TEXT HYPERNYM is a correct hypernym
0	Otherwise

DISAMBIGUATED HYPERNYM CORRECT

valuecondition 1 Description of the DISAMBIGUATED HYPERNYM includes a supersense for the entity 0Otherwise. The exact error type is indicated by 1 in one of the DIFFERT SENSE, OVERLAPPING SENSE and REDIRECT ISSUE fields.

Examples

ARTICLE	PLAIN TEXT HYPERNYM	DISAMBIGUATED HYPERNYM	PLAIN TEXT HYPERNYM CORRECT	DISAMBIGUATED HYPERNYM CORRECT
http://en.wikipedia.org/wiki/Thomas_Clifford,_8th_Baron_de_Clifford	member	http://dbpedia.org/ontology/Person	0	1

The correct PLAIN TEXT hypernym is "son", however, the DISAMBIGUATED HYPERNYM is correct.

ARTICLE	PLAIN TEXT HYPERNYM	DISAMBIGUATED HYPERNYM	PLAIN TEXT HYPERNYM CORRECT	DISAMBIGUATED HYPERNYM CORRECT
http://en.wikipedia.org/wiki/Lakes_96FM	station	http://dbpedia.org/ontology/Station	1	0

The DISAMBIGUATED HYPERNYM definition covers public transport station (train, bus), while the entity describes radio station.

Note: For more detailed guidelines when the DISAMBIGUATED HYPERNYM points at a disambiguation concept (i.e. the corresponding Wikipedia page is a disambiguation page), refer to the description provided for the DISAMBIGUATED HYPERNYM IS AMBIGUOUS field.

DIFFERENT SENSE

value	condition
1	DISAMBIGUATED HYPERNYM does not include even a close “supersense” for the entity.
Empty	Otherwise.

Examples

ARTICLE	PLAIN TEXT HYPERNYM	DISAMBIGUATED HYPERNYM	PLAIN TEXT HYPERNYM CORRECT	DISAMBIGUATED HYPERNYM CORRECT	DIFFERENT SENSE
http://en.wikipedia.org/wiki/Lakes_96FM	station	http://dbpedia.org/ontology/Station	1	0	1

Note: This field is evaluated only if DISAMBIGUATED HYPERNYM CORRECT = 0

OVERLAPPING SENSE

value	condition
1	Description of the type partly, but not completely, fits the role of the supersense for the entity .
Empty	Otherwise.

Examples

ARTICLE	PLAIN TEXT HYPERNYM	DISAMBIGUATED HYPERNYM	PLAIN TEXT HYPERNYM CORRECT	DISAMBIGUATED HYPERNYM CORRECT	OVERLAPPING SENSE
http://en.wikipedia.org/wiki/KM3	language	http://dbpedia.org/ontology/Language	1	0	1

The Wikipedia article describes a computer language, while the definition of the disambiguated hypernym covers natural language.

Note: This field is evaluated only if DISAMBIGUATED HYPERNYM CORRECT = 0

REDIRECT SENSE SHIFT

value	condition
1	The DISAMBIGUATED HYPERNYM is redirected to another, less relevant concept.
Empty	Otherwise.

Examples

ARTICLE	PLAIN TEXT HYPERNYM	DISAMBIGUATED HYPERNYM	PLAIN TEXT HYPERNYM CORRECT	DISAMBIGUATED HYPERNYM CORRECT	REDIRECT SENSE SHIFT
http://en.wikipedia.org/wiki/Nightwatchman_(cricket)	batsman	http://dbpedia.org/resource/Batsman	1	0	1

The PLAIN TEXT HYPERNYM is correct, the disambiguated hypernym is an existing DBpedia concept. If this concept had a definition, it would probably match the use of batsman in the Wikipedia article. However, Dbpedia redirects http://dbpedia.org/resource/Batsman to http://dbpedia.org/resource//Batting_%28cricket%29, which is no longer a correct sense.

Note: This field is evaluated only if DISAMBIGUATED HYPERNYM CORRECT = 0

ENTITY NOT INSTANCE

value	condition
1	The Wikipedia article does not describe an instance. This is most commonly the case when the article is a disambiguation page.
Empty	Otherwise.

Examples

ARTICLE	PLAIN TEXT HYPERNYM	DISAMBIGUATED HYPERNYM	PLAIN TEXT HYPERNYM CORRECT	DISAMBIGUATED HYPERNYM CORRECT	ENTITY NOT INSTANCE
http://en.wikipedia.org/wiki/Mahari	acronym	http://wikipedia.org/wiki/Acronym	1		1

HYPERNYM IS AMBIGUOUS

value	condition
1	DISAMBIGUATED HYPERNYM is a disambiguation concept (the corresponding Wikipedia page is a disambiguation page).
Empty	Otherwise.

Examples

ARTICLE	PLAIN TEXT HYPERNYM	DISAMBIGUATED HYPERNYM	PLAIN TEXT HYPERNYM CORRECT	DISAMBIGUATED HYPERNYM CORRECT	DISAMBIGUATED HYPERNYM IS AMBIGUOUS
http://en.wikipedia.org/wiki/Personal_account	account	http://wikipedia.org/wiki/Account	1	1	1

Note: if the value is 1 then the value of the DISAMBIGUATED HYPERNYM CORRECT is determined based on the content of all the senses referenced. The senses are assessed based on the content of the respective Wikipedia pages. For the DISAMBIGUATED HYPERNYM CORRECT is set to 1 if and only if at least one of the senses matches. If DISAMBIGUATED HYPERNYM CORRECT is set to 0, the type of error is indicated as follows: If none of the senses is correct, but at least one is overlapping, the OVERLAPPING SENSE is set to 1. If none of the senses is correct or overlapping, DIFFERENT SENSE is set to 1.

DISCOVERED HYPERNYM NOT FIRST

THD algorithm aims to extract the first hypernym in the Wikipedia article. This field provides supplemental information on the count of cases when other than the first hypernym was retrieved.

value	condition
1	PLAIN TEXT HYPERNYM is not the first hypernym in the article (hypernyms that are part of the article name are not considered).
Empty	Otherwise.

Examples

ARTICLE	PLAIN TEXT HYPERNYM	DISAMBIGUATED HYPERNYM	PLAIN TEXT HYPERNYM CORRECT	DISCOVERED HYPERNYM NOT FIRST
http://en.wikipedia.org/wiki/Pluridens	masasaur	1	1

The first hypernym in the article is lizard, nevertheless, the masasaur hypernym is also correct. The hypernym is extracted from the first sentence in the Wikipedia article: Halisaurus ("Ocean Lizard") is an extinct genus of marine lizard belonging to the mosasaur family.

NO HYPERNYM IN ARTICLE

value	condition
1	The article does not contain any hypernym (hypernyms that are part of the article name are not considered).
Empty	Otherwise.

Note: If the value is 1 then fields PLAIN TEXT HYPERNYM CORRECT and DISAMBIGUATED HYPERNYM CORRECT are set to empty string.

Examples

ARTICLE	PLAIN TEXT HYPERNYM	DISAMBIGUATED HYPERNYM	PLAIN TEXT HYPERNYM CORRECT	DISAMBIGUATED HYPERNYM CORRECT	NO HYPERNYM IN ARTICLE
http://en.wikipedia.org/wiki/SCSI_command	unit	http://dbpedia.org/ontology/Organisation			1