Entity Stamping Precision Judgment Guidelines

1 Task Overview
The goal of this task is to evaluate if an entity is one of the primary entities for a web page. A “primary entity” is the entity that is one
of the main topics for the web page. There could be multiple primary entities per page.

Once you log in to UHRS and have launched the HitApp, you will see a screen similar to this example below:

The HitApp shows two windows side-by-side:

• On the left is a web page.
• On the right is the entity pane with an information card for entity (People, Place, Organization, Event, etc.)
You need to judge if the entity in the right window is one of the primary entities of the web page on the left.

2 Judgment Instructions
2.1 Process Steps

1. Read the web page on the left.

2. Read the entity card on the right.
3. Determine whether the right entity is one of the main topics for the web page on the left.
4. Choose an option from below.
 a. Primary main subject: any of three conditions (i, ii or iii below) are met
i. If the webpage is clearly a homepage of the entity displayed or dedicated to it.
ii. If using one sentence to summarize the webpage and the entity is one of the main subjects of this sentence;
iii. If the main content of the webpage is a forum/discussion thread, the opening post of the thread should be
primarily about the entity. If the main content is a Q&A thread, either the question post or the best answer
post (if exists) should be primarily about the entity.
b. Mentioned but not Primary: any of the following conditions are met
i. The entity is mentioned outside of its main content area like navigation menu, sidebar, footer, or
advertisement OR
ii. The entity is mentioned in the left web page in the main content area BUT the entity is not the primary
main subject as explained in (a)
c. Not Mention: any of the following conditions are met
i. The entity is not mentioned in the main content area of the left web page OR
ii. The entity displayed on the right is not the entity referenced in the main content area.
d. Can’t tell: The following conditions are met:
i. If the left web page content is not loaded/displayed (in HitApp OR via click) OR
ii. If the entity information card is not loaded/displayed (in HitApp OR via click) OR
iii. The page is in a foreign language that you don’t understand; OR
iv. There is not enough information for you to make a judgment

5. Click “Submit”.

2.2 Note
 If the URL links in entity card do not load for any reason, please follow the URL link to view the webpage.
 Some webpages refer to an entity by an alternative name that is different from the one presented in the entity information
card. For example, http://www.screenjunkies.com/tag/the-rock-106/ refers to Dwayne Johnson by his ring name “The Rock”;
http://www.yelp.com/biz/the-bush-man refers to David Johnson (http://en.wikipedia.org/wiki/World_Famous_Bushman)
as “The Bush Man”. They are considered as the same entity. http://www.yelp.com/biz/the-bush-man-san-francisco

3 Rating Examples

Rating Description Examples

Entity Information Left Web Page (content URL)

Card

(a) Web page is Skyfall https://www.youtube.com/watch?

Primary primarily about or "Skyfall" is the theme song of the 2012 James Bond film v=q-gLRp5bSpw or
Main dedicated to the Skyfall, performed by British singer Adele. It was co- http://www.007. com/adeleskyfall-
Subject entity. written by Adele and producer Paul Epworth and features themesong-is-live/
orchestration by J. A. C. Redford. Film company Eon
Productions invited the singer to work on the theme song
in early 2011, a task that Adele accepted after reading the
film's script. While composing the song, Adele and
 Epworth aimed to capture the mood and style of the
other Bond themes, including dark and moody lyrics
descriptive of the film's plot....
http://www.lyrics.com/skyfall-lyrics-adele.html

Adele http://www.adele.tv/ (Adele’s official

Adele Laurie Blue Adkins MBE is an English singer- website)
songwriter. After graduating from the BRIT School for http://www.metr olyrics.com/adel e-
Performing Arts and Technology in 2006, Adele was given lyrics.html (A page dedicated to Adele
a recording contract by XL Recordings after a friend with the lyrics of her songs)
posted her demo on Myspace the same year. In 2007, she
received the Brit Awards "Critics' Choice" award and won
the BBC Sound of 2008 poll. Her debut album, 19, was
released in 2008 to commercial and critical success. It is
certified seven times platinum in the UK, and three times
platinum in the US. The album contains her first song,
"Hometown Glory", written when she was 16, which i...
https://en.wikipedia.org/wiki/Adele

Danny Sullivan http://www.linke din.com/in/dann

Danny Sullivan is the Chief Content Officer at Third Door ysullivan
Media, and co-founder of Search Engine Land, an industry http://twitter.co m/dannysullivan (main
publication that covers news and information about twitter page of Danny Sullivan)
search engines, and search marketing, SEO and SEM http://daggle.com/ (Blog of Danny
topics. Third Door Media also produces Marketing Land, a Sullivan)
sister website that covers broader digital marketing topics
including social media, display advertising, email
marketing, analytics, mobile, and marketing technology.
Search Engine Land and Marketing Land are owned by
Third Door Media, of which Danny Sullivan is partner and
chief content officer....
http://en.wikipe dia.org/wiki/Dan ny_Sullivan_(tec
hnologist)

(b) Danny Sullivan http://daggle.com/my-fall-of-the-berlin-

Not The entity is Danny Sullivan is the Chief Content Officer at Third Door wall-story-3509 (Danny Sullivan’s blog
Mentio mentioned in Media, and co-founder of Search Engine Land, an industry post about berlin wall
n the left web page publication that covers news and information about )
main content but search engines, and search marketing, SEO and SEM
the page is not topics. Third Door Media also produces Marketing Land, a
primarily about sister website that covers broader digital marketing topics
the entity OR the including social media, display advertising, email
entity is
marketing, analytics, mobile, and marketing technology.
mentioned
Search Engine Land and Marketing Land are owned by
outside of its Third Door Media, of which Danny Sullivan is partner and
main content chief content officer....
area (such as http://en.wikipe dia.org/wiki/Dan ny_Sullivan_(tec
navigation hnologist)
menu,
sidebar, footer, or Tim Tebow
advertisement) Timothy Richard Tebow is a former professional American
OR the entity is football player and current professional baseball
NOT mentioned outfielder in the New York Mets organization. He played
at all. college football for the University of Florida, winning the
Heisman Trophy in 2007 and appearing on BCS National
Championship-winning teams during the 2006 and 2008
http://www.linkedin.com/pub/tim-
seasons. Tebow was selected by the Denver Broncos in
tebow/13/28/b50 (A different person)
the first round of the 2010 NFL Draft and spent two
seasons with the team. He also played for the New York
Jets in 2012. Additionally, he had preseason stints with
the New England Patriots and the Philadelphia Eagles in
http://en.wikipedia.org/wi
ki/Tim_Tebow

Movies
"Movies" is a song by Alien Ant Farm, released as the first
single from their album Anthology in 2001, then re-
released to a larger audience after the success of
"Smooth Criminal". Though it peaked only at No. 18 on http://www.mov ies.com/ (A different
the US Modern Rock chart, it remained on the chart for entity)
thirty-two weeks, five weeks longer than "Smooth
Criminal" which hit No.
http://en.wikipedia.org/wiki/Movies_(song)

Michael Jordan
Michael Jeffrey Jordan, also known by his initials, MJ, is
an American retired professional basketball player,
businessman, and principal owner and chairman of the
Charlotte Hornets. Jordan played 15 seasons in the
National Basketball Association for the Chicago Bulls and
Washington Wizards. His biography on the NBA website http://www.cs.b erkeley.edu/~jor dan/
states: "By acclamation, Michael Jordan is the greatest (A different person)
basketball player of all time." Jordan was one of the most
effectively marketed athletes of his generation and was
considered instrumental in popularizing the NBA around
the world in the 1980s and 1990s....
http://en.wikipedia.org/wiki/Michael_Jordan
 UPS Airlines
https://www.ups.com/
UPS Airlines is an American cargo airline owned by United
(note that the page shown has
Parcel Service Inc.. The company is headquartered in
interaction needed to see actual page, so
Louisville, Kentucky. Its international all-points air hub,
you are expected to click the url in
Worldport, is based at Louisville International Airport. The
HitApp and judge that.).
pilots are represented by the Independent Pilots
In this case the entity is specific but page
Association....
is general and does not mentioned the
http://en.wikipedia.org/wiki/UPS_Airlines
entity in topic area.

4 FAQ
a) Q: The page sometimes fails to load or content is blocked with a popup/overlay.

IF the left web page fails to load or content is blocked with popup/overlay, please open the page via content link given and
continue judging.

b) Q: I am using Internet Explorer 11 and I see the following message pop up when I open the HitApp, what should I do?

A: Click “Options for this site” button, and then click “Always allow” in the drop-down menu.

If see the “Are you sure you want to leave this page” dialog afterwards, click “Leave this page”, and the page will be
refreshed.

c) Q: I am using chrome and I see the alert icon in the address bar, what should I do?
 A: Click the red alert button, and then click “Load unsafe script” in the message pop up.

--You are ready judge now---