Effective ways for querying images by content

over the Internet

M. Vlachos1, G. Vardangalos2, C. Tatsiopoulos3

Abstract Content-based image retrieval, image database, keywords,

Increasing amounts of images have made explicit the need
for more effective and efficient image indexing and search of
not only the metadata associated with an image but also
retrieval directly on the content of the image. The scope of the
COBWEB project (Context-Based Image Retrieval on the
WEB - EP28773) is to create a pioneer system that will provide
a solution to the problem of automatically storing and
retrieving images from large image databases dispersed over
the Internet. The project focuses on the following issues:
• Development of effective algorithms for the automatic
storing of the visual features of the images that can be
used as a means for retrieving similar images.
• Creation of a simple, cross-platform user interface that
helps the user to query the database in the most natural
way.
• Development of a robust communication subsystem that
guarantees satisfying retrieval times.
In this paper, we will describe the architecture and
implementation details of the Human Computer Interface
(HCI) and the Intelligent Search Component (ISC), which are
two of the main components of the COBWEB system.
The HCI consists of a graphical interface, which enables the
user to interact with the system. It provides the user with
various approaches in querying the database including
keywords specification and sample image submission. The ISC
main tasks are to handle and translate the query criteria
submitted by the user through the HCI, navigate the database
and find images that satisfy those criteria and return the
results to the HCI.
Index Terms
visual features, algorithms, cross-platform, Java, Internet,
feature extaction, query, HCI, ISC, FEC
I. INTRODUCTION
Increasing amounts of images have made explicit the
need for more effective and efficient image indexing and
search of not only the metadata associated with an image
(e.g. annotations, captions, keywords) but also retrieval
directly on the content of the image (visual features).
The scope of the COBWEB project (Context-Based
Image Retrieval on the WEB - EP28773) is to create a
pioneer system that will provide a solution to the problem
of automatically storing and retrieving images from large
image databases dispersed over the Internet. The project
focuses on the following issues:
• Development of effective algorithms for the automatic
storing of the visual features of the images that can be
used as a means for retrieving similar images.
• Creation of a simple, cross-platform user interface that
helps the user perform queries and retrieve images in
the most natural way.
• Development of a robust communication subsystem
that guarantees satisfying retrieval times.
In this paper, we will describe the architecture and
implementation details of the Human Computer Interface
(HCI) and the Intelligent Search Component (ISC), which
are two of the main components of the COBWEB system.

II. OVERALL SYSTEM ARCHITECTURE

1
Epsilon Software SA, Figure 1 shows a simplified diagram of the overall
377 Syngrou Ave. COBWEB system structure. It presents in a simplified
175 64 Palaio Faliro Athens, Greece manner the physical architecture of the approach and
Tel: +30-1-9408370, Fax: +30-1-9427719 clarifies how functionality is distributed across the different
mike13@otenet.gr elements of the system. Furthermore, it shows how those
2
National Technical University of Athens, elements communicate with each other.
Dept. of Electrical and Computer Engineering,
Division of Computer Science,
Zografou Campus, GR-15773,
Tel: +30- 1- 7722425, Fax:+30 - 1 - 7722534
gvard@telecom.ntua.gr

3
Senior Software Engineer, IT Consultant,
11B Kodrigktonos Str. Athens, Greece,
Tel: +30 - 1 - 8230165
ctatsio@hol.gr
 Output Human
Environment
Computer
Interface
COBWEB user COBWEB user
results
thumbnails (image IDs)
return 1
(through Web Feature
Server)
4
query Extraction
criteria
Internet Component
6
uses
algorithms
COBWEB Intelligent
Administrator
Search 2

Component uses
Intelligent Search algorithms
mechanism
COBWEB 5
Web Server
thumbnail
3 Feature
request queries Comparison
images
image
database
Component
image preview visual
thumbnails images features

COBWEB
Image Database

COBWEB images
internal keywords

system Database Component

Figure 1

COBWEB is implemented as an Internet / Intranet- Figure 2

based application consisting of three logical tiers. The user
does not interact directly with the image database but
through a user-friendly, graphical interface that
communicates with a search engine. The search engine’s
intelligence lies in its ability to navigate the image database
and find images that can possibly correspond to the user’s
demands. The thumbnails of those images (and full-sized
previews) are available to the user interface for
downloading through the COBWEB’s web server. Thus, the
search engine functions as an intermediate tier between the
user interface and the image database.
Figure 2 explains the purpose of each functional
component of COBWEB in the context of the system’s
general architecture. Interactions between the components
of the system are also shown.
The Human Computer Interface (HCI) is the user’s
view of the COBWEB system. The user submits keywords,
sample images or sketches as criteria for his queries. The
above criteria are sent directly to the Intelligent Search
Component (ISC) i.e. the search engine. The ISC uses the
algorithms provided by the Feature Extraction
Component (FEC) to “translate” those sketches or images
into meaningful (for the ISC) feature vectors. Then, the ISC
searches into the database (Database Component -DC) for
images that satisfy the criteria using the algorithms
provided by the Feature Comparison Component (FCC).
Specifically, the ISC compares the keywords and the visual
feature vector of the sample image with the respective
keywords and feature vectors of the database’s images.
After the interaction with the DC has finished, the ISC
returns to the HCI a ranked list of image IDs. Finally, the
HCI fetches those images’ thumbnails (or larger, full-size
preview images) from the web server and presents them to
the user.
At the following paragraphs we will present the architecture
and implementation details of two of the main components
of the COBWEB system, the Human Computer Interface
(HCI) and the Intelligent Search Component (ISC).
III. HUMAN COMPUTER INTERFACE
A. Overview
The HCI consists of a graphical interface, which enables
the user to interact with the database. It is targeted to
inexperienced people who do not have extensive knowledge
about matters such as computers or image processing. Thus,
it provides an intermediate solution between effectiveness
and simplicity. At later phases of the software
implementation the HCI will enable experienced users to
choose to follow more sophisticated and accurate query
procedures.
B. Function / Processing
The HCI provides the user with various approaches in
querying the database including: specification of keywords,
submission of sample images (possibly loaded from the
local file system), rough sketching of the image’s general
view (with the use a simple, embedded drawing utility) and
a query refinement mechanism, based on the results of the
previous retrieval step.
The query procedure consists of the following steps:
1) Submission of query criteria to the ISC – The HCI
transmits through the network to the ISC the keywords
and the sample image the user has submitted.
2) Retrieval of images and presentation of results to the
user - The HCI receives the IDs of the matching
images from the ISC. Then, it requests the resulting
image thumbnails from the web server and presents
them to the user. Obviously, the best matches are
presented first.
After that first retrieval step, the user can do one of the
following:
• Query refinement
The user navigates the results from the previous retrieval
step, chooses the images that match best to the image he is
searching for and asks for a query refinement.
• Alteration of a previously retrieved image
The user selects an image from the results, uses the
embedded drawing utility to alter it and make it resemble to
the image he is looking for and re-queries using it as a new
sample image.
C. Implementation
A basic requirement for the HCI was to be developped as
a cross-platform application. Therefore, it has been
implemented as an independent Java application
conforming to Sun’s Java 2 standards.
D. Future work
One of COBWEB’s main targets is to implement the HCI
as a downloadable Java applet. For the time being, the
above solution is not feasible because of the following
reasons:
• Currently, browsers do not support the Java 2
platform.
• The embedded drawing application makes the
code quite lengthy for downloading.
• Java 2 execution times are long, especially when
dealing with complicated user intefaces.
IV. INTELLIGENT SEARCH COMPONENT
A. Overview
The Intelligent Search Component (ISC) is the search
engine of the COBWEB system and is responsible for the
execution of the queries to the image database. The retrieval
process is triggered by the query request, submitted through
the HCI. Functionally, the ISC serves as an intermediate tier
between the HCI client and the database.
ISCs power lies on the use of effective and efficient
mahematical algorithms, capable of extracting a number of
well-defined features (visual features) from images of
arbitrary size. A brief description of the nature of the
algorithms integrated in the first prototype of the COBWEB
system follows:
• Global color histogram
• Local color layout
• Coarse level color distribution of homogeneous
regions
• Large scale color texture
Detailed description of the algorithms mentioned above is
beyond the scope of this paper. For more information see
references 3,4,5 and 6.
B. Function / Processing
The ISC receives the user’s query criteria from the HCI
and navigates the database to retrieve the images that satisfy
the user’s needs. A detailed description of the query
procedure follows:
• Query reception from the HCI
The ISC receives query criteria from the HCI. As
mentioned before the query criteria consist of a sample
image and a set of keywords.
• Extraction of visual features of sample image
The ISC uses the algorithms provided by the Feature
Extraction Component (FEC) to create a feature vector
containing visual information about the sample image.
• Database navigation
Initially, the ISC searches the database to find a subset of
images whose keywords match the ones selected by the
user. After the selection has been done, the ISC retrieves
from the database the feature vectors of the images of that
subset. Then, the feature vector of the sample image is
compared with the respective one of each image of that
subset. The above goal is accomplished with the help of the
similarity algorithms contained in the Feature Comparison
Component (FCC).
• Score sorting
The ISC sorts the similarity scores the FCC created for each
target image of the subset.
• Results aggregation
The ISC selects the images with the best scores and returns
the results to the HCI, in the form of image IDs. The
maximum number of returned images is subject to the end –
user’s preferences.
Figure 3 is a visual representation of the procedure
described before:
 Feature [4] Kovalev, V., and Volmer, S., Color Co-occurance Descriptors
Extraction for Querying-by-Example, IEEE Conf. Multimedia Modelling,
Component pp. 32-38, October 1998.
[5] Volmer, S., Tracing Images in Large Databases by Comparison
4
of Wavelet Fingerprints, Proc. of the 2nd Int'l Conf. on Visual
feature
sample Information Systems, pp. 163-172, December, 1997.
vector
image [6] Pass, G., and Zabih, R., Histogram Refinement for Content-
5 based Image Retrieval, IEEE Workshop Applications of
keywords,
sample image 1 Computer Vision, December 1996
Human Intelligent [7] Vardangalos, G., and Vlachos, M. COBWEB Deliverable D1.2:
Feature
Search “Revised Requirements Report”, February 1999
Computer 5 Comparison
Component [8] Vardangalos, G., and Vlachos, M. COBWEB Deliverable D1.3:
Interface matching Component
6
image IDs
“Early Prototype Specification”, April 1999
similarity comparisons [9] COBWEB Web site http://cobweb.eunet.no/
between sample image &
3 target images

target
images
subset SQL query
through
process keywords
data 2

Database
Component

Figure 3

C. Implementation
The ISC is implemented with the help of the Microsoft’s
Visual J++ environment. Microsoft’s solution has been
adopted instead of Sun’s pure Java solution (Java 2) in
order to ensure the communication with the FEC and FCC
components, which follow the standards of Microsoft’s
Component Object Model (COM).

D. Future work
Obviously, feature extraction and feature comparison
procedures are time-consuming procedures and the ISC is
considered a time-critical application. Moreover, Java is not
considered the best platform for the execution of
computational tasks. Therefore, it has been decided that the
next version of the ISC engine will be implemented with
the help of a more powerful environment / language such as
Microsoft’s Visual C++.

V. REFERENCES
[1] Bach, J.R. and Fuller, C. and Gupta, A. and Hampapur, A. and
Horowitz, B. and Humphrey, R. and Jian, R. and Shu, C.F., The
Virage Image Search Engine: an Open Framework for Image
Management, Storage and Retrieval for Image and Video
Databases IV, SPIE, pp. 76-87, 1996.
[2] Flickner, M. and Sawhney, H.S. and Niblack, W. and Ashley, J.
and Huang, Q. and Dom, B. and Gorkani, M. and Hafner, J. and
Lee, D. and Petkovic, D. and Steele, D. and Yanker, P., Query
by Image Content: the QBIC System, IEEE Computer, vol.28,
no. 9, pp. 23-32, September, 1995.
[3] Strickner, M, and Orengo, M., Similarity of Color Images,
SPIE Proc. Storage and Retrieval for Image and Video
Databases III, February 1995, pp. 381-392