A Fuzzy Expert System for

Concept-based Image Indexing and Retrieval

I.A. Azzam C.H.C. Leung J.F. Horwood

Computer Science and Mathematics
Victoria University of Technology, Ballarat Road,
Melbourne 3001, Victoria, Australia
azzamiaa@sci.vu.edu.au

ABSTRACT

Image indexing and retrieval using a concept-based approach involves extraction, modelling and indexing of image
content information. Computer vision offers a variety of techniques for searching images in large collections. We
propose a method that enables components of an image to be categorised on the basis of their relative importance in
combination with filtered representations. Our method concentrates on matching subparts of images, defined in a variety
of ways, in order to find particular objects. These ideas are illustrated with a variety of examples. We focus on Concept-
based Image Indexing and Retrieval (CIIR), using a fuzzy expert systems, density measure, supporting factors and other
attributes of image components to identify and retrieve images accurately and efficiently.

1. INTRODUCTION image is or what it is about. The approach taken in this paper is

Image retrieval is the one of the most exciting and fastest
growing research areas in the field of visual information systems
(VIS). Visual information, incorporating image retrieval, has
become a preferred source of information for many consumers
[1]. Efficient indexing and retrieval of multimedia data are
necessary to a successful multimedia system. A competent
retrieval system should retrieve images that satisfy the user’s
needs. It should measure image similarity in a manner consistent
with human perception [2], [3], [4].
A Variety of features has been used to index and retrieve
images on contents, keywords, textual description or captions
[5], [6], [7]. The limitation of the current image analysis
techniques necessitates that most image retrieval systems use
some form of text description provided by the users as the basis
to index and retrieve images. These techniques are rather
primitive at present and they need further development and
refinement [8], [9], [10], [11].
Research in image retrieval has, so far, been divided between
two main categories; "concept-based" image retrieval, and
"content-based" image retrieval. The former focuses on using
words to retrieve images (e.g. title, subject heading, keyword,
and caption), while the latter focuses on the visual features of
the image (e.g. size, colours, textures, etc.) [12], [13], [14].
On a conceptual level, it is clear that humans employ a variety
of socio-cognitive processes, as well as sensory skills, in the
retrieval and evaluation of visual information. Content-based
indexing has the advantage of providing a higher-level analysis
of the image content but is expensive to implement and suffers
from a lack of inter-index consistency due to the subjective
nature of image interpretation. Concept-based indexing uses
controlled vocabulary or natural language to express what an
concept-based [15], [16].
Our goal in this paper is to provide a methodology that
concentrates on matching subparts of images, defined in a
variety of ways, in order to find particular objects. The
approach used is to integrate our concept-based image indexing
and retrieval method with the use of a fuzzy expert system.
From the initial returned images, users can select image using
partial or complete objects related to images [17].
Fuzzy logic is multi-valued logic that deals with degrees of
membership and degrees of truth. It uses the continuum of logic
values between zero and one. Classical binary logic can be
considered as a special case of multi-valued fuzzy logic. In this
paper, fuzzy logic is used in order to compute the object
densities, which are used for indexing and retrieving the images
[18].
2. IMAGE ORGANIZATION
The following example illustrates the method to manipulate the
index organization: We shall make use of an image in the
database (Figure 2.1), with the following objects as shown in
Table 2.1.
Figure 2.1 Image of Wedding

Proceedings of the 11th International Multimedia Modelling Conference (MMM’05)

1550-5502/05 $20.00 © 2005 IEEE
 Table 2.1 Image Descriptions people. We can create mid-level objects “plants” using some of
them as in Rule 1 and “wedded couple” as in Rule 2. The
Image Descriptions created mid-level objects, plants and wedded couple, and the
Bride remaining object(s) of the image that are not used to create the
Groom mid-level object, can be used to create a high-level object
Flowers “wedding”, as in Rule 3. In this way we structure the index
Trees representations into a hierarchy. Therefore, several primitive
indices will be able to define mid-level indices and several mid-
People
level indices will be able to define high-level indices and so on.
The benefit of this method is the reusability of the rules. Once a
2.1 BUILDING THE FACTS LEVELS rule that defines an object is created; it can be used to define
We develop techniques for the categorization of image several other higher-level objects/concepts [19], [20], [21].
components, and the assignment of different attributes To evaluate the rules, the inference engine requires that all of
(weights, density, supporting factors, etc) to these components the conditions be satisfied. It is possible that more than one rule
on a proportional scale based on their relative importance will apply to an image.
within the image [13].
The image index consists of many discrete fact types: low-
level facts (primitive objects only), mid-level facts (can be Database
primitive facts; they can also consist of two or more low-level
Fact: A is x
facts), high-level facts (high-level facts consist of two or more
mid-level facts) and so on. In this paper, we use the ternary fact Fact: B is y
model by expanding and restructuring low-level facts, mid-level
facts, and high-level facts into a hierarchy [1] (Figure 2.2).
Match Fire
Low-level fact Mid-level fact High-level fact
Knowledge Base
Bride
W-Couple Rule: IF A is x THEN B is y

Groom
People Wedding Figure 2.3 Inference Engine Match-fire Cycles using a
Flowers Match-fire Procedure
Plants An example of an inference chain is given below:
Trees

Figure 2.2 Facts Levels Rule 1:

If F is true
AND G is true A
2.2 FORWARD AND BACKWARD CHAINING THEN H is true
C
B
In a rule-based expert system, the domain knowledge is
represented by a set of IF-THEN production rules and data are Rule 2: D F
represented by a set of facts about the current situation. The If C is true
AND D is true
inference engine compares each rule stored in the knowledge H
AND E is true
base with facts contained in the database, when the IF THEN F is true G
E
“condition” part of the rule matches a fact, the rule is fired and
its THEN “action” part is executed. The matching of the rule Rule 3:
IF part to the facts produces inference chains; the inference If A is true
chain indicates how an expert system applies the rules to reach AND B is true
a conclusion. THEN C is true
Suppose that we have created a knowledge-based system
using the following rules:
x Rule 1: IF  Flowers and Trees, then the object is Plants. Figure 2.4 Inference Chain
x Rule 2: If  Bride and Groom, then the object is Wedded
Couple 2.2.1 FORWARD CHAINING (FC)
x Rule 3: IF  Wedded Couple and People, then the object Forward Chaining is data-driven reasoning. The reasoning starts
is Wedding from the known data and proceeds forward with that data, each
Upon the execution of these rules, we will generate new time only the topmost rule is executed, and when fired, the rule
indices that represent mid-level or high-level indices. As an adds a new fact to the database. Any rule will be executed only
example, consider image1 in Figure 2.1. It contains the once; the match-fire cycle stops when no further rules can be
following primitive objects: trees, flowers, bride, groom, and fired.

Proceedings of the 11th International Multimedia Modelling Conference (MMM’05)

1550-5502/05 $20.00 © 2005 IEEE
 many rules may be executed that have nothing to do with the
Knowledge Base
established goal. Therefore, if our goal is to infer only one
If there are trees and flowers, then the object is plants particular fact, the forward chaining inference technique would
not be efficient.
If there are bride and groom, then the objects is wedded couple
2.2.2 BACKWARD CHAINING (BC)
If there are wedded couple and people, then the object is wedding
Backward Chaining is goal-driven reasoning. In backward
chaining, an expert system sets the goal and the inference
engine attempts to find the evidence to establish it. First, the
knowledge base is searched to find rules that might have the
Database desired solution. Such rules must have the goal in their THEN
(action) parts. If such a rule is found and the IF (condition) part
Trees Flowers Groom Bride People
matches data in the database, then the rule is fired and the goal
is proved. However, this is rarely the case. Thus the inference
Plants engine puts aside (stacks) the rule it is working with the rule
and sets up a new goal, a sub-goal, to prove the IF part of this
rule, then the knowledge base is searched again for rules that
Figure 2.5 Rule 1 can prove the sub-goal. The inference engine repeats the
process of stacking the rules until no rules are found in the
Knowledge Base
knowledge base to prove the current sub-goal.
If there are trees and flowers, then the object is plants
3. FACT DENSITY
If there are bride and groom, then the objects is wedded couple
Fact density ( D i ) is related to the fact(s) repetition/width in
If there are wedded couple and people, then the object is wedding
the image, and its height, which is equivalent to the area
occupied by the fact(s) in the image. The fact density provides
users with a means to give priorities to selected query
components in order to optimise the results of the query.
Applying higher-level fact density values will increase the
Database value of the fact recall rate, while higher-level and link density
Trees Flowers Groom Bride People values will increase the value of the precision rate. Users may
want to further tailor the query results by widening the scope of
the result or narrowing the result.
The fact density takes one of three values as below:
Plants W-Couple
[ 0 . 0 ] to [ 0 . 4 ] ^ for low density
Di [ 0 . 3 ] to [ 0 . 7 ] ^ for medium density
Figure 2.6 Rule 2
[ 0 . 6 ] to [1 . 0 ] ^ for high density
Knowledge Base The above values are considered to be reasonable in the
present context. In general, of course, these can be fine-tuned or
If there are trees and flowers, then the object is plants modified, which may take on the form:
If there are bride and groom, then the objects is wedded couple [ 0 . 0 ] to [ D ] ^ for low density
If there are wedded couple and people, then the object is wedding Di [ E ] to [ J ] ^ for medium density
[ G ] to [1 . 0 ] ^ for high density
Where 0  E d D  1 , G d J

3.1 DENSITY AND FUZZY INDEXING & RETRIEVAL

Database Fuzzy indexing and retrieving has four main steps, and they
are: 1) Specifying the problem and defining the linguistic
Trees Flowers Groom Bride People
variables. 2) Determining the fuzzy sets. 3) Constructing
fuzzy rules. 4) Encoding the fuzzy sets, fuzzy rules and
procedures. These steps are illustrated in following
Plants W-Couple Wedding paragraphs.
3.1.1 SPECIFY THE PROBLEM AND DEFINE
Figure 2.7 Rule 3 LINGUISTIC VARIABLE
FC is a technique for gathering information and then inferring In building any expert system, we need to determine problem
from it whatever can be inferred; however, in forward chaining, input and output variables and their ranges in order to

Proceedings of the 11th International Multimedia Modelling Conference (MMM’05)

1550-5502/05 $20.00 © 2005 IEEE
 determine the density level [22]. Table (3.1) specifies the For three inputs and one output, the representation takes the
range of our linguistic variables, which has the overall range shape of an M x N x K cube. If we increase the input to three
of [0, 1]. inputs and one output, we have a fuzzy associative memory
Table 3.1 Linguistic Variables and their Ranges (FAM), which enables us to derive twenty-seven rules that
represent complex relationships between all variables used in
Linguistic variable: Fact’s weight the expert system [17]. Since images can be understood and
Linguistic value Notation Numerical range interpreted at multiple levels, they can also be indexed at
Low L [0.00, 0.40] multiple levels.
Medium M [0.30, 0.70] 3.1.4 ENCODING THE FUZZY SETS, FUZZY RULES
High H [0.60, 1.00] AND PROCEDURE
To encode the fuzzy sets, fuzzy rules and procedures, we may
3.1.2 DETERMINING THE FUZZY SETS choose one of two options: using a programming language, or
Determining the initial value of fact density D depends on its to apply a fuzzy logic development tool to encode the fuzzy
repetition/width and height in image. Triangle or trapezoidal sets, fuzzy rules and procedures to perform fuzzy inference in
shapes can often provide an adequate representation of the the expert system. Most experienced fuzzy system builders
expert knowledge and at the same time significantly simplifies prefer the programming language approach [23], [24], [25].
the process of computation. One of the key points is to
maintain sufficient overlap in adjacent fuzzy sets, Figure 3.1.
3.2 COMPUTING THE FACTS’ DENSITY
The equation below computes the fact density using the product
Degree of membership
of the repetition/width and hight values with its degree of
membership(s).
b x k
1
¦
x a
M h ( x1 ) x1  ¦
x c
M r (x2 )x2
… (ME9)
Di b x k
0.8
¦
x a
M h ( x1 )  ¦
x c
M r ( x2 )
0.6
Suppose that we have a number of objects (trees) / width with
Low Medium High
value of 0.32 at point “a” on the x-axis, and with height value of
0.4 0.8 at point “b” in the x axis (both values can take the value
between 0.0 and 1.0).
0.2 A1 A2 A3 The degree of membership for the number of trees in A1 =
0.38 on y-axis, and in A2 = 0.2 on y-axis. The height degree of
membership for the trees in A3 = 0.91 on y-axis, as shown in
0.2 a 0.4 0.6 b 0.8 1
Figure 3.1. The density of trees will be computed as follows:
Figure 3.1 Density (Normalized) Height/repetition 0 . 1 u 0 . 32  0 . 38 u 0 . 32  0 . 91 u 0 . 8
D Trees | 0 . 63
0 . 1  0 . 38  0 . 91
3.1.3 CONSTRUCTING FUZZY RULES
We consider an example, having two inputs and one output; it 4. DATA INDEXING AND RETRIEVING
is often convenient to represent fuzzy rules in a matrix form.
The representation takes the shape of an M x N matrix. We Once we input all of the facts, the search engine is ready to
make use of a very basic relation between the height “h” and perform a search on the database. Figure 4.1 and Figure 4.2
the number of objects “r” (repetition/width of the same object shows the steps needed to index and retrieve the images.
in the image), assuming that other input variables is fixed.
This relation can be expressed in the following form: 4.1 DATA INDEXING
If r increases, and h increase, the fact density-concept will Tables 4.1, show how to store information for three levels of
increase. Thus we could write nine rules like: facts, and their attributes. The information shown in these tables
If the repetition/Width is X, height Y, then D is Z, and so on as are based on Figure 4.1.
in Figure 3.2, where X, Y can each take on the values L, M, or
H. Table 4.1 Query Processing Tables
r Fact Data Table
Fact ID Fact name Type ID Density
H M M H
001 Trees 1 0.5
002 Flowers 1 0.1
M L M M 003 Bride 1 0.3
004 Groom 1 0.3
L L L M 005 Plants 2 0.5
006 W-Couple 2 0.3
L M H h 007 People 2 0.5
Figure 3.2 Density Representations 008 Wedding 3 0.5

Proceedings of the 11th International Multimedia Modelling Conference (MMM’05)

1550-5502/05 $20.00 © 2005 IEEE
 Fact Details Table User
FID FID Image Weight Supporting Weight
Child Parent factor in
image Enter Image name/s and Density
001 005 001.jpg 0.41 0.99 0.184
002 005 001.jpg 0.59 0.77 0.265
003 006 001.jpg 0.50 1.00 0.225 Data Processing
004 006 001.jpg 0.50 1.00 0.225
005 008 001.jpg 0.10 0.40 0.100 Fact Identification Dictionary
006 008 001.jpg 0.45 1.00 0.450
007 008 001.jpg 0.45 1.00 0.450
008 008 001.jpg 1.00 1.00 1.000
Semantic concept & Name
fact values DB Match
Here, we have made use of other fact attributes detailed in
our previous paper [1]. Briefly they are:
1. Facts’ weight ( W i ): It is a measure of the importance of
an object to the set where it belongs in an image. Fact Match
2. Facts’ weight in image ( W II ): It is a measure of the
importance of an object in an image. It is computed by
multiplying the fact’s weight by the weight of the set in
which it is a component of and the result follow the same List of Images User Feedback
procedure until the last level.
3. Facts’ supporting factor ( S i ): It is a measure of the Figure 4.2 Retrieval Modules
dependence of the appearance of a high-level fact on this Our retrieving algorithm consists of the following steps:
fact. The value of the support factor appears beside the 1. Enter the facts names or image name and its desired
arrow connection between facts on figures, which density.
originally was named inclusion factor in our previous 2. The query processing starts to identify the fact/s, the
paper. semantic concept and facts values from the database.
Data Extracting 3. Initially access the different facts tables depending on the
Attributes user query.
Facts in
computing 4. When the user’s query matches results, the matched
images
images will be displayed and the process will stop.

Densities
Store in DB
Images
collection
Other
attributes Indexing
Figure 4.1 Indexing Modules
Figure 4.1 illustrates the algorithm needed to store the data of
the fact attributes for all levels of facts.
Thus our indexing algorithm consists of the following steps:
1. Collect the suitable images and store them in an image
folder.
2. Determine the primitive fact contained within the image.
3. Extract the textual feature and create the mid-level and the
high-level fact, which are not primitive.
4. Compute the attributes values (densities, weights,
supporting factors, and weights in image when required).
5. Save the fact information and the computed values in the
database and perform the indexing.
4.2 QUERY PROCESSING
Figure 4.2 illustrates the algorithm needed to retrieve the data
of the fact attributes for all levels of facts.
4.3 EXPERIMETAL RESULTS
Before discussing the experimental results, we would like to
explain what we mean by supporting object. For example, if the
user searches for plant (which means factory), the system may
retrieve some other images not relevant like plant life and
garden.
Using supporting object like smoke, which have a high value
of supporting factor with the plant (factory) and low value of
supporting factor with the plants (plant life, garden), this will
help to reduce the ambiguity, and retrieve the most relevant
images.
We have conducted experiments using different data sets to
measure the performance of our system. Table 4.2 shows the
results of our queries. When searching using images name
without supporting object(s) having different levels of densities,
the result was giving a large number of images, some of them
may not be relevant. However, with one supporting object
having different levels of densities, the number of images
decreases, and finally with two supporting objects having
different levels of densities, the number of displayed images
was decreased further and gives the most relevant images.
Our system checked the supporting factors for every object in
addition to the densities; both the density and the supporting
factor play a major role in determining the retrieved images in
any query.

Proceedings of the 11th International Multimedia Modelling Conference (MMM’05)

1550-5502/05 $20.00 © 2005 IEEE
 Table 4.2 Retrieved Images [9] Barber, R. et.all., “A Guided Tour of Multimedia Systems
and Application: Query by Content for Large On-Line
Density Level Number of retrieved images Image Collection”, IEEE Computer, 1995, pp. 357-378.
(SO=Supporting Object) [10] R.J. Hwarth and H. Buxton, “Conceptual-Description from
(repetition/height) Without With % of With % of Monitoring and Watching Image Sequences”, Image and
SO one without two without Vision Computing 18, 2000, pp 105-135.
SO OS SO SO [11] Hsin-Chih Lin, Ling-Ling Wang and Shi-Nine Yang,
H (H/H) 50 31 62 16 32 “Regular-Texture Image Retrieval based on Texture-
M (H/M) 30 20 66 11 36 Primitive Extraction”, Image and Vision Computing 17,
M (M/H) 29 19 65 11 37
1999, pp 51-63.
M (M/M) 25 16 64 8 32
[12] Tat-Seng Chua, Hung-Keng Pung, Guo-Jun Lu and Hee-
M (H/L) 24 16 66 7 29
M (L/H) 20 14 70 6 30 sen Jong, “A Concept-based Image retrieval system”,
L (M/L) 15 11 73 6 40 IEEE computer, Jan 1994, pp 590-598.
L (L/M) 15 11 73 6 40 [13] Corinne Jorgensen, “Attributes of Images in Describing
L (L/L) 10 5 50 5 50 Tasks”, Information Processing and Management, 1998,
Vol.34, No. 2/3, pp 161-174.
[14] Dwi Sutano and C.H.C.Leung, “Automatic Index
5. CONCLUDING REMARKS Expansion for Concept-based Image Query”, Proceedings
Our method concentrates on matching subparts of images, of the International Conference on Visual Information
defined in a variety of ways, in order to find particular objects. Systems, June 1999, pp 399-408.
These ideas have been illustrated with a variety of examples in [15] Ma, W. & Manjanath, B. ”Netra: A toolbox for navigating
this paper. large image databases”, Proceedings of IEEE International
It is obvious that the use of fuzzy logic and supporting factors Conference on Image Processing (ICIP97), 1, 1998,
lead to improvements of both precision and recall. pp568-571.
This approach provides a new and innovative way to specify [16] Zheng, M. (1999). Metadata elements for object
and view visual data across the whole spectrum of applications description and representation: A case report from a
that involve the processing of visual data. This approach has digitized historical fashion collection project. Journal of
significantly increased the performance of the retrieval the American Society for Information Science, 50(13),
algorithm, as measured by the proportion of actual retrieved 1999, pp1193-1208.
images versus the total number of images that satisfy the [17] Turksen, I.B., Tain, Y. and Berg, M. “A Fuzzy expert
retrieval criteria. system for servis centre of spare parts, Expert Systems with
Applications, 1992, 5, pp447-464.
[18] Cox E. 1994 “The fuzzy systems handbook: A
6. REFERENCES Practitioner’s guide to building using, and maintaining
Fuzzy Systems Academic Press”, Cambridge, 1994.
[1] I. A. Azzam, C.H.C.Leung, J.F. Horwood, “Implicit
[19] Leung, C.H.C. and Zheng, Z.J., “Image Data Modeling for
Concept- based Image Indexing and Retrieval”, The 10th
Efficient Content Indexing”, Proc IEEE International
International Multimedia Modeling Conference
Workshop on Multi-Media Database Management
MMM2004”, Jan-2004, Brisbane, pp 354-359.
Systems, 1995, pp. 143-150.
[2] H.Z.Long and W.K. Leow, “Perceptual texture pace
[20] R.J. Hwarth and H. Buxton, “Conceptual-Description from
improves perceptual consistency of computational
Monitoring and Watching Image Sequences”, Image and
features”. In Proc. IJCAI, 2001.
Vision Computing 18, 2000, pp 105-135.
[3] H.Z.Long and W.K. Leow, “Perceptual consistency
[21] A. Tam and C.H.C. Leung “Structured natural-Language
improves images retrieval Performance” ACM,
content retrieval of visual materials”, Journal of American
SIGIR01,Sep 9-12, 2001
Society for information Science and Technology, 2001, pp.
[4] J.S. Payne. L. Hepplewhite and T.J. Stonham “Perceptually
930-937.
based metrics for the evaluation of textual image retrieval
[22] John Determan and Greg Becker “A prototype Generic
methods”. In Proc. IEEE ICMCS’99, pp 793-797.
Expert System Architecture for Data Review/Validation”,
[5] Alan f. Smeaton and Ian Qigley. “Experiments on Using
7th Non-destructive Assay Waste Characterization
Sematic Distances Between Words in Image Caption
Conference, Salt Lake City, Utah, May 22-26, 2000, pp.
Retrieval”. In Proceedings of ACM SIGIR Conference,
261-270.
1996.
[23] Li, H and Gupta, M., “Fuzzy logic and intelligent
[6] Ishwar K. Sethi, “Image Computing for Visual Information
Systems”, Kuluwe Academic Publishers, Boston, 1995.
Systems”, IEEE computer, May 1995, pp 06-10.
[24] Ian H. Witten, Alistair Moffat, Timothy C.Bell “Managing
[7] Leung, C.H.C., D. Hibler and N. Mwara, “Image retrieval
Gigabyte? Evaluation retrieval effectivness”, Morgan
by Content Description”, Journal of Information Science,
Kaufman publishers Inc.1999, pp 188-190.
1992, Vol (18), pp. 111-119.
[25] Mohamad Obeid, Bruno Jedynak and Mohamed Daoudi
[8] Barber, R. et.all., “MULTIMEDIA MANAGER: Query
“Image Indexing & Retrieval Using Intermediate
By Image Content and its Applications”, IEEE Spring
Features”, ACM, 2001, pp 531-533.
Compcon ‘94, 1994, pp. 424-429.

Proceedings of the 11th International Multimedia Modelling Conference (MMM’05)

1550-5502/05 $20.00 © 2005 IEEE