CN520 Project 3

Backpropagation

Due date: Tuesday, March 2

Assignment
You are to train two networks to recognize bitmapped images of Arabic numerals and to convert
them to a binary representation. One network will not contain a hidden layer, the other will contain a
20-unit hidden layer. The input patterns consist of 15 bits arranged in a 3x5 grid, as shown in Figure 1.
The target output patterns are 4-bit binary representations of the digits (0000, 0001, 0010, 0011, 0100
etc. for the digits 0, 1, 2, 3, 4, etc.).

Please remember that all assignments have an 8 page limit, including diagrams but not
including the appendix that contains your MATLAB code.

Part 1 No hidden layer
a) Program a multilayer perceptron containing 15 input nodes (one for each bit of the input pattern),
NO HIDDEN LAYER, and 4 output nodes. Use the logistic function to convert each nodes activation
into its output. (See the hints section).
b) Train this network using 800 epochs, where in each epoch all 10 input-output pairs are presented.
You must use BATCH MODE and a MOMENTUM TERM.
c) Find values for the learning rate (eta) and the momentum parameter (alpha) that allow quick and
robust learning. Write down in your report what they are.
c) Your report should contain a diagram of the network, the equation that relates its output (i.e.
classification) to its input and weights, AND the weight update equation you used. Remember to define
ALL your symbols. The more working you show and the more you explain what you are doing, the
more likely you will get a high mark.
d) While training the network, keep track of BOTH the mean-square error and the mean classification
error for each epoch, calculated over the entire epoch. Plot these results as a function of epoch and
include these plots in your report.

Part 2 20 unit hidden layer
Now repeat each and every step (including, for instance, drawing the new network diagram) of part 1
but this time using a multilayer perceptron that contains 20 hidden units.

Part 3 Bonus Section (not compulsory)
Do further analysis of your own choosing. Full marks can be obtained without doing this section it
is just a bonus. (Possible ideas include studying what happens if the hidden layer only contains 2 units,
the XOR problem, adding noise to the inputs and outputs, or any inspiration you can find in the lecture
notes).

Part 4 - Discussion
a) Explain why the mean-square error continues to decrease even when the network can classify all
the inputs correctly.
b) Explain what happens if you make either the learning rate or the momentum parameter too large.
CN520 Project 3 `


2
c) Which of the two networks (i.e. part 1 or part 2) performed better (consider both the average time
to converge to 100% correct classification AND the number of extra calculations required to train the
network).

Hints
1) If your network does not converge, you may have a bug in your code or the learning rate may be too
high or too low.

2) Be sure to start this project early so that you can ask for help if you need it.

3) The following MATLAB code will output the logistic function. Save it in a separate file named
logistic.m and place this file in the directory from which you are running MATLAB. You will then be
able to use this function in your code. (You can verify this by calling the function from the command
line in MATLAB).

function Y = logistic(X)
% Logistic function. Note that Y can take on values only between 0 and
% 1. X is the nodes activation (i.e. the weights multiplied by the inputs)
% and Y is the nodes output.

Y = 1./(1+exp(-X));
end

4) The following MATLAB code will output the first derivative of the logistic function. Like before,
save it in a separate file named diffLogistic.m and place this file in the directory from which you are
running MATLAB.

function Y = diffLogistic(X)
% Differential of logistic function

Y = logistic(X).*(1-logistic(X));
end



CN520 Project 3 `


3
IN: 010101101101010
OUT: 0000
IN: 010010010010010
OUT: 0001
IN: 111001010001110
OUT: 0011
IN: 110001010100111
OUT: 0010
IN: 001101111001001
OUT: 0100
IN: 111100111001110
OUT: 0101
IN: 111001001010010
OUT: 0111
IN: 011100110101010
OUT: 0110
IN: 010101010101010
OUT: 1000
IN: 010101011001001
OUT: 1001


Figure 1 The input and output patterns.