Hamming Codes

11/17/04
History
In the late 1940s Richard Hamming recognized
that the further evolution of computers required
greater reliability, in particular the ability to not
only detect errors, but correct them. His search
for error-correcting codes led to the Hamming
Codes, perfect 1-error correcting codes, and the
extended Hamming Codes, 1-error correcting
and 2-error detecting codes.
Uses
Hamming Codes are still widely used in
computing, telecommunication, and other
applications.
Hamming Codes also applied in
Data compression
Some solutions to the popular puzzle The Hat Game
Block Turbo Codes
A [7,4] binary Hamming Code
Let our codeword be (x
1
x
2
x
7
) F
2
7
x
3
, x
5
, x
6
, x
7
are chosen according to the message
(perhaps the message itself is (x
3
x
5
x
6
x
7
)).
x
4
:= x
5
+ x
6
+ x
7
(mod 2)
x
2
:= x
3
+ x
6
+ x
7

x
1
:= x
3
+ x
5
+ x
7

[7,4] binary Hamming codewords
A [7,4] binary Hamming Code
Let a = x
4
+ x
5
+ x
6
+ x
7
(=1 iff one of these
bits is in error)
Let b = x
2
+ x
3
+ x
6
+ x
7

Let c = x
1
+ x
3
+ x
5
+ x
7

If there is an error (assuming at most one) then
abc will be binary representation of the subscript
of the offending bit.
A [7,4] binary Hamming Code
If (y
1
y
2
y
7
) is received and abc 000, then
we assume the bit abc is in error and switch it. If
abc=000, we assume there were no errors (so if
there are three or more errors we may recover
the wrong codeword).

Definition: Generator and Check
Matrices
For an [n, k] linear code, the generator matrix is
a kn matrix for which the row space is the
given code.
A check matrix for an [n, k] is a generator matrix
for the dual code. In other words, an (n-k)k
matrix M for which Mx = 0 for all x in the
code.
A Construction for binary Hamming
Codes
For a given r, form an r 2
r
-1 matrix M, the
columns of which are the binary representations
(r bits long) of 1, , 2
r
-1.
The linear code for which this is the check
matrix is a [2
r
-1, 2
r
-1 r] binary Hamming Code
= {x=(x
1
x
2
x
n
) : Mx
T
= 0}.

Example Check Matrix
A check matrix for a [7,4] binary Hamming
Code:

Syndrome Decoding
Let y = (y
1
y
2
y
n
) be a received codeword.
The syndrome of y is S:=L
r
y
T
. If S=0 then
there was no error. If S 0 then S is the binary
representation of some integer 1 t n=2
r
-1
and the intended codeword is
x = (y
1
y
r
+1 y
n
).
Example Using L
3
Suppose (1 0 1 0 0 1 0) is received.




100 is 4 in binary, so the intended codeword was
(1 0 1 1 0 1 0).
Extended [8,4] binary Hamm. Code
As with the [7,4] binary Hamming Code:
x
3
, x
5
, x
6
, x
7
are chosen according to the message.
x
4
:= x
5
+ x
6
+ x
7

x
2
:= x
3
+ x
6
+ x
7
x
1
:= x
3
+ x
5
+ x
7
Add a new bit x
0
such that
x
0
= x
1
+

x
2
+ x
3
+ x
4
+ x
5
+ x
6
+ x
7
. i.e., the new
bit makes the sum of all the bits zero. x
0
is called a
parity check.
Extended binary Hamming Code
The minimum distance between any two
codewords is now 4, so an extended Hamming
Code is a 1-error correcting and 2-error
detecting code.
The general construction of a [2r, 2r-1 - r]
extended code from a [2r 1, 2r 1 r] binary
Hamming Code is the same: add a parity check
bit.
Check Matrix Construction of
Extended Hamming Code
The check matrix of an extended Hamming
Code can be constructed from the check matrix
of a Hamming code by adding a zero column on
the left and a row of 1s to the bottom.
Decoding Extended Hamming Code

q-ary Hamming Codes
The binary construction generalizes to
Hamming Codes over an alphabet A={0, , q},
q 2.
For a given r, form an r (q
r
-1)/(q-1) matrix M
over A, any two columns of which are linearly
independent.
M determines a [(q
r
-1)/(q-1), (q
r
-1)/(q-1) r]
(= [n,k]) q-ary Hamming Code for which M is
the check matrix.
Example: ternary [4, 2] Hamming
Two check matrices for the some [4, 2] ternary
Hamming Codes:
Syndrome decoding: the q-ary case
The syndrome of received word y, S:=My
T
, will
be a multiple of one of the columns of M, say
S=m
i
,

scalar, m
i
the i
th
column of M. Assume
an error vector of weight 1 was introduced y = x
+ (0 0), in the i
th
spot.
Example: q-ary Syndrome
[4,2] ternary with check matrix ,
word (0 1 1 1) received.



So decode (0 1 1 1) as
(0 1 1 1) (0 0 2 0) = (0 1 2 1).

Perfect 1-error correcting
Hamming Codes are perfect 1-error correcting
codes. That is, any received word with at most
one error will be decoded correctly and the code
has the smallest possible size of any code that
does this.
For a given r, any perfect 1-error correcting
linear code of length n=2
r
-1 and dimension n-r
is a Hamming Code.
Proof: 1-error correcting
A code will be 1-error correcting if
spheres of radius 1 centered at codewords cover the
codespace, and
if the minimum distance between any two
codewords 3, since then spheres of radius 1
centered at codewords will be disjoint.
Proof: 1-error correcting
Suppose codewords x, y differ by 1 bit. Then
x-y is a codeword of weight 1, and M(x-y) 0.
Contradiction. If x, y differ by 2 bits, then
M(x-y) is the difference of two multiples of
columns of M. No two columns of M are
linearly dependent, so M(x-y) 0, another
contradiction. Thus the minimum distance is at
least 3.
Perfect
A sphere of radius centered at x is
S

(x)={y in A
n
: d
H
(x,y)

}. Where A is the
alphabet, F
q
, and d
H
is the Hamming distance.
A sphere of radius e contains
words.
If C is an e-error correcting code then
, so .
Perfect
This last inequality is called the sphere packing
bound for an e-error correcting code C of length
n over F
m
:
where n is the
length of the code and in this case e=1.
A code for which equality holds is called perfect.


Proof: Perfect



The right side of this, for e=1 is q
n
/(1+n(q-1)).
The left side is q
n-r
where n= (q
r
-1)/(q-1).
q
n-r
(1+n(q-1)) = q
n-r
(1+(q
r
-1)) = q
n
.
Applications
Data compression.
Turbo Codes
The Hat Game
Data Compression
Hamming Codes can be used for a form of lossy
compression.
If n=2
r
-1 for some r, then any n-tuple of bits x
is within distance at most 1 from a Hamming
codeword c. Let G be a generator matrix for
the Hamming Code, and mG=c.
For compression, store x as m. For
decompression, decode m as c. This saves r bits
of space but corrupts (at most) 1 bit.
The Hat Game
A group of n players enter a room whereupon
they each receive a hat. Each player can see
everyone elses hat but not his own.
The players must each simultaneously guess a
hat color, or pass.
The group loses if any player guesses the wrong
hat color or if every player passes.
Players are not necessarily anonymous, they can
be numbered.
The Hat Game
Assignment of hats is assumed to be random.
The players can meet beforehand to devise a
strategy.
The goal is to devise the strategy that gives the
highest probability of winning.
Source
Notes on Coding Theory by J.I. Hall
http://www.mth.msu.edu/~jhall/classes/coden
otes/coding-notes.html