You are on page 1of 13

KYBERNETIK

BAND I D E Z E M B E R 1961 HEFT 3

Studies with Artificial Neurons, I: Properties and Functions of an Artificial Neuron


By LEON D. HARMON
Bell Telephone Laboratories, Inc., Murray ]=[ill, New Jersey, USA
With 21 Figures in the Text
(Received 7. Sept. 1961)
Summary. 1. A brief outline of the evolution of neuron view that the essential role of neural structure is to
modeling is given and an argument is presented for studying manipulate information; neural events represent the
neural behavior by "black-box" logical equivalence.
2. An electronic circuit incorporating many of the digi- language of this process. Such an approach is not
tal and analog properties of neurons is described. Having concerned with detailed electrical, chemical or ionic
such properties as variable threshold, summation, all-or-none events within a cell, but only with their ultimate
output, absolute and relative refractoriness, and inhibition, operational effects. Neural functions can (in principle)
it exhibits a considerable amount of functional equivalence
to biological structures. as readily be specified with eleetrordc circuits or mathe-
3. Properties of the model are described in detail and matical symbols as with ion-membrane mechanisms.
relevance to biological neuron measurements are shown. Such I t is essential only to preserve the fundamental logical
behavior as time-intensity trade, repetitive firing and temporal operations. Thus, chemical, electronic and mathe-
summation are readily acMeved and are shown to appro-
ximate in vivo behavior. matical systems m a y all be expressions of identical
4. The model is sufficiently flexible so that with simple functions and m a y be used interchangeably as state-
parameter changes and external circuit configurations, a ments of the same mechanism.
wide variety of neurological phenomena can be exhibited Modeling of neural properties is not new. Sixty
and studied closely. By adding stimulus-derived inhibition
for example, accommodation and adaptation are obtained. years ago H E ~ A N N (1899) considerer nervous activity
5. Experiments with the model have suggested new rela- in terms of electrical cable behavior. LmLIE (1936)
tionships which nervous structures may exhibit. A linear described a chemical model of axon conduction which
dependence of burst pulse number on accommodation time was subsequently greatly elaborated by others. A
constant, and a summation-division phenomenon are ex-
amples of such findings. large number of mathematical models have appeared
6. Models of this sort have utility not only for studying in the literature, of which those of McCuLLOCH &
single unit properties but also for investigating group inter- PITTS (1943), HODGKI2~ & HUXLEY (1952b), MINSXY
actions. Such studies may be relevant to elucidation of (1953), K L E E ~ (1956) and YON NEUMA~N (1956) are
neural network behavior.
representative. Electronic analogs of neural properties
have been described by BtmNs (1955), Htd~MON (1959),
Introduction CRANE (1960) and KffPFMOLLER& JENIK (1961). Com-
I n attempting to understand nervous systems, one puter simulation studies such as those by ROCHESTER
m a y ask questions on at least four different levels. (1956), WILLIS (1959) andFARLEY & CL~d~K(1960) et al.
At the subcellular level are studies such as those of have been directed to investigating neural network
ion transport mechanisms and membrane properties. behavior.
Secondly, single cells m a y be studied for their input- Electronic analogs of neurons provide a flexible
output functions, the stimulus-response relationships. means for exploring the logics of the nervous system.
At another level, groups of cells are investigated in I t is relatively easy to incorporate a large number of
terms of their interactive behavior; this ranges from parameters representative of the biological cell. The
subsystems such as reflex arcs to neural pools and model m a y then literally be used as an analog com-
the organized mass actions. Finally, there are the puter to examine functions that m a y be exceedingly
psychophysical investigations of the entire organism difficult if not impossible to predict or to fully analyze.
viewed as a "black-box". The studies described in this paper take for their
The research reported here takes for its locus the premise that such simulation can have utility and
cellular input-output functions level, seeking to ex- that continued exploration with such models can lead
plore the "logical" properties of the single neuron by to a better comprehension of neural events.
the use of analog models. I t is also directed in part There has been a sharp increase in neural modeling
to the third of the foregoing categories, i.e., to an activity during the last few years, and it is important
investigation of the interactions of small numbers of to point out that two distinct philosophies are in-
these units. I t seeks to explore some of the information volved. One school, of which the present work is
processes encoded in neural activity. representative, seeks to make its "neurons" simulate
In the past, biological systems have been regarded biological functions as closely as possible. Input-output
in such terms as metabolism, electrical activity, re- relationships are made to be consistent with what is
production, and movement. I t is increasingly apparent known of the biological parameters. The intent is to
that in addition, the processing and flow of in/ormation study the probable information-processing functions
is a relevant property. This is especially true for of neurons and in so doing to elucidate further the
nervous systems. The present work is based on the operations of the biological system.
Kybernetik,Band 1
90 L~o~ D. HARMON: Studies with Artificial Neurons. I Kybernetik

Among the second group of modelers (by far the diodes. An earlier version was described (HARMON,
most populous) the idea is to explore the large-scale 1959) in which the functions were less accurate and
behavior of m a n y quasi-neuron elements arranged in flexible than they are presently. A detailed circuit
randomly connected nets. Inasmuch as the properties analysis of t h a t version appears in HARMON & WOLFE
of these elements are greatly simplified abstractions (1959). The present version, which retains the basic
of known neural functions, and the network connec- structure of the earlier one, is shown in Fig. 1.
tivity bears no necessary resemblance to biological Five excitatory inputs are integrated by their as-
structure, it seems unlikely t h a t results bearing real sociated 150 K~9 resistors and the 0.02/~F capacitor
physiological significance will emerge. on the base of the first transistor T 1. Depending on
There is reason to confine our present studies to how the input (dendritic) leads are connected to
peripheral mechanisms. Little enough is known of various (axon or receptor) sources, the integrating
function at the transducer levels and the early relay time constant can be varied from 0.46 to 1.2 mscc.
stages; firm knowledge of cortical events is even less Threshold is determined by the relationship between
detailed and secure. The peripheral structures are the potential on the base of T~ (corresponding to the
simpler, are better known anatomically and physio- time course of input signals) and the potential on the
logically, and are more accessible to experimentation emitter of T 1 (depending upon the recent firing history
than are the central structures. Furthermore, it would of the unit). T 1 and T~ together form a monostable
appear useful to understand the mechanisms of the multivibrator whose function is to produce a single
retina and the optic nerve signal encodings, for ex- pulse if threshold is exceeded. T~ and T 4 form an
ample, before attempting to comprehend more central amplifier and low impedance output source such t h a t
events. Certain recent studies (KUFFL]~R, 1953; HUBEL as m a n y as 100 other units m a y be driven directly
& WIESEL, 1959; LETTWN, MATURA~A, McCuLLOCH & if desired. The function of T5 is to invert an other-
1)ITTS, 1959) illustrate the utility of this view. wise excitatory signal. The inverted signal is summed
with the five excitatory inputs to produce inhibition.
The Model
The refractory function is obtained from the time
Most of the models proposed to date have incor-
constant associated with T 1 and T2, whose pulse-
porated the classical axonal properties of temporal
emitting activity m a y be inhibited by a graded change
summation, variable threshold, inhibition, refractory in their virtual threshold. The circuit provides an
recovery and all-or-none output. I t has become appar- absolute refractory period which is approximately
ent, however, t h a t the properties of the entire neuron 1.0 msec long (the output pulse duration) and which
must be simulated. This means t h a t adequate pro- is immediately followed by a decaying exponential
vision must also be made for the graded dendritic recovery to resting threshold having a time constant
functions having longer time constants and decre- of 2.7 mscc. As will be shown, this and the input
mental conduction. time constant m a y be readily modified over a wide
Although a considerable amount of important range to simulate a variety of neural mechanisms.
activity is assignable to these continuous-variable in- The neuromime has a transmission delay t h a t
put functions (as opposed to axonal discrete-variable varies from 0.1 to 1.0 msec depending on input con-
behavior), there is no compelling reason to believe nections and on firing frequency. Because of the
t h a t axon action potentials do not represent the prin- nature of the active electronic components, the unit
cipal language of the peripheral nervous system. We permits unidirectional propagation only, i.e., no anti-
recognize BULLOCK'S (1959) observation that axon dromic stimulation is possible.
spikes are not seen in some neural structures and his Inhibition is a continuous (graded) and essentially
conclusion t h a t they m a y be incidental and super- linear function of the potential applied to the inhibi-
fluous to function. Nevertheless, if we take as a func- tion input. Two quite different kinds of inhibition
tional neural unit the system from dendritic ramifica- have been described for biological preparations. In
tions to axonal termini, it is clear t h a t however diffuse, one, an inhibitory impulse increases resting membrane
graded, and smeared in time the input signals m a y potential, thereby increasing the threshold for ex-
be, the functional output is unique, standardized (to eitatory impulses (B~ocK, CooM~s, & ECCL~S 1952).
a first-order approximation) and is crisply defined. I n the other, inhibitory impulses diminish any local
All-or-none activity assuredly accounts for more than polarization or depolarization which m a y be present,
long-distance transmission; for example, the concomi- tending to restore resting potential (FATT & KATZ,
t a n t refractory phenomena clearly play an important 1953). Thus in the first case inhibition m a y be viewed
role in modifying signals. as an active potentiating process while in the second
Thus, it is useful to consider t h a t there are two it is effectively a passive attenuation. The former
kinds of information processing present. I n one, the process is subtractive, the latter multiplicative. In
intracellular continuous-variable processes are signifi- the present design of the neuromime the first process
cant. I n the other, discrete but variable signals be- is modeled.
tween cells operate to transmit and to modify in- Any number of excitatory or inhibitory inputs are
formation. Both are essential to a complete description possible, of course, since multiple inputs can be con-
of neural events. verged into any combination of the five excitatory
The model 1 we use is electronic, being a circuit of and one inhibitory leads. Each input can have addi-
five transistors and associated resistors, capacitors and tional resistive networks appended to obtain decre-
In a proposal for a systematic nomenclature of devices mental conduction, and auxiliary capacitors m a y be
which simulate biological functions, VAN BERGEIJK (1960) added to obtain long time constant, graded dendritic
suggested the term neuromime for neuron models. We adopt
this designation in subsequent references to the artificial behavior. This leads to latencies which act in addition
neuron. to the simple internal delay mentioned above. Diode
Band 1, :Heft3
Dezember 1961
LEON D. HARMOn: Studies with Artificial Neurons. I 91

isolation between units can be arranged, if required, of the pulse generator, excitation does not grow after
to minimize interference or loading between multiple removal of a stimulus. Therefore, the model is, with
driving units. t h a t exception, passive. A third omission is the pos-
There appears to be some evidence t h a t the trans- sibility of making a given input lead excitatory at
mission properties of synapses in the CNS v a r y with one time and inhibitory at another; thus, there is
use (EcoLES & MCI~TYRE, 1953). Although we have no equivalent here to the presumed functional change
developed models for obtaining variable synaptic im- of presynaptic fibers dependent on post-synaptic
pedance, they are not being employed in present membrane potential (FATT, 1954; GRUNDFEST, 1959).
experiments and will not be described here. We do We shall show, however, t h a t despite these deficiencies,
not introduce them for two reasons: first, the evidence the model is sufficiently complete to simulate a number
for variable synaptic behavior is incomplete and of biological phenomena.
controversial; second, peripheral (e.g. retinal) structure The neuromime has three primitive input-output
m a y be assumed to be relatively invariant, and there properties. They are one-to-one, where a single spike
seems little necessity at present to invoke synaptic elicits a single spike; one-to-many, where a step func-
changes as functions of experience at this level. tion or steady potential results in a train of output
The primitive functions of the neuromime are sum- pulses (as in peripheral transducers) ; and many-to-one,
marized in Table 1.
Excitatory /.,V~-]=
Table 1

1. Resting threshold --1.5 V for steady, transducer-like


input (rheobase)
--3.0V for typical single spike
lOk~ i2I
input
2. Temporal summation Time constant: 0.46 to 1.2 msee,
depending on input configura-
tion
3. Delay 0.1 to 1.0 msec, depending on firing
frequency and input configura-
tion zsok~ I Imok~ 2zke 5 ke .~__J.~
:- IIVI00
4. Spike output 0.8 to 1.2 msec pulse of --10V
amplitude
5. Refractoriness [/.SV
Absolute: ~ 1.0 msec
Relative : 2.7 msee time constant -~- ~, Tz, T3,Tq,Ts are
6. Firing frequency 0--600 pulses per second Inhibitory\--~---r = 2N3G9Or equiva/ent
7. Inhibition Graded subtraction from excita- input
tion Fig. 1. Circuit diagram of artificial neuroll
8. Power consumption 81 mW

The magnitudes of the voltages employed in the in which spatial or temporal summation occurs. (We
model do not correspond to biological levels. This is ignore for the present a fourth possibility, namely
not important, however, because the voltage ratios those outputs which result from no input, as in spon-
are similar to the natural ratios. For example, the taneous firing.) These properties which include such
6.7 : 1 ratio of spike amplitude to rheobase in the model time-varying characteristics as excitation, threshold
approximates the 7:1 safety-factor described for bio- and inhibition are sufficient to produce a vast number
logical preparations (I-IoDoKI~ 1937, 1948). The tem- of logical functions. These include "and", "or", " n o t " ,
poral relationships closely approximate those in natural "if-and-only-if", addition, subtraction and division, as
neurons. While it m a y be argued t h a t a rectangular well as more complex operations such as filtering,
output pulse of 1 msec duration having no subnormal differentiation, and counting.
or supernormal phases is' too gross a representation
of the spike potential, our feeling is t h a t the discre- Some properties o[ the model
pancies introduce negligible differences in over-all Some of the typical characteristics of the neuro-
functional (logical) behavior. Should the discrepan- mime are illustrated in this section. These are the
cies turn out to be significant, it would be relatively primitive properties of the circuit shown in Fig. 1 and
easy, as will be shown, to modify the model to provide do not include functions such as accommodation and
a differentiated output or post-output polarizations. adaptation. I t will be shown later how these more
Several natural neural properties are not incor- complex operations can be built up from the primitive
porated in the model. For one, conduction velocity ones.
along input or output leads can be made neither slow Due to component tolerances, there is some spread
nor variable. This means t h a t there is no " e v e n t " (on the order of 10%) in performance between units.
delay other than two simple types of internal latencies The following results show the measured responses
which will be described. However, the neuromime of an average unit.
can also be considered as a section of myelinated Time-intensity trade. For a given pulse input to
axon rather t h a n as a complete cell. If a number a biological neuron, there is a minimum time required
of units are connected serially, then conduction is for a given intensity to elicit a response (A~VANITA_KI,
saltatory; each unit represents a node, while the con- 1938). The behavior of the model closely approximates
nections between them represent rapidly conducting A~VAN~TAKI'S strength-duration results as is shown in
internodes. I n this way, various transmission delays Fig. 2. We see, for example, t h a t a 2.7 V stimulus
m a y be achieved. A second omission is that, with the must be at least 1.0 msec long to cause firing, while
exception of the very rapid dynamic flip-flop action one having an amplitude of 1.68 V must last for a
7*
92 L~oN D. H~MON: Studies with Artificial Neurons. I Kybernetik

minimum of 3.0 msec. The horizontal asymptote of Latency. Latency in the model is conventionally
this curve, roughly 1.5 V, represents the rheobase. defined as the delay of impulse initiation in response
The shape of the curve is a direct consequence of to a stimulus of long duration. Since there is no
the exponential input characteristic. The instanta- internal regenerative process contributing significantly
neous potential E (t) available at the summing input is to firing delay after the removal of a stimulus, latency
E (t) = Era= (1-- ~-tl~) (]) is principally due to the gradual build-up of potential
in the input integrator. Firing occurs as soon as
~5 threshold is exceeded. Hence, the strength-duration
curve of Fig. 2 is also a strength-latency curve. These
results are similar to those described for motoneurons
(F~ANK & FUOI~TES, 1956). The curve is asymptotic
to about 0.5 msec which represents the minimum
firing delay in the pulse generating circuit.
Temporal summation. The time-intensity relation-
ships required for the sum of two subthreshold pulses
to elicit firing are given in Fig. 3. Each point along
the abscissa indicates the elapsed time between the
end of the first pulse and the onset of the second.
Points along the ordinate indicate the minimum pulse
amplitude (equal for both pulses) required to just
elicit a response. This behavior too, is the straight-
forward result of the input integration characteristic,
/'~0 / 2 3 g 5 and it is similar to the synaptic summation described
Time in milliseconds by ECCLES (1946).
Fig. 2. Tinle-intens~ty trade for threshold firing. Since firing occurs just I t is seen that time and intensity in the model
at the termination of the stimulus, the curve also represents the
strengthlatency relationship can be traded over a range of about one volt, which
amounts to approximately 10% of the amplitude of
Time between fwo fu/se~ in msec
a typical output pulse. Summation can be achieved
-2.0 0.~ 0.8 lg 1.6 2.0 gq
over a time ranging from 0 (temporal coincidence)
(o -22
to about 3 msec, for which time the required amplitude
of the first pulse is sufficient to exceed resting thresh-
old. Longer time constants of facilitation may be
readily obtained by introducing additional integrating
elements (R and C) at the input.
Re/raetory recovery. Typical recovery after firing
is shown in curve A of Fig. 4. This is similar to
-3.0 ~ ~-~ recovery of biological neurons after inactivation (Era
L A ~ G : E ~ & G A S S E R , 1937; H O D G K I N & H U X L E Y , 1952a).
-3.2 The neuromime is absolutely refractory for approxi-
Threshold for s in~zle-
O.eTasec pulse mately 1 msec, the duration of the outpu~ pulse; the
Fig. 3. Temporal summation for two 0.8 msee pulses of equal amplitude. relative refractory state, beginning at t = 0 in the
Indicated times are from the termination of the first Imlse to the onset illustration, starts at the cessation of the pulse
of the second. Amplitudes are those required to just elicit firing
(ADI~IAN, 1921). This exponential recovery toward
resting threshold has a time constant of 2.7 msec.
Recovery is effectively complete in about 20 msec
since a rheobasic stimulus applied at this time will
once again elicit firing.
This curve results from the use of one input lead.
In B, three leads, used in parallel, increase the effect
of a stimulus. Thus, the refractory level will appear
reduced while its time constant remains unchanged.
Output pulse. The all-or-none rectangular output
-'~0 / 2 3 z/ 5 6'
pulse has relatively constant amplitude; its width,
T/me in milliseconds however, changes somewhat as a function of firing
Fig. 4. Exponential refractory recovery of artificial neuron. This relatively frequency. This is shown in Fig. 5. This behavior
refractory phase is preceded by the absolutely refractory time (output is a consequence of several interacting circuit para-
pulse duration) which terminated at t =0. Curve A results from the use
of one input lead; the recovery in B is due to more effective excitation meters and was not deliberately designed. I t has a
derived from three input leads used in parallel slight effect on temporal integration (reducing its
efficiency with increasing frequency up to 500 pps I)
where Emax is the amplitude of the pulse stimulus, and but probably negligibly alters the neuromime's be-
k is the time constant of the input integrator (1.2 msec). havior. Diminution of spike amplitude with increasing
Each point on the curve represents the condition frequency has been described by RENSHAW (1942),
E (t) ----Erheobase= const. Thus as the intensity (Emax) KUFFL]~R (1953) and others. If the subsequent effec-
is increased, t must decrease. This is the equivalent tiveness of a spike is determined by its energy (time-
of the familiar relationship amplitude integral), then reduction of pulse-width with
Irheobase~-~/stimulus (1 -- 8-tlk ). 1 pps = pulses per second.
Band 1, Heft
Dezember 19613 LEON ~D.HARMON: Studies with Artificial Neurons. I 93

increasing frequency can have the same end result as Diode isolation 1 is required to prevent C from dis-
amplitude change. Such duration-amplitude inter- charging into the output line. The effect of this
changeability is known to hold, for example, in the circuit is simply to prolong the relative refractory time.
well documented time-intensity trade relationships. For R = 0.5 MY2 and C = 1.O/~F, stable repetitive
D e l a y . This term refers to the time between the firing continues smoothly down to 5 pps. This is
beginning of a driving pulse and the onset of an output shown as dotted extensions of the curves of Fig. 7.
pulse during continuous pulse stimulation. I t is not ]./
latency in the usual sense. Latency, generally measur-
ed for single pulse or for step-function stimulation,
does not involve post-firing refractory phenomena. zo
Delay in the model is a function both of excitatory
processes (input integration) and of refractory reco-
very. Typical behavior is shown in Fig. 6. "~ 03
Curve A depicts the delay as firing frequency is
varied and applied to a single input lead (k ~- 1.2 msec). "~ a8
As a driving frequency of about 250 pps is reached,
the driven unit "drops out", and its firing frequency
falls discontinuously to a lower value. This pheno-
menon is due to some interesting relationships between I00 200 300 ~00 500 600
F/ping frequency in ~.p.s.
the input and refractory time constants which will be Fig. 5. Variation of output pulse width (duration) with firing frequency.
discussed later. Pulse a m p l i t u d e remains constant
Curve B illustrates the behavior for all five inputs I.C
strapped together making k-----0.46 mscc. Here the
driven unit "follows" the input pulses up to its
m a x i m u m rate. In addition, the amount and varia-
bility of the firing delay is reduced. In both cases A
and B delay increases with frequency because pro-
o.8

.~ 0.6
I A

gressively less refractory recovery is encountered at


the onset of each driving pulse.
R e p e t i t i v e / i r i n g . The generation of a continuous
train of impulses for a constant stimulus is readily
achieved by the model. Repetitive firing in neurons O.2
/ J
has been attributed to interaction of a constant J
stimulus with refractory recovery (ADgTA~, 1930;
KATZ, 1936). In another view (HoDGKIN, 1948), it is o /oo 200 300 r 500 800
a function of a periodic internal excitation process Firinq Pre~uency in p.p.s.
which must be repeatedly "charged-up" by the con- Fig. 6. Total delay from the beginning of a stimulus pulse in a repetitive
stant applied stimulus. The repetitive firing charac- t r a i n to the firing of a typical unit. Characteristic A holds for a long
time constant i n p u t i n t e g r a t i o n ; B results from a shorter time constant
teristic of the present model combines both viewpoints and concomitant stronger excitation 9 See text for details
since the input integrator is partially discharged with
each firing; this works in combination with the ex- 700
ponential refractory characteristic to determine re- ~_ SO0
petition rate, although the latter parameter is t h e
dominant one. ,~ 5oo
Fig. 7 shows firing frequency as a function of ~
steady input voltage for two different input configura-
tions. Curve A results from the strapping together ~" 300
of three of the five excitatory input leads to obtain .,~ 2O0
an integration time constant of 0.66 msec. Curve B
is obtained by using a single input lead where k ~ IO~
1.2 msec. n~ I I I [ I I I I I I I
0 2 r 6 8 /0 /2
I~ /6 /8 20 22 2r
When driven by a steady input, both configurations Input in Volfs
produce reliable firing frequencies over a range of Fig. 7. R e p e t i t i v e firing frequency as a function of stimulus level for two
about 50--600 pps. When a pulse source is used (such different i n p u t configurations. A - - Three i n p u t leads used in parallel.
B - - One i n p u t lead. Stable firing down to 50 pps is obtained 9 This lower
as another unit) the response frequency has no mini- l i m i t can be extended by a simple circuit a d d i t i o n
mum.
If an extremely low firing frequency for small, Repetitive firing frequency in biological prepara-
steady excitation levels is required, an external feed- tions has been shown to be approximately logarith-
back loop from output to inhibitory input readily pro-
duces it. Fig. 8 shows the arrangement. 1 The only elements besides the basic model circuit used
in any of the experiments described are resistors, capacitors,
Each output pulse is integrated with a time con- and diodes. The first two components are generally employed
stant ~ which is determined b y the choice of R and C, to provide additional integrative mechanisms and decremental
and is applied to the inhibition input. Consequently, conduction. Diodes may be considered as rough analogs of
a decaying exponential voltage appears at the inhibi- some synaptic mechanisms and are introduced only where
their logical function does no violence to neurophysiological
tion input after termination of the output pulse. evidence.
Kybernetik, Band 1
7a
94 LEo~ D. HARMON: Studies with Artificial Neurons. I Kybernetik

mically related to the stimulus intensity on receptors system, for i n s t a n ~ minute variations in threshold
(HARTLINE,1938 ; GRANIT,1947 ; KATZ,1950) ; in other or excitation.
studies firing frequency has been found to be linearly I n the case of repetitive firing of the model under
dependent on generator potential up to 60 pps d.c. stimulus, the lowest firing frequency is about
(FuORTES, 1959) and up to 300 pps (KATz, 1950). The 50 pps. J u s t below this minimal stimulus, firing ceases
frequency-excitation curves of the neuromime (Fig. 7) abruptly. The effective recovery time thus is about
20 msec. Since the refractory time constant T is
Excile-,,,x~ 2.7 msec, recovery is complete in about 7.5 time con-
In Out
stants. W h a t does this imply about noise in the
system ? W h y is the firing frequency not arbitrarily
low ?
The threshold recovery function, defined as R (t), is
R (t) = R (1 --s--t~'), (2)
where R is the threshold increase at the start of the
relative refractory period. We can calculate R(t) for
different values of t in order to find the extent of
Fig. S. Circuit arrangement to extend low-frequency limit of stable firing. refractory recovery at various times. For example,
Self-inhibition prolongs the refractory period
at 3 v,
R (t) = R (1 __~-3) : .950 R ;
are quasi-logarithmic. The low-frequency sections are
recovery is complete to within 5 % of resting threshold.
essentially linear; characteristic A lies within 3 % of
I f the refractory recovery spans, say 10.5 V (see A,
linearity up to 300 pps, while B lies within this limit
Fig. 4), then the distance to resting threshold at 3 T
up to about 250 pps. When driven b y nonlinear trans-
is 5% X 10.5 V = 525 mV. At this point a signal, an
ducer elements such as photo-resistors, the over-all
uncertainty, or a noise having a magnitude of 525 m V
response of the neuromime is very close to log-linear
(and the proper polarity) superimposed on the recover-
over more than four decades of light intensity change.
ing threshold will bring it to the asymptotic (resting)
/0 t level. A plot of such points, based on a threshold of
I
Volts
RffilO.5
1.5 V and a refractory time constant of 2.7 msec is
/0 7 _~m.+ shown in Fig. 9.
Rather than have an abscissa calibrated in time,
~ me /. it is more revealing to indicate frequency. Thus the
1
33 point discussed above appears at-~--~-E~ m s e c ~
E 105
.~ / 123 pps.
The low-frequency cutoff of the neuromlme
( ~ 5 0 pps) corresponds to a recovery within 6 mV
(0.4%) of resting threshold. Thus, any fluctuation
I0 I Equiv.alen/
noi3e in the " s t e a d y " stimulus, or noise in the circuit will
cause firing at this time if it exceeds 6 inV. If reliable
repetitive firing down to 25 pps is required, then no
more than 4/,V noise m a y be present, a most un-
iO t likely situation.
I Low
.~-freauencj On the average, the measured peak noise signals
i0 o ]cutoff at the input in the absence of a stimulus are typically
/0-~ ~ iOp.p.s 1.5 mV with occasional peaks of 2 or 3 times this
amplitude. The stimulus lead was found to have
0 gO 80 120 160 200 2~0 2000 320 360
Firing frequency in "p.p.s, about i mV peak noise on its "steady" signal. This
Fig. 9. The lower limit of firing frequency is determined by noise in the
establishes the theoretical low-frequency cutoff at
system. During exponential refractory recovery, the difference between between 45 and 50 pps, in good agreement with ob-
instantaneous threshold and resting threshold at any instant can be stated served behavior.
as a voltage. Firing will occur whenever a superimposed signal {referred
to as equivalent noise) exceeds this difference. If time is expressed as This approach m a y have some utility in the in-
frequency, then for any level of equivalent noise there is a corresponding vestigation of internal fluctuations in biological neu-
repetitive firing frequency. See text for detailed explanation
rons. Some experimental and theoretical work has
already been done on threshold variations and internal
Repetitive firing as a function of exponential re- noise (P~cH~R, 1939; FATT & KATZ, 1952; FRISrr-
fractory recovery exemplifies an interesting charac- KOPF & ROSEZ~BLITI~, 1958); the technique described
teristic of dynamic systems which is often neglected, above might be used to amplify such results.
namely, the role played b y noise. Consider, for ex-
ample, a neuron which has an exponential refractory
period having a time constant of 2.7 msec. Theoreti- Complex properties o/the model
cally, resting threshold is approached only asymptoti- I n order to exhibit some of the more complicated
cally and is reached in an infinite time. Practically, functions of neurons, external circuits must be ap-
of course, refractory recovery is measured as complete pended to the neuromime. Such phenomena as ac-
in only a few time constants, generally less than commodation, adaptation, self-sustained discharge and
20 msec. This is true only because of noise in the post-spike polarization m a y be achieved.
B a n d 1, H e f t 8
D c z e m b e r 1961
LEON D. HARMON: Studies with Artificial Neurons. I 95

A n u m b e r of these effects are d e r i v e d from a S i m i l a r results f o u n d in t h e n e u r o m i m e are shown


circuit in which a stimulus b o t h excites a n d inhibits. in Fig. 12. E a c h curve was t a k e n for a different con-
This follows from HILL'S (1936) p r o p o s e d i n t e r n a l s t a n t s u b t h r e s h o l d v o l t a g e a p p l i e d a t t = 0. T h e ordi-
a c c o m m o d a t i o n process which is d e p e n d e n t u p o n b u t n a t e v a l u e of a given curve a t a n y t i m e is t h e ampli-
which proceeds a t a r a t e different from t h a t of a t u d e of a s u p e r i m p o s e d 0.5 msec t e s t pulse r e q u i r e d
s i m u l t a n e o u s e x c i t a t i o n process. HILL showed t h a t
m a n y aspects of n e u r a l e x c i t a b i l i t y could be r a t h e r
well a c c o u n t e d for b y i n v o k i n g a t i m e c o n s t a n t k for E ]-
e x c i t a t i o n a n d a c o n s i d e r a b l y larger 2 for t h e t i m e e V e
c o n s t a n t of d e p e n d e n t a c c o m m o d a t i o n (threshold
change).
A n e q u i v a l e n t c o n c e p t can be p r o p o s e d as follows: o o
O) !
let t h r e s h o l d r e m a i n c o n s t a n t , a n d s u b s t i t u t e i n h i b i t i o n
for a c c o m m o d a t i o n . Consider i n h i b i t i o n to be t h e
n e g a t i o n of e x c i t a t i o n a n d t h a t l i n e a r s u p e r p o s i t i o n
+J E
holds. Since e x c i t a b i l i t y is m e a s u r e d b y t h e difference
b e t w e e n e x c i t a t i o n a n d v a r i a b l e t h r e s h o l d in HILL'S e V
model, i t is e q u i v a l e n t l y m e a s u r e d b y t h e difference
b e t w e e n net e x c i t a t i o n a n d fixed t h r e s h o l d 1 in t h e
a l t e r n a t i v e view. I f t h e e x c i t a t i o n a n d i n h i b i t i o n 0 r

m e c h a n i s m s h a v e a p p r o p r i a t e t i m e constants, t h e n t h e
t w o p o i n t s of view are identical. - (2)
These two e q u i v a l e n t concepts are s c h e m a t i z e d in V
+
Fig. 10. The t h r e e sets of curves on t h e left i l l u s t r a t e
t h e Hill m o d e l of a c c o m m o d a t i o n where V is t h e e b'
v a r i a b l e e x c i t a t i o n a n d U is t h e r e s u l t a n t threshold.
F i r i n g occurs o n l y if V>U, n o t therefore p r e s e n t for
c o n d i t i o n (1), j u s t b a r e l y occurring in (2), a n d lasting
IL
for t h e i n t e r v a l s h a d e d in (3). (3)
I n t h e curves on t h e right, t h r e s h o l d T is fixed.
F o r firing to occur, e x c i t a t i o n E less i n h i b i t i o n I m u s t
result in a n e t e x c i t a t i o n N which exceeds threshold. Fig. 1O. E q u i v a l e n t concepts of f i r i n g criteria. I n ttIZI,'s m o d e l (left).
excitation V m u s t exceed a c c o m m o d a t e d threshold U for f i r i n g to occur,
A g a i n t h e r e is no response in (1), m o m e n t a r y firing A n a l t e r n a t i v e v i e w (right) a s s u m e s a c o n s t a n t threshold. The criterion
in (2) where N ~ T , a n d s u s t a i n e d firing in (3) as for f i r i n g is t h a t n e t excitation N exceeds threshold T. N e t excitation
is the difference b e t w e e n a n e x c i t a t o r y stimulus E a n d a concurrent
long as N > T. inhibition I
B o t h e x c i t a t i o n a n d i n h i b i t i o n are o b t a i n e d from
a s t i m u l u s b y t h e circuit of Fig. 11. T h e i n h i b i t i o n to cause firing. One i n p u t l e a d was used (k = 1.2 msec),
t i m e c o n s t a n t , d e t e r m i n e d b y t h e choice of R a n d C, a n d t h e t i m e c o n s t a n t of i n h i b i t i o n (analogous to
is m a d e large c o m p a r e d to t h e e x c i t a t i o n t i m e c o n s t a n t . HILL'S 2) was 3.7 msec 1.
A n u m b e r of well-known n e u r a l p h e n o m e n a are RvsHrO~'S curves were a s y m p t o t i c to resting
r e p r o d u c e d b y using this circuit. T h e y are described t h r e s h o l d ; no residual a c c o m m o d a t i o n a p p e a r e d . This
in t h e following sections.
Transient change o/excitability. The " a c c o m m o d a -
t i o n " process d e s c r i b e d in t h e foregoing section can
o Exc/r ,_
be used to s i m u l a t e m a n y of t h e w e l l - d o c u m e n t e d
t r a n s i e n t e x c i t a b i l i t y p h e n o m e n a . This follows d i r e c t l y
from t h e f a c t t h a t a s u b t h r e s h o l d s t i m u l u s will p r o d u c e
an e a r l y a n d fast r e d u c t i o n in t h r e s h o l d as t h e ex-
c i t a t o r y i n p u t i n t e g r a t o r b u i l d s u p p o t e n t i a l . Con-
s e q u e n t l y , t h e a d d i t i o n a l stimulus r e q u i r e d d u r i n g this
p e r i o d to exceed t h r e s h o l d a n d cause firing is reduced.
2
T h r e s h o l d s u b s e q u e n t l y rises (more slowly) to its rest- Fig. 11. The circuit configuration used to obtain s t i m u l u s - d e r i v e d inhibi-
ing v a l u e or b e y o n d as i n h i b i t i o n (accommodation) tion. P a r t of a n y stimulus (excitation) voltage is i n t e g r a t e d a n d applied
to the inhibition i n p u t
supervenes.
One m e t h o d of e x p l o r i n g t r a n s i e n t e x c i t a b i l i t y uses
a s u b t h r e s h o l d c o n s t a n t stimulus. RUSttTON'S (1932) is due to t h e fact t h a t t h e c o n d i t i o n i n g s t i m u l u s was
results are t y p i c a l . H e showed t h a t d u r i n g t h e applica- r e m o v e d j u s t a f t e r t h e t e s t s t i m u l u s was applied. T h e
t i o n of a s u b t h r e s h o l d c o n s t a n t current, t h e a m p l i t u d e curves of Fig. 12 a p p r o a c h a c o n s t a n t i n h i b i t i o n level
of a brief t e s t pulse r e q u i r e d to elicit firing decreases. r e p r e s e n t i n g a h i g h e r - t h a n - r e s t i n g t h r e s h o l d ; t h e con-
F o r t h e p a r t i c u l a r n e r v e tested, this decrease h a d a 1 We adopt the symbol ~t for the inhibition time constant.
s h a r p onset, r e a c h e d its m a x i m u m v a l u e a t a b o u t To calculate ;t, the 53 Ks total resistance of the inhibition
1.0 msec, a n d t h e n slowly subsided t o i n i t i a l conditions input circuit (comprising the 24 Kg2 resistor and the base
in 5 - - 1 0 msec. resistance of Ts) must be included. For C=0.1/~F and
R = 120 KD,
1 Threshold is constant in this view only for prefiring 53- 10-s x 120.10 -~
(resting) processes. I t is considered to be a variable, however, ;t •
173 • 10 -3
with respect to the refractory period. 3.7 msec.
96 L~oN D. HARMON':Studies with Artificial Neurons. I Kybernetik

ditioning stimulus was left on indefinitely in this case. follows, having a peak (threshold maximum) at about
Any asymptote can be chosen simply by modification 0.5 msec for the class of fiber tested, which gradually
of the inhibition network. Similarly the time at which returns to resting threshold within 3 or 4 msec. Similar
peak excitability occurs also is variable, gheobase behavior in the neuromime follows from the mech-
anisms operating for a constant conditioning stimulus
as described in the preceding section. These changes
in threshold have been converted into proportional
excitability and plotted in the curves of Fig. 13. Each
3 kga? point on a curve represents the percent ratio of resting
threshold to instantaneous threshold.
A subthreshold conditioning pulse of 0.5 msec dura-
-~.~ tion was terminated at t = 0 , and a variable amplitude
test pulse 0.5 msee long was used to determine thresh-
old at a number of time intervals. Curve A was derived
from a very weak conditioning stimulus; curves B
and C were taken for increasing amplitudes of con-
ditioning pulse. I n D, the conditioning stimulus ex-
\ ~ ~eobase 2.5V ceeded threshold and the unit fired. This curve then
represents refractory recovery. These results closely
parallel those of ERLANGER & GASSER. Recovery in
o I 2 3 ~ 5 each case is to resting threshold (100% excitability)
"]'/'me in m i l l i s e c o n d s
Fig. 12. T r a n s i e n t change of excitability u p o n the application of a constant
since there is no long-lasting stimulus to maintain
subthreshold stimulus. Ordinate values represent the a m p l i t u d e s required accommodation.
of a 0.5 msec test pulse a t v a r i o u s t i m e s to cause firing. M a x i m u m facili- P u l s e trains. For a stimulus of indefinite duration
tation occurs a t ~ 1.3 msec. E a c h c u r v e represents different levels of
conditioning s t i m u l u s whose level is above threshold but below some critical
value, a burst of output pulses will occur for a time t,
becomes 2.5 V for the configuration shown, due to the during which accommodation has not yet overtaken
concurrent inhibition. excitation (HILL, 1936; KATZ, 1936; HODOKIN, 1948).
Another technique for exploring transient excita- KATZ demonstrated a linear relation between the
bility uses a brief pulse rather than a constant current logarithm of relative excitation (in I/Io) and burst
duration, obtaining a family of such lines for various
20O accommodation time constants 2.
This behavior in the neuromime also follows di-
rectly from the stimulus-derived inhibition described
above. The results shown in Fig. 14 closely approxi-
160 mate those described by KATZ.
The measured average excitation time constant k
was 1.0 msec for the units tested; this value was main-

t
tained for all results described below. The inhibition
"~/20
time constant 2 was adjusted by varying C (Fig. l l)
from 0.5ttF ( 2 = 2 1 m s e c ) to 5 . 0 # F ( 2 = 2 1 0 m s e c ) ,
holding R constant at 200 KD. I t is seen that the burst
J duration is a linear function of In E / E o where E is
f
the stimulus level used to elicit the burst and E 0 is
rheobase. As KATZ pointed out, I-~ILL'Sresults imply
J that the relationship of burst duration to stimulating
r
f- current, threshold, and excitation and accommodation
time constants is
~tla _ ] _I/Is~/~ (3)
/
0 l 2 3 # 5
D'me /n milliseconds and if k/~ ~ 0, as is found in nerves showing repetitive
Fig. 13. T r a n s i e n t change of excitability following a 0.5 msec conditioning response, then
stimulus. Ordinate vahles indicate excitability relative to resting threshold t ~ 2 In I / I o. (4)
(100%). E x c i t a b i l i t y is established b y m e a s u r i n g the a m p l i t u d e of a
second 0.5 msec test pulse r e q u i r e d to elicit firing. Threshold is lowered This is the result obtained from the model and plotted
i m m e d i a t e l y following the conditioning stimulus, passes t h r o u g h rheobase
in less t h a n 0.4 msec, a n d reaches a m a x i m u m a t ~ 0.8 msee, g r a d u a l l y
in Fig. 14.' We expect to find that when I = 610, t = 2 .
r e t u r n i n g to its resting value. Curve A results f r o m a low a m p l i t u d e The points shown in the figure are located at these
stimulus; B a n d C represent increasing levels. I n D the s t i m u l u s w a s expected values. The measurements were made for
suprathreshold, t h u s the t i m e course of excitability follows the r e f r a c t o r y
r e c o v e r y characteristic voltages E rather than for currents I.
Burst duration lengthens as excitation is increased
as a subthreshold conditioning stimulus. This is fol- to a critical level, beyond which sustained firing occurs.
lowed by a test pulse to determine threshold. Typi- This occurs because the inhibition circuit saturates
cally it has been found (E~LANOER & GASSIER, 1937) at a somewhat lower level than does the excitation
that the facilitating effect of the conditioning pulse circuit. Consequently at very high stimulus levels,
is proportional to its strength, diminishing rapidly steady-state net excitation is above threshold, and
after the stimulus is removed. A subnormal phase the burst becomes indefinitely long.
B a n d 1, H e f t 3
D e z e m b e r 1961
LEoN D. HARMON: Studies with Artificial Neurons 9 I 97

Increased duration of burst implies greater num- steady-state the net excitation lies just below thresh-
bers of pulses per burst, since frequency of firing also old. Both excitation and inhibition are exponential
goes up with increased excitation. The discrete steps functions; thus, net excitation
observed in this process are shown in Fig. 15A. E(t) ~ Emax (1--e -tl~) -- Emax (1--e -t/~) (6)
For an inhibition time constant of 5.2 msec there where Emax is the stimulus amplitude, and k and
are only four levels before sustained firing occurs9 As )~ respectively are the excitation and inhibition time
is increased so that accommodation lags more and
constants 9 For the production of bursts, accommoda-
more, greater numbers of levels occur9
'
In Fig. 15B it is seen that for 2~-210 msec, bursts
of more than 1 l0 pulses occur9 Individual steps have 8
not been plotted for these curves beyond the first
few levels. Sample points were taken at the higher 7
levels and smooth curves drawn for simplicity.
For an excitation time constant which is short
relative both to typical interpulse periods and to in- G
hibition time constant, net excitation rises quickly and
2.5 I
~s
)~=~//2 / 8 5 /B5 2/o
2

/.5 // ./ !

0
2 3 q 5 6 7 8310 15 gO
Excitation in Volts
/ Fig. 15A. The nunlber of pulses in a t r a n s i e n t burst increases with ex-
citation a n d with inhibition t i m e constant 2. Above a critical stimulus
level, continuous firing occurs
I
50 I00 150 200 250 300 350 400 #50 120
Bur# dural/on in milliseconds Cont/'nuau~ i ~ - ~
Fig. 14, Relation between multiple-response (burst) d u r a t i o n a n d relative II0
excitation for v a r i o u s t i m e constants of inhibition (accommodation). The i

points shown are theoretically predicted vahles of b u r s t length when the


100
excitation E is e t i m e s the rheobase E0
9O
subsides slowly. This results in adaptation which
consists of a high initial firing frequency which dimin- ~8o
ishes as the burst progresses. When output frequency
is measured as a function of time for several different
inhibition time constants, the curves of Fig. 16 result.
Each of these curves was derived from a constant
applied stimulus 7.4 • rheobase, (ln E/E o ~- 2), hence
they all start at essentially the same frequency. The #0
form of each curve is exponential, reflecting the expo-
36
nential course of the growing inhibition9 Firing ceases
at about 50 pps, the low frequency cutoff described 20
earlier and shown in Fig. 7.
Examination of the total number of pulses in a 10
burst for each value of ~ suggested a relationship
which has not been previously obvious9 As shown in 2 3 #- 5 6 8910 /5 20
the curves of Fig. 17, it appears that the pulse number Excitation in Volts
Fig. 15B, The n u m b e r of pulses in a burst as a functiou of excitation
is a linear function of inhibition time constant. This for large values of 2. Smooth curves r a t h e r t h a n discrete steps h a v e been
holds over a wide range of stimulus levels. Curve A d r a w n for simplicity where the pulse count becomes large. O v e r 110 pulses
results from In E/Eo= 2; in B, In E/Eo~-1.5, and C h a v e been observed in a b u r s t for a n inhibition t i m e c o n s t a n t of 210 msec

represents the condition In E/Eo~-1.0.


That this relationship must hold is evident if the tion must rise much more slowly than excitation, hence
functions shown in Fig. 16 are considered more care- for the case of interest k ~ 2 . Hence Eq. (6) can be
fully9 Since each curve represents firing f r e q u e n c y / simplified to
as a function of time, the area under each curve must E(t) ~ Emax e-t/~. (7)
be equal to the total number of pulses, N. That is, From Fig. 7 we have the approximate relationship
T
N = f / (t)dt. (5) /(t) = cE(t) (8)
0
where/(t) is instantaneous frequency and c is a con-
The net excitation available to the neuromime is
stant of proportionality. Thus we can rewrite Eq. (5),
the stimulus-derived excitation less the stimulus-
the expression for the sum of the pulses, as
derived inhibition. In the circuit used, these two
T
components are approximately equal ~ so that in the N ~ cf Ema x e-t/,~dt (9)
1 This is equivalent to HILL'S (1936, p. 319) assumption 0
that Ul - V1= U0-- V0. .~ cEmax ~ (1--cT/~). (10)
98 L~oN D. H~Mo~r Studies with Artificial Neurons. I Kybernetik

T is the time at which firing ceases. This occurs I t is implicit in the work of KATZ and in the exponen-
at a definite firing frequency, /c, which is the same tial nature of accommodation, as was shown above.
for all 2 (about 50 pps) as shown in Fig. 16. Hence Confirmation could be useful, since counting the num-
from Eqs. (7) and (8) ber of spikes in a burst may provide an index of
/~ ~ cEmax 8-Tl~. (11) accommodation time constant more easily obtained
sloe; than by the use of conventional experimental techni-
ques. I t should also be noted that for the conditions
discussed, adaptation mirrors accommodation; their
~ Total No. time courses are identical. Hence in this case the
6~
~ 300 of pulses accommodation time constant can be inferred from a
k~ 21 8
~t2 /5 measurement of adaptation.
By varying the inhibition parameters (level and
125 q-G time constant), a wide range of accommodation and
~ 2o0 210 77
adaptation can be exhibited. Without inhibition, no
accommodation and no adaptation occurs; rheobase
stimulation elicits repetitive discharge which is con-
~- zoo t\W\ \ \ stant in frequency. For moderate inhibition, as de-
scribed, rheobase stimulation elicits one or two pulses ;
increased levels of stimulation evoke bursts or repeti-
~--21 q2 85 125 270 tive discharges which exhibit accommodation and
adaptation. This variety of behavior is seen in the
0 /0o 200 300 ~00 500 results described by HODGKIN (1948) for a large class
Time in milliseconds of motor neurons and sensory receptors.
F i g . 16. A d a p t a t i o n d u r i n g a b u r s t for v a r i o u s inhibition t i m e constants.
A step-function stimulus of 7.4 • rheobase causes initial firing r a t e s of Sell-sustained discharge. There are two basic kinds
~ 3 5 0 pps in each case; concurrent inhibition s u p e r v e n i n g a t a rate de- of mechanisms which can give rise to continuous acti-
p e n d e n t on ~ e v e n t u a l l y t e r m i n a t e s each b u r s t a t ~ 5 0 pps, the
lowfrequency cutoff vity (i.e., oscillation) in the absence of external stimuli.
I n one, a closed chain of neurons propagates reverbera-
Substituting (11) into (10), we have tory activity around a loop (LoI~NT~ DE N6, 1949).
N ~ Jt ( c E m a x - - l c ) " The minimum number of units required in the loop
Since c, Emax, and /c are constant, is determined by pulse amplitudes and durations and
N :r 4, approximately. by refractory time constants. Any given unit must re-
8O
cover sufficiently before the next stimulus comes along.
The second class of self-sustained oscillators de-
pends on positive feedback around only one neuron.
This feedback m a y be due to internal mechanisms
-o
GO i ;i or to a feedback loop around the entire unit, as may
occur in recurrent axon collaterals. An output-derived
"~ 50 I potential in the neuromime can easily be integrated
and fed back to form an adequate excitation stimulus
% for self-sustained discharge. In this case, maximum
;/ J firing frequency is limited by the refractory time con-
stant and the available output level.
..o
The two kinds of circuits are shown in Figs. 18
~ 2o f and 19. I n Fig. 18, where curve A is for one excitation
- r Z input lead, a minimum of seven units is required for
oscillation. As more units are added, oscillation con-
tinues, but because transit time through the loop in-
0 50 I00 150 200 250 creases, firing frequency decreases. If all five input
Inh/bl'f/on //me cons/an/ )u
Fig. 17. The total n u m b e r of pulses in a t r a n s i e n t burst is a linear function
leads are strapped together (curve B), the more effec-
of the inhibition t i m e c o n s t a n t for a n y g i v e n stimulus level. A results tive excitation causes firing earlier in the recovery
f r o m a stimulus E where In E/Eo = 2. I n B, In B/E~,= 1.5; in C, In E/Eo = cycle; hence fewer units and higher frequencies are
1.0. T h i s is predictable f r o m the n a t u r e of the t i m e course of the
s t i m u l u s d e r i v e d inhibition. See t e x t for d e t l i l e d analysis possible.
In this type of self-sustained discharge circuit, the
The approximate nature of the relationship arises from upper firing frequency is determined by the transit
the nonlinear firing frequency characteristic, the fact time in the shortest possible loop. There is no limit
that final net excitation is not exactly zero, and the on the lowest firing frequency; this is determined
fact that k is not vanishingly small. For the cases solely by the total number of units in the loop.
considered, the approximation is accurate to within I n the single-unit oscillator of Fig. 191, the firing
5%, hence with this limitation we can write freqqency is independent of C as C is made large;
N~, (12) 1 Note how this circuit differs from that of Fig. 8. In
that is, the total number of pulses in a burst is directly both circuits a potential derived from the output is fed back
proportional to the inhibition time constant. The close- to an input. However, in the case of the oscillator (Fig. 19),
ness of the approximation is evident in the linearity the feedback is positive; an output signal is applied to an
excitatory input. This action is regenerative. In contrast,
of the curves in Fig. 17. the circuit of Fig. 8 utilizes negative feedback since the output,
I t would be interesting to know if this result can applied to the inhibitory input, tends to depress further
be verified experimentally for biological preparations. activity.
B a n d 1, H e f t 3
D e z e m b e r 1961
L~oN D. HARMON: Studies with Artificial Neurons. I 99

a sufficiently large capacitor insures a fully smoothed was originally unity, progresses integrally l, 2, 3, 4 . . . .
o u t p u t waveform. T h a t is, the o u t p u t pulses are This is w h a t one reasonably expects to happen, b u t
integrated to yield a feedback excitation voltage which in fact it does not. W h a t does occur is shown in
is relatively constant, being the peak amplitude of Fig. 20. The ratio of /2//1 has been plotted for con-
the o u t p u t pulse. As C is decreased, the average venience. I t is seen t h a t for decreasing excitation
feedback voltage is diminished, and firing frequency at the second unit, the integral steps expected show
drops off until available excitation does not exceed up, but considerably more complex behavior also
instantaneous threshold. Thus the lowest possible appears. Thus the predicted steps are seen as l : l ,
firing rate for this t y p e of reverberatory circuit is 1 : 2, 1 : 3 . . . 1 : l0 (frequency division has been oh-
fixed, unlike the multiple unit loop circuit. The upper
800
limit of firing frequency, which is determined b y the
peak o u t p u t voltage, is likewise bounded.
Curve A depicts the case for three input leads
strapped together. Oscillation becomes impossible if • 600
more leads are connected in parallel, i.e., if the input
time constant ]c is made smaller t h a n 0.66 msec. There
are two reasons for this. As /c is reduced, latency is
decreased. Hence excitation derived from an o u t p u t #00
pulse occurs earlier, when refractory recovery is less
complete. Secondly, reducing/c means t h a t C is dis-
charged sooner, with the result t h a t the feedback A
~. 200
excitation is drained more rapidly. Curve B results
from using one i n p u t ; here latency is increased, hence
the repetition rate is lower.
A special case o / s u m m a t i o n . During the course of
these experiments an unusual and unsuspected pheno- 0 2 ~ 6 8 /0

m e n o n was found. I t m a y have no relevance to bio- Number of un/fs


Fig. 18. Self-sustained discharge in a chain of _~r units. A m i n i m m n of 7 is
logical behavior, but it does illustrate an interesting required w h e n a single i n p u t lead is used (A). W i t h m o r e effective ex-
additional p r o p e r t y of neuron-like entities. citation obtained b y five i n p u t leads in parallel (B), only 4 units are re-
quired to sustain oscillation. I n each ease, as m o r e units are a d d e d to
If one unit is used to drive another directly, and the chain, f i r i n g frequency diminishes
the pulse amplitude is sufficiently high, then one would
expect the second to "follow" the first, pulse for pulse. 70O
This is, in part, w h a t is observed. At relatively high
frequencies, however, the second unit begins to "drop 9~ 5oo
o u t " ; i.e., at some point the pulse amplitude of the voo
first is insufficient to exceed the relative refractory
threshold of the second. B u t if three (or more) of the
excitatory inputs of the second unit are strapped
together to utilize the driving pulses more effectively,
then following is complete up to the m a x i m u m possible -01 aOg
L
aOg ~06 a08 a l aZ a#
firing rate. This action was mentioned briefly in an C /n rnicrofarads
earlier section dealing with delay. Fig. 19. Self-sustained discharge obtained by a feedback loop around
one unit. I n c r e a s i n g the output-pulse i n t e g r a t i n g capacitor provides a
Consider now the following experiment. Suppose g r e a t e r a v e r a g e level of self-excitation. F i r i n g frequency thus increases
t h a t at some relatively high frequency (say 400 to up to the limit imposed b y the m a x i m u m available output-pulse amplitude.
Oscillation becomes impossible when the filtering is inadequate to m a i n t a i n
600 pps) one unit is following another, pulse for pulse. sufficient stimulus level
Such one-for-one following has been described b y BUL-
LOCK (1946) and b y FURSI~PAN & POTTER (1957). Now
served up to 1:13). However a m u c h larger n u m b e r
let the o u t p u t amplitude of the driving unit be mono-
of nonintegral steps (such as 3:5, 5:16, 3:19, etc.)
tonically reduced, " s t a r v i n g " the driven unit. This
also appears. This cannot be explained simply on the
could represent a case either of decremental conduc-
basis either of summation or refractory recovery alone
tion of spikes or a change in threshold of the recipient
or in combination.
neuron. As the driven unit receives progressively less
I t was found t h a t the mechanism responsible for
excitation, there will be a critical point at which a
this curious behavior depends on phase relationships
given driving pulse has insufficient energy to fire the
driven unit. However, it turns out t h a t when the recovery additionally comes into play. Note that a pulse
next driving pulse comes along, it is integrated with in such a continuous train of reduced amplitude pulses fails
the preceding one to cause firing. I n other words, to excite because it is subliminal with respect to instantaneous
(refractory) threshold, not to resting threshold. Hence some
temporal s u m m a t i o n occurs, and the firing frequency subsequent driving pulse will elicit firing when refractory
of the driven unit is half t h a t of the driver. One recovery has progressed sufficiently. Therefore the 4th, 5th
would expect t h a t as the driving pulse amplitudes or nth pulse in a train can be effective. In this way a pulse
are reduced still more, every third pulse would be frequency division occurs, every nth driving pulse causing
the driven unit to fire. "Summation" can then be considered
effective, then every fourth 1 etc., so t h a t ]1]]0. which as a composite process; classical axon summation occurs for
nonrepetitive stimuli, where each of a few subrheobase pulses
1 Summation becomes ineffective for more than n pulses are integrated to exceed quiescent threshold. In the case of
when the time from the first to the n th pulse exceeds the repetitive stimuli, n - - 1 pulses in each train of n pulses fail
effective integrating time at the input of the driven unit. to exceed the instantaneous dynamic threshold; this is
Typically this limit is 2 or 3 pulses. However, refractory iterated subtraction, i.e., division.
100 LEON D. HARMON: Studies with Artificial Neurons. I Kybernetik

between pulses in both N 1 and N 2. This can be seen as indicated by the arrow. The voltage at the base
in Fig. 21. of T 1 in the driven unit falls exponentially to a parti-
The top line shows the output of the driving unit. cular level as the pulse from 371 charges up the input
Seven down-going pulses appear. The bottom row of integrating capacitor. Meanwhile the emitter of T 1
16
(3rd trace) rises from a deep, fully inhibited level,
O~ heading for the resting threshold level at a rate
0.6---- governed by the relative refractory time constant (the
unit is just recovering from having been fired by the
07 .12:3 previous N 1 pulse). At the time that the potential
at the base of T 1 is equal to the potential at the
l~ O5
1:2 I emitter, the firing threshold condition is satisfied, and
an output pulse in N 2 is initiated. This occurs just
before the end of the second N 1 pulse. The base of T 1
reflects this firing by terminating the input pulse
integrating process and by being rapidly pulled in the
positive (upward) direction.
.g At termination of the output pulse, the base of T,
is again free to integrate the next pulse out of N 1.
O.2 I ]3..~e I I I Meanwhile the relative refractory phase, reflected in
the slow rise of potential on the emitter of T1, has
begun again as soon as the N~ output pulse terminates.
Note, however, t h a t as this action repeats, the time
at which hr2 fires becomes progressively later with
respect to the N 1 driving pulse. This follows from the
d~O 1 2 3 it 5 fact t h a t the instantaneous magnitude of the N~
E.Yci/ation in Volis threshold (refractory curve) is not phase-locked to the
Fig. 20. Firing response of one unit being driven by another. "Following"
is precise at high levels of excitation. As the driven unit receives pro-
N 1 driving pulses. Thus a phase slippage can and
gressively smaller stimuli, it begins to drop pulses. The 1:2 and 1:3 steps does occur. Eventually (at the fifth pulse) this slip
result from ordinary temporal summation. Other ratios of firing frequencies accumulates to the point t h a t by the time the driving
stem from another phenomenon, caused by asynchrony between driving
pulses and the refractory period of the driven unit pulse ends (the base of T 1 has reached its m a x i m u m
excitation level), the refractory recovery has not been
pulse respresents the output of the driven unit. Note sufficient to allow firing. Consequently a pulse is
that, including the partial pulse at the right end, only missed.
six appear. One pulse, corresponding to the fifth one Refractory recovery continues during this blank
from N 1 has dropped out. This sequence repeats (not interval, more and more closely approaching resting
threshold. Then the next N 1 pulse comes along. Since
threshold is now quite low, firing occurs relatively
early. Again the base of T 1 is discharged and awaits
the next input signal. Thus the cycle is completed;
this point in time corresponds to the situation at the
second pulse where the tracing of the sequence began.
I t would be interesting to examine biological pre-
parations for the presence of this type of coding.
Auditory volleying, for example, m a y utilize such
mechanisms.

Uses/or the model


There are two distinct types of explorations which
can be made with the described model. I n one, single
unit properties can be studied in detail, and new or
unsuspected relationships m a y be suggested or dis-
closed. Such characteristics as the linear relation of
number of pulses in a burst to accommodation time
constant, and the "summation-division" pulse-follow-
Fig. 21. Phase relationships between driving and driven units which explain ing function are examples of such results.
the firing frequency relationships of Fig. 20. See text for detailed At a higher level, group interaction properties such
explanation
as the behavior of small networks and transducer
encodings, for example, m a y be studied. I t has been
shown) in such a way t h a t there are only r pulses possible to explore systems at the retinal and cochlear
from N e for every 5 pulses from Nf. This corresponds levels with profit (VAN BERGEIJK & HARMO~r 1960).
to the 4:5 step shown in Fig. 20. Such mechanisms as on, o][, and on-o/~ receptor
The reason for this behavior can be seen by con- systems, and dynamic range extension in spiral in-
sidering the second and third traces of Fig. 21. In the nervation networks have been examined with these
second, the base of T 1 (input transistor, see Fig. 1) electronic models. Other studies have been made of
in N 2 reflects the (integrated) input pulses coming the possible neurological origins of flicker-fusion pheno-
from N 1 . Consider the second pulse as a starting point, mena and of primitive pattern recognition.
Band 1, Hef~ 3
Dezember 1961 LEON D. H ~ o ~ : Studies with Artificial Neurons. I 101

T h e t w o p a p e r s f o l l o w i n g d e s c r i b e in d e t a i l t w o Theory in Biology. New York: Pergamon Press 1958. - -


of t h e s e e x p l o r a t i o n s . C u r r e n t a n d c o n t i n u i n g w o r k FUORTES, M. G. F. : Initiation of impulses in visual cells of
Lit~ulus. J. Physiol. (Lend.) 148, 14 (1959). --FURSm'A~,
is i n t e n d e d to e l u c i d a t e f u r t h e r s o m e of t h e i n f o r m a - E. J., and D. D. POTTER: Mechanism of nerve-impulse trans-
t i o n - p r o c e s s i n g m e c h a n i s m s of p e r i p h e r a l n e u r a l s t r u c - mission at a crayfish synapse. Nature (Lend.) 180, 342
tures. (1957). - - GRANIT, R.: Sensory Mechanisms of the Retina,
p. 157. London: Oxford Univ. Press 1947. - - GRUNDFEST, H. :
I n c r e a s i n g l y d e t a i l e d k n o w l e d g e of n e u r a l f u n c t i o n Synaptic and ephaptic transmission. In: Handbook of Physio-
h a s m a d e m o r e a n d m o r e c o m p l e x o u r p i c t u r e of t h e logy, Section I: Neurophysiology, Vol. 1, ed, J. FIELD,
n e u r o n . I n a v e r y r e a l sense a u n i f i e d a n d c o m p r e h e n - pp. 147--197. Washington: American Physiological Society
sive understanding appears to become more, rather 1959. - - HARMON, L . D . : Artificial neuron. Science 129,
962 (1959). - - HARMON, L . D . , and R. M. WOLFE: An elec-
t h a n less, r e m o t e . H o w e v e r , i t m a y v e r y w e l l be pos- tronic model of a nerve cell. Semiconductor Products 2, 36
sible t h a t a s m a l l n u m b e r of b a s i c p r o p e r t i e s a c c o u n t s (1959). - - HAI~TLINE, H . K . : The response of single optic
for t h e e s s e n t i a l c o m p u t i n g f u n c t i o n s of t h i s e l e m e n t . nerve fibers of the vertebrate eye to illumination of the retina.
C e r t a i n l y t o a f i r s t - o r d e r a p p r o x i m a t i o n t h e cha- Amer. J. Physiol. 121, 400 (1938). - - HERMANN, L.: Zur
Theorie der Erregungsleitung und der elektrischen Erregung.
r a c t e r i s t i c s d i s c u s s e d in t h i s p a p e r u n d e r l i e m u c h Pfliigcrs Arch. ges. Physiol. 75, 574 (1899). - - HILL, A. V.:
of o b s e r v e d n e u r a l a c t i v i t y . A s m a l l n u m b e r of p r i m i - Excitation and accommodation in nerve. Prec. cap. Soc.
t i v e r e l a t i o n s h i p s d e s c r i b e a l a r g e b o d y of e x p e r i - B l l g , 305 (1936). - - HODOKIN, A. L. : Evidence for electrical
mental results. transmission in nerve, Part II. J. Physiol. (Lend.) 90, 211
(1937). - - The local electric changes associated with repetitive
I n t h e d e s i g n of t h e e l e c t r o n i c m o d e l , a f e w simpli- action in a non-medullated axon. J. Physiol. (Lend.) 107,
fied a priori assumptions were seen to have implicit 165 (1948). - - HODGKIN, A. L., and A. F. HUXLEY: The dual
i n t h e m m a n y f u n c t i o n s of b i o l o g i c a l n e u r o n s . Con- effect of membrane potential on sodium conductance in the
giant axon of Loligo. J. Physiol. (Lend.) 116, 497 (19529); - -
s e q u e n t l y , a f u n d a m e n t a l t h e o r y of n e u r a l a c t i o n m a y A quantitative description of membrane current and its
u l t i m a t e l y b e m u c h less c o m p l i c a t e d t h a n t h e e n o r m o u s application to conduction and excitation in nerve. J. Physiol.
b o d y of d e t a i l e d s t u d y p r e s e n t l y l e a d s o n e t o s u s p e c t . (Lend.) 117, 500 (1952b). --HUEEL, D. H., and T. N. WIESEL:
A s i m p l i f i e d , c o h e r e n t v i e w of n e u r a l e v e n t s is h i g h l y Receptive fields of single neurones in the cat's striate cortex.
J. Physiol. (Lend.) 148, 574 (1959). - - KATZ, B.: Multiple
d e s i r a b l e ; n e u r a l a n a l o g s c a n p r o v i d e a u s e f u l t o o l in response to constant current in frog's medullated nerve. J.
a p p r o a c h i n g t h a t goal. Physiol. (Lend.) 88, 239 (1936); - - Depolarization of sensory
terminals and the initiation of impulses in the muscle spindle.
J. Physiol. (Lend.) 111, 261 (1950). - - K L E E ~ E , S.C.: Re-
R e l e r e n c e s . ADRIAn, E . D . : The recovery process of ex- presentation of events in nerve nets and finite automata.
citable tissues, Part 1I. J. Physiol. (Lend.) 55, 193 (1921); - - In: Automata Studies, ed. C. E. S~ANNON and J. McCARTHY,
The effects of injury on mammalian nerve fibres. Prec. Roy. pp. 3--41. Princeton: Princeton Univ. Press 1956. - - KiieF-
Soc. B 106, 596. - - ARVAN~TAK~,A. : Les variations gradu4es r~iiLLER, K., u. F. JENIK: ~ b e r die Nachriehtenverarbeitung
de la polarisation des syst6mes excitables. Paris: Herman in der Nervenzelle. Kybernetik 1, 1 (1961). - - KUFFLER,
1938. - - BERGEIJK, W . A . WiN: Nomenclature of devices S . W . : Discharge patterns and functional organization of
which simulate biological functions. Science 132, 1248 mammalian retina. J. Neurophysiol. 16, 37 ( 1 9 5 3 ) . -
(1960). - - BEaOE~JK, W. A. vA~, and L. D. HARMON: What LETTVIV, J . Y . , H . R . MATUaA~A, W. S. McCuLLOCH and
good are artificial neurons ? Bionics Symposium, WADD W. H. PITTS: What the frog's eye tells the frog's brain.
Tech. Rcp. No60-600, 395 (1960). - - BROCK, L. G., J. S. Prec. Inst. Radio Eng. 47, 1940 (1959). - - LILLIE, R. S.:
COO~BS and J. C. ECCLES: The recording of potentials from The passive iron wire model of protoplasmic and nervous
motoneurones with an intracellular electrode. J. Physiol. transmission and its physiological analogues. Biol. Rev. 11,
(Lend.) 117, 431 (1952). - - BULLOCK, T . H . : A preparation 181 (1936). - - LORENTE DE NL, R.; Cerebral cortex: Archi-
for the physiological study of the unit synapse. Nature (Lend.) tecture, intracortical connections, motor projections. In:
158, 555 (1946); - - Neuron doctrine and electrophysiology. Physiology of the Nervous System, ed. J. F. FULTON, 3rd ed.,
Science 129, 997 (1959). - - B u a N s , B. D.: The mechanism of pp. 288--312. New York: Oxford Univ. Press 1949. - -
after-bursts in cerebral cortex. J. Physiol. (Lend.) 127, 168 McCuLLOCH, W. S., and W. H. PITTS: A logical calculus of
( 1 9 5 5 ) . - CRANE, H. D.: Neuristor Studies. Stanford Elec- the ideas immanent in nervous activity. Bull. Math. Biophys.
tronics Laboratories technical report 1506-2, pp. 1--191. 5, 115 (1943). - - MINSKY, M . L . : Theory of neural-analog
Stanford, Calif. 1960. - - ECCLES, J. C. : Synaptic potentials reinforcement systems and its application to the brain-model
of motoneurones. J. Neurophysiol. 9, 87 (1946). - - ECCLES, problem. P h . D . diss., Princeton University 1953. - - NEU-
J . C . , and A . K . MCINTYRE: The effects of disuse and of ~ANN, J. v. : Probabalistic logics and the synthesis of reliable
activity on mammalian spinal reflexes. J. Physiol. (Lend.) organisms from unreliable components. In: Automata Studies,
121, 492 (1953). - - ERLANOE~, J., and H. S. G~SSEa: Elec- cd. C. E. S H A ~ o ~ and J. McCARThY, pp. 43--98. Princeton:
trical Signs of Nervous Activity. Philadelphia: Univ. of Penn- Princeton Univ. Press 1956. - - PECHER, C.: La fluctuation
sylvania Press (1937). - - FARLEY, B. G., and W. A. CL~mK: d'cxcitabilit~ de la fibre nerveuse. Arch. int. Physiol. 49,
Activity in networks of neuron-like elements. Prec. 4th Lend. 129 (1939). - - RENSHAW, B.: Effects of presynaptic voUeys
Symp. Information Theory, 1960 (in the press). - - FATT, P. : on spread of impulses over the soma of the motoneuron.
Biophysics of junctional transmission. Physiol. Rev. 34, 674 J. Neurophysiol. 5, 235 (1942). - - ROCUESTER, N., J . H .
(1954). - - FATT, P., and B. KATZ: Spontaneous subthreshold HOLLAND, L . H . HAIBT and W. L. DUDA: Tests on a cell
activity at motor nerve endings. J. Physiol. (Lend.) 117, assembly theory of the action of the brain, using a large
109 (1952); - - The effect of inhibitory nerve impulses on a digital computer. Inst. Rad. Eng. Trans. Inf. Theory IT-2
crustaceaR muscle fibre. J. Physiol. (Lend.) 121, 374 ( 1 9 5 3 ) . - No 3, 80 (1956). - - RUSHTO~r, W. A. It.: A new observation
F~K, K., and M. G . F . F u o ~ E S : Stimulation of spinal of the excitation of nerve and muscle. J. Physiol. (Lend.)
motoneurones withintracellular electrodes. J. Physiol. (Lend.) 7~, 16P (1932). - - ~VILLIS, D. G. : Plastic neurons as memory
184, 451 (1956). - - FRm~KOrF, L. S., and W. A. ROSENBLITn: elements. Inst. Rad. Eng. WESCON Cony. Record, Vol. 3,
Fluctuations in neural thresholds. Symposium on Information pt. 4, 55 (1959).

You might also like