2D1432 Artificial Neural Networks and Other Learning Systems
!

Plasticity vs. Stability Dilemma
Plasticity: Network needs to learn new patterns Stability: Network needs to memorize old patterns Human brain: face recognition

Adaptive Resonance Theory (ART)

!

!

!

Plasticity vs. Stability Dilemma
Backpropagation
!

ART Characteristics
Goal: Design a neural network that preserves its previously learned knowledge while continuing to learn new things. Biologically plausible: ART has a selfregulating control structure that allows autonomous recognition and learning no supervisory control or algorithmic implementation.

!

!

!

New patterns require retraining of the network No Stabilization Stabilization achieved by decreasing learning rate Decreasing learning rate reduces plasticity

!

!

Kohonen maps (SOM)
!

!

other Neural Networks
ART Online learning Self-organizing (unsupervised) Maintains permanent plasticity Learn in approximate match phase Non-stationary world Other ANN (BP) Offline learning supervised Plasticity regulated externally Learn in mismatch phase (error based) Stationary world

ART Terminology
STM : Short term memory
!

!

Refers to the dynamics of neural units (recognition, matching) Refers to the adaptation of weights (learning) control structure to activate/deactivate search and matching

!

LTM : Long term memory
!

!

Gain control :
!

1

ART Basic Architecture
F2 gain + !

ART Basic Architecture
F2 layer:
! ! !

STM F2
LTM +
!

Each node represents a prototype (template) Competitive dynamics (winner takes all) Performs template selection Performs template matching Matches template from layer F2 with the input Just stores the input to F1, no dynamics

+

STM F1
F1 gain

F2 reset

!

F1 layer:
! !

Input layer :
!

Inputs

ART Basic Algorithm
! !

ART Algorithmic Level

!

Presentation of input vector to layer F1 Until resonance between F1 and F2 ! Determine winning node in F2 that has the highest activation in response to input. (adaptive filter) ! Matching between template and input ! If Mismatch is severe enough ! Inhibit template node Layers F1 and F2 are in resonance: Adapt topdown and buttom-up LTM weights

neurons vj Templates: Top-down weights zij neurons xi Input I

STM F2
LTM

wij

Adaptive filter: Bottom-up weights

STM F1

Inputs

ART Algorithmic Level
! ! ! ! !

ART Algorithmic Level
! ! ! ! !

X = {x1,…,xM} , xi ∈{0,1} (F1 nodes) |X| = Σi xi Zj = {zj1,…,zjM} , zij ∈{0,1} (templates) {wj1,…,wjM} , wij ∈[0,1] (adaptive filters) A = B ∩ C bitwise and
!

ai=1 if bi=1 and ci=1
! !

Initialize top-down weights zji=1 Initialize bottom-up weights wij randomly ensuring wij ρ (resonance occurs, ρ = vigilance parameter) ! Else delete vj from J until resonance or J is empty If resonance update template Zj and adaptive filter Wj ! top-down expectations zji = zji ∩ xi ! bottom-up filter wij= 1/|X| if xi=1, else wij=0

2

ART Algorithm Demo
Neural Network Demo ART1 Demo

ART System Implementation gain control + + +

Attentional subsystem

Orienting subsystem
+ + STM Reset signal +

!

STM F2
LTM

+

+ gain control +

STM F1
+

Inputs

ART 2/3-Rule
!

ART System Implementation
Input pattern I generates a STM activity pattern X at F1. I also activates F1 gain control and the orienting subsystem.

!

!

Nodes in layer F1 are supraliminally activated if they receive a signal from at least two out of three possible sources ! Bottom up input I ! Top down input Z ! Gain control A node in F1 that receives input from only one of the sources is subliminally activated.

!

+
+ + +

X=0110101
+

A
+

I=0110101

ART System Implementation
!

ART System Implementation
!

!

Pattern X inhibits A and generates the bottom up signal S, which is transformed by the weights W into signal T. F2 is a competitive network so the node vj which receives the largest total input is activated.

!

Pattern V at F2 generates the top-down signal pattern U which is transformed by the top-down templates into the expectation pattern Z. Pattern also V inhibits F1 gain control.

+

V=0010 S X=0110101
+ +

T

+ A
+

V=0010 Z=1100001

+

+

+ +

S X=0110101
+

U T

A +

I=0110101

I=0110101

3

ART System Implementation
!

ART System Implementation
!

!

As a result only those nodes X* in F1 that represent bits in intersection of input I and expectation Z remain supraliminally activated. If Z mismatches I this results in a decrease in the total inhibition from F1 to A.

!

If the mismatch is severe enough A can no longer be prevented from releasing a reset signal to F2. This resets the active node at F2. The vigilance parameter ρ determines how much mismatch will be tolerated.

+

- Z=1100001 + +

S X*=0100001 + I=0110101

V=0010 U T

+ A +

- Z=1100001 + +

S X*=0100001 + I=0110101

V=0010 U T

+ A +

ART System Implementation
! !

ART System Implememtation
!

!

After the F2 node is inhibited its top down expectation is eliminated and X can be reinstated at F1. The cycle begins again. X once again generates input pattern T to F2, but a different node is activated. The previous chosen node at F2 remains inhibited until F2’s gain control is disengaged by removal of the input pattern.

!

Once a match is found that allows F1 to inhibit the orienting subsystem the attentional subsystem is in resonance and the activations remain stable. Learning (adapting LTM) only takes places when layers F1 and F2 are in resonance.

+

V=1000 S X=0110101
+ +

T

A
+

- Z=1100101 + +

+

+

S LTM learning A X*=0100101 + + I=0110101

V=1000 U T

I=0110101

Leaky Integrator Neuron
! !

Shunting Model
! !

ε dn(t)/dt = -n(t) + p(t) Fix point n=p
+ dn(t)/dt 1/ε n(t)

ε dn(t)/dt = -n(t) + (b+-n(t))p+(t) – (n(t)+b-) p-(t) Fix point: n=(b+p+-b-p-)/(1+p+-p-) b+(t) X + dn(t)/dt 1/ε X b-(t) + + n(t) + +

p+(t)

p(t)

-

p-(t)

4

Input Normalization
! !

Input Normalization ε dn(t)/dt = -n(t) + (b+-n(t))W+p(t) – (n(t)+b-)W-p(t) Excitatory input: W+ = I , w+ij=δij Inhibitory input: W- , w-ij=1-δij On-center/Off-surround pattern n(t) + b+ n p + + -

ε dn(t)/dt = -n(t) + (b+-n(t))W+p(t) – (n(t)+b-)W-p(t) Fix point: n=(b+W+-b-W-)p)/(1+(W+-W-)p) b+ W+ X + dn(t)/dt 1/ε X

SxS p +

Sx1 SxS
W-

-

Input Normalization
! !

LTM Learning Law
Hebbian learning rule with decay dwij(t)/dt = α (-wij(t) + ni(t) nj(t)) For ART learning and forgetting should be turned off if the post-synaptic neuron ni is not active. dwij(t)/dt = α ni(t) (-wij(t) + nj(t)) Choose LTM time constant α much smaller than STM time constant ε ni w11 nj w21 w12 w22

! ! ! ! ! !

Assume inhibitory bias b-=0, and excitatory bias identical for all elements b+i=b+ ε dni(t)/dt = -ni(t) + (b+-ni(t))pi – ni(t) Σj≠i pj Fixpoint: 0 = -ni(t) + (b+-ni(t))pi – ni(t) Σj≠i pj ni = (b+pi)/(1+Σi pj) Total input P= Σi pj relative intensity pi* = pi/P ni = (b+P)/(1+P) pi

!

!

ART Layer 1
+ +

ART Layer 1
W2:1 b+ +
!

Expectation

a2

!

ε dn1(t)/dt = -n1(t) + (b+-n1(t))(p+W2:1 a2(t)) - (n1(t)+b-)W- a2(t)
Thresholded output 1 1 ! a = 1 if n > 0 1 1 ! a = 0 if n ≤ 0 p+W2:1 a2(t) sum of input vector p and L2-L1 expectation Assume the j-th neuron in layer 2 won the competition p+W2:1 a2(t) = p+wj2:1 W- a2(t) : gain control term , inhibitory input to layer 1 ! w ij = 1 ! Since there is only one winning neuron in layer 2, gain control will be 1 if layer 2 is active and 0 otherwise

X p + dn1(t)/dt 1/ε X

+
! !

Sx1
-

n1 +

a1
!

Gain control

W-

a2

b-

+

5

ART Layer 1
! ! ! ! !

ART Layer 1
! !

Case layer 2 is inactive a2j=0 ε dni1(t)/dt = -ni1(t) + (b+-ni1(t)) pi Fixpoint: ni1=b+pi / (1+pi) hardlim transfer function : ai=pi Summary: If layer 2 is inactive the output a of layer 1 is the same as the input p.

! ! ! !

!

Case layer 2 is active a2j=1, a2k=0 for k≠j ε dni1(t)/dt = -ni1(t) + (b+-ni1(t)) (pi+wij2:1 ) - (ni1(t)+b-) Fixpoint : ni1 =(b+ (pi+wij2:1 ) – b- )/(2+pi+ wij2:1 ) Choose b+=1, b-=1.5 ai1 = pi ∩ wij2:1 Layer 1 performs a logical AND operation between the input p and the layer 2 expectation wj Matlab demo with a12=0, a22=1

ART Layer 2
!

Other ART Architecture
!

! ! ! ! ! !

ε dn2(t)/dt = -n2(t)+(b+-n2(t))(W+ σ(n2(t))+W1:2 a1(t)) - (n2(t)+b-) W- σ(n2(t)) Excitatory input: W+ σ(n2(t))+W1:2 a1(t)) Inhibitory input: W- σ(n2(t)) W+, W-, on-center, off-surround feedback connections W1:2 prototype patterns b+ =1, b- =1 ni2(t) ∈[-1,1] Fixpoint behavior: 2 ! ai = 1 if wi1:2 a1 = max [ wk1:2 a1 ] 2 = 0 otherwise ! ai

!

!

!

Matlab demo: with W1:2 = [ 0.5 0.5, 1.0 0.0]

ART2 : handles continuous input patterns (Carpenter&Grossberg 1987) ARTMAP: hetero-associative memory suitable for supervised training, consists of two ART modules, one that receives the input, the other one receiving the desired output (similar to hetero-associative Hopfield network see laboration 2) (Carpenter&Grossberg&Reynolds 1991) Fuzzy ART and fuzzy ARTMAP : incorporates fuzzy logic, templates can assume fuzzy values (degree of membership) rather than binary membership

6