Pentti Kanerva: UC Berkeley VS298 - Neural Computation Lecture 2008-12-09
<p>Dec 22, 2010 05:12 PM  Pentti's lecture at  
 
1 hour and 20 mins at Berkeley  in Bruno O's  seminar.
 
similar work by  Rachkovskij and Kuseel  Context Dependent Thinning
 
Plate:  Holographic Reduced Representation
 
Olshuasen  Shifter Circuits
 
Arathorn  Mapping Circuits
 
his work in SDM  and Spatter Code
 
big reference on LSA  is  Landauer and Dumair  1997  A solution to Plato's problem:
The Latent Semantic Analysis theory of acquisition, induction and repres of knowledge
PSYCHOL REView  104 (2)  211-240.
 
he shows random indexing = RI.
 
but can extend with  neighboring words  (with good results)
or equival classes for words  (discvoer grammar)
 
deriving meanings of words:
 
look at 37,000  "documents"  each documents is 166 words long.  (from high school texts in a variety of subjects:
science, art, business, history etc
 
total vocabulary is 54,000 words...
 
the big matrix  has 54,000 rows and 37,000 columns
 
say one row is for liquid... if it appears twice in the article about water then   a  2 will be stored at that position.
A document is said to activate that column.
 
That is the raw data matrix.
Raw semantic vector corresponds to each row:  eg the vector for liquid and the vector for liquids should be
very similar  in that they will appear in exactly the same types of articles.
and same for  man and woman or furniture and wood.
 
now you need a measure for measuring the similarity of vectors:
eg dot product or  cosine (normalized to unit length or correlation)
 
test it by synonymy
using  TOEFL  (test of English for foreign language)
 
test of 80 words  eg
 
prominent is a synonym of a) battered  .... d) conspicuous
 
zenith is a synon of  b) pinnacle
flawed is a syn of  b) imperfect
 
now:  check the vector for prominent  (with 37,000 entries) and
the vector for  conspicuous    that is a good test of how well it works.
 
You can do LSA with either  PCA or with ICA
LSA is the standard method for doing dimension reduction.  using SVD singular value decomp[ostion.
 
but the  raw semantic vectors are very sparse  (cuz most words are not in most documents)
Pentti  does random indexing   (the standard method is  LSA  latrent semantic analysis  ... early 1990s).
 
here's how SVD works:
 
SVD of matrix F  (F is the big frqeuncy matrix.
 
F (w times d) that is  words * documents  but can be exactly repres as product of three matrices
 
r is the rank of the matrix
 
A (w*r)  times  B (r *r)  times  C  (r * d)
 
B  is a diagonal  matrix of  eigen-values.
 
can approx F  by just using the first p eigen-vectors/ principal compoents
 
F prime  =  Aprime times Bprime times C prime
 
where rows of A prime are semantic vectors for the words
 
perhaps  liquid  =  .7 fluid , .3  cold,  etc.  (my exmaple).
 
300 is the optimal number;  when u go beyond, the score goes down cuz you're adding noise  (overfitting).
*
 
his method derives semantic vectors instead by random indexing.
 
so in SVD  thy go from  text to the F matrix  54k by 37k  then go to SVD  then get  54k by 300 matrix  (A prime)
where each row is a semantic vector.
 
instad in  RI  never collect the F matrix
go direclty from text to a RI (random index)  that is  54k  by 10k
(or can go to 10k by 10k by doing both dimensions)
 
here's how to do it with RI
 
start with  empty F or G matrix with all zeroes.
 
in the standard method  SLA  whenver you hit the word in the document, increment the cell by 1  (that gives you
the count of "liquid"  in the ith article.
 
in contrast in  Random Indexing RI:  each doument activates SEVERAL RANDOMLY CHOSEN columns
out of 10,000 in the RI-matrix  G;
10 positive and 10 negative coumns
randomly  10 positive and 10 negative columns
for each document.
if liquid appears in the document then add 1 to ALL the positive coumns
if it does not then subtract 1 from those columns.
 
 
 
 
 
 
 
He shows two method for reducing the dimensions of that matrix
 
one is called LSA  latent semantic analysis  LSA... it uses PCA  principal components analsysis
 
to go from  37,000 columns down to only 300 columns (that contain most of the meaning (I suppose)) of each word.
 
 
 
 
 
 
 </p>