GIROD, Bernd, Mobile Visual Search, April 2, 2014
<p class="MsoNormal" >GIROD, Bernd, Mobile Visual Search, April 2, 2014</p>
<p>
intro by Joyce Wandell;</p>
<p>
great bkgrnd in academia and assoc with many start-ups...</p>
<p>
linking the Virtual and Physical World: Mobile Visual Search..</p>
<p>
great cartoon from Qualcomm.. the next stage of human evolution
(homo sapiens with cell phone)</p>
<p>
Landmark recog... eg, where are you, based on looking at Ferry Bldg in SF.</p>
<p>
how abt Media Cover Recog (in a vinyl record store on Telegraph Ave in Berkeley)
take a pic of the cover and the music begins to play...</p>
<p>
Motorola Droid Smartphone...</p>
<p>
he takes video of his book shelf and has it labled with AR with price and Amazon info.</p>
<p>
wine lables, field guide (with leaves)</p>
<p>
real estate...</p>
<p>
just like hyperlinks but using the real world...</p>
<p>
Image-based retrieval...</p>
<p>
but... are you going to serially do a comparison to all the images? NO... too slow..</p>
<p>
OUTLINE for today</p>
<p>
comp vision for image-based retrieval
feature compression
phone-based vs server-based processes
and toward mobile AR =augmented reality...</p>
<p>
A Bag of Words</p>
<p>
an example: self-evident, Liberty, truths, happiness, endowed, inalienable, pursuit, Creator, Life</p>
<p>
retrieves Declar of Independence...</p>
<p>
but must pick the salient words...</p>
<p>
can we do the same thing with images (analogous to Bag of words)</p>
<p>
so, how to represent = repr an image as a "bag of visual features"</p>
<p>
eg golgen gate bridge... get the towers, the cable</p>
<p>
(but u need features that are robust to illumination changes, scale,etc)</p>
<p>
visual features: key point detection</p>
<p>
usu just work with grayscale image</p>
<p>
DoG = difference of gaussian filter...</p>
<p>
eg detect extremes (corners, edges)</p>
<p>
get canonical image patches...</p>
<p>
get image gradients and then extract features...</p>
<p>
SIFT is the most famous... use a 128 vector (David Lowe)</p>
<p>
can even just count the features matched btwn the query image and the DB of images.</p>
<p>
but still plagued by serial search slowness.</p>
<p>
so how to arrange these hierarchically...</p>
<p>
eg can do hierarchial k means with 3 clusters...</p>
<p>
then you want to compare tree histograms...</p>
<p>
so then do you compute a Query tree histogram and
compare that to whole DB of tree histograms.</p>
<p>
Architecture A: send an image thru the wireless network</p>
<p>
say, take an image with cellphone and send to server where all the computing is done</p>
<p>
but sending a frame at 20 kbps will take 20 seconds to get adeq image</p>
<p>
so, much faster to send the Features </p>
<p>
or architect C: do all computing on the cellphone</p>
<p>
(ie do feature extraction and feature matching on the client (and do not use cloud at all)</p>
<p>
MPEG proposal in 2011</p>
<p>
say 2 k bytes for image: say 50 bits per descriptor...
(and store 100K images right ON the phone)
say 300 features</p>
<p>
want SOTA= state of the art image matching and retrieval...</p>
<p>
want a descriptor that’s smaller than a jpeg...</p>
<p>
SIFT is the gold standard in comp vision</p>
<p>
usu abt 1024 bits per descriptor * # of features (is > jpeg file)</p>
<p>
so, insead use canonical patches...</p>
<p>
use ROC to evaluate performance (TP vs false positives)</p>
<p>
compare L2 distance (computed by SIFT)</p>
<p>
u want to have low false postiives...</p>
<p>
CHoG compressed histogram of gradients</p>
<p>
use distrib of gradients...</p>
<p>
with 3 descriptors can map the probs onto an equilat triange (guaranteed to add to 1)</p>
<p>
CHog (pron chog) with Lattice coding is best (so far)</p>
<p>
can xmit features in ANY order... so waht order to use to send them...</p>
<p>
can do location histogram coeding (he shows building with many windows)</p>
<p>
compressed feature descriptor</p>
<p>
say the original SIFT descr requries 1088 bits...
can get it down to 59 bits</p>
<p>
hugfe gain by sending only the features say just 2.2 Kbutes take 3 second versus</p>
<p>
sending jpeg = 40 kbytes that takes 7 seconds...</p>
<p>
(server is typicall 10X as fast as mobile device)</p>
<p>
but, if u have fasgt connection, may want to just send the image....</p>
<p>
really expensive past is sending the data (in terms of energy consump by the phone)
the computing is relativ. free.</p>
<p>
also if u have a poor connex, must send it over and over again.</p>
<p>
*</p>
<p>
Augmented Reality..</p>
<p>
(really wants Terminator vision)</p>
<p>
he shows scene from Terminator 2 where Arnold is looking for clothes in the biker bar
in the opening scene...</p>
<p>
goes all the way back to Ivan Sutherland in 60s.</p>
<p>
1997: “The TOURING machine” with matches to bldgs.</p>
<p>
maybe contact glasses...</p>
<p>
shows video (not real)</p>
<p>
say without pressing a button (with Google Glasses... recog books as u look around)</p>
<p>
shows ratings of books... (they did implement that... )</p>
<p>
must track camera pose...</p>
<p>
he shows motion-adaptive query selection...
(when u go slow... it picks up features (but not when u move the camera)</p>
<p>
onto Architecture C (doing it ALL on the cellphone)</p>
<p>
problem: is RAM storage ont he phone.. must be really big to store DB of features on phone*</p>
<p>
we need vocab tree say with 1M leaf nodes... takes abt 70 MB...</p>
<p>
but hen also need to store compressed Inverted Index see Chen DCC 2010
say for 100K iamge4 is total of 140 MB. (with lots of compression)</p>
<p>
so add these = 210 MB ( a lot for a phone)</p>
<p>
David Chan’s work (grad student of his here)</p>
<p>
generating feature RESIDULAS (what was NOT matched)</p>
<p>
Fisher vectors and fisher information... (suggests this should be superior to BoW approach)</p>
<p>
and uses smaller codebook...</p>
<p>
REVV = residual enhanced viual vector..</p>
<p>
extract local features then use PCA and the use codebook to quantize feature descripsw and
use LDAS = linear discrim analys... and ultima. get
weighed correl scroes...</p>
<p>
performance is compar to vocab tree but his usess much smaller (25X smaller.. storage reqruied)</p>
<p>
he shows performa cne with Samsung Galaqxy s# smartphoen with 1.4 GHz proc
and 1 GB RAM and DB of 100K images
takes only abt 1 second!!! to match... eg in grocery store for labels.</p>
<p>
</p>
<p>
Hybrid Query Mode... occas (if no match) then send it to the cloud (send REVV signature)</p>
<p>
this is much faster then sending images (abt 100X as fasty to send REVV descrips as sending Jpeg)</p>
<p>
this is akin to vidoe coding in 1980... mainly uncharted territory...</p>
<p>
MPEG has been standardized</p>
<p>
but Video is next CDVS is going to CDVidS</p>
<p>
see Signal Process Magazine from July 2011... a mobile revolution IEEE publicaiton
(but 3 yrs old)</p>
<p>
Q power consump of feature extraction (most consumptive.. cuz have to touch all the pixels)</p>
<p>
perhaps can move feature extaction into the CAMERA module (so preprocess it even before it hits cpu of cellphone)</p>
<p>
Q difft lighting conditions (illum variations)
Q why not store those features also (under foreshortening or diff lighting)</p>
<p>
BW Q: building DB of images and features (annotated DB of images)</p>
<p>
harvesting from the web (and perhaps annotate with Mech Turk)</p>
<p>
Q and comment; we expect incorporation into the camera (of the phone) or a camera in a car...</p>
<p>
comment on phone: must be less than 3 watts (to hold n hand) typical power use is....
1W for radio, 1 W for cpu and 1 W for display.</p>
<p>
</p>
<p>
</p>