Bill Dally
<p>Nov 9, 2009 7:58 PM
 
Bill DALLY  (as in Sally)  INtoducer says  Bill Dally also made  7 passses on the Death Ride!!!  in Tahoe... cycling.
 
was a prof at MIT in the 80s.  now at Stfd  was Ch. of CSD;  now also Chief Sci. at  <b>NVIDIA</b> as of 2009.
 
from  1980 until 2000  Moore's Law was in full sway for  single-thread processor...
got a speed up of  50% per year.
 
Moore's prediction in 1965 was a doubling of # of xstrs every year...  ; later revised to q 18 months.
 
More xstrs  was translated by computer architects like him into  MORE Performance  and the software guys
would tranlate more performance into  more complex applications and More Value with the XS cycles.
 
He shows  picoseconds per instruction falling by 52% per year from  1980 to 2000 for a single thred cpu.
After 2000 it has only been falling by  19% per year.
 
So, now getting speed up of only 20% per year.
 
Also got 4 instruction per clock cycle  and  fewer gates to traverse per cycle.
 
Another scaling factor was decreasing power consumption  ( to dissipate that power and to move all that power onto
the chip that fast.   Currently is at about 150 watts on the chip.
 
So,  as a software guy you could do NOTHING in the old days ... run the same crappy code yr after yr and
get a 52% incr in performance .  Therefore, in those days it DID NOT PAY to optimize code and architecture.
 
NOW it will pay to optimize code and archit...  cuz we'll be able to get a  300X speed up in a decade by
recoding  the single-thread code to parallelize it.
 
Chips are power limited.  MOST Power is NOT spent in the ALU but rather in moving data around on the chip ...
having to switch the voltage on all those lines from 0 to 1 all the way across the chip.
 
Another big source of heat now is LEAKAGE heat cuz the gates are so thin..
 
He notes  today is  BERLIN WALL 20th anniversary day....  however  this  speed up by  300X by
parallelization will be another revoltuion.
 
He's an artist on silicon ... chip is  20 mm by 20 mm.  Can put whatever you want on it.
 
Say,  draw  4,000  64 bit  FPU's  =  300 watts of dissip and 12 Teraflops.
 
However, he says,  kids and pets are easy to acquire but hard to feed and maintain.
So, those FPU's  are easy to acquire but to feed them with data and keep them busy is tough.
 
25 picojoules per 64 bit word  (to move data?)
 
His big lessons:  EFFICIENCY = LOCALITY ...  keep the data u need local;  whenever u need to
get data from a more distant cache it takes time and heat.   Also  PERFORMANCE = PARALLELISM.
 
APPLICS:  SIMULATE PHYSICAL WORLD ...  fluid dynamics,  molec dynamics.
Video Processing  (eg  Video Conferencing  a la  Sirikada)  Video decoding;  Video Surveillance.
Sensory stuff.
 
He is trying to do with genl purpose chip what others do with a special purpose dedicated chip.
 
Lots of  chips are embeeded applications, eg  ARM core in iPHONE's   for video, codecs, modems, image processing.
 
Amdahl's Law:  CAn't go faster than the SERIAL PORTION of the task.
 
single-thread processors take 20 nanojoules per instruction?
His systems take  200 picojoules per instruct  (2 order of mag better).
 
He shows a diagram of a tree ...
leaves at bottom are each a processor with its own  L1 memory  those are linked together by a network and that
network (the L2 level) also has its own memory and  many of those are linked together to form the L3 level with its
own memory  and so on.
 
Then use explicit memory management.
 
Applic  in  physical simulations (FEA)  is in receiving data across the faces of the element then doing computation to simul.
interior of the element.
 
StreamFem3D  look this up to see pdf of Bill's 2005 talk.
 
Imagine Chip  ...  Merrimack  Storm-1
 
Explicit Memory Mgmt is important.
 
Core2 Duo is a 20 nanojoule machine  ?  per instruc.  very inefficient.
 
1997  started with StreamC and Kernel C
 
2002  Brook  (get it.. like a stream)
works with Pat Hanrahan
 
2007  CUDA and Sequoia   oil and gas explor  seismic analysis  all these guys use stuff like this.
 
HD video;  Pricing bonds doing Monte Carlo sim.
Ultrasound  bioinformatics.
 
N body problem  gravity;  convolutions;   Throughput computing
 
So by 2017 will have  40 TeraFlop machine ... right now  Jaguar is fastest genl purpose machine
CUDA Sequoia programming system.
 
First Q from audience:  how abt  desktop machines:  his answer they will be helped by this for
video applic  (Photoshop)  Adobe Creative Systems  etc.
*
Below my Q's for  Bill Dally ... I did not ask these
Kwabena Boahen - Neuromorphic Neurons
R U still at Stfd or full-time at NVIDIA
Is Moore's Law on Track ... prospects for next 20 years
After CMOS, what's next
Do u work with vision guys:  Fei-Fei Li, Andrew Ng
Robotics guys:  Sebast  Thrune ...  great applic ...  will get  300X thruput with genl purpose computers.
Data Mining...  yes ... great applic
Will multi-core processors be a net win in the home for the typical mix of applications.  yes for video communic/ sensory
Can those applications be parallelized?
 
 </p>