TENENBAUM, Josh

Mar 17, 2011 01:59 PM  email is  jbt@mit.edu
 
see Josh's videos at MIT and NIPS;

Humans demonstrate a remarkable ability to infer physical properties of objects and predict physical events in dynamic scenes. These abilities have been modeled as probabilistic simulations of a mental physics engine akin to 3D physics engines used in computer simulations and video games (Battaglia, Hamrick & Tenenbaum 2013; Sanborn, Mansinghka & Griffiths 2013), but it is unknown if and how such a physics engine is implemented in the brain. Does the brain represent quantities corresponding to the key latent variables of physical objects that contribute to their dynamics?

To find out, we used multivariate pattern classification analyses of fMRI data from subjects viewing videos of dynamic objects. The masses of depicted objects could be decoded from parietal and frontal brain regions previously implicated in intuitive physics (Fischer et al., 2016). Crucially, this decoding was invariant to the scenario revealing the object’s mass, as well as the the material, friction, and amount of motion of the object. These regions may support a generalized engine for intuitive physics where this invariant representation of mass serves as a key variable.

Engaging with the world requires a model of its physical structure and dynamics – how objects rest on and support each other, how much force would be required to move them, and how they behave when they fall, roll, or collide. This intuitive understanding of physics develops early in childhood and in a consistent order; by 3 to 4 months of age, infants understand that the world is composed of bounded, continuous objects (Kestenbaum & Spelke, 1987; Spelke, Kestenbaum, Simons & Wein, 1995), by 5 months, they can differentiate liquids and solids using expectations about the behavior of nonsolid objects (Hespos, Ferry & Rips 2009; Hespos, Ferry, Anderson, Hollenbeck, Rips 2016), and by 11 months they can infer an object’s weight based on its compression of a soft material (Hauf, Paulus, & Baillargeon, 2012). By adulthood, humans are capable of making sophisticated physical predictions in many different tasks requiring implicit physical reasoning or simulation (Battaglia, Hamrick & Tenenbaum, 2013). This body of behavioral evidence suggests that the brain contains detailed knowledge of the physical attributes of objects and the laws

of physical interactions between them. We consider these concepts and the laws relating them to constitute an intuitive theory of physics that enables us to successfully interact with objects, make predictions about physical events, and perform novel physical tasks. Recent computational efforts have explained human physical reasoning via an intuitive physics model that is quantitative, approximate, compositional, and probabilistic. Battaglia et al. propose a computational architecture shared by many physics engines, with two core parts: an object based representation of a 3D scene (which encodes static variables such an object’s size and mass), and a model of physical forces that govern the scene’s dynamics. This type of simulation-based model can make robust inferences about configurations of many rigid objects subject to gravity and friction, with varying numbers, sizes, and masses (Battaglia, Hamrick & Tenenbaum, 2013). If such a physics model were implemented in the brain, we would expect underlying brain regions to represent relevant physical dimensions as concepts that generalize across scenarios. Here we applied pattern classification methods to fMRI data obtained from subjects viewing videos of dynamic objects, to test for invariant representations of mass in brain regions previously implicated in intuitive physical inference (dorsal premotor cortex/supplementary motor area, and bilateral parietal regions; Fischer et al, 2016).

Experiments

Participants were scanned with fMRI while performing physical inference, prediction, and orthogonal tasks on visually-presented stimuli.

“Localizer” scans enabled us to identify key nodes of the candidate physics network in each subject individually, following Fischer et al. (2016). We then conducted three experiments to (i) test whether we can decode relevant physical variables from candidate physics fROIs with various degrees of invariance and (ii) test the automaticity of these representations. Design

Each scanning session included two runs of a localizer task from Fischer et al. (2016) which used a univariate contrast

Results

In Experiment 1, situation-invariant mass decoding in the candidate physics system had group mean accuracy 0.64, which was significantly above chance (p<0.05), and was found numerically in 6 out of 6 subjects. Critically, this representation of object mass does not depend on whether the object is splashing into water, being blown by a hair dryer, or being dropped onto a pillow: to obtain a decoding accuracy greater than 50%, the classifier must generalize a representation learned from two scenarios to a third scenario left out of training. Mean classification accuracies as well as classification accuracies for each left out scenario were greater than 50% in all subjects. Further, mass representations are not confounded with shape or color, as colors and shapes were represented in equal proportions for both masses in the training and testing data. While result mirrors the situation-invariant representations expected in a physics engine, an alternative hypothesis is that we may be decoding a prepared response to the explicit mass task (“Light or Heavy?” which is constant across scenarios). To test this hypothesis, as well as the automaticity of the mass representation, in Experiment 2 we used a design that interleaves blocks of the physics task and a color task on the same stimuli. This design enabled us to ask whether a situation-invariant mass representation can also be decoded from multivoxel activity during blocks where subjects perform the orthogonal color task where mass was not relevant.

In 6 new subjects, we replicated the findings of Experiment 1: mean decoding accuracy 0.63 was significantly above chance (p<0.05), and present numerically in each subject individually during the mass task. More importantly, mass decoding was also significantly above chance (mean = 0.61, p<0.05), and present numerically in each subject individually, during the color task. This result shows that mass is represented even when the task does not require it, and further that the decoding of mass we observe cannot be explained as an abstract response code. Further evidence against the idea that the mass representations reflect response codes comes from the fact that color decoding from the same voxel activity during the color task was at chance in all subjects. Thus the candidate physics engine does not represent all task-relevant dimensions and may be more specific to physical variables. In Experiment 3 we asked whether this representation can be decoded during a physical prediction task that requires an understanding of mass but does not explicitly suggest subjects attend to it, and tested the invariance of this representation to friction and motion. Experiment 3 replicated once again our finding that mass can be decoded from candidate physics regions (mean accuracy of 0.60 was significant, p<0.05, and numerically present in each of 13 out of 14 participants individually). Further, this experiment demonstrates an important new invariance of these mass representations beyond those already found in Experiments 1 and 2: the mass decoding in Experiment 3 required generalization across the friction and material of the object shown (lego to cork for heavy, and cardboard to aluminum for light).
 
Discussion

This work tested for a physics engine in the human brain by asking whether brain regions previously implicated in intuitive physical reasoning (Fischer et al., 2016) contain information about the physical properties of objects, specifically mass.

Indeed, we showed using fMRI decoding methods that the candidate brain regions for physical inference contain information about mass in 25/26 subjects tested. Importantly, this mass information is present even when mass is irrelevant to the task (Experiment 2) or when mass is relevant but the participant is not asked to report it
 explicitly (Experiment 3). Further, mass information is invariant to the dynamic scenario in which the mass of the object is revealed (Experiments 1 and 2), as well as to the coefficient of friction of the object, the material it is made of, and the overall amount of motion in the scene (Experiment 3). Taken together these results show that the brain regions previously implicated in intuitive physical reasoning represent mass in an invariant manner that would be expected for an intuitive physics engine. 
 Importantly, the brain regions implicated in intuitive physical reasoning resemble those previously implicated in action planning, highlighting the tight link between these two functions. Indeed, recent work in robotics (Todorov 2017) combined optimization with a physics model to design a highly efficient goal-directed action planning system. And fMRI studies in which humans perform actions (Johansson & Flanagan 2009; Loh et al. 2010; Chouinard, Leonard & Paus 2005; van Neunen, et al. 2012) reveal representations of object mass in brain regions where we report mass decoding in inference tasks, further strengthening the link between physical inference and action. A model-based account of physics in the brain could support both physical inference and action planning in the same underlying brain regions, which may serve as the seat of a neural physics engine. The suggestion of overlapping activation could be strengthened in future work testing action planning and physical inference in the same subjects.

TENENBAUM, Josh

Attachments