Interactive dynamic video

Abe Davis from MIT has developed a technique for creating a realistic dynamic simulation of an object by analysing vibrations in regular 2D video.

Objects have vibration modes — different patterns of vibration — that have different natural frequencies. This means that the kind of small, random-looking agitation you can capture on video (eg, from wind or physical contact) is made up from the effects of these vibration modes. Abe’s insight is the possibility of decomposing the random motion to understand the individual modes, allowing you to compute the effect of specific stimulus on the object by finding compositions of those modes that fit.

The resulting simulations are not perfect but they are visually compelling and require many times less effort to create than 3D scanning and rendering, and because they require only regular 2D video the source material can be existing footage shot by someone else. He talks about their use in engineering and in visual effects, as well as in augmented reality (AR). It seems to me that this technique is also useful in VR environments, allowing hitherto static 360º backdrops to have realistic interaction with the user.