Interactive Physically-based Sound Synthesis

 

While visual feedback is often the main focus in Virtual Environments, sound is another important modality. Auditory feedback can enhance the immersion and believability of virtual scenes. However, the common approach of using pre-recorded sound samples is often not feasible for highly interactive environments, because too many sounds would have to be recorded or designed upfront. Especially with physically simulated objects, a lot of sounds are generated by collisions between objects. The sound of such collisions depends on different factors: the material and shape of the objects, as well as the positions, strength and temporal distribution of the collision forces. Especially the latter aspects are difficult to reproduce with sound samples.

To overcome these problems, physically-based sound synthesis is used to interactively generate contact sounds. Collision forces excite vibrations in an object, which are simulated to determine the resulting sound. This approach takes the objects’ geometry, material properties, and impact forces into account, and is thus able to generate interaction-dependent sounds for arbitrary objects without an explicit sound recording step. However, the explicit physical simulation of vibrating objects is computationally expensive and not yet possible in real-time. Thus, modal analysis is used to calculate an object’s characteristic modes (i.e. frequency and decay) in a pre-processing step. At runtime, forces acting on the object excite a position-dependent set of modes. The sound of the whole object is then generated in real-time by accumulating the sound of the individual modes.

  Example of an modal synthesis VR Group Different impact positions on the sound mesh excite a specific set of modes, which are accumulated to generate a collision sound.

While visual feedback is often the main focus in Virtual Environments, sound is another important modality. Auditory feedback can enhance the immersion and believability of virtual scenes. However, the common approach of using pre-recorded sound samples is often not feasible for highly interactive environments, because too many sounds would have to be recorded or designed upfront. Especially with physically simulated objects, a lot of sounds are generated by collisions between objects. The sound of such collisions depends on different factors: the material and shape of the objects, as well as the positions, strength and temporal distribution of the collision forces. Especially the latter aspects are difficult to reproduce with sound samples.

To overcome these problems, physically-based sound synthesis is used to interactively generate contact sounds. Collision forces excite vibrations in an object, which are simulated to determine the resulting sound. This approach takes the objects’ geometry, material properties, and impact forces into account, and is thus able to generate interaction-dependent sounds for arbitrary objects without an explicit sound recording step. However, the explicit physical simulation of vibrating objects is computationally expensive and not yet possible in real-time. Thus, modal analysis is used to calculate an object’s characteristic modes (i.e. frequency and decay) in a pre-processing step. At runtime, forces acting on the object excite a position-dependent set of modes. The sound of the whole object is then generated in real-time by accumulating the sound of the individual modes.

Pre-processing of the modal analysis enables interactive sound synthesis. However, some objects may be generated at runtime (e.g. shards from a breaking window). Furthermore, dynamic damping and coupling effects influence an object’s sound; e.g. a wine glass standing on a table sounds different than one held in the hand. Thus, we are working on methods to calculate the modal analysis interactively. To achieve this, we accelerate the analysis process using parallelization and level-of-detail approaches. Furthermore, we examine the use of prediction to estimate future simulation states, so that more time is available for the analysis computation.

 

Cooperation

  1. Joint work with the RWTH Institute of Technical Acoustics