Spectrogram and other time-frequency analysis methods transfer an audio file into an auditory image. When signal processing-based analysis and interpretation is performed on these auditory images instead of an audio signal, spectrographic analyses can identify interesting patterns that focus on very different aspects of the signal compared to an audio-based analysis. To facilitate an auditory image-based study, a quantitative analysis and interpretation framework is implemented for exploring the spectrographic images in multiple time and frequency scales and for automatically identifying image features that are relevant to human auditory perception. This analysis framework is applied to two social media datasets: (1) soundtracks from video commercials and “hit” music excerpts from social media platforms, and (2) soundtracks from television and film. Analysis results from social media are also compared with audience subjective evaluations to validate the perceptual relevance of the identified spectrographic patterns.

Nguyen Le Thanh Nguyen, Hyunhwan Lee, Joseph Johnson, Mitsunori Ogihara, Gang Ren, James W. Beauchamp. A new auditory image for social media: Moving towards correlation of spectrographic analysis and interpretation with audience perception. The Journal of the Acoustical Society of America October 2019 146(4):2846  https://asa.scitation.org/doi/abs/10.1121/1.5136876