Modern oscilloscopes, digitizers and data loggers generate a large amount of waveform data for long-duration waveform capturing and analysis. The contrast of time scales of long-duration waveform capturing (e.g., hours or days in high sampling rate) and analysis (e.g., signal fragments of several microseconds) produces unique big data challenges. The proposed long-duration waveform clustering algorithms are designed for signal waveform analysis and user interaction for various “big-data” waveform analysis scenarios. To cope with the real-time processing demand and the hardware constraints of the target platforms, the proposed algorithm utilizes multiple layers of data pre-sorting, database query, and waveform similarity-based clustering for versatile speed-precision tradeoffs. We integrated the system as an intuitive big waveform data analytics framework which provides unprecedented performance and productivity to engineers and scientists. Experimental result shows superb speed and data volume capability.

This paper was presented at the 2019 IEEE International Conference on BIG DATA, Los Angeles, CA, Dec. 9-12, 2019. See Program Schedule .pdf (page 34 / may be slow to load).

M. Goto, N. Kobayashi, G. Ren and M. Ogihara, Scaling Up Heterogeneous Waveform Clustering for Long-Duration Monitoring Signal Acquisition, Analysis, and Interaction: Bridging Big Data Analytics with Measurement Instrument Usage Pattern, 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, 2019, pp. 1794-1803, doi: 10.1109/BigData47090.2019.9006208.  https://ieeexplore.ieee.org/document/9006208