Peak Spark ODS Adapter: Gain more information from the existing measurement channels in ASAM ODS

A widely used standard for the persistent storage of test data is ASAM ODS. Automotive manufacturers and suppliers use it primarily to set up overarching test data management solutions for different measurement systems and data formats. In this way, they make test data usable in the long-term, regardless of the tools of individual vendors. To gain more information from the existing measurements, Peak Solution has developed the Peak Spark ODS Adapter.

The solution allows the user to directly access the mass data of ASAM ODS (= external components). After selecting a set of measurement channels, e.g. via the Peak Test Data Workplace, he can execute arbitrarily complex queries on it. The solution returns the query result to the user as a list. He can either cache this list or export it as a csv file, e.g. in order to apply algorithms and statistical methods for detecting patterns, relationships, and clusters.

The implementation of the adapter is based on Apache Spark. This  makes it possible to perform complex data queries and analyzes on large amounts of data with high speed. Peak Solution has developed a special plug-in which provides the ODS data structure of the measurement channels as so-called RDDs (Resilient Distributed Datasets). Out of this the application then creates temporary tables (= data frames) on which the mentioned algorithms and methods can be user-friendly executed.

The user can use web-based notebooks like Apache Zeppelin or Jupyter for the interactive analysis of the measurement channels. The great strength of this tools are explorative evaluations (= data mining). This allows the user first to gain an overview of the data and then approach his analysis goal gradually. He can easily visualize and share intermediate results. In order to do this, the user only has to release his "notebook" as a web page. Authorized persons can then view and continue to use it.

An important advantage of the Peak Spark ODS Adapter is its scalability: neither Spark nor Zeppelin require a Hadoop Eco-System. As a result, companies can leverage the solution within their existing ODS infrastructure without extensive IT investments before gradually building it into a powerful big data cluster for managing and analyzing huge amounts of test data. According to the motto: Start small, think tall.