EXA4MIND: AI-Powered Data Analytics Tools for Supercomputers

The European research project EXA4MIND provides tools and preconfigured workflows that make it easier to collect, manage, and analyse extremely large and diverse datasets.

More than 25 scientific papers and studies over nearly four years, 30 training courses and workshops on artificial intelligence (AI)-based data analysis and processing, as well as an online platform featuring links to useful tools and extensive documentation: this is the track record of the European research project “Extreme Analytics for Mining Data Spaces” (EXA4MIND). Ten universities, research institutes, and companies leveraged the resources of European High Performance Computing (HPC) centres and developed tools and interfaces that use open-source software and libraries to create modular workflows. These solutions enable researchers to access databases, manage research data and results, and analyse big data using AI methods. The project was coordinated by the Czech National Supercomputing Center, IT4Innovations, with support from the Leibniz Supercomputing Centre (LRZ).

“EXA4MIND brings together high-performance and cloud computing, AI and data analytics, as well as professional data management across Europe to unlock datasets that are more diverse and larger than conventional big-data collections,” explains Dr. Stephan Hachinger, Head of Research Data Management at LRZ. “Researchers and small and medium-sized enterprises can find tools on the platform that allow them to automate their own analytics workflows across different computing resources and storage solutions, while professionally indexing, cataloging, and publishing their data according to the FAIR principles.” This makes research data and analytical results discoverable online for both humans and machines.

AI Workflows for HPC Systems

One of the main drivers behind these efforts is the continuous growth in data volumes and the increasing use of AI in science and industry. At the same time, more researchers in companies and universities are working with cloud technologies and require additional expertise to establish their own data analytics workflows while integrating supercomputers into their processes. Today, most high-performance computing systems rely on Graphics Processing Units (GPUs), which support not only traditional simulation workloads but also AI applications such as computer vision and machine learning. While industry often uses databases, object storage systems, and data-streaming technologies to support AI, HPC centres typically focus on adapting file systems, network connections, and input/output libraries. They generally assume that experienced users can efficiently navigate these environments. However, AI is attracting new user groups to HPC centres, many of whom use pre-existing models to analyse research data. As a result, there is growing demand for interoperable tools and standardised workflows that simplify entry into AI-driven supercomputing.

For work at supercomputing centres, EXA4MIND provides guidance on installing data backends such as vector databases and object storage systems, as well as recommendations on when these solutions offer advantages over traditional storage approaches. EXA4MIND’s Advanced Query and Indexing System (AQIS) supports the setup of high-performance analytics pipelines based on the open-source frameworks Airflow and Dask. AQIS also supports AI inference and querying data using natural language. In doing so, it helps make the benefits of AI and HPC accessible to startups and SMEs across Europe. “EXA4MIND opens opportunities for researchers and small companies to explore the services of supercomputing centres and work more efficiently with massive datasets,” says Hachinger. “The modules of the EXA4MIND/EDD platform can be deployed on systems ranging from laptops to HPC clusters, depending on the computational requirements.” Through the interfaces developed by the project partners, European and international data spaces, as well as data from the European Open Science Cloud (EOSC) and EUDAT, can be accessed using EXA4MIND tools. “We explicitly encourage users,” Hachinger adds, “to connect their workflows with these environments in order to search for data and share results.”

Data Analytics Workflows in Practice

To demonstrate the effectiveness of its workflows and their applicability across different domains, EXA4MIND validated them through practical use cases from research and industry:

For molecular dynamics simulations, the researchers established a community data platform. Using IDA4SIMS and ADAMS4SIMS, scientists can compare their simulations with real-world observations, enrich their research data in compliance with FAIR principles, and improve simulation quality. The platform also enables testing of force-field descriptions that determine molecular behavior.
Automotive supplier Valeo uses AI to enhance driver assistance systems. Through EXA4MIND, the company developed processes for enriching and annotating data generated by such systems. Project partners also built a database of driving situations and created tools for analysing traffic scenarios in which assistance systems frequently fail, providing important foundations for more autonomous driving.
Agriculture also benefits from AI, particularly in irrigation and fertilisation management. These technologies can increase yields while conserving water. With Aquaview, a system developed by Terraview/Gamaya, vineyard operators can monitor soil moisture levels and make weather-dependent irrigation decisions. The service is based on satellite data. By applying EXA4MIND tools for caching, data conversion, and filtering, workflows were streamlined and data volumes reduced, saving both processing time and storage capacity.

These applications, analytics tasks, and the tools and technical processes behind them are thoroughly documented and linked on the EXA4MIND platform. Interested users can also access studies, research papers, and workshop materials to learn about the technologies and adapt them for their own needs. In addition, the EXA4MIND team developed workflows specifically for supercomputing centres, enabling their computing power to be used more effectively for AI applications and scaled as required.

EXA4MIND represents an important contribution to Europe’s data-driven research landscape. As with its predecessor project LEXIS, which created a platform for orchestrating geographically distributed HPC and cloud systems, the practical experience gained with workflows and technologies can feed into larger strategic IT initiatives. The EuroHPC Federation Platform, currently being developed by the EuroHPC Joint Undertaking, aims to provide future access to supercomputers, quantum computers, and AI systems. “We are convinced,” concludes Hachinger, “that European supercomputing, particularly within the framework of the EuroHPC Joint Undertaking, will benefit from the concepts developed in EXA4MIND and LEXIS, further expanding the range of applications available at European computing centres.” vs | LRZ

Picture credit: Illustrative image – stock.adobe.com