In this complex context, the instrument of clusters emerges as a very useful tool not only to become effective in the aforementioned activities, but also to develop them in an efficient way.

Graph-Massivizer understood this premise from the beginning, and instead of setting up many new channels with a very limited reach, we created a strategy that would revolve around the principles of collaboration and networking. This has been materialized by the set up of two clusters that offer complementary opportunities to the project.

The first cluster, extremely critical to the success of Graph-Massivizer, has been labelled as DataNexus. It brings together all the Research and Innovation Actions addressing the topic of Extreme data mining, aggregation and analytics technologies and solutions. These projects aim to provide ground-breaking advances in the performance, speed and/or accuracy as well as usefulness of data discovery, collection, mining, filtering and processing when “extreme data” is involved.

Extreme data is defined here as “data that exhibits one or more of the following characteristics, to an extent that makes current technologies fail: increasing volume, speed, variety; complexity/diversity/multilingualism of data; the dispersed data sources; sparse/missing/insufficient data/extreme variations in values”.

According to IDC, the volume of data created each year is forecast to increase at a CAGR of 24.9% from 2024 to 2029 (faster unstructured data). In the case of Data Integration and Intelligence SW, the revenue at worldwide level is projected to nearly double from $6.4B in 2024 to $12.2B in 2029 (11,8% for EMEA), and interestingly enough, the AI Life-Cycle Software showcases CAGRs above 27% 2024-2029 (EMEA from $3B to $11B). Extreme data will influence all these new solutions, and has a huge potential market, as a great percentage of data falls under the former definition of extreme data. A lot of challenges come with those opportunities, such as complexity and integration, rising costs, regulatory and security concerns, talent and ecosystem gaps and ROI and value extraction, to name a few. Addressing these challenges requires a holistic approach and collaboration between those initiatives that focus on the different “pieces” of the big problems. Graph-Massivizer, for example, targets the whole lifecycle of graph-based data.

DataNexus, as a cluster that connects the different projects working on extreme scale data challenges has been instrumental in creating a knowledge base of technologies and developments, allowing project partners to collaborate in common aspects and understanding complementary views of addressing similar problems. In addition, the entire portfolio enables a more complete picture of the challenges that arise when dealing with the computing continuum, the processing of data in different computing infrastructures or issues associated to diverse vertical sectors. Furthermore, extreme data as a research topic has been highlighted by the strength of the cluster, which is more powerful than that of a single project. The following table summarizes key aspects about the positioning of the different projects and diversity of use cases covered, as well as contributions to the cluster.

DataNexus Cluster Overview

The DataNexus cluster has produced a lot of materials that provide more elaborated insights of this work. See links below for reference [1].

EUDATA+ Cluster Overview

The EUDATA+ cluster has produced a lot of materials that provide more elaborated insights of this work. See links below for reference [2].

Conclusion

The set up of the DataNexus and EUDATA+ clusters and activities therein have been instrumental to give visibility to the outcomes generated by Graph-Massivizer to a wide audience. In addition to the increased number of dissemination opportunities (and thus Graph-Massivizer exposure), they have enabled a more clear positioning of our project in a complex ecosystem of projects that work in related fields, allowing us to derive concrete messages to our target audiences and to define a more accurate and finetuned value proposition, both aspects of great importance to foster the adoption of project results, which is one of the ultimate goals of the committed investments.

Author: Nuria de Lama (Consulting Director, IDC)

References

[1] https://www.youtube.com/watch?v=CLBs7Si0MNo;
https://www.youtube.com/watch?v=7kTdhwvELB4&t=139s
https://extract-project.eu/introducing-datanexus/
https://emeralds-horizon.eu/synergies/data-nexus-cluster

[2] https://datamite-horizon.eu/eudata/
Working Groups of the EUDATA+ cluster (link to zenodo)

GraphMa is now public, fully documented, and ready for developers to explore. You can now explore, clone, and contribute to GraphMa on GitHub:

• https://github.com/graph-massivizer/graph-inceptor-graphma

The repository includes:

• Core implementation of GraphMa’s pipeline-oriented graph processing model.
• Benchmark suites for ingestion and traversal performance.
• Examples and starter pipelines to help developers get up and running quickly.

GraphMa is fully open source under the Apache License, Version 2.0.

What Makes GraphMa Different?

GraphMa introduces four key innovations for large-scale graph processing:

1. Pipeline Representation – Centralised construction and execution with type safety, defined as “blueprints” and evaluated lazily, enabling optimisations such as operator switching and resource-aware execution.
2. Operator Model – Provides an extensible catalogue of graph analytics (e.g., centrality, clustering, structural metrics), including stateless transformations, stateful operations, and terminal triggers, supporting both generic and domain-specific logic.
3. Directed Data Transfer – Implements a protocol for clear producer-consumer roles and deterministic message flow across pipeline stages.
4. Higher-Order Traversal – Abstracts traversal mechanics into a unified protocol with modes for fine-grained, bulk, and conditional iteration, supporting scalable processing across diverse graph formats and large datasets.

These innovations enable high throughput and low latency for graph ingestion and traversal.

Getting Started

To start using GraphMa:

1. Clone the repository: git clone https://github.com/graphmassivizer/graph-inceptor-graphma
2. Follow the setup guide in the README.
3. Explore the sample pipelines and benchmarks.

The Graph-Massivizer project reimagines how large-scale graph data is processed by leveraging the computing continuum, a seamless integration of edge, cloud, and high-performance computing (HPC) environments. Unlike traditional siloed infrastructures, the continuum enables cross-layer orchestration, where graph workloads dynamically shift between infrastructure tiers based on real-time latency, energy, and cost trade-offs. This shift is crucial for modern data-intensive applications, where graph-based workloads, ranging from social network analysis to anomaly detection and recommendation systems, demand high throughput, low latency, and adaptive execution environments.

Why Graph Workloads Need the Continuum

Graph analytics pose unique challenges: irregular computation patterns, data dependencies, and unpredictable workloads. A single computation can touch millions of nodes and edges, triggering cascading execution chains. The continuum provides the flexibility needed to manage this complexity:

• Latency-aware execution: Low-latency components like subgraph filtering or anomaly inference run directly on edge nodes (e.g., Jetson, Raspberry Pi), minimizing data transfer.
• Sustainability-driven scheduling: Carbon-intensive operations, such as matrix multiplications or full-graph traversals, are routed to green cloud regions or HPC clusters with monitored energy efficiency.
• Cost-efficient scalability: By adopting serverless computing, resources scale elastically with demand, avoiding idle infrastructure and reducing operational overhead.

Graph-Choreographer: The Serverless Orchestrator of the Continuum

At the heart of this adaptive orchestration lies Graph-Choreographer, the execution backbone of the Graph-Massivizer toolkit. It bridges graph workflows, known as basic graph operations (BGOs), with serverless, Kubernetes-native deployment across heterogeneous nodes.

Key Features

• Declarative to Executable: Transforms high-level BGO workflows into executable DAGs using orchestration services like HEFTLess and EnergyLess.
• Backend Agnosticism: Dynamically selects between Argo Workflows and OpenFaaS, enabling stateful or stateless execution depending on runtime conditions.
• Real-time Adaptation: Monitors energy draw, CO2 intensity, and performance KPIs via Prometheus, Kepler, and PowerJoular, enabling runtime adaptations like reassigning tasks, throttling concurrency, or reprioritizing stages.

Graph Processing Continuum Cycle

The Next Frontier

The computing continuum challenges us to rethink the boundaries of infrastructure, not as isolated layers, but as interconnected zones of opportunity. For graph processing, this means more than just faster analytics; it means smarter deployments, energy-aware execution, and systems that respond to both workload and world conditions. As data grows in scale and complexity, our tools must evolve to be not only scalable and sustainable but also intelligent. Embracing the continuum is not just a technical decision; it is a strategic shift toward architectures that are adaptive by design and responsible by default. By uniting serverless computing, edge intelligence, and green HPC under a common orchestration model, projects like Graph-Massivizer pave the way for a new generation of data-driven systems that are both high-performing and environmentally conscious. The path forward is clear: if our data spans the continuum, so must our computation.

Author:
Dr. Reza Farahani
University of Klagenfurt, Austria

Synthetic financial data generation is emerging as a core capability rather than an experimental add-on. When engineered correctly, synthetic data enables quantitative teams to scale research, stress models beyond observed regimes, train AI systems robustly, and satisfy regulatory and compliance requirements, all without compromising market realism.

This blog, reflecting the work done and concluded in the Graph-Massivizer EU project, outlines how market-consistent synthetic time series can be engineered and generated using a graph-centric approach, and why this methodology represents a step change for quantitative research, trading, and regulatory validation.

Why Real Market Data Is No Longer Sufficient

While historical market data remains indispensable, it exhibits several structural limitations:

Synthetic data, when naively generated, risks compounding these problems. When engineered with market structure awareness, however, it becomes a strategic asset.

Defining “Market-Consistent” Synthetic Financial Data

Market-consistent synthetic data is not defined by point-wise similarity to historical prices, but by preservation of structural, statistical, and relational properties that govern real markets.

Achieving these properties simultaneously requires moving beyond purely parametric models or black-box generative AI.

Graph-Massivizer Approach: A Graph-Centric Paradigm for Synthetic Data Engineering

Graph-Massivizer financial use case was built on the premise that financial markets are naturally relational systems, not independent time series collections. Assets, time steps, market regimes, and derived features form a structured network of dependencies that can be explicitly modeled.

Historical financial data across assets, instruments, and time resolutions is first ingested and transformed into a graph representation:

• Nodes can represent time points, instruments, regimes, or derived states.
• Edges can encode temporal transitions, cross-asset dependencies, and statistical constraints.
• Multi-layer graphs capture interactions across different time scales.

This representation preserves information that is typically lost in flat tabular datasets. Then, before any generation occurs, the source data undergoes structural analysis to define what must be preserved and where variability is allowed. Next, synthetic data is then generated by expanding the graph under explicit constraints such as local randomness, correlations preservation where specific behaviors may be amplified, as well as reduced similarity to the original historic data to the point where reverse engineering is not possible.

The objective is original plausible novelty: data that is statistically consistent yet not traceable to any original observation, given that a critical compliance requirement is that synthetic data must not allow reconstruction of original data. This is particularly relevant for regulatory audits and third-party model validation.

Applications in Quantitative Research and Trading

Performance metrics derived from synthetic data are diagnostic, not predictive, highlighting fragility and structural bias.

Synthetic data is a pre-training and stress-training substrate and not a replacement for real data.

This shifts validation from retrospective justification to proactive resilience testing.

Synthetic data becomes a compliance enabler

There are various challenges as not all synthetic financial data is fit for purpose. Common failure modes include over-fitting synthetic data to historical distributions, ignoring cross-asset and temporal dependencies, excessively smooth or overly random time series or lack of quantitative validation metrics. A graph-centric, constraint-driven approach mitigates these risks by design.

Conclusion

Synthetic financial data generation is no longer an experimental direction. When engineered with structural awareness, statistical rigor, and regulatory foresight, it becomes a core infrastructure capability for modern quantitative organizations.

Graph-Massivizer demonstrates that markets can be expanded, not merely replayed, producing market-consistent time series in extreme volumes that support deeper research, more resilient trading systems, and more credible regulatory validation.

The future of quantitative finance will belong not only to those who analyze history best, but to those who can systematically explore what history did not contain, while following the rules that markets, mathematics and regulators impose.

Laurentiu Vasiliu, founder, Peracton Ltd

19/12/2025

Buy-side quantitative teams face a structural constraint: financial market data remains scarce, fragmented, expensive, and often unsuitable for large-scale AI training. Historical datasets are finite, heavily reused, biased by survivorship and regime persistence, and increasingly subject to restrictive licensing terms. As a result, many AI-driven trading initiatives stall not due to lack of modeling sophistication, but due to insufficient, contaminated, or non-scalable data.

Synthetic financial data is emerging as a strategic solution to this bottleneck, enabling a transition from data scarcity to data scale, while preserving market realism and regulatory relevance.

Why Traditional Market Data No Longer Scales for AI Training

Quantitative trading strategies based on machine learning and deep learning differ fundamentally from traditional statistical or factor-based models. They require large volumes of diverse training data, exposure to multiple market regimes, including rare and extreme events, a clean separation between training, validation, and stress-testing datasets and continuous refresh without historical leakage or overfitting.

Traditional market data is limited on several of these dimensions:

These constraints are structural, not incremental. They cannot be solved by marginally better data sourcing or vendor negotiation.

Synthetic Financial Data: From Approximation to Market-Consistent Engineering

Modern synthetic financial data is not a simplistic resampling or noise-augmented replica of historical prices. When engineered correctly, it represents a market-consistent multiverse of financial time series that preserves statistical properties across time scales, cross-asset and cross-market dependencies, microstructure dynamics (order flow, spreads, volatility clustering), regime transitions and structural breaks This can be achieved through a combination of stochastic and regime-switching models, graph-based dependency modeling and constraint-driven generation aligned with real market invariants

The result is not one synthetic dataset, but thousands—or millions—of plausible market trajectories that extend far beyond what history alone can provide.

Powering AI Training at Scale

Synthetic financial data fundamentally changes how AI models are trained and validated in quantitative trading.

Business Impact for Buy-Side Quantitative Teams

From a business perspective, the adoption of synthetic financial data is less about experimentation and more about competitive positioning. Quant teams can iterate models faster without waiting for new historical data or negotiating incremental licenses. Then, synthetic data decouples AI scaling from data vendor pricing, enabling predictable and controllable cost structures. Further on, exposure to a broader market multiverse improves resilience across regimes, directly impacting drawdown control and long-term performance stability.

Synthetic datasets support explainability, reproducibility, and scenario-based validation that are key concerns for internal model risk committees and external regulators.

What is changing today is not just the quality of synthetic financial data, but its role in the quantitative stack. It is evolving from an augmentation tool into core data infrastructure for AI-driven trading.

Scaling Artificial Intelligence, Not Just Data

Synthetic financial data enables this shift from scarcity to scale, by providing the foundation required for industrial-grade AI training in finance. For buy-side quantitative teams, it represents not only a technical advancement, but a strategic lever: accelerating innovation while improving robustness, compliance readiness, and long-term performance sustainability.

The evolution of quantitative trading will not be determined solely by better models or faster hardware, but by the ability to systematically train AI across diverse, realistic, and unbiased market environments.

In an environment where alpha is increasingly driven by adaptability rather than historical coincidence, synthetic financial data is rapidly becoming a requirement rather than an option.

Laurentiu Vasiliu, founder, Peracton Ltd

26/12/2025

However, powering such large numbers of computers takes a significant and growing amount of energy. The global energy demand of data centres is estimated to reach 8% in 2030 [1]. Unfortunately, burning fossil fuels remains an important and widely used source of energy. This makes data centre construction and operation a significant contributor to greenhouse gas (GHG) emissions.

In this post, we work towards sustainable massive graph-processing workloads by using Graph Greenifier to analyse the effect of temporal shifting (a technique to run workloads with sustainability in mind) on the carbon emission of data centres.

What is Carbon Intensity?

Carbon intensity is a metric that quantifies the (un)sustainability of an energy source by computing the amount of CO2 emitted per unit of energy. The table below presents an overview of the carbon intensity of four highly popular energy sources [1].

Today, in practice, the electricity that is generated from both renewable and non-renewable sources is combined and offered to electricity consumers through an electricity network, also known as the energy grid. This means that electricity consumers on the grid do not use entirely renewable or non-renewable electricity, but rather use whatever combination is put onto the grid by electricity producers.

By knowing how much electricity on an energy grid comes from each source, we can compute the carbon intensity of that grid. To do so, we use the following formula:

Formula for computing grid carbon intensity.

Here, CIg is the carbon intensity of the grid, S is the collection of energy sources used on the grid, CIs is the carbon intensity of one specific energy source, Es is the amount of energy obtained from that energy source, and Eg is the total amount of energy on the grid. In plain English, this formula computes the carbon intensity of each energy source and then computes the weighted sum of those sources.

Knowing the carbon intensity of a grid allows electricity consumers, such as data centres running massive graph-processing workloads, to compute the carbon emissions of their activities by multiplying their electricity use by the carbon intensity of the grid they use to obtain their electricity. Once the carbon emission of a data-centre workload is known, we can start exploring approaches such as temporal shifting to reduce it.

What is Temporal Shifting?

The carbon intensity of grids that obtain energy from renewable sources can change significantly over time because the amount of renewable energy is highly variable and depends on factors such as the time of day and the weather. For example, the image below shows the Dutch energy grid over the course of one month. The top plot shows the amount of available renewable (green) and non-renewable (gray) energy on the grid, and the bottom plot shows the carbon intensity of the grid.

Energy mix and carbon intensity of the grid in the Netherlands during October 2023 [6].

Intuitively, we can reduce GHG emissions by using electricity when its carbon intensity is low. We can implement this approach for graph processing in data centres by delaying the execution of incoming workloads when carbon intensity is high and starting execution when carbon intensity is low. This effectively moves the workload in time, which we call temporal shifting. This idea has been suggested in related scientific work [3, 4, 5], but Graph Greenifier allows us to simulate, and therefore quantify, its effect.

The Graph Greenifier Approach

Graph Greenifier can simulate what happens in data centres during massive (graph-)processing workloads and supports simulating operational techniques such as temporal shifting. Operational techniques are actions data centres take to influence their operation. For example, selecting when and where to schedule a task. This allows data center operators, designers, and researchers to understand the impact of such techniques on both data center performance and sustainability. The figure below shows a simulation result from Graph Greenifier for this scenario.

Simulation results comparing FCFS scheduling and Carbon-Aware scheduling.

Massive graph-processing workloads and other large workloads consist of many small tasks that need to be executed. The top plot in the figure shows the number of actively running tasks for two different scheduling approaches. The blue curve shows a traditional First-Come-First-Served (FCFS) scheduler, which schedules tasks to be executed as soon as possible, and in the order they arrive. The orange curve shows the Carbon-Aware scheduler, which delays the execution of incoming tasks when carbon intensity is high. The green curve in the bottom plot shows the carbon intensity of the grid over time.

We can see that the Carbon-Aware scheduler effectively delays tasks until carbon intensity is low by looking at the orange and green curves. Specifically, we see that the peaks in the orange curve (high number of active tasks) align with the valleys in the green curve (low carbon intensity). For this particular workload, the reduction in carbon emissions is 2.5%, but this can increase depending on the workload and the carbon intensity (variation) of the energy grid to which the data centre is connected.

Next Steps for Graph Greenifier

Temporal shifting is but one technique in a large collection of commonly used operational techniques in data centres. These include spatial shifting, checkpointing, and active-active replication, to name but a few. Additionally, changing the data-centre scheduling policy and other operational techniques can affect not only carbon emissions, but also workload performance and other non-functional properties.

By supporting these techniques in Graph Greenifier, scientists, data centre operators, and other stakeholders can explore “what-if” scenarios and perform a wide range of deep analyses using arbitrary combinations of these techniques to make a trade-off between sustainability and performance for their graph-processing workloads.

References

[1] Anders S. G. Andrae and Tomas Edler. 2015. On Global Electricity Usage of Communication Technology: Trends to 2030. Challenges 6, 1 (2015), 117–157. LINK

[2] Udit Gupta, Mariam Elgamal, Gage Hills, Gu-Yeon Wei, Hsien-Hsin S. Lee, David
Brooks, and Carole-Jean Wu. 2022. ACT: designing sustainable computer systems with an architectural carbon modeling tool. In Proceedings of the 49th Annual
International Symposium on Computer Architecture (New York, New York) (ISCA
’22). Association for Computing Machinery, New York, NY, USA, 784–799. LINK

[3] T. Sukprasert, A. Souza, N. Bashir, D. Irwin, and P. Shenoy, “On the limitations of carbon-aware temporal and spatial workload shifting in the cloud,” in EuroSys, 2024.

[4] Philipp Wiesner, Ilja Behnke, Dominik Scheinert, Kordian Gontarska, and Lauritz Thamsen. 2021. Let’s wait awhile: How temporal workload shifting can reduce carbon  missions in the cloud. In Proceedings of the 22nd International Middleware Conference. 260–272.

[5] Jiechao Gao, Haoyu Wang, and Haiying Shen. 2020. Smartly handling renewable energy instability in supporting a cloud datacenter. In 2020 IEEE international parallel and distributed processing symposium (IPDPS). IEEE, 769–778

[6] D. Niewenhuis, S. Talluri, A. Iosup, and T. De Matteis, “Footprinter: Quantifying data center carbon footprint,” in HotCarbon, 2024.

Authors: Prof.Andrea Bartolini (Associate Professor at University of Bologna) and Junaid Ahmed Khan (PhD student and Research Fellow at the University of Bologna)

Modern data centers and high-performance computing (HPC) systems generate extraordinary volumes of telemetry data. The CINECA Marconi100 supercomputer alone produced approximately 49 terabytes of uncompressed operational data, with individual systems containing up to one million unique sensors sampling at 20-second intervals [1]. This data flows continuously from compute nodes, memory subsystems, power delivery infrastructure, cooling systems, and storage components—creating an unprecedented wealth of operational intelligence. Yet despite this abundance, extracting actionable insights remains remarkably difficult.

The fundamental challenge lies not in data collection but in data access. Operators seeking to understand system behavior must navigate three simultaneous barriers: deep domain expertise in HPC operations, intimate knowledge of the specific monitoring framework architecture, and proficiency in the query languages and APIs of underlying NoSQL databases. This triple requirement effectively restricts analytical capabilities to a small group of specialists, leaving the vast potential of operational telemetry largely untapped.

Researchers at the University of Bologna, in collaboration with CINECA (Italy’s largest supercomputing center), have spent several years addressing this challenge through the EU-funded Graph-Massivizer project. Their work traces an evolution from complex manual queries through structured knowledge representations to an intelligent natural language assistant capable of answering operational questions with over 92% accuracy—transforming how facility managers and engineers interact with their data.

Figure 1: Examon’s massive scale and data heterogeneity

The Marconi100 (M100) system at CINECA utilizes a holistic monitoring framework for operational data analytics called “ExaMon”. It is designed to collect data from various sources, including hardware sensors, software logs, and performance metrics, and stores this data in a NoSQL database (Cassandra, with KairosDB for time-series) in a centralized repository. Figure 1 shows the complexity of the data collected at the M100 system using the ExaMon framework. It integrates data from nine specialized plugins that collect a wide spectrum of information—from air-conditioning and power-distribution data (Vertiv, Schneider, Logics) to node-level sensor telemetry (IPMI), cluster-wide performance metrics (Ganglia, SLURM, Nagios), and external environmental conditions (Weather). Each plugin contributes its own set of metrics and plugin-specific fields, resulting in a highly heterogeneous and multidimensional dataset. This diversity of data sources underscores the complexity of the monitoring environment.

Structured Understanding Through Domain Ontology

The research began by recognizing that the schema-less nature of NoSQL databases—while enabling flexibility and scalability—creates fundamental obstacles for complex analytical queries. Without predefined schemas, users must manually establish connections between different data sources, navigate vendor-specific naming conventions, and construct multi-step query chains that require extensive domain knowledge.

The team’s first innovation was developing a domain-specific ontology for operational data analytics telemetry [2]. Unlike existing data center ontologies that focus primarily on inventory cataloging and infrastructure documentation, this ontology captures the critical relationships between topological components, hardware systems, and job execution data. The representation mirrors how domain experts actually conceptualize HPC operations: racks contain compute nodes positioned in three-dimensional space, plugins organize sensors that produce timestamped readings, and submitted jobs connect directly to the computational resources they utilize. Figure 2 shows the developed ODA ontology.

Figure 2: Operational data analytics (ODA) Ontology

This structured representation enables queries through SPARQL, a graph query language whose logical structure follows the natural relationships in the data. Comparative analysis revealed dramatic simplifications: complex queries requiring dozens of lines of Python code with multiple sub-queries in NoSQL approaches could be expressed in just five to fifteen lines of SPARQL. More significantly, these queries follow paths that non-experts can understand by tracing relationships through the ontology—from rack to node to plugin to sensor to reading—without requiring intimate knowledge of database internals.

The approach proved particularly valuable for queries involving relationships across data sources. Calculating the average power consumption during a specific job’s execution, for instance, traditionally requires querying the job table to identify execution times and allocated nodes, then separately querying sensor tables for power readings during those intervals, and finally correlating the results through manual data manipulation. With the knowledge graph approach, this becomes a single query that traverses the explicit job-to-node-to-sensor-to-reading pathway defined in the ontology.

Confronting the Scalability Challenge

While the ontology approach proved effective for query simplification, a fundamental scalability challenge emerged during implementation. Converting time-series telemetry data into RDF triples resulted in storage requirements approximately 745 times larger than equivalent NoSQL representations. For a single month of data from just one monitoring plugin on the Marconi100 system, this translated to nearly three terabytes of graph storage versus four gigabytes in compressed Parquet format. Such overhead rendered full materialization impractical for production environments.

The solution emerged through virtual knowledge graphs—a technique that constructs graph representations on-demand, containing only the data relevant to answer a specific user query. Rather than materializing the entire dataset as a persistent graph, the system dynamically extracts entities from user questions, fetches relevant data from the NoSQL datalake based on identified time ranges and metrics, constructs a temporary lightweight graph, and executes queries against this focused representation.

Figure 3: Data Analytics (DA) Chatbot

Figure 3 shows the proposed end-to-end architecture for the Data Analytics (DA) chatbot for data centers and HPC [3]. The architecture contains five components: (1) Frontend, built using streamlit library, that provides a chatbot interface to the user, (2) Backend, that is built using FlaskAPI, once it receives a user question, two concurrent processes of SPARQL generation and Virtual Knowledge Graph (VKG) generation is started. SPARQL is generated using the (3) LLM inference service, that takes as input the user input prompt in natural language along with the ODA ontology, to guide the LLM about the logical data model, and a set of few-shot examples as context. Meanwhile, the second concurrent task of VKG takes the user prompt, and using natural language processing tools, such as pattern matching and rule-based critertias to extract the necessary entities from the user text that guides the system of which parameters and sensor data to fetch from the (5) IoT datalake component of the system. Once these entities are extracted, using templated query codes, specific to the IoT datalake, the system fetches the data and then maps it according to the ODA ontology to create the virtual knowledge graph and then this graph is stored inside the (4) Graph database. Inside the graph database, their is already a Base-KG that contains the system’s topological and spatial metadata and combining these with the incoming virtual knowledge graph, the system now has the required data to get the answer to the initial natural language user question. Finally, the generated SPARQL query is executed on the SPARQL endpoint of the (4) Graph Database, and the corresponding answer is then returned back to the user on the (1) Frontend, as a table, or in case of a large table with more than 50 rows, as a CSV downloadable link.

This approach maintained the semantic advantages of knowledge graphs while reducing storage overhead to manageable levels. The maximum observed virtual knowledge graph size across evaluated queries was just 179 megabytes—trivial compared to multi-terabyte full materializations. The team further optimized the generation pipeline through several technical refinements: adopting the Polars library for data processing (achieving 92.75% faster data fetching than the initial Pandas implementation), selecting N-Triples serialization format for its speed advantages, and implementing batch triple generation rather than incremental graph construction.

Natural Language Meets Graph Intelligence

The culmination of this research trajectory is EXASAGE—the first operational data analysis assistant for HPC systems [4]. The figure 4 shows the architecture diagram of EXASAGE framework. EXASAGE combines the structured power of knowledge graphs with large language models to create a natural language interface for telemetry data, enabling operators to ask questions in ordinary language and receive accurate, contextually appropriate responses.

The system architecture reflects careful integration of complementary technologies. An input validator extracts key entities from natural language questions—identifying nodes, racks, jobs, metrics, and time ranges through rule-based algorithms aligned with the domain ontology. The LLM-query generator translates these validated questions into SPARQL queries, grounded by the ontology schema and few-shot examples that demonstrate correct query patterns. Simultaneously, the virtual knowledge graph generator constructs a query-specific graph containing precisely the data needed to answer the question. A query refinement stage then corrects common LLM-generated syntax errors through pattern matching before execution.

Figure 4: EXASAGE: The first data center operational data analysis assistant, shown as a block diagram.

Evaluation across one thousand queries demonstrated remarkable effectiveness. The system achieved 93.6% accuracy in generating correct SPARQL queries and retrieving accurate answers—compared to just 25% accuracy when large language models attempted to generate equivalent NoSQL queries directly. This dramatic difference stems from the knowledge graph’s explicit encoding of relationships between data sources: information that NoSQL approaches require users to establish manually through multiple coordinated queries.

The knowledge graph approach also proved more efficient in practice. SPARQL queries generated by the system were significantly more concise, with 56.89% fewer output tokens on average, and executed faster despite the additional virtual knowledge graph construction step. Optimization efforts reduced complete end-to-end latency from 20.36 seconds to just 3.03 seconds—an 85% improvement that makes the system practical for interactive analytical sessions rather than batch processing alone [3].

Toward Universal Interoperability

The most recent advancement extends this work toward cross-system analytics through a unified ontology designed to support multiple heterogeneous HPC environments [5]. The figure 5 shows the developed ontology, comprising 164 axioms: 104 logical axioms defining semantics (e.g., domains, ranges, characteristics, inverses) and 60 declarations introducing named entities. The ontology defines 12 classes representing core HPC concepts—jobs, compute nodes, racks, sensors, users—and includes 23 object properties for inter-class relations (e.g., job-tonode, rack-to-node) and 25 data properties linking individuals to literals (e.g., timestamps, metrics). Designed for interoperability across heterogeneous HPC systems, the schema supports multiple facilities via DataCenter and HPCSystem, captures workloads through User, Job, and JobMetric, and models infrastructure layout with Rack, ComputeNode, and Position. Monitoring data is represented via Sensor and SensorReading, while temporal dynamics are abstracted with the Time class, enabling representation of time-dependent events and relationships. The Plugin class models software components involved in monitoring and analysis. The research team validated this generalized schema on the two largest publicly available operational datasets from top-ranked supercomputers: the Marconi100 dataset [1] from CINECA in Italy and F-DATA [6] from the Fugaku supercomputer at RIKEN in Japan.

Figure 5: Unified ODA Ontology

This unification required addressing fundamental differences in how facilities conceptualize and record operational data. While the Marconi100 telemetry follows a sensor-centric model where time-series readings attach to physical monitoring devices, Fugaku’s F-DATA dataset is fundamentally job-centric—recording performance metrics per job rather than per sensor. The unified ontology accommodates both paradigms through new classes that capture data center and HPC system hierarchies, user-specific workload patterns, and job-level metrics alongside traditional sensor readings.

The team also addressed storage efficiency through ontology design optimizations. By eliminating redundant classes, centralizing timestamp representations through a dedicated Time class with Unix encoding, and relocating unit specifications from individual readings to parent sensor definitions, the unified ontology achieves 38.84% storage reduction compared to the previous approach. Additional deployment configurations using blank nodes for sensor readings provide a further 26.82% reduction when global addressability of individual readings is not required.

The unified ontology was validated against 36 competency questions spanning system topology, sensor monitoring, job execution analysis, user activity patterns, scheduling efficiency, and cross-system comparative analytics. Questions that would be essentially impossible with traditional approaches—such as comparing average job execution times across different HPC systems or identifying which facility achieves better energy efficiency per job—become answerable through standardized SPARQL queries operating over the unified schema.

Implications for Operational Intelligence

This research progression—from complex NoSQL queries through unified ontologies and virtual knowledge graphs to intelligent natural language assistants—represents a fundamental shift in how data center operators can interact with telemetry data. The implications extend beyond high-performance computing to any IoT environment generating heterogeneous time-series data at scale.

The core insight is that knowledge graphs, when combined with modern language model capabilities and careful architectural optimization, can bridge the gap between massive operational datasets and actionable insights. By encoding domain semantics explicitly and leveraging the reasoning capabilities of language models, these systems enable facility managers, system administrators, and engineers to query telemetry data using questions they would naturally ask—rather than requiring mastery of database internals and query languages.

The accuracy improvements are particularly striking. A 92-93% query accuracy rate versus 25% for direct LLM-to-NoSQL approaches reflects more than incremental optimization; it demonstrates that providing structured semantic context transforms language models from unreliable query generators into effective analytical partners. The knowledge graph supplies precisely the relational information that large language models struggle to infer from unstructured database schemas.

As data centers grow in complexity to meet AI-driven computational demands, these graph-powered approaches offer a scalable path toward intelligent, accessible operational analytics. The research demonstrates that the barrier between operational data and operational insight need not remain as high as current practices suggest—and that thoughtful integration of semantic technologies with modern AI can democratize access to the intelligence hidden within facility telemetry.

Conclusion

In conclusion, thanks to the Graph-Massivizer project, CINECA and UNIBO have demonstrated that graph-based data representation is a key enabler for future autonomous and sustainable data centres: democratizing data and trasforming them from a storage overhead into actionalble insight. This research will be a valuable asset for the future development of data centres, enabling simpler, more effective, and more intelligent operational optimization.

References

[1] Borghesi, Andrea, et al. “M100 Exadata: a data collection campaign on the CINECA’s Marconi100 Tier-0 supercomputer.” Scientific Data 10.1 (2023): 288.

[2] Junaid Ahmed Khan, Martin Molan, Matteo Angelinelli, and Andrea Bartolini. 2024. ExaQuery: Proving Data Structure to Unstructured Telemetry Data in Large-Scale HPC. In Companion of the 15th ACM/SPEC International Conference on Performance Engineering (ICPE ’24 Companion). Association for Computing Machinery, New York, NY, USA, 127–134. https://doi.org/10.1145/3629527.3652898

[3] Junaid Ahmed Khan, Hiari Pizzini Cavagna, Andrea Proia, Andrea Bartolini. “From Data Center IoT Telemetry to Data Analytics Chatbots — Virtual Knowledge Graph is All You Need.” arXiv preprint, https://doi.org/10.48550/arXiv.2506.22267

[4] Khan, Junaid Ahmed, Martin Molan, and Andrea Bartolini. “EXASAGE: The First Data Center Operational Data Analysis Assistant.” Future Generation Computer Systems (2025): 108185.

[5] Khan, Junaid Ahmed, and Andrea Bartolini. “A Unified Ontology for Scalable Knowledge Graph-Driven Operational Data Analytics in High-Performance Computing Systems.” arXiv preprint arXiv:2507.06107 (2025)

[6] Antici, F., Bartolini, A., Domke, J. et al. F-DATA: A Fugaku Workload Dataset for Job-centric Predictive Modelling in HPC Systems. Sci Data 12, 1321 (2025). https://doi.org/10.1038/s41597-025-05633-1

The Challenge: Extracting Insights from Massive Knowledge Graphs

Knowledge graphs are a useful tool to model information that represents complex relationships in your data. Typically, with knowledge graphs, there are standard symbolic approaches and algorithms that work well to address your needs for useful data analytics.

When knowledge graphs become massive, however, and the relationships and semantics in your data become more intricate, it is often necessary to re-examine the traditional approaches to find strategies that can be tailored for massive graphs. This need can be due to memory issues, where a graph is too big to store on a normal computer or even a server, or too big for algorithms to handle when they need to look at lots of data at the same time.

In industry and academia, there are very large graphs that represent publications, authors, domains and the relationships between them.

In this article, we will investigate an example of such a massive research knowledge graph and show how AI approaches can be leveraged to address the unique challenges posed when developing algorithms to utilize a massive knowledge graph.

Massive Knowledge Graphs: what are they?

Massive knowledge graphs are knowledge graphs (KGs) that are, for some reason or another, much too large to manage with strategies that are typically effective for smaller graphs. Often, a knowledge graph is associated with an ontology, which schematizes the intended structure of the data in the knowledge graph in a formal way that is understandable by a computer or a human. When we say too large, this can mean a few different things, such as being simply too big to fit on a standard computer or too massively complex to process with typical algorithms, to give a few examples.

Scalable analytics for a massive research knowledge graph: exploring the SemOpenAlex use case

Now let’s look at an existing use case that combines AI with massive knowledge graphs.

Researchers and engineers at metaphacts are contributing to an EU project called Graph-Massivizer, where scalable solutions are being developed to process and extract insights from massive graphs with the Graph-Massivizer Toolkit. Composed of 12 partners from eight EU countries, the Graph-Massivizer project brings together the world-leading roles of European researchers in graph processing and serverless computing and uses leadership-class European infrastructure in the computing continuum. This project develops optimizations and AI/ML tools connected with the use cases described here, and motivates the bigger-picture topics we’ve discussed.

One of the use cases for this project is a development and integration scenario where test cases are developed for the massive SemOpenAlex knowledge graph of open academic publication data. The SemOpenAlex knowledge graph is challenging to design scalable algorithms for graph analytics due to its sheer size, so the project aims to use it as a test case to demonstrate the functionality under development. The use case features multiple interesting applications of these techniques for automated workflow execution and agentic AI that will be shown in the following sections.

An example of this use case examines a scholarly research collaboration network, answering the following query: ”How do I find the shortest path from myself to the most popular researcher in my field?”. The academic knowledge graph use case using SemOpenAlex that shows co-authorship relations for publications, attempting to find connections between influential researchers in the same field by tracing co-authorship relations. The knowledge graph includes additional metadata that can be leveraged by learning algorithms, such as publication information, including titles, authors, venues, and authors’ research fields. The use case defines “popular researcher” based on a metric considering extensive publications and co-authors.

Fig. 1: The workflow diagram for the SemOpenAlex use case.

A workflow (as shown in the above diagram) for the use case is schematized in the user interface above with an ontology that is discussed in the next section. Tasks in the workflow include algorithms such as loading a subgraph using a SPARQL query or computing the Betweenness Centrality values for nodes in the graph, to give some examples. Using the metaphactory interface allows a user to select predefined algorithms based on available hardware and programming languages to execute their workflow. Because the workflow is expressed in RDF, it can be understood by a human as well as the computer responsible for executing it. The workflow contains pointers to data and functions that it describes, provided by the technical user who sets it up; a program or agent can execute it automatically with minimal input.

Executable workflows for graph algorithms

Workflows can be represented explicitly in RDF according to an ontology that also connects to a data source. This allows an agent, whether human or AI, to interact with and execute user-specified workflows dynamically while a system is given access to the required algorithms. If the workflow also specifies data flow, it can be used to execute an entire pipeline of different functions, sending input from one function to another automatically and showing the user the end result. This is advantageous when integrating knowledge graphs with AI since it abstracts away the information that is already known about how to use algorithms, which allows a developer or user to customize the execution of a workflow without always giving it lots of very specific instructions on what to do.

Fig. 2: The graph-processing workflow ontology.

The workflows used in the SemOpenAlex use case are structured according to the ontology shown above. At the top, you can see the workflow class, which must contain at most one first task, which itself can have any number of next tasks. This can model a sequential or even tree-like structure based on user needs, where the first task is the root of the tree and all next tasks branch from there. This tree-like structure is intended to correspond roughly to the directed acyclic graph (DAG) that is required by the toolkit to define its workflows, ensuring that operations are well defined and not running in an infinite loop. When an end-user interacts with the system, all they are required to do is specify the workflow of tasks and associate them with available algorithms preloaded according to the other parts of the ontology.

For a developer who sets up the system, they specify available algorithms using the class BGO, or basic graph operation. This class represents a type of graph algorithm, such as breadth-first search. Each of these abstract BGOs is then connected with at least one implemented algorithm. It is possible to include multiple implementations of the same algorithm in various languages or with different inputs and outputs so that different execution pathways and scenarios can be made available to support diverse use cases. The system is then able to dynamically choose the algorithm and runtime environment best suited for execution of the task.

The motivation behind modeling and developing these workflows is that they enable an expert or team of experts to configure the Graph-Massivizer Toolkit in advance with algorithms and workflows that they wish to make available to an end user. The end user can then utilize these advanced AI techniques via a simple interface in metaphactory that builds and executes workflows correctly without needing to already have a deep technical understanding of AI techniques. Next, we will see an example of how the same type of approach can be integrated with a neuro-symbolic system that has its own unique advantages.

Analytics and Agentic AI with metis

Graph analytics and Agentic AI can be combined with neuro-symbolic approaches to achieve unique advantages. One such system, metis, is a knowledge-driven AI platform combining large language models and knowledge graphs to deliver AI agents that provide generative power, semantic precision and contextual, explainable insights.

metis, which sits on top of metaphactory, is capable of exploring existing data in a knowledge graph and is also able to guide users through the modeling process when they design their ontology, or can be customized or selected from an agent repository. The AI component aids a user in their interaction with the knowledge graph in a conversational interface by translating natural language into machine-readable actions and instructions. We will see how this tool can be integrated with the use case discussed previously so that a user is able to perform analytics using a simple conversational interface.

Because metis allows interaction with a knowledge graph, users can utilize AI tools to improve productivity without sacrificing the essential trust that comes with using a well-designed ontology to manage their data.

Algorithms for graph analytics are supported as extensions to metaphactory applications and can also be integrated with metis. This makes the adoption of analytics features seamless with the platform usage in general. An app can act as a tool for metis, allowing it to answer complex questions over the knowledge graph that require advanced machine learning algorithms.

Fig. 3: A sample conversation in metis executing the SemOpenAlex usecase.

In the image above, you can see a sample conversation where metis is used to execute the same workflow as we saw in the previous section. Using only natural language, a user can request complex workflows such as finding the shortest path between an instance and the instance with the maximum centrality in a graph. In order to answer this question, the agent not only needs to execute the algorithms but also must understand how to generate workflows in case multiple operations are requested in sequence. And with this functionality, users don’t need to compose workflows themselves since they can simply ask the agent using natural language.

Conclusion

We have looked at various methods for using AI and knowledge graphs together in complementary ways. The SemOpenAlex use case from the Graph-Massivizer project featured multiple interesting applications of these techniques for automated workflow execution and agentic AI. There are many new, exciting possibilities being developed every day to support AI and large knowledge graphs at metaphacts in this as well as other projects and domains, so look forward to more updates from our team about new applications for AI with your knowledge graphs.

Explore solutions

After seeing some examples of how knowledge graphs and AI can work together, you may already have an idea for how these technologies can benefit your organization.

Speak with an expert to discuss your organization and specific use case, and learn how metaphactory can support your knowledge graph and AI use case. You can also request a demo of metis to see it in action

Artificial intelligence is reshaping modern society, but this technological revolution comes at an environmental cost that we can no longer ignore. Although AI offers innovative ways to tackle climate change, it also contributes to carbon emissions and the consumption of resources.

The environmental impact of AI is staggering. Research published in Nature Sustainability reveals that implementing AI servers in the United States could generate between 24 and 44 million metric tons of CO₂ equivalent emissions annually by 2030 — comparable to adding 5 to 10 million cars to American roads. The United Nations emphasises that data centres hosting AI servers produce electronic waste, consume vast amounts of water, and use huge quantities of electricity, thereby fuelling greenhouse gas emissions. Golestan Radwan, UNEP Chief Digital Officer, states that we must ensure the net effect of AI on the planet is positive before it is implemented on a large scale. AI is what researchers call “double-edged technology”.

Supercomputers and AI models have their own carbon footprint, which varies depending on the type of AI and the training methods used. However, AI can also play a key role in reducing emissions through climate change modelling and smart grid design. Microsoft Research has demonstrated that AI-based systems can integrate renewable energy more effectively into stable electrical grids and reduce carbon capture costs by accelerating the discovery of new materials.

In this critical context, Graph Massivizer shows how innovation and sustainability can go hand in hand. The project aims to improve the efficiency of data analysis and reduce the energy impact of extract, transform, and load operations. It seeks to enhance data centre energy efficiency by a factor of two and reduce greenhouse gas emissions associated with graph-organised database operations.

The Data Centre Digital Twin use case, involving CINECA and the University of Bologna, exemplifies this vision by creating a digital graph representation of CINECA’s supercomputers. This representation allows system operations to be studied and understood and enables efficiency and sustainability to be optimised for the next generation of exascale supercomputers.

Recent research from the University of Bologna and CINECA showcases AI designed to support sustainable development. The ExaQuery project proposes an innovative ontology for operational data in HPC systems that organises and queries telemetry data more efficiently, reducing computational load. This knowledge graph-based approach facilitates the identification of complex relationships between hardware components, computational jobs and performance metrics, paving the way for intelligent resource optimisation.

Building on this foundation, researchers have developed a Virtual Knowledge Graph system that provides natural language access to heterogeneous IoT data in data centres. Combining Large Language Models with Knowledge Graphs achieves 92.5% query accuracy for this system, compared to 25% for traditional LLM-to-NoSQL approaches, while reducing latency by 85%. This breakthrough demonstrates that intelligent data organisation can dramatically improve accessibility and efficiency when managing complex telemetry systems.

As Graph Massivizer highlights, although data analysis and processing have a significant environmental impact, they can also be invaluable tools for achieving environmental sustainability. The key lies in using technology responsibly by focusing on efficiency, renewable energy and intelligent computational architectures. The message is clear: AI can and must be part of the solution to the climate crisis, not the problem. Projects like Graph Massivizer show that technology’s future can be powerful and sustainable if we make the right choices today.

The Essence of GraphMa

At its core, GraphMa is a conceptual framework that seamlessly merges the principles of pipeline computation with the intricacies of graph processing. It introduces a series of powerful abstractions that empower developers to decompose complex graph operations into modular, composable functions. These functions can then be orchestrated into streamlined pipelines, facilitating the systematic development and execution of graph algorithms.

The Building Blocks of GraphMa

• Computation as Type: This foundational abstraction elevates computation units to first-class entities, encapsulating them within a well-defined interface. This approach ensures type safety and promotes modularity, enabling the creation of reusable and composable pipeline stages.
• Higher-Order Traversal Abstraction: This abstraction provides a versatile mechanism for navigating and accessing data within graphs. It defines methods for traversing various data sources, empowering developers to manipulate and process graph data with flexibility and efficiency.
• Directed Data-Transfer Protocol: This protocol governs the seamless and efficient transfer of data between computational stages. It adheres to functional programming principles, ensuring clear directionality and optimized data flow throughout the pipeline.
• Operator Model: This model introduces a comprehensive set of constructs for managing the lifecycle and states of operators within the pipeline. It facilitates a wide array of data processing operations, from transformations to aggregations, enabling the construction of sophisticated graph algorithms.
• Pipeline Abstraction: This abstraction serves as the overarching framework that orchestrates the entire graph processing workflow. It encapsulates the complexities of data transformation and transmission, providing a high-level blueprint for defining and executing graph processing pipelines.

Embracing Established Computational Models

GraphMa’s versatility shines through its ability to seamlessly integrate well-established computational models for graph processing. Whether it’s the vertex-centric model, where computations are centered around individual nodes, or the edge-centric model, which focuses on the relationships between nodes, GraphMa provides a flexible platform for implementing and executing these models within its pipeline-oriented architecture.

The Promise of GraphMa

GraphMa represents a significant leap forward in the field of graph processing. By combining the power of pipeline computation with graph-specific abstractions, it offers a structured and modular approach to tackling the challenges of graph data analysis. Its potential to enhance scalability, efficiency, and expressiveness in graph processing tasks positions it as a valuable tool for researchers and practitioners navigating the complexities of interconnected data. As GraphMa continues to evolve, we can anticipate its widespread adoption and its transformative impact on the way we understand and leverage the power of graphs in the digital age.

References

Schroeder, Daniel Thilo,    Tobias Herb,    Brian Elvesæter, and Dumitru Roman “GraphMa: Towards new Models for Pipeline-Oriented Computation on Graphs.” In Companion of the 15th ACM/SPEC International Conference on Performance Engineering, pp. 98-105. 2024.