In this complex context, the instrument of clusters emerges as a very useful tool not only to become effective in the aforementioned activities, but also to develop them in an efficient way.

Graph-Massivizer understood this premise from the beginning, and instead of setting up many new channels with a very limited reach, we created a strategy that would revolve around the principles of collaboration and networking. This has been materialized by the set up of two clusters that offer complementary opportunities to the project.

The first cluster, extremely critical to the success of Graph-Massivizer, has been labelled as DataNexus. It brings together all the Research and Innovation Actions addressing the topic of Extreme data mining, aggregation and analytics technologies and solutions. These projects aim to provide ground-breaking advances in the performance, speed and/or accuracy as well as usefulness of data discovery, collection, mining, filtering and processing when “extreme data” is involved.

Extreme data is defined here as “data that exhibits one or more of the following characteristics, to an extent that makes current technologies fail: increasing volume, speed, variety; complexity/diversity/multilingualism of data; the dispersed data sources; sparse/missing/insufficient data/extreme variations in values”.

According to IDC, the volume of data created each year is forecast to increase at a CAGR of 24.9% from 2024 to 2029 (faster unstructured data). In the case of Data Integration and Intelligence SW, the revenue at worldwide level is projected to nearly double from $6.4B in 2024 to $12.2B in 2029 (11,8% for EMEA), and interestingly enough, the AI Life-Cycle Software showcases CAGRs above 27% 2024-2029 (EMEA from $3B to $11B). Extreme data will influence all these new solutions, and has a huge potential market, as a great percentage of data falls under the former definition of extreme data. A lot of challenges come with those opportunities, such as complexity and integration, rising costs, regulatory and security concerns, talent and ecosystem gaps and ROI and value extraction, to name a few. Addressing these challenges requires a holistic approach and collaboration between those initiatives that focus on the different “pieces” of the big problems. Graph-Massivizer, for example, targets the whole lifecycle of graph-based data.

DataNexus, as a cluster that connects the different projects working on extreme scale data challenges has been instrumental in creating a knowledge base of technologies and developments, allowing project partners to collaborate in common aspects and understanding complementary views of addressing similar problems. In addition, the entire portfolio enables a more complete picture of the challenges that arise when dealing with the computing continuum, the processing of data in different computing infrastructures or issues associated to diverse vertical sectors. Furthermore, extreme data as a research topic has been highlighted by the strength of the cluster, which is more powerful than that of a single project. The following table summarizes key aspects about the positioning of the different projects and diversity of use cases covered, as well as contributions to the cluster.

DataNexus Cluster Overview

The DataNexus cluster has produced a lot of materials that provide more elaborated insights of this work. See links below for reference [1].

EUDATA+ Cluster Overview

The EUDATA+ cluster has produced a lot of materials that provide more elaborated insights of this work. See links below for reference [2].

Conclusion

The set up of the DataNexus and EUDATA+ clusters and activities therein have been instrumental to give visibility to the outcomes generated by Graph-Massivizer to a wide audience. In addition to the increased number of dissemination opportunities (and thus Graph-Massivizer exposure), they have enabled a more clear positioning of our project in a complex ecosystem of projects that work in related fields, allowing us to derive concrete messages to our target audiences and to define a more accurate and finetuned value proposition, both aspects of great importance to foster the adoption of project results, which is one of the ultimate goals of the committed investments.

Author: Nuria de Lama (Consulting Director, IDC)

References

[1] https://www.youtube.com/watch?v=CLBs7Si0MNo;
https://www.youtube.com/watch?v=7kTdhwvELB4&t=139s
https://extract-project.eu/introducing-datanexus/
https://emeralds-horizon.eu/synergies/data-nexus-cluster

[2] https://datamite-horizon.eu/eudata/
Working Groups of the EUDATA+ cluster (link to zenodo)

GraphMa is now public, fully documented, and ready for developers to explore. You can now explore, clone, and contribute to GraphMa on GitHub:

• https://github.com/graph-massivizer/graph-inceptor-graphma

The repository includes:

• Core implementation of GraphMa’s pipeline-oriented graph processing model.
• Benchmark suites for ingestion and traversal performance.
• Examples and starter pipelines to help developers get up and running quickly.

GraphMa is fully open source under the Apache License, Version 2.0.

What Makes GraphMa Different?

GraphMa introduces four key innovations for large-scale graph processing:

1. Pipeline Representation – Centralised construction and execution with type safety, defined as “blueprints” and evaluated lazily, enabling optimisations such as operator switching and resource-aware execution.
2. Operator Model – Provides an extensible catalogue of graph analytics (e.g., centrality, clustering, structural metrics), including stateless transformations, stateful operations, and terminal triggers, supporting both generic and domain-specific logic.
3. Directed Data Transfer – Implements a protocol for clear producer-consumer roles and deterministic message flow across pipeline stages.
4. Higher-Order Traversal – Abstracts traversal mechanics into a unified protocol with modes for fine-grained, bulk, and conditional iteration, supporting scalable processing across diverse graph formats and large datasets.

These innovations enable high throughput and low latency for graph ingestion and traversal.

Getting Started

To start using GraphMa:

1. Clone the repository: git clone https://github.com/graphmassivizer/graph-inceptor-graphma
2. Follow the setup guide in the README.
3. Explore the sample pipelines and benchmarks.

The Graph-Massivizer project reimagines how large-scale graph data is processed by leveraging the computing continuum, a seamless integration of edge, cloud, and high-performance computing (HPC) environments. Unlike traditional siloed infrastructures, the continuum enables cross-layer orchestration, where graph workloads dynamically shift between infrastructure tiers based on real-time latency, energy, and cost trade-offs. This shift is crucial for modern data-intensive applications, where graph-based workloads, ranging from social network analysis to anomaly detection and recommendation systems, demand high throughput, low latency, and adaptive execution environments.

Why Graph Workloads Need the Continuum

Graph analytics pose unique challenges: irregular computation patterns, data dependencies, and unpredictable workloads. A single computation can touch millions of nodes and edges, triggering cascading execution chains. The continuum provides the flexibility needed to manage this complexity:

• Latency-aware execution: Low-latency components like subgraph filtering or anomaly inference run directly on edge nodes (e.g., Jetson, Raspberry Pi), minimizing data transfer.
• Sustainability-driven scheduling: Carbon-intensive operations, such as matrix multiplications or full-graph traversals, are routed to green cloud regions or HPC clusters with monitored energy efficiency.
• Cost-efficient scalability: By adopting serverless computing, resources scale elastically with demand, avoiding idle infrastructure and reducing operational overhead.

Graph-Choreographer: The Serverless Orchestrator of the Continuum

At the heart of this adaptive orchestration lies Graph-Choreographer, the execution backbone of the Graph-Massivizer toolkit. It bridges graph workflows, known as basic graph operations (BGOs), with serverless, Kubernetes-native deployment across heterogeneous nodes.

Key Features

• Declarative to Executable: Transforms high-level BGO workflows into executable DAGs using orchestration services like HEFTLess and EnergyLess.
• Backend Agnosticism: Dynamically selects between Argo Workflows and OpenFaaS, enabling stateful or stateless execution depending on runtime conditions.
• Real-time Adaptation: Monitors energy draw, CO2 intensity, and performance KPIs via Prometheus, Kepler, and PowerJoular, enabling runtime adaptations like reassigning tasks, throttling concurrency, or reprioritizing stages.

Graph Processing Continuum Cycle

The Next Frontier

The computing continuum challenges us to rethink the boundaries of infrastructure, not as isolated layers, but as interconnected zones of opportunity. For graph processing, this means more than just faster analytics; it means smarter deployments, energy-aware execution, and systems that respond to both workload and world conditions. As data grows in scale and complexity, our tools must evolve to be not only scalable and sustainable but also intelligent. Embracing the continuum is not just a technical decision; it is a strategic shift toward architectures that are adaptive by design and responsible by default. By uniting serverless computing, edge intelligence, and green HPC under a common orchestration model, projects like Graph-Massivizer pave the way for a new generation of data-driven systems that are both high-performing and environmentally conscious. The path forward is clear: if our data spans the continuum, so must our computation.

Author:
Dr. Reza Farahani
University of Klagenfurt, Austria

Synthetic financial data generation is emerging as a core capability rather than an experimental add-on. When engineered correctly, synthetic data enables quantitative teams to scale research, stress models beyond observed regimes, train AI systems robustly, and satisfy regulatory and compliance requirements, all without compromising market realism.

This blog, reflecting the work done and concluded in the Graph-Massivizer EU project, outlines how market-consistent synthetic time series can be engineered and generated using a graph-centric approach, and why this methodology represents a step change for quantitative research, trading, and regulatory validation.

Why Real Market Data Is No Longer Sufficient

While historical market data remains indispensable, it exhibits several structural limitations:

Synthetic data, when naively generated, risks compounding these problems. When engineered with market structure awareness, however, it becomes a strategic asset.

Defining “Market-Consistent” Synthetic Financial Data

Market-consistent synthetic data is not defined by point-wise similarity to historical prices, but by preservation of structural, statistical, and relational properties that govern real markets.

Achieving these properties simultaneously requires moving beyond purely parametric models or black-box generative AI.

Graph-Massivizer Approach: A Graph-Centric Paradigm for Synthetic Data Engineering

Graph-Massivizer financial use case was built on the premise that financial markets are naturally relational systems, not independent time series collections. Assets, time steps, market regimes, and derived features form a structured network of dependencies that can be explicitly modeled.

Historical financial data across assets, instruments, and time resolutions is first ingested and transformed into a graph representation:

• Nodes can represent time points, instruments, regimes, or derived states.
• Edges can encode temporal transitions, cross-asset dependencies, and statistical constraints.
• Multi-layer graphs capture interactions across different time scales.

This representation preserves information that is typically lost in flat tabular datasets. Then, before any generation occurs, the source data undergoes structural analysis to define what must be preserved and where variability is allowed. Next, synthetic data is then generated by expanding the graph under explicit constraints such as local randomness, correlations preservation where specific behaviors may be amplified, as well as reduced similarity to the original historic data to the point where reverse engineering is not possible.

The objective is original plausible novelty: data that is statistically consistent yet not traceable to any original observation, given that a critical compliance requirement is that synthetic data must not allow reconstruction of original data. This is particularly relevant for regulatory audits and third-party model validation.

Applications in Quantitative Research and Trading

Performance metrics derived from synthetic data are diagnostic, not predictive, highlighting fragility and structural bias.

Synthetic data is a pre-training and stress-training substrate and not a replacement for real data.

This shifts validation from retrospective justification to proactive resilience testing.

Synthetic data becomes a compliance enabler

There are various challenges as not all synthetic financial data is fit for purpose. Common failure modes include over-fitting synthetic data to historical distributions, ignoring cross-asset and temporal dependencies, excessively smooth or overly random time series or lack of quantitative validation metrics. A graph-centric, constraint-driven approach mitigates these risks by design.

Conclusion

Synthetic financial data generation is no longer an experimental direction. When engineered with structural awareness, statistical rigor, and regulatory foresight, it becomes a core infrastructure capability for modern quantitative organizations.

Graph-Massivizer demonstrates that markets can be expanded, not merely replayed, producing market-consistent time series in extreme volumes that support deeper research, more resilient trading systems, and more credible regulatory validation.

The future of quantitative finance will belong not only to those who analyze history best, but to those who can systematically explore what history did not contain, while following the rules that markets, mathematics and regulators impose.

Laurentiu Vasiliu, founder, Peracton Ltd

19/12/2025

Buy-side quantitative teams face a structural constraint: financial market data remains scarce, fragmented, expensive, and often unsuitable for large-scale AI training. Historical datasets are finite, heavily reused, biased by survivorship and regime persistence, and increasingly subject to restrictive licensing terms. As a result, many AI-driven trading initiatives stall not due to lack of modeling sophistication, but due to insufficient, contaminated, or non-scalable data.

Synthetic financial data is emerging as a strategic solution to this bottleneck, enabling a transition from data scarcity to data scale, while preserving market realism and regulatory relevance.

Why Traditional Market Data No Longer Scales for AI Training

Quantitative trading strategies based on machine learning and deep learning differ fundamentally from traditional statistical or factor-based models. They require large volumes of diverse training data, exposure to multiple market regimes, including rare and extreme events, a clean separation between training, validation, and stress-testing datasets and continuous refresh without historical leakage or overfitting.

Traditional market data is limited on several of these dimensions:

These constraints are structural, not incremental. They cannot be solved by marginally better data sourcing or vendor negotiation.

Synthetic Financial Data: From Approximation to Market-Consistent Engineering

Modern synthetic financial data is not a simplistic resampling or noise-augmented replica of historical prices. When engineered correctly, it represents a market-consistent multiverse of financial time series that preserves statistical properties across time scales, cross-asset and cross-market dependencies, microstructure dynamics (order flow, spreads, volatility clustering), regime transitions and structural breaks This can be achieved through a combination of stochastic and regime-switching models, graph-based dependency modeling and constraint-driven generation aligned with real market invariants

The result is not one synthetic dataset, but thousands—or millions—of plausible market trajectories that extend far beyond what history alone can provide.

Powering AI Training at Scale

Synthetic financial data fundamentally changes how AI models are trained and validated in quantitative trading.

Business Impact for Buy-Side Quantitative Teams

From a business perspective, the adoption of synthetic financial data is less about experimentation and more about competitive positioning. Quant teams can iterate models faster without waiting for new historical data or negotiating incremental licenses. Then, synthetic data decouples AI scaling from data vendor pricing, enabling predictable and controllable cost structures. Further on, exposure to a broader market multiverse improves resilience across regimes, directly impacting drawdown control and long-term performance stability.

Synthetic datasets support explainability, reproducibility, and scenario-based validation that are key concerns for internal model risk committees and external regulators.

What is changing today is not just the quality of synthetic financial data, but its role in the quantitative stack. It is evolving from an augmentation tool into core data infrastructure for AI-driven trading.

Scaling Artificial Intelligence, Not Just Data

Synthetic financial data enables this shift from scarcity to scale, by providing the foundation required for industrial-grade AI training in finance. For buy-side quantitative teams, it represents not only a technical advancement, but a strategic lever: accelerating innovation while improving robustness, compliance readiness, and long-term performance sustainability.

The evolution of quantitative trading will not be determined solely by better models or faster hardware, but by the ability to systematically train AI across diverse, realistic, and unbiased market environments.

In an environment where alpha is increasingly driven by adaptability rather than historical coincidence, synthetic financial data is rapidly becoming a requirement rather than an option.

Laurentiu Vasiliu, founder, Peracton Ltd

26/12/2025

However, powering such large numbers of computers takes a significant and growing amount of energy. The global energy demand of data centres is estimated to reach 8% in 2030 [1]. Unfortunately, burning fossil fuels remains an important and widely used source of energy. This makes data centre construction and operation a significant contributor to greenhouse gas (GHG) emissions.

In this post, we work towards sustainable massive graph-processing workloads by using Graph Greenifier to analyse the effect of temporal shifting (a technique to run workloads with sustainability in mind) on the carbon emission of data centres.

What is Carbon Intensity?

Carbon intensity is a metric that quantifies the (un)sustainability of an energy source by computing the amount of CO2 emitted per unit of energy. The table below presents an overview of the carbon intensity of four highly popular energy sources [1].

Today, in practice, the electricity that is generated from both renewable and non-renewable sources is combined and offered to electricity consumers through an electricity network, also known as the energy grid. This means that electricity consumers on the grid do not use entirely renewable or non-renewable electricity, but rather use whatever combination is put onto the grid by electricity producers.

By knowing how much electricity on an energy grid comes from each source, we can compute the carbon intensity of that grid. To do so, we use the following formula:

Formula for computing grid carbon intensity.

Here, CIg is the carbon intensity of the grid, S is the collection of energy sources used on the grid, CIs is the carbon intensity of one specific energy source, Es is the amount of energy obtained from that energy source, and Eg is the total amount of energy on the grid. In plain English, this formula computes the carbon intensity of each energy source and then computes the weighted sum of those sources.

Knowing the carbon intensity of a grid allows electricity consumers, such as data centres running massive graph-processing workloads, to compute the carbon emissions of their activities by multiplying their electricity use by the carbon intensity of the grid they use to obtain their electricity. Once the carbon emission of a data-centre workload is known, we can start exploring approaches such as temporal shifting to reduce it.

What is Temporal Shifting?

The carbon intensity of grids that obtain energy from renewable sources can change significantly over time because the amount of renewable energy is highly variable and depends on factors such as the time of day and the weather. For example, the image below shows the Dutch energy grid over the course of one month. The top plot shows the amount of available renewable (green) and non-renewable (gray) energy on the grid, and the bottom plot shows the carbon intensity of the grid.

Energy mix and carbon intensity of the grid in the Netherlands during October 2023 [6].

Intuitively, we can reduce GHG emissions by using electricity when its carbon intensity is low. We can implement this approach for graph processing in data centres by delaying the execution of incoming workloads when carbon intensity is high and starting execution when carbon intensity is low. This effectively moves the workload in time, which we call temporal shifting. This idea has been suggested in related scientific work [3, 4, 5], but Graph Greenifier allows us to simulate, and therefore quantify, its effect.

The Graph Greenifier Approach

Graph Greenifier can simulate what happens in data centres during massive (graph-)processing workloads and supports simulating operational techniques such as temporal shifting. Operational techniques are actions data centres take to influence their operation. For example, selecting when and where to schedule a task. This allows data center operators, designers, and researchers to understand the impact of such techniques on both data center performance and sustainability. The figure below shows a simulation result from Graph Greenifier for this scenario.

Simulation results comparing FCFS scheduling and Carbon-Aware scheduling.

Massive graph-processing workloads and other large workloads consist of many small tasks that need to be executed. The top plot in the figure shows the number of actively running tasks for two different scheduling approaches. The blue curve shows a traditional First-Come-First-Served (FCFS) scheduler, which schedules tasks to be executed as soon as possible, and in the order they arrive. The orange curve shows the Carbon-Aware scheduler, which delays the execution of incoming tasks when carbon intensity is high. The green curve in the bottom plot shows the carbon intensity of the grid over time.

We can see that the Carbon-Aware scheduler effectively delays tasks until carbon intensity is low by looking at the orange and green curves. Specifically, we see that the peaks in the orange curve (high number of active tasks) align with the valleys in the green curve (low carbon intensity). For this particular workload, the reduction in carbon emissions is 2.5%, but this can increase depending on the workload and the carbon intensity (variation) of the energy grid to which the data centre is connected.

Next Steps for Graph Greenifier

Temporal shifting is but one technique in a large collection of commonly used operational techniques in data centres. These include spatial shifting, checkpointing, and active-active replication, to name but a few. Additionally, changing the data-centre scheduling policy and other operational techniques can affect not only carbon emissions, but also workload performance and other non-functional properties.

By supporting these techniques in Graph Greenifier, scientists, data centre operators, and other stakeholders can explore “what-if” scenarios and perform a wide range of deep analyses using arbitrary combinations of these techniques to make a trade-off between sustainability and performance for their graph-processing workloads.

References

[1] Anders S. G. Andrae and Tomas Edler. 2015. On Global Electricity Usage of Communication Technology: Trends to 2030. Challenges 6, 1 (2015), 117–157. LINK

[2] Udit Gupta, Mariam Elgamal, Gage Hills, Gu-Yeon Wei, Hsien-Hsin S. Lee, David
Brooks, and Carole-Jean Wu. 2022. ACT: designing sustainable computer systems with an architectural carbon modeling tool. In Proceedings of the 49th Annual
International Symposium on Computer Architecture (New York, New York) (ISCA
’22). Association for Computing Machinery, New York, NY, USA, 784–799. LINK

[3] T. Sukprasert, A. Souza, N. Bashir, D. Irwin, and P. Shenoy, “On the limitations of carbon-aware temporal and spatial workload shifting in the cloud,” in EuroSys, 2024.

[4] Philipp Wiesner, Ilja Behnke, Dominik Scheinert, Kordian Gontarska, and Lauritz Thamsen. 2021. Let’s wait awhile: How temporal workload shifting can reduce carbon  missions in the cloud. In Proceedings of the 22nd International Middleware Conference. 260–272.

[5] Jiechao Gao, Haoyu Wang, and Haiying Shen. 2020. Smartly handling renewable energy instability in supporting a cloud datacenter. In 2020 IEEE international parallel and distributed processing symposium (IPDPS). IEEE, 769–778

[6] D. Niewenhuis, S. Talluri, A. Iosup, and T. De Matteis, “Footprinter: Quantifying data center carbon footprint,” in HotCarbon, 2024.

The Challenge: Extracting Insights from Massive Knowledge Graphs

Knowledge graphs are a useful tool to model information that represents complex relationships in your data. Typically, with knowledge graphs, there are standard symbolic approaches and algorithms that work well to address your needs for useful data analytics.

When knowledge graphs become massive, however, and the relationships and semantics in your data become more intricate, it is often necessary to re-examine the traditional approaches to find strategies that can be tailored for massive graphs. This need can be due to memory issues, where a graph is too big to store on a normal computer or even a server, or too big for algorithms to handle when they need to look at lots of data at the same time.

In industry and academia, there are very large graphs that represent publications, authors, domains and the relationships between them.

In this article, we will investigate an example of such a massive research knowledge graph and show how AI approaches can be leveraged to address the unique challenges posed when developing algorithms to utilize a massive knowledge graph.

Massive Knowledge Graphs: what are they?

Massive knowledge graphs are knowledge graphs (KGs) that are, for some reason or another, much too large to manage with strategies that are typically effective for smaller graphs. Often, a knowledge graph is associated with an ontology, which schematizes the intended structure of the data in the knowledge graph in a formal way that is understandable by a computer or a human. When we say too large, this can mean a few different things, such as being simply too big to fit on a standard computer or too massively complex to process with typical algorithms, to give a few examples.

Scalable analytics for a massive research knowledge graph: exploring the SemOpenAlex use case

Now let’s look at an existing use case that combines AI with massive knowledge graphs.

Researchers and engineers at metaphacts are contributing to an EU project called Graph-Massivizer, where scalable solutions are being developed to process and extract insights from massive graphs with the Graph-Massivizer Toolkit. Composed of 12 partners from eight EU countries, the Graph-Massivizer project brings together the world-leading roles of European researchers in graph processing and serverless computing and uses leadership-class European infrastructure in the computing continuum. This project develops optimizations and AI/ML tools connected with the use cases described here, and motivates the bigger-picture topics we’ve discussed.

One of the use cases for this project is a development and integration scenario where test cases are developed for the massive SemOpenAlex knowledge graph of open academic publication data. The SemOpenAlex knowledge graph is challenging to design scalable algorithms for graph analytics due to its sheer size, so the project aims to use it as a test case to demonstrate the functionality under development. The use case features multiple interesting applications of these techniques for automated workflow execution and agentic AI that will be shown in the following sections.

An example of this use case examines a scholarly research collaboration network, answering the following query: ”How do I find the shortest path from myself to the most popular researcher in my field?”. The academic knowledge graph use case using SemOpenAlex that shows co-authorship relations for publications, attempting to find connections between influential researchers in the same field by tracing co-authorship relations. The knowledge graph includes additional metadata that can be leveraged by learning algorithms, such as publication information, including titles, authors, venues, and authors’ research fields. The use case defines “popular researcher” based on a metric considering extensive publications and co-authors.

Fig. 1: The workflow diagram for the SemOpenAlex use case.

A workflow (as shown in the above diagram) for the use case is schematized in the user interface above with an ontology that is discussed in the next section. Tasks in the workflow include algorithms such as loading a subgraph using a SPARQL query or computing the Betweenness Centrality values for nodes in the graph, to give some examples. Using the metaphactory interface allows a user to select predefined algorithms based on available hardware and programming languages to execute their workflow. Because the workflow is expressed in RDF, it can be understood by a human as well as the computer responsible for executing it. The workflow contains pointers to data and functions that it describes, provided by the technical user who sets it up; a program or agent can execute it automatically with minimal input.

Executable workflows for graph algorithms

Workflows can be represented explicitly in RDF according to an ontology that also connects to a data source. This allows an agent, whether human or AI, to interact with and execute user-specified workflows dynamically while a system is given access to the required algorithms. If the workflow also specifies data flow, it can be used to execute an entire pipeline of different functions, sending input from one function to another automatically and showing the user the end result. This is advantageous when integrating knowledge graphs with AI since it abstracts away the information that is already known about how to use algorithms, which allows a developer or user to customize the execution of a workflow without always giving it lots of very specific instructions on what to do.

Fig. 2: The graph-processing workflow ontology.

The workflows used in the SemOpenAlex use case are structured according to the ontology shown above. At the top, you can see the workflow class, which must contain at most one first task, which itself can have any number of next tasks. This can model a sequential or even tree-like structure based on user needs, where the first task is the root of the tree and all next tasks branch from there. This tree-like structure is intended to correspond roughly to the directed acyclic graph (DAG) that is required by the toolkit to define its workflows, ensuring that operations are well defined and not running in an infinite loop. When an end-user interacts with the system, all they are required to do is specify the workflow of tasks and associate them with available algorithms preloaded according to the other parts of the ontology.

For a developer who sets up the system, they specify available algorithms using the class BGO, or basic graph operation. This class represents a type of graph algorithm, such as breadth-first search. Each of these abstract BGOs is then connected with at least one implemented algorithm. It is possible to include multiple implementations of the same algorithm in various languages or with different inputs and outputs so that different execution pathways and scenarios can be made available to support diverse use cases. The system is then able to dynamically choose the algorithm and runtime environment best suited for execution of the task.

The motivation behind modeling and developing these workflows is that they enable an expert or team of experts to configure the Graph-Massivizer Toolkit in advance with algorithms and workflows that they wish to make available to an end user. The end user can then utilize these advanced AI techniques via a simple interface in metaphactory that builds and executes workflows correctly without needing to already have a deep technical understanding of AI techniques. Next, we will see an example of how the same type of approach can be integrated with a neuro-symbolic system that has its own unique advantages.

Analytics and Agentic AI with metis

Graph analytics and Agentic AI can be combined with neuro-symbolic approaches to achieve unique advantages. One such system, metis, is a knowledge-driven AI platform combining large language models and knowledge graphs to deliver AI agents that provide generative power, semantic precision and contextual, explainable insights.

metis, which sits on top of metaphactory, is capable of exploring existing data in a knowledge graph and is also able to guide users through the modeling process when they design their ontology, or can be customized or selected from an agent repository. The AI component aids a user in their interaction with the knowledge graph in a conversational interface by translating natural language into machine-readable actions and instructions. We will see how this tool can be integrated with the use case discussed previously so that a user is able to perform analytics using a simple conversational interface.

Because metis allows interaction with a knowledge graph, users can utilize AI tools to improve productivity without sacrificing the essential trust that comes with using a well-designed ontology to manage their data.

Algorithms for graph analytics are supported as extensions to metaphactory applications and can also be integrated with metis. This makes the adoption of analytics features seamless with the platform usage in general. An app can act as a tool for metis, allowing it to answer complex questions over the knowledge graph that require advanced machine learning algorithms.

Fig. 3: A sample conversation in metis executing the SemOpenAlex usecase.

In the image above, you can see a sample conversation where metis is used to execute the same workflow as we saw in the previous section. Using only natural language, a user can request complex workflows such as finding the shortest path between an instance and the instance with the maximum centrality in a graph. In order to answer this question, the agent not only needs to execute the algorithms but also must understand how to generate workflows in case multiple operations are requested in sequence. And with this functionality, users don’t need to compose workflows themselves since they can simply ask the agent using natural language.

Conclusion

We have looked at various methods for using AI and knowledge graphs together in complementary ways. The SemOpenAlex use case from the Graph-Massivizer project featured multiple interesting applications of these techniques for automated workflow execution and agentic AI. There are many new, exciting possibilities being developed every day to support AI and large knowledge graphs at metaphacts in this as well as other projects and domains, so look forward to more updates from our team about new applications for AI with your knowledge graphs.

Explore solutions

After seeing some examples of how knowledge graphs and AI can work together, you may already have an idea for how these technologies can benefit your organization.

Speak with an expert to discuss your organization and specific use case, and learn how metaphactory can support your knowledge graph and AI use case. You can also request a demo of metis to see it in action

Artificial intelligence is reshaping modern society, but this technological revolution comes at an environmental cost that we can no longer ignore. Although AI offers innovative ways to tackle climate change, it also contributes to carbon emissions and the consumption of resources.

The environmental impact of AI is staggering. Research published in Nature Sustainability reveals that implementing AI servers in the United States could generate between 24 and 44 million metric tons of CO₂ equivalent emissions annually by 2030 — comparable to adding 5 to 10 million cars to American roads. The United Nations emphasises that data centres hosting AI servers produce electronic waste, consume vast amounts of water, and use huge quantities of electricity, thereby fuelling greenhouse gas emissions. Golestan Radwan, UNEP Chief Digital Officer, states that we must ensure the net effect of AI on the planet is positive before it is implemented on a large scale. AI is what researchers call “double-edged technology”.

Supercomputers and AI models have their own carbon footprint, which varies depending on the type of AI and the training methods used. However, AI can also play a key role in reducing emissions through climate change modelling and smart grid design. Microsoft Research has demonstrated that AI-based systems can integrate renewable energy more effectively into stable electrical grids and reduce carbon capture costs by accelerating the discovery of new materials.

In this critical context, Graph Massivizer shows how innovation and sustainability can go hand in hand. The project aims to improve the efficiency of data analysis and reduce the energy impact of extract, transform, and load operations. It seeks to enhance data centre energy efficiency by a factor of two and reduce greenhouse gas emissions associated with graph-organised database operations.

The Data Centre Digital Twin use case, involving CINECA and the University of Bologna, exemplifies this vision by creating a digital graph representation of CINECA’s supercomputers. This representation allows system operations to be studied and understood and enables efficiency and sustainability to be optimised for the next generation of exascale supercomputers.

Recent research from the University of Bologna and CINECA showcases AI designed to support sustainable development. The ExaQuery project proposes an innovative ontology for operational data in HPC systems that organises and queries telemetry data more efficiently, reducing computational load. This knowledge graph-based approach facilitates the identification of complex relationships between hardware components, computational jobs and performance metrics, paving the way for intelligent resource optimisation.

Building on this foundation, researchers have developed a Virtual Knowledge Graph system that provides natural language access to heterogeneous IoT data in data centres. Combining Large Language Models with Knowledge Graphs achieves 92.5% query accuracy for this system, compared to 25% for traditional LLM-to-NoSQL approaches, while reducing latency by 85%. This breakthrough demonstrates that intelligent data organisation can dramatically improve accessibility and efficiency when managing complex telemetry systems.

As Graph Massivizer highlights, although data analysis and processing have a significant environmental impact, they can also be invaluable tools for achieving environmental sustainability. The key lies in using technology responsibly by focusing on efficiency, renewable energy and intelligent computational architectures. The message is clear: AI can and must be part of the solution to the climate crisis, not the problem. Projects like Graph Massivizer show that technology’s future can be powerful and sustainable if we make the right choices today.

The Essence of GraphMa

At its core, GraphMa is a conceptual framework that seamlessly merges the principles of pipeline computation with the intricacies of graph processing. It introduces a series of powerful abstractions that empower developers to decompose complex graph operations into modular, composable functions. These functions can then be orchestrated into streamlined pipelines, facilitating the systematic development and execution of graph algorithms.

The Building Blocks of GraphMa

• Computation as Type: This foundational abstraction elevates computation units to first-class entities, encapsulating them within a well-defined interface. This approach ensures type safety and promotes modularity, enabling the creation of reusable and composable pipeline stages.
• Higher-Order Traversal Abstraction: This abstraction provides a versatile mechanism for navigating and accessing data within graphs. It defines methods for traversing various data sources, empowering developers to manipulate and process graph data with flexibility and efficiency.
• Directed Data-Transfer Protocol: This protocol governs the seamless and efficient transfer of data between computational stages. It adheres to functional programming principles, ensuring clear directionality and optimized data flow throughout the pipeline.
• Operator Model: This model introduces a comprehensive set of constructs for managing the lifecycle and states of operators within the pipeline. It facilitates a wide array of data processing operations, from transformations to aggregations, enabling the construction of sophisticated graph algorithms.
• Pipeline Abstraction: This abstraction serves as the overarching framework that orchestrates the entire graph processing workflow. It encapsulates the complexities of data transformation and transmission, providing a high-level blueprint for defining and executing graph processing pipelines.

Embracing Established Computational Models

GraphMa’s versatility shines through its ability to seamlessly integrate well-established computational models for graph processing. Whether it’s the vertex-centric model, where computations are centered around individual nodes, or the edge-centric model, which focuses on the relationships between nodes, GraphMa provides a flexible platform for implementing and executing these models within its pipeline-oriented architecture.

The Promise of GraphMa

GraphMa represents a significant leap forward in the field of graph processing. By combining the power of pipeline computation with graph-specific abstractions, it offers a structured and modular approach to tackling the challenges of graph data analysis. Its potential to enhance scalability, efficiency, and expressiveness in graph processing tasks positions it as a valuable tool for researchers and practitioners navigating the complexities of interconnected data. As GraphMa continues to evolve, we can anticipate its widespread adoption and its transformative impact on the way we understand and leverage the power of graphs in the digital age.

References

Schroeder, Daniel Thilo,    Tobias Herb,    Brian Elvesæter, and Dumitru Roman “GraphMa: Towards new Models for Pipeline-Oriented Computation on Graphs.” In Companion of the 15th ACM/SPEC International Conference on Performance Engineering, pp. 98-105. 2024.

• k-bisimulation can be used to create a condensed version of a graph. This condensed version is a graph summary, keeping specific properties of the original
• k-bisimulation partitions the nodes of the graph in equivalence classes which we call blocks
• We create the summary by creating one node for each block. Then, for each edge in the original graph, we connect the corresponding blocks with an edge, with the same label.
• To speed up the computation of the k-bisimulation, we
  • use a partition refinement approach
  • implemented everything in C++, making use of the boost libraries
  • treat singleton blocks separately
  • then treat blocks with only 2 nodes
  • … and only then the rest
  • devised a new step in the algorithm which remembers which parts might need to be refined from step k-1 to compute the blocks at step k
• Doing all this, we obtained a speedup of 20X compared to the already improved python implementation.

Introduction

When graphs get very large, they can become difficult to work with. One way to deal with such a graph is by reducing it to a smaller data structure which maintains the properties you want to preserve. In this specific case, we want to create a quotient graph based on a k-bisimulation. This summary graph preserves paths which were in the original graph, but can be much smaller than the original. For very large graphs, computing k-bisimulation is itself a challenge. There are existing frameworks, but they have their limitations. Some are either hard to set up and the overhead of the framework makes them less scalable. Often these frameworks trade efficiency for broader applicability; they have capabilities to produce a wider variety of summaries. In this blog post, we first define k-bisimulation and look at a naive algorithm to compute it. Then we will look into partition refinement, which is a faster way to compute the same thing. Finally, we will look at further optimizations of partition refinement and discuss how we implemented it.

k-bisimulation

We start from a labeled graph G = (V,E,L). where V is the set of vertices or nodes, L is a set of labels and E ⊂ {(v₁,l,v₂)|v₁, v₂ ∈ V and l ∈ L} is the set of labeled edges, also called the triples, of the graph. v₁ is the source vertex of the edge, v₂ is the target vertex. This definition implies that there can be multiple labeled edges between two vertices, but not two edges with the same label.

Now, we will start talking about paths in a graph. A path is a sequence of edges where the target vertex of the previous edge is the source vertex of the next edge. If we call the source of the first edge v_start and target of the last edge v_end, then we say that this is a path from v_start to v_end. The length of the path is the number of edges in the path. In some definitions it is assumed that an edge occurs only once in a path; we make no such assumption. Commonly, we are only interested in the labels of the edges in the path and not in the path itself. Therefore, we introduce the term labelpath to mean the sequence of labels of the path defined above. We define the set of all outgoing labelpaths of length k as paths_k, out(v_A) to be the set of all labelpaths of at most length k starting at vertex v_A (and ending anywhere in the graph).

Now we can find our k-bisimulation by first explaining when two vertices are bisimilar. Two vertices v_A and v_B are k-bisimilar if paths_k, out(v_A) = paths_k, out(v_B), i.e., they have the same set of labelpaths up to length k.

Side note: to be precise, we are working with forward-bisimilarity which deals with outgoing paths only. Analogously, backward-bisimulation deals with paths ending in a specific node. Forward-backward bisimulation deals with both at the same time, meaning that both incoming and outgoing paths must be equal.

What we now do to create the summary is first fixing the parameter k. Then, we use the bisimilarity as an equivalence relation between nodes, i.e., we consider two nodes equivalent if the are bisimilar. This equivalence relation identifies a partition on the vertices of the graph G, i.e., we can split the vertices E into subsets such that

1. None of the subsets is empty and each vertex is in precisely one of the subsets.
2. In each set, each vertex is bisimilar to all other vertices in that set.
3. No vertex from one subset is bisimilar to a vertex from another subset.

We call each of these subsets a block of the partition.

Now we are ready to create our summary as follows. Given a graph G = (V,E,l) create a summary graph S = (V_S,E_S,l), where

1. V_S = {v_B|B is a block in the partition}, i.e., we create one supernode for each of the blocks.
2. E_S = {(v_A,l,v_B)|(v_a,l,v_b) ∈ E and A, B ∈ V_S and v_a ∈ A and v_b ∈ B}, i.e., for each edge in the original graph, we create a new edge, with the same label, between the vertices representing their blocks in the summary graph.

This kind of summary graph is a quotient graph.

To create these summaries the main algorithmic step is to compute the partition, i.e., find the subsets. After that, creating the edges is trivial.

A naive partition function.

A naive way to find the blocks is by computing all outgoing labelpaths for all nodes. This works as follows (python pseudo-code):

def find_paths (v: Vertex, k: int):
    labelpaths = set()
    for label, target in v.outgoing_edges():
        if k == 1:
            labelpaths.add([label])
        else:
            for deeperpath in find_paths(target, k - 1):
                labelpaths.add([label].extend(deeperpath))
    return paths


equivalence_map = defaultdict(list)
for v in V:
    paths = find_paths(v)
    equivalence_map[paths].append(v)

In this algorithm, for each vertex, we compute the set of all outgoing labelpaths. Then, we use the equivalence_map to make sure all vertices with the same set of paths get grouped together. In the end, the values in the map are the blocks we are looking for.

The problem with this implementation is that in the worst case the speed and memory use become quadratic in the depth of the path. This happens, for example, with graphs which look like the one in the figure.

Here, the depth of the paths is only 3, which will not cause an issue. An issue arises when we encounter such structures with longer paths. If we make a larger graph with the same structure as the one above but with depth k instead of 3, we observe that the number of paths becomes 2^k, which for a large k means very many paths. The following figure shows the outcome of an experiment for increasing depths.

What we see is that the time to execute becomes ever longer with an increasing depth (note the logarithmic scale on the y-axis). In general, we can see that this type of algorithm can behave exponentially. In effect, this implementation is not scalable for deep paths. Also when paths are shorter, this implementation is far from scalable for large graphs because of the large amount of data stored for the paths.

Towards a more efficient implementation

As usual in computer science, everything has been solved in the 80-ies. Also in this case. In the paper

@article{doi:10.1137/0216062,
    author = {Paige, Robert and Tarjan, Robert E.},
    title = {Three Partition Refinement Algorithms},
    journal = {SIAM Journal on Computing},
    volume = {16},
    number = {6},
    pages = {973-989},
    year = {1987},
    doi = {10.1137/0216062},
    URL = {https://doi.org/10.1137/0216062}
}

the partition refinement algorithm was used to compute bisimulations. The lingo used in the paper is rather different, but the algorithm almost directly applies. There are a few differences, though. In that work, the authors did not care about k, but were only interested in the case where k reaches infinity, meaning that all paths, independent on length must be the same for vertices to be bisimilar. We hence adapted the algorithm to our use case. Here, I explain the intuition behind the algorithm.

Partition refinement

As mentioned the algorithm is called partition refinement. It works by creating a partitioning for a depth k − 1, and then refining that partition to become the one for level k of the bisimulation. In other words, when we need to compute k-bisimulation, we assume (k-1)-bisimulation has already been computed. In other words, the algorithm works inductively.

For k = 0, meaning paths of length 0, all vertices are equivalent. So we define the partition to contain one block that contains all vertices.

For k > 0, we assume the (k−1)-bisimulation has been computed already. This one has a number of blocks. The vertices within each block are pairwise (k−1)-bisimilar.

Now, we work block by block through the blocks at level (k−1). For a given block A, we compute a signature for each of the vertices. For a vertex v_a, this signature consists of the set of all (l,B) for which (v_a,l,v_b) ∈ E and v_b ∈  block B on level (k−1).

Based on these signatures, we are splitting block B, into smaller blocks, where each new block contains the vertices which have the same signature. These blocks are added to the collection of blocks for level k. After this, we throw away the signatures and continue with the next block.

Correctness of partition refinement

It is important to realize that the partition refinement algorithm will result in precisely the same final partition as the original algorithm. You can either believe this, and skip this section, or follow the informal argument. If that does not convince you, you could read a more formal proof in the original paper, specifically section 3 ‘Relational coarsest partition’.

We need to show the three properties:

1. None of the blocks is empty and each vertex is in precisely one of the blocks.
2. In each block, each vertex is k-bisimilar to all other vertices in that block.
3. No vertex from one block is k-bisimilar to a vertex from another block.

First, it is trivial that none of the blocks is empty, and that each vertex is in precisely one block because we encounter each vertex only once iterating over the blocks.

For the second property, let’s look at two vertices in a block at level k. The vertices which ended up here had the same signature. If we look at one element (l,B) of the signature, we realize that both vertices have one (or more) outgoing edges with label l which end up in a vertex in block B. But, by induction, the vertices in block B are (k−1)-bisimilar, meaning that all of them have the same sets of outgoing paths. Prepending all these paths with l still results in two sets with the same paths. So, each part of the signature results in a set of paths which are the same for both vertices. Hence, this pair of vertices, and by extension all pairs of vertices in the block are k-bisimilar. We chose an arbitrary block, so in all blocks on level k the property holds.

We can show the third property by contradiction. Imagine the property holds for the previous round and now we find two vertices v_a and v_b which are k-bisimilar and in different blocks in the current round. To be k-bisimilar, these two vertices need to be (k−1)-bisimilar. So, in the previous round, they must have been in the same block B. And therefore, their signatures were directly compared. The only way they could have ended up in different blocks is if their signatures were not the same. This can have two causes.

1. One of the labels might be different. This, however, means that the vertices are not k-bisimilar which is a contradiction.
2. We find a part of the signature for v_a which has the same label, but ends in a block A, rather than the corresponding part in the signature for v_b, which refers to block B. (if this is not the case, invert the roles of v_a and v_b). However, we assumed all was fine until the previous round. This means that v_a has an edge to a vertex in block A, while v_b does not have an edge to a vertex in block A. Since the vertices in block B are not (k−1)-bisimilar to the vertices in block A, v_a and v_b are not k-bisimilar, which is a contradiction.

So, given that all three conditions are fulfilled, we are guaranteed a correct k-bisimulation with this algorithm.

Efficient implementation in Python

The theoretical result is interesting, but we apply a few additional insights to speed up the computation.

1. When no refinement is happening, we know that we are done, and can stop. This happens at the latest when the number of rounds becomes equal to the diameter of the graph.
2. When a block only contains one element, ie., it becomes a singleton, we never have to look at it again. It can also be stored more efficiently: instead of storing a list to contain all vertices in the block, we can have one list containing all vertices which are in a singleton block at this round. These can just be extended with new singletons in the next round.

With these tricks applied, we reduced the runtime of the bisimulation algorithm significantly. The following figure shows the runtime on the same type of graphs which illustrated the exponential behavior above. Now, we see that running a graph with depth 20 takes under 0.10 seconds, rather than the 100 seconds needed before. Even using a depth of 1000 results in a runtime of less than 2 seconds.

Now, we were ready to run this algorithm on a large graph. We chose a DBpedia dump which contains 8 million entities, and about 22 million edges. And ran it on a laptop with a i7-1280P CPU. The bisimulation ran until it reached k=146 before it finished. It took about 12 minutes 39 seconds, including about 40 seconds to load the data. It used about 40GB RAM.

To scale this up further, we had some more ideas, some of which would need more control on the memory management. We therefore moved to implement this in C++.

Efficient implementation in C++

For the implementation in C++ we heavily relied on the Boost libraries which provide efficient unordered container types like flat sets and flat maps. We also made use of emplacement to avoid object copying when possible. Besides the optimizations done for the Python version, we further optimized the following:

1. We keep track of which blocks have been split at the previous round. Only blocks which have vertices with edges to these blocks can be split at the next round. As far as we know, this is a novel addition to the algorithm.
   • We also experimented with a reverse index which keeps track of these edges directly, this did however not lead to significant speedups
2. We keep a mapping from the vertices to the block in which they are.
   • We specialize this for level zero because everything is in the same block.
   • We do not keep singletons explicitly. Rather, we map the corresponding vertices to negative integers, which indicates that they are singletons. To keep track of singletons blocks, we only need to remember how many there were.
   • After the round, the index of the previous round is cleared as is is no longer needed.
3. To create the blocks at level k, we first create a shallow copy of the blocks at level k-1. Only the modified block will occupy additional memory, others will only occupy one pointer.
4. When splitting blocks, we try to not move all the data around. Rather, we put the first part of the split in place of the old block and put the other new parts in the back, then we update only the necessary mappings.
   • In some cases, a split results in only singletons. In that case, we add a special empty block to the result. As soon as possible, that block will be overwritten by other splits, which will be put in these places, rather than appended to the back. Note that in a rare case an empty part could remain. This might contradict the requirements. A final cleanup step, which is not yet implemented, could take care of this.
   • We deal with blocks of size 2 first, because when they split, they cause two singletons, and always an empty block. The heuristic is that by doing these first that there is a larger chance that larger blocks will not just split into singletons and fill this gap.

With these additional steps, we could further reduce the running time for the DBpedia dataset. This now runs in 56 seconds, including 20 seconds for reading, meaning 36 seconds for the actual computation. To compare, the python version needed 720 seconds (excluding reading). The speedup is about a factor 20. The memory usage went down from 40 GB to 10.7GB, so roughly by a factor 4.

Further details

Both implementations also have a support parameter, which defaults to 1. If the size of a block goes under that size, then the block will no longer be a candidate for splitting.

It would be possible to port some of the optimizations from the C++ version back to the python version.

Further possible optimizations

It would be possible to only partially compute the signatures until enough of it is computed to notice a difference where a node gets into a singleton block. This would, however, require quite some bookkeeping and that is most likely outweighing potential gains.

Michael Cochez – Assistant Professor, Vrije Universiteit Amsterdam