Graph Choreographer

Graph-Choreographer offers comprehensive resource orchestration, scheduling, and provisioning capabilities for graph processing tasks employing serverless computing on the computing continuum.

The core of Graph-Choreographer’s process lies in its use of distributed computing, workflow management systems, intelligent load balancing, adaptive auto-scaling, and comprehensive real-time analytics, culminating in a robust, scalable, and eco-friendly solution for complex, large-scale graph data management challenges. It integrates seamlessly with leading open source serverless platforms, employing advanced optimization algorithms and (meta)heuristics to optimize resource allocation while minimizing environmental impact in processing large-scale graph structures. It operates in concert with the Graph-Massivizer suite, enabling dynamic data exchange for enhanced real-time performance and sustainability analytics.

Graph Greenifier

The Graph-Greenifier is in charge of simulating the execution of graph processing workloads on a large-scale infrastructure, reporting sustainability metrics, and proposing alternative execution plans.

To simulate execution plans, the Graph-Greenifier leverages information given by other components of the Graph-Massivizer toolkit. First, the Greenifier requires a description

of all operations that can be used for a given job. These algorithms are described in terms of Basic Graph Operations (BGO). Second, the Graph-Optimizer provides models that describe the performance of the BGOs on different types of hardware. Finally, the Graph-Choreographer provides an overview and the status of the currently available hardware.

The Greenifier uses all this information and combines it with forecast of available energy sources to simulate several execution plans, and deriving their performance and sustainability impact.

Graph Optimizer

Given the strong correlation between graph processing workloads’ performance and energy consumption, one cannot ignore the strong correlation between a given graph’s properties and the algorithms chosen to implement a workflow.

The Graph-Optimizer tool is responsible for searching for the right combination of BGO implementations using performance and energy models, micro-benchmarking for calibrating these models, and a design-space exploration process for a given input dataset. This optimization is fundamental to the overall performance, because orders of magnitude performance degradation are possible when choosing the “wrong” algorithms, implementations, or infrastructure.

Graph Scrutinizer

Graph-Scrutinizer offers comprehensive capabilities for scalable graph processing, employing graph sampling, partitioning, and summarization to handle large-scale graphs efficiently within single computer systems. It introduces advanced functionalities, including heuristics and neural query execution engines, interoperability between time series and graphs, and generative capabilities, targeting a wide range of use cases.

Graph-Scrutinizer’s design aims to avoid processing entire massive-scale graphs by obtaining relevant samples, thus addressing the challenges of analyzing and mining large-scale graph data, as well as balancing execution time, exactness of results, and energy efficiency. This tool will be integrated with other tools under the Graph-Massivizer project to support efficient and environmentally friendly graph processing across diverse environments, including cloud, edge, and HPC.

Graph Inceptor

Graph-Inceptor is a tool for creating KGs and processing and storing graph data. It offers two main services:

1. KG creation pipeline that
   • supports various source data file formats;
   • uses declarative mappings (i.e., no programming required) to transform source data to graph data;
   • stores the resulting graph data in a graph database.
2. Graph processing framework that
   • reads source or graph data in various formats;
   • offers in-memory access and processing of graph data in various representations;
   • saves the resulting graph data in various graph file formats;

This figure depicts a component architecture diagram of the Graph-Inceptor tool. The two services are implemented by two components:

1. metaphacts ETL Pipeline provides a blueprint implementation for a KG creation pipeline. The pipeline can read various data sources, map them to RDF, and pre- and post-process the data. The result is a (virtual) KG stored in the graph storage. It is not necessary to convert all source data to RDF. A virtualization approach allows referencing original data in other databases, such as time series or binary files. Well-defined and standardised interfaces access and query the KG through SPARQL endpoints.
2. GraphMa is a graph processing framework that reads, assembles, and processes raw or graph data from different formats, offering in-memory access and integration across various processes, and saving the resulting graph data in various graph file formats. Graphma supports a set of BGOs such LoadNetwork, Filter, Data2Graph, Interface2Graph, StoreGraph and Graph2File.