Networks Processing Performance Report

At Unum, we develop a neuro-symbolic AI, which means combining discrete structural representations of data and semi-continuous neural representations. Think of it as building a huge Knowledge Graphs. Such graphs have an extremely irregular structure, which makes data access patterns very unpredictable. Sounds like an ultimate workload for a serious DBMS benchmark.

Reproduce Our Results

Setup

Datasets

• Patent Citation Network.
  • Size: 272 MB.
  • Edges: 16,518,947.
  • Average Degree: 8.
• Mouse Gene Regulatory Network.
  • Size: 295 MB.
  • Edges: 14,506,199.
  • Average Degree: 670.
• Human Brain Network.
  • Size: 4 GB.
  • Edges: 87'273'967.
  • Average Degree: 186.

Device

• CPU:
  • Model: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz.
  • Cores: 8 (16 threads @ 2.3 Ghz).
• RAM Space: 16.0 GB.
• Disk Space: 931.5 GB.
• OS Family: Darwin.
• Python Version: 3.7.7.

Databases were configured to use 512 MB of RAM for cache and 4 cores for query execution.

Sequential Writes: Import CSV (edges/sec)

Every datascience project starts by importing the data. Let’s see how long it will take to load an adjacency list into each DB. But before comparing DBs, let’s see what our SSD is capable of by simply parsing the list (2 or 3 columns CSV). This will be our baseline for estimating the time required to build the indexes in each DB.

Most DBs provide some form functionality for faster bulk imports, but not all of them where used in benchmarks for various reasons.

• Neo4J supports CSV imports, but it requires duplicating the imported file and constantly crashes (due to Java heap management issues).
• Postgres and MySQL dialects of SQL have special functions for importing CSVs, but their functionality is very limited and performance gains aren’t substantial. A better approach is to use unindexed table of incoming edges and later submit it into the main store once the data is absorbed. That’s how we implemented it.
• MongoDB provides a command-line tool, but it wasn’t used to limit the number of binary dependencies and simplify configuration.

The benchmarks were repeated dozens of times. These numbers translate into following import duration for each dataset.

Those benchmarks only tell half of the story. SSDs have a relatively short lifespan, especially new high-capacity technologies like TLC and QLC. Most DBs don’t have high-performance bulk I/O options. It means, that when you import the data there is no way to inform the DB about various properties of the imported dataset. Which in turn results in huge write-amplification. Combine this with inefficient and slow built-in compression and prepare to give all your money to AWS!

Once the data is imported, it’s on-disk representation has different layouts in each DB. Some are more compact than others. For comparison, let’s take the HumanBrain 4 GB graph. According to the chart above, a total of 5.3 GB was written during the import. However, thanks to our compression, the resulting DB size is only 0.8 GB. The same graph uses ~3.5 GB in MongoDB, ~15 GB in MySQL, ~15 GB in Postgres, and ~15 GB in SQLite.

Read Queries

Following are simple lookup operations. Their speed translates into the execution time of analytical queries like:

• Shortest Path Calculation,
• Clustering Analysis,
• Pattern Matching.

As we are running on a local machine and within the same filesystem, the networking bandwidth and latency between server and client applications can’t be a bottleneck.

Random Reads: Find Specific Edge

• Input: 2 vertex identifiers (order is important).
• Output: edge that connects them in a given direction.
• Metric: number of queries per second.

Random Reads: Find Connected Edges

• Input: 1 vertex identifier.
• Output: all edges attached to it.
• Metric: number of queries per second.

Random Reads: Find Ingoing Edges

• Input: 1 vertex identifier.
• Output: all edges incoming into it.
• Metric: number of queries per second.

Random Reads: Find Friends

• Input: 1 vertex identifier.
• Output: the identifiers of all unique vertexes that share an edge with the input.
• Metric: number of queries per second.

Random Reads: Count Friends

• Input: 1 vertex identifier.
• Output: the total number of attached edges and their accumulated weight.
• Metric: number of queries per second.

Random Reads: Count Followers

• Input: 1 vertex identifier.
• Output: the total number of incoming edges and their accumulated weight.
• Metric: number of queries per second.

Write Operations

We don’t benchmark edge insertions as those operations are uncommon in graph workloads. Instead of that we benchmark *upserts- = inserts or updates. Batch operations have different sizes for different DBs depending on memory consumption and other limitations of each DB. Concurrency is tested only in systems that explicitly support it.

Random Writes: Upsert Edge

• Input: 1 new edge.
• Output: success/failure indicator.
• Metric: number inserted edges per second.

Random Writes: Upsert Edges Batch

• Input: 500 new edges.
• Output: 500 success/failure indicators.
• Metric: number inserted edges per second.

Random Writes: Remove Edge

• Input: 1 existing edge.
• Output: success/failure indicator.
• Metric: number removed edges per second.

Random Writes: Remove Edges Batch

• Input: 500 existing edges.
• Output: 500 success/failure indicators.
• Metric: number removed edges per second.

• Tweet
• Subscribe to receive similar articles 📨
• Reach Us to try the fastest data processing software 🔥