Unum | Unifying CS and HPC for the future of AGI

Stanford Large Network Collection

SNAP is the go-to place for everyone analyzing graphs and networks. It contains a broad set of massive datasets, including social networks, citation networks, logistical networks, brain networks, gene-expression graphs, online reviews, user interactions and many more. For this case study, we picked the US Patent citation network.

Most databases don't natively support graphs. DBs designed specifically for graphs are even worse at handling them than general-purpose DBMSs from our testing. So we adapted the most common battle-tested general-purpose DBs and tested them head-to-head against UnumDB:
• - Postgres.
• - MySQL.
• - MongoDB.

Uber Rides Dataset

Since 2009 Uber has released billions of data points about Taxi and Uber rides in New York City. This helps data scientists better understand the transportation market and explore the city's neighbourhoods, nightlife, and traffic from an entirely new angle.

Given the sheer size of the dataset, it became a popular target among High-Performance engineers. Today, only purely analytical databases like BrytlytDB and OmniSci shine in those workloads. Unfortunately, their universal solution for every problem is to throw more money on servers for horizontal scaling. UnumDB is different.

Processing a Billion Tweets

Twitter is arguably the best social network for data mining. Loved and used by developers all over the planet, everyone wants to access their API to analyze the social dynamics, sentiment around polarising topics and even to predict future stock prices!

We share the love for that dataset and consider it a perfect example of real-world web-application data. Every Tweet is a hierarchical JSON document with roughly 120 nested fields in it. It includes strings, integers, real numbers, geo coordinates, hash-tags, UTF-8 codes and other fuzzy components, making it a common accessible and fair benchmarking choice!

Indexing 300 TB of CommonCrawl Data

CommonCrawl is by far the biggest publicly available database in history. It's a dump of billions of HTML web pages scraped from all over the internet. Today that dataset is used to train the biggest Transformer Neural Networks for Natural Language Processing. GPT-3, the most famous model making headlines since 2020, was trained on just a fraction of that data!

GPT-3 looked at only 410 Billion tokens from CommonCrawl, yet its training is estimated to cost over $25 Million. It required 285,000 CPU cores, 10,000 GPUs and absurd networking bandwidth across the training cluster. By integrating UnumDB with UnumKL, we can significantly reduce training costs, simultaneously accelerating the entire process.

Nearest Neighbors Search in High-Dimensional Metric Spaces

No matter what kind of Deep Learning you do, your intermediate representations live in some high-dimensional space. Every point there is a vector, and you must be able to search for its nearest neighbours as fast as possible. That's how modern Semantic Search works at Google, Bing, Yandex, Baidu and everywhere else.

This kind of functionality is crucial for building intelligent services. That's why Facebook has made FAISS. Unfortunately, their system stores the indexes in memory and can only use GPUs for quantization, not the whole pipeline. This limits the applicability to Billion point clusters, or more realistically, Million point clusters. UnumDB doesn't have those bottlenecks!