What if transactions and analytics shared One Store?

Separating storage from compute was the defining idea of the last decade of systems design, and it earned it: capacity and compute scale on their own, object storage is durable and cheap, and a scan does not care about latency. It runs out where reads depend on each other. A transaction validates what it read, a graph query follows edges, a training loop samples at random, and every hop pays a network round trip, so systems split into an OLTP database, an OLAP warehouse and a change-data-capture pipeline that always trails the writes.

The hardware that forced the split is gone. One server holds two dozen PCIe 5.0 drives and GPUs on the same PCIe fabric, and UStore puts transactions, scans and random sampling over one copy on that box.

The LSM tree is the right structure for write-heavy storage, and RocksDB is its embedded default; UStore runs it as an engine behind the same ABI. Its compaction is where the time goes: the CPU merges and rewrites every level through the page cache, while a GPU can read the same runs straight from NVMe and merge them in parallel.

A merge of sorted runs is bulk, regular work. UStore folds large merges on GPUs, from sorted adjacency runs to JSON patches, with GPUDirect Storage reading the runs from the drives into GPU memory, io_uring and SPDK bypassing the page cache, and SIMD kits keeping point lookups on the CPU. The engine links into your process, with no server or RAID layer.

Per-collection atomicity and column families are enough while a record is one row. A user is a document, a vertex, an avatar blob and an embedding, and today those live in four systems that agree only eventually.

Blobs, documents, graphs, strings and pages share one 16-byte key with USearch embeddings, and one consistency model spans all five on the RAM, NVMe and RocksDB engines. Transactions watch touched keys and conflict rather than block; staging validates writes and hides them until every engine, log or shard is ready, then commit publishes them together.

import ustore

db = ustore.Database.open_nvme("/var/lib/ustore/")
alice, bob = 1, 2

txn = db.transaction()
txn["payments"].docs[alice] = {"amount": 10, "currency": "USD"}
txn["connections"].graph.add_edge(alice, bob)
txn["avatars"].blobs[alice] = png_bytes
txn.commit()

# one key reads every modality back
payment = db["payments"].docs[alice]
followed = list(db["connections"].graph.successors(alice))

Rust

use ustore::{jsontape, Database, Durability, Key, NVMEOptions};

let database = Database::open_nvme(&NVMEOptions::at("/var/lib/ustore/"))?;
let [alice, bob] = [1u64, 2].map(Key::from);
let payment = jsontape::parse(br#"{"amount":10,"currency":"USD"}"#)?;

let mut transaction = database.transaction()?;
transaction.collection("payments")?.docs().set(alice, &payment)?;
transaction.collection("connections")?.graph().add_edge(alice, bob)?;
transaction.collection("avatars")?.blobs().set(alice, &png_bytes)?;
transaction.commit(Durability::Buffered)?;

C++

#include <ustore/ustore.hpp>
using namespace unum::ustore;

auto db = *database_t::open_nvme({.directory = bytes_of("/var/lib/ustore/")});
auto const alice = key_from_integer(1), bob = key_from_integer(2);

auto transaction = *db.transaction();
auto payments = *transaction.find<docs_collection_t>("payments");
auto connections = *transaction.find<graph_collection_t>("connections");
auto avatars = *transaction.find("avatars");

payments[alice].assign(R"({"amount": 10, "currency": "USD"})");
connections.write_edge(edge_t {alice, bob, key_from_integer(100)});
avatars[alice].assign(png_bytes);
transaction.commit();

Columnar warehouses are right for scans and aggregates. Training is neither: it rereads a dataset in random order, so a shuffle of a petabyte is a second copy, and the metadata alone may exceed RAM.

Analytics and training read the copy transactions write. Sample keys uniformly, resolve JSONPath and regular expressions over the drawn documents on GPUs, and gather the matches into typed columns that Arrow, NumPy and UForm’s loader read without a copy.

import ustore
from ustore.table_view import Table
from ustore.graph_view import DiGraph

arena = ustore.Arena()
keys = db.sample(1024, arena, collection="pictures")
values = db.read(keys, ustore.Arena(), collection="pictures")
loss = model.train_step(values)

columns = db.docs_gather(
    product_keys, ["price", "rating"],
    [ustore.FieldType.F64, ustore.FieldType.F32], arena,
    collection="products",
)
frame = Table(db, "products").astype({"price": "float64"}).to_pandas(page=1 << 16)
graph = DiGraph(db, "follows")

Rust

let mut pictures = database.collection("pictures")?;
let mut arena = ustore::Arena::new()?;

while loss > threshold {
    let batch = pictures.sample(1024, &mut arena)?;
    loss = model.train_step(batch.keys(), batch.values().tape())?;
}

A query language is the right boundary between teams. Inside one process it formats, parses, plans and serializes on every call. UStore takes one argument struct per call and returns typed results into an arena you own.

Every modality is implemented once over the in-memory, NVMe, RocksDB and LevelDB engines. Pick one at open and keep the calls; the docs cover each engine’s limits.

import ustore

db = ustore.Database.open_nvme("/var/lib/ustore/", transport=ustore.Transport.LINUX_URING, gpu_devices=range(8))
avatar, titanic = 1, 2

txn = db.transaction()
txn["movies"].docs[avatar] = {"title": "Avatar", "year": 2009}
txn["similar"].graph.add_edge(avatar, titanic)
txn.commit(ustore.Durability.FLUSHED)

from ustore.table_view import Table
frame = Table(db, "movies").to_pandas()

Python · RAM

import ustore

db = ustore.Database.open_ram("/var/lib/ustore/", isolation=ustore.Isolation.SNAPSHOT)
avatar, titanic = 1, 2

txn = db.transaction()
txn["movies"].docs[avatar] = {"title": "Avatar", "year": 2009}
txn["similar"].graph.add_edge(avatar, titanic)
txn.commit(ustore.Durability.FLUSHED)

from ustore.table_view import Table
frame = Table(db, "movies").to_pandas()

The reference engine: ordered multi-version stores in memory, persisted as one image when it names a directory.

Python · RocksDB

import ustore

db = ustore.Database.open_rocksdb("/var/lib/ustore/")
avatar, titanic = 1, 2

txn = db.transaction()
txn["movies"].docs[avatar] = {"title": "Avatar", "year": 2009}
txn["similar"].graph.add_edge(avatar, titanic)
txn.commit(ustore.Durability.FLUSHED)

from ustore.table_view import Table
frame = Table(db, "movies").to_pandas()

RocksDB keeps its own storage format behind the same calls: its transactions read the snapshot their first access took and lock nothing until they stage, through its pessimistic Prepare.

Rust · RAM

use ustore::{jsontape, Database, Durability, Key, Isolation, RAMOptions};

let database = Database::open_ram("/var/lib/ustore/", &RAMOptions {
    isolation: Isolation::Snapshot,
    ..RAMOptions::default()
})?;
let [avatar, titanic] = [1u64, 2].map(Key::from);

let mut transaction = database.transaction()?;
transaction.collection("movies")?.docs().set(avatar, &jsontape::parse(br#"{"title":"Avatar","year":2009}"#)?)?;
transaction.collection("similar")?.graph().add_edge(avatar, titanic)?;
transaction.commit(Durability::Flushed)?;

The reference engine: ordered multi-version stores in memory, persisted as one image when it names a directory.

Rust · NVMe

use ustore::{jsontape, Database, Durability, Key, NVMEOptions, Transport};

let database = Database::open_nvme(&NVMEOptions {
    transport: Transport::LinuxUring,
    gpu_devices: &[0, 1, 2, 3, 4, 5, 6, 7],
    ..NVMEOptions::at("/var/lib/ustore/")
})?;
let [avatar, titanic] = [1u64, 2].map(Key::from);

let mut transaction = database.transaction()?;
transaction.collection("movies")?.docs().set(avatar, &jsontape::parse(br#"{"title":"Avatar","year":2009}"#)?)?;
transaction.collection("similar")?.graph().add_edge(avatar, titanic)?;
transaction.commit(Durability::Flushed)?;

Each collection is an LSM tree whose point lookups run on SIMD kits and whose compactions merge sorted runs on the GPUs it is given.

Rust · RocksDB

use ustore::{jsontape, Database, Durability, Key, RocksDBOptions};

let database = Database::open_rocksdb(&RocksDBOptions::at("/var/lib/ustore/"))?;
let [avatar, titanic] = [1u64, 2].map(Key::from);

let mut transaction = database.transaction()?;
transaction.collection("movies")?.docs().set(avatar, &jsontape::parse(br#"{"title":"Avatar","year":2009}"#)?)?;
transaction.collection("similar")?.graph().add_edge(avatar, titanic)?;
transaction.commit(Durability::Flushed)?;

RocksDB keeps its own storage format behind the same calls: its transactions read the snapshot their first access took and lock nothing until they stage, through its pessimistic Prepare.

C++ · RAM

#include <ustore/ustore.hpp>
using namespace unum::ustore;

auto db = *database_t::open_ram({.directory = bytes_of("/var/lib/ustore/"), .isolation = ustore_isolation_snapshot_k});
auto const avatar = key_from_integer(1), titanic = key_from_integer(2);

auto transaction = *db.transaction();
auto movies = *transaction.find<docs_collection_t>("movies");
auto similar = *transaction.find<graph_collection_t>("similar");
movies[avatar].assign(R"({"title": "Avatar", "year": 2009})");
similar.write_edge(edge_t {avatar, titanic, key_from_integer(100)});
transaction.commit(ustore_durability_flushed_k);

The reference engine: ordered multi-version stores in memory, persisted as one image when it names a directory.

C++ · NVMe

#include <ustore/ustore.hpp>
using namespace unum::ustore;

int const gpus[8] = {0, 1, 2, 3, 4, 5, 6, 7};
auto db = *database_t::open_nvme({
    .directory = bytes_of("/var/lib/ustore/"),
    .transport = ustore_transport_linux_uring_k,
    .gpu_devices = gpus,
    .gpu_devices_count = 8,
});
auto const avatar = key_from_integer(1), titanic = key_from_integer(2);

auto transaction = *db.transaction();
auto movies = *transaction.find<docs_collection_t>("movies");
auto similar = *transaction.find<graph_collection_t>("similar");
movies[avatar].assign(R"({"title": "Avatar", "year": 2009})");
similar.write_edge(edge_t {avatar, titanic, key_from_integer(100)});
transaction.commit(ustore_durability_flushed_k);

Each collection is an LSM tree whose point lookups run on SIMD kits and whose compactions merge sorted runs on the GPUs it is given.

C++ · RocksDB

#include <ustore/ustore.hpp>
using namespace unum::ustore;

auto db = *database_t::open_rocksdb({.directory = bytes_of("/var/lib/ustore/")});
auto const avatar = key_from_integer(1), titanic = key_from_integer(2);

auto transaction = *db.transaction();
auto movies = *transaction.find<docs_collection_t>("movies");
auto similar = *transaction.find<graph_collection_t>("similar");
movies[avatar].assign(R"({"title": "Avatar", "year": 2009})");
similar.write_edge(edge_t {avatar, titanic, key_from_integer(100)});
transaction.commit(ustore_durability_flushed_k);

RocksDB keeps its own storage format behind the same calls: its transactions read the snapshot their first access took and lock nothing until they stage, through its pessimistic Prepare.