Configuring the RocksDB Backend

Overview

RocksDB is an embedded LSM-tree key-value store. With the rocksdb backend, HugeGraph-Server keeps all graph data in RocksDB instances that live inside the server process, so there is no separate storage service to deploy. This is the backend used by the shipped conf/graphs/hugegraph.properties.

Since version 1.7.0 the server accepts only memory, rocksdb, hbase and hstore as the backend. The rocksdb backend stores data on the local disks of one server: it does not support shared storage, so a graph cannot be served by several servers over the same data directory. For a distributed deployment use the hstore backend with PD and Store.

The RocksDB JNI library is pinned to version 8.10.2 by hugegraph-rocksdb/pom.xml, so the on-disk format and the option semantics are those of RocksDB 8.10.

The backend driver version reported by this store is 1.11, and it is written into the meta table of the system store when the graph is initialized.

Selecting the backend

Set the backend and the serializer in the graph properties file (conf/graphs/<graph>.properties):

gremlin.graph=org.apache.hugegraph.HugeFactory

backend=rocksdb
serializer=binary

store=hugegraph

# rocksdb backend config
#rocksdb.data_path=/path/to/disk
#rocksdb.wal_path=/path/to/disk

• backend=rocksdb selects the RocksDB store provider.
• serializer=binary is the serializer the shipped template uses for this backend. The built-in serializers are binary, binaryscatter and text.
• store is the database namespace of the graph, and it is also part of the graph name that the provider passes down to the store.

Run bin/init-store.sh once before the first start to create the stores, then start the server. Both bin/init-store.sh and bin/hugegraph-server.sh load the RocksDB library, so the data directories are created on the machine that runs them.

The distribution registers the option space and the store provider for each backend listed in the packaged backend.properties, whose value comes from the hugegraph.backends build property. A default build registers rocksdb, hbase, hstore; building with -Drocksdb-only activates the rocksdb-only profile and produces a distribution that registers only rocksdb. A backend that is not registered fails at startup with Not exists BackendStoreProvider.

The provider registration also adds a second name, rocksdbsst, for the store that writes SST files instead of a live database. That name is not in the list of allowed backends, so backend=rocksdbsst is rejected with backend is illegal: rocksdbsst. To load SST files into a normal rocksdb graph, use rocksdb.sst_path as described below.

Data directory layout

Two directories matter: rocksdb.data_path (default rocksdb-data/data) and rocksdb.wal_path (default rocksdb-data/wal). Relative paths resolve against the working directory of the server, which is the installation directory.

Each graph opens three stores: m for schema, g for graph data, and s for the system store. The store name is appended to both configured paths, so a default single-graph installation looks like this:

rocksdb-data/
  data/
    m/    # schema store: property keys, vertex/edge/index labels, counters
    g/    # graph store: vertices, edges, index tables, olap tables
    s/    # system store: tasks, server info, backend meta (driver version)
  wal/
    m/
    g/
    s/

Every backend table becomes a RocksDB column family inside the store it belongs to, named <database>+<table>, where the database is derived from the graph name. Column families of existing data directories are always reopened, so tables created by an older version stay readable.

Other points to keep in mind:

• Two graphs must not share a data path. When a graph is created by cloning an existing configuration through the API, the provider appends _<newGraph> to both rocksdb.data_path and rocksdb.wal_path. Deleting such a graph deletes both directories.
• Snapshots are created beside the data directory: the last two segments of the data path are rewritten with a prefix, so with the default paths the snapshot of the graph store goes to rocksdb-data/<prefix>_data/g. Resuming a snapshot closes the instance, deletes the data directory and moves the snapshot into its place.
• With rocksdb.data_disks set, the tables named there are opened as separate RocksDB instances under the given paths instead of under rocksdb.data_path. The server opens up to 8 instances in parallel, waits at most 600 seconds for the open to finish and 30 seconds for sessions to close.

Path and log options

Compaction and compression options

Database level options

Memtable options

Level sizing and write stall options

File I/O options

SST table format and block cache options

Bloom filter options

The options in this group are read only when rocksdb.bloom_filter_bits_per_key is 0 or greater. With the default value of -1 there is no bloom filter and none of the other options in this table take effect, including the index and filter block caching ones.

How the options are applied

The server builds the RocksDB option objects once per store and per column family, so a change to any of the options above takes effect on the next server start.

• rocksdb.optimize_mode=true applies presets before the values in the tables above: at the database level it raises parallelism to half of the available processors (at least one), allows concurrent memtable writes and enables the write thread adaptive yield; at the column family level it calls the RocksDB level-style and universal-style compaction presets. The explicit options are applied afterwards, so any value you set in the properties file wins over the preset.
• rocksdb.bulkload_mode=true disables automatic compaction, raises the three level-0 triggers to the maximum integer and the two pending compaction limits to the maximum long value. Turn it off and restart after the load, otherwise compaction never runs.
• rocksdb.block_cache_capacity=0 turns the block cache off completely rather than making it unbounded.
• rocksdb.prefix_extractor_n_bytes greater than 0 installs a capped prefix extractor of that length.
• Every column family uses the uint64add merge operator, which is what the counter table relies on.
• The database is created if it is missing, and avoid_unnecessary_blocking_io and write_dbid_to_manifest are always on.

Memory notes

The caches and write buffers of RocksDB are native allocations, so they are not part of the JVM heap sizing in bin/hugegraph-server.sh. The GET /metrics/backend endpoint reports what the store uses: the memory number is the sum of the block cache usage, the pinned block cache usage, the estimated table reader memory (index and filter blocks) and the size of all memtables, taken from the RocksDB properties of every open column family.

Two option values multiply with the number of column families:

• rocksdb.block_cache_capacity creates one cache instance per column family, so the total block cache of a server is roughly this value times the number of open tables across the m, g and s stores of every graph, plus the instances opened for rocksdb.data_disks.
• rocksdb.write_buffer_size times rocksdb.max_write_buffer_number bounds the memtable memory of one column family. rocksdb.db_write_buffer_size caps the total across all column families of one store, and its default of 0 means there is no such cap.

rocksdb.row_cache_capacity is different: it is one cache per store, and 0 disables it.

Ingesting SST files

Setting rocksdb.sst_path turns on ingestion. When a store is opened, and again whenever tables are created, the server walks <sst_path>/<column family>/, collects every non-empty *.sst file below it and ingests those files into the matching column family. The files are moved rather than copied, so the source directory is consumed by the ingestion.

Raft mode

The RocksDB backend can still run behind the raft state machine: with raft.mode=true the store provider of any local backend is wrapped by the raft provider. The wrapper rejects backends with shared storage, so rocksdb is accepted while hbase is not. Under raft mode a RocksDB session writes with the WAL disabled and without sync, because the state machine can restore from a snapshot plus the raft log, and snapshots are supported by this backend.

Notes for anyone using it:

• bin/init-store.sh forces raft.mode=false while it initializes the backend, so initialization never goes through raft.
• The shipped conf/graphs/hugegraph.properties marks the raft options as deprecated. Distributed deployments of 1.7.0 and later use the hstore backend with PD and Store instead.
• The raft peer endpoints are served under graphspaces/{graphspace}/graphs/{graph}/raft/, with list_peers, get_leader, set_leader, transfer_leader, add_peer and remove_peer. bin/raft-tools.sh wraps the same operations, but it still builds URLs without the graphspace segment, so the path has to be adjusted for a 1.7.0 server.
• The remaining raft.* options are listed in the Server Complete Configuration Manual.

Backend capabilities

The feature flags of this backend affect what the server can push down to the store:

• Scans by key prefix and by key range, paged queries, range conditions and order-by are supported.
• There is no index inside RocksDB, so querying schema by name, querying by label and deleting edges by label are done by the server instead of the store.
• Transactions are supported through RocksDB write batches.
• Snapshots are supported, which is what raft mode and backup rely on.
• Shared storage is not supported, so one data directory belongs to one server.
• Olap properties are supported, and their tables are created as extra column families.
• The store does not expire data by itself, so the server filters out elements whose TTL has passed when it reads them.
• in, contains and contains_key conditions, aggregate properties and vertex or edge property updates in place are not supported at the store level.

Platform note for riscv64

On Linux riscv64 the RocksDB JNI library needs libatomic.so.1. bin/util.sh looks for it and adds it to LD_PRELOAD before bin/hugegraph-server.sh, bin/init-store.sh and bin/dump-store.sh start the JVM. If it is missing, those scripts stop with RISC-V RocksDB requires libatomic.so.1; install libatomic1, and installing the libatomic1 package fixes it.