Skip to content

1 - Server Startup Guide

1 Overview

The directory for the configuration files is hugegraph-release/conf, and all the configurations related to the service and the graph itself are located in this directory.

The main configuration files include gremlin-server.yaml, rest-server.properties, and hugegraph.properties.

The HugeGraphServer integrates the GremlinServer and RestServer internally, and gremlin-server.yaml and rest-server.properties are used to configure these two servers.

  • GremlinServer: GremlinServer accepts Gremlin requests and invokes the graph engine.
  • RestServer: It provides a RESTful API that, based on different HTTP requests, calls the corresponding Core API. If the user’s request body is a Gremlin statement, it will be forwarded to GremlinServer to perform operations on the graph data.

Now let’s introduce these three configuration files one by one.

2. gremlin-server.yaml

The main structure of gremlin-server.yaml is shown below. Some imports are omitted from this example; refer to the file included in the release package for the complete content.

# host and port of gremlin server, need to be consistent with host and port in rest-server.properties
#host: 127.0.0.1
#port: 8182

# timeout in ms of gremlin query
evaluationTimeout: 30000

channelizer: org.apache.tinkerpop.gremlin.server.channel.WsAndHttpChannelizer
# don't set graph at here, this happens after support for dynamically adding graph
graphs: {
}
scriptEngines: {
  gremlin-groovy: {
    staticImports: [
      org.opencypher.gremlin.process.traversal.CustomPredicates.*',
      org.opencypher.gremlin.traversal.CustomFunctions.*
    ],
    plugins: {
      org.apache.hugegraph.plugin.HugeGraphGremlinPlugin: {},
      org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},
      org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {
        classImports: [
          java.lang.Math,
          org.apache.hugegraph.backend.id.IdGenerator,
          org.apache.hugegraph.type.define.Directions,
          org.apache.hugegraph.type.define.NodeRole,
          org.apache.hugegraph.masterelection.GlobalMasterInfo,
          org.apache.hugegraph.util.DateUtil,
          org.apache.hugegraph.traversal.algorithm.CollectionPathsTraverser,
          org.apache.hugegraph.traversal.algorithm.CountTraverser,
          org.apache.hugegraph.traversal.algorithm.CustomizedCrosspointsTraverser,
          org.apache.hugegraph.traversal.algorithm.CustomizePathsTraverser,
          org.apache.hugegraph.traversal.algorithm.FusiformSimilarityTraverser,
          org.apache.hugegraph.traversal.algorithm.HugeTraverser,
          org.apache.hugegraph.traversal.algorithm.JaccardSimilarTraverser,
          org.apache.hugegraph.traversal.algorithm.KneighborTraverser,
          org.apache.hugegraph.traversal.algorithm.KoutTraverser,
          org.apache.hugegraph.traversal.algorithm.MultiNodeShortestPathTraverser,
          org.apache.hugegraph.traversal.algorithm.NeighborRankTraverser,
          org.apache.hugegraph.traversal.algorithm.PathsTraverser,
          org.apache.hugegraph.traversal.algorithm.PersonalRankTraverser,
          org.apache.hugegraph.traversal.algorithm.SameNeighborTraverser,
          org.apache.hugegraph.traversal.algorithm.ShortestPathTraverser,
          org.apache.hugegraph.traversal.algorithm.SingleSourceShortestPathTraverser,
          org.apache.hugegraph.traversal.algorithm.SubGraphTraverser,
          org.apache.hugegraph.traversal.algorithm.TemplatePathsTraverser,
          org.apache.hugegraph.traversal.algorithm.steps.EdgeStep,
          org.apache.hugegraph.traversal.algorithm.steps.RepeatEdgeStep,
          org.apache.hugegraph.traversal.algorithm.steps.WeightedEdgeStep,
          org.apache.hugegraph.traversal.optimize.ConditionP,
          org.apache.hugegraph.traversal.optimize.Text,
          org.apache.hugegraph.traversal.optimize.TraversalUtil,
          org.opencypher.gremlin.traversal.CustomFunctions,
          org.opencypher.gremlin.traversal.CustomPredicate
        ],
        methodImports: [
          java.lang.Math#*,
          org.opencypher.gremlin.traversal.CustomPredicate#*,
          org.opencypher.gremlin.traversal.CustomFunctions#*
        ]
      },
      org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {
        files: [scripts/empty-sample.groovy]
      }
    }
  }
}
serializers:
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1,
      config: {
        serializeResultToString: false,
        ioRegistries: [org.apache.hugegraph.io.HugeGraphIoRegistry]
      }
  }
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0,
      config: {
        serializeResultToString: false,
        ioRegistries: [org.apache.hugegraph.io.HugeGraphIoRegistry]
      }
  }
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV2d0,
      config: {
        serializeResultToString: false,
        ioRegistries: [org.apache.hugegraph.io.HugeGraphIoRegistry]
      }
  }
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0,
      config: {
        serializeResultToString: false,
        ioRegistries: [org.apache.hugegraph.io.HugeGraphIoRegistry]
      }
  }
metrics: {
  consoleReporter: {enabled: false, interval: 180000},
  csvReporter: {enabled: false, interval: 180000, fileName: ./metrics/gremlin-server-metrics.csv},
  jmxReporter: {enabled: false},
  slf4jReporter: {enabled: false, interval: 180000},
  gangliaReporter: {enabled: false, interval: 180000, addressingMode: MULTICAST},
  graphiteReporter: {enabled: false, interval: 180000}
}
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 65536
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536
ssl: {
  enabled: false
}

In most cases, you only need to pay attention to channelizer, host, and port. Graphs are not loaded from the Gremlin Server graphs section. Whether local graph configurations are loaded is controlled by graph.load_from_local_config in rest-server.properties.

  • channelizer: The default WsAndHttpChannelizer supports both WebSocket and HTTP. Gremlin Console uses WebSocket, while HugeGraph Client, Loader, and Hubble use HTTP.

By default, the GremlinServer serves at 127.0.0.1:8182. If you need to modify it, configure the host and port settings.

  • host: The hostname or IP address of the machine where the GremlinServer is deployed. GremlinServer is not directly exposed to users, the RestServer forwards Gremlin requests to it.
  • port: The port number of the machine where the GremlinServer is deployed.

Additionally, you need to add the corresponding configuration gremlinserver.url=http://host:port in rest-server.properties.

3. rest-server.properties

The following is an example of the available rest-server.properties options. The current upstream release template does not include graph.load_from_local_config, whose source-code default is false; set it explicitly to true when using local graph configurations under conf/graphs.

# bind url
# could use '0.0.0.0' or specified (real)IP to expose external network access
restserver.url=http://127.0.0.1:8080
#restserver.enable_graphspaces_filter=false
# gremlin server url, need to be consistent with host and port in gremlin-server.yaml
#gremlinserver.url=127.0.0.1:8182

graphs=./conf/graphs
graph.load_from_local_config=true

# The maximum thread ratio for batch writing, only take effect if the batch.max_write_threads is 0
batch.max_write_ratio=80
batch.max_write_threads=0

# configuration of arthas
arthas.telnetPort=8562
arthas.httpPort=8561
arthas.ip=127.0.0.1
arthas.disabledCommands=jad

# authentication configs
#auth.authenticator=org.apache.hugegraph.auth.StandardAuthenticator
# for admin password, By default, it is pa and takes effect upon the first startup
#auth.admin_pa=pa
#auth.graph_store=hugegraph

# use pd
# usePD=true

# slow query log
log.slow_query_threshold=1000
# bytes of request body recorded as-is (may contain sensitive literals), 0 to disable
log.slow_query_body_limit=512

# jvm(in-heap) memory usage monitor, set 1 to disable it
memory_monitor.threshold=0.85
memory_monitor.period=2000
  • restserver.url: The URL at which the RestServer provides its services. Modify it according to the actual environment. If you can’t connet to server from other IP address, try to modify it as specific IP; or modify it as http://0.0.0.0 to listen all network interfaces as a convenient solution, but need to take care of the network area that might access.
  • graphs: The directory containing graph configuration files. The default is ./conf/graphs. init-store scans this directory; the Server loads its properties files only when graph.load_from_local_config=true.
  • graph.load_from_local_config: Whether the Server reads local graph configurations at startup. Its default value in the source code is false.

The current upstream template still uses arthas.telnet_port, arthas.http_port, and arthas.disabled_commands, but ServerOptions reads the camelCase names shown in the example above. Custom configurations should use arthas.telnetPort, arthas.httpPort, and arthas.disabledCommands.

The gremlinserver.url configuration option is the URL at which the GremlinServer provides services to the RestServer. By default, it is set to http://127.0.0.1:8182. If you need to modify it, it should match the host and port settings in gremlin-server.yaml. The value may omit the scheme, as the template does, because http:// is prepended when it is missing.

4. hugegraph.properties

hugegraph.properties is a type of file. If the system has multiple graphs, there will be multiple similar files. This file is used to configure parameters related to graph storage and querying. The default content of the file is as follows:

# gremlin entrance to create graph
# auth config: org.apache.hugegraph.auth.HugeFactoryAuthProxy
gremlin.graph=org.apache.hugegraph.HugeFactory

# cache config
#schema.cache_capacity=100000
# vertex-cache default is 1000w, 10min expired
vertex.cache_type=l2
#vertex.cache_capacity=10000000
#vertex.cache_expire=600
# edge-cache default is 100w, 10min expired
edge.cache_type=l2
#edge.cache_capacity=1000000
#edge.cache_expire=600


# schema illegal name template
#schema.illegal_name_regex=\s+|~.*

#vertex.default_label=vertex

# NOTE: since 1.7.0, only hstore, rocksdb, hbase, memory are supported for backend.
# if you want to use Cassandra/MySql/PG... as backend, please use version < 1.7.0
backend=rocksdb
serializer=binary
# The process-wide max capacity of one serialization buffer in bytes
#serializer.buffer_max_capacity=134217728

store=hugegraph

# pd config
#pd.peers=127.0.0.1:8686

# task config
task.schedule_period=10
task.retry=0
task.wait_timeout=10

# search config
search.text_analyzer=jieba
search.text_analyzer_mode=INDEX

# rocksdb backend config
#rocksdb.data_path=/path/to/disk
#rocksdb.wal_path=/path/to/disk

# hbase backend config
#hbase.hosts=localhost
#hbase.port=2181
#hbase.znode_parent=/hbase
#hbase.threads_max=64
# IMPORTANT: recommend to modify the HBase partition number
#            by the actual/env data amount & RS amount before init store
#            It will influence the load speed a lot
#hbase.enable_partition=true
#hbase.vertex_partitions=10
#hbase.edge_partitions=30

# WARNING: These raft configurations are deprecated, please use the latest version instead.
# raft.mode=false

# memory management config
#memory.mode=off-heap
#memory.max_capacity=1073741824
#memory.one_query_max_capacity=104857600
#memory.alignment=8

Pay attention to the following uncommented items:

  • gremlin.graph: The entry point for GremlinServer startup. Users should not modify this item, except to switch it to org.apache.hugegraph.auth.HugeFactoryAuthProxy when authentication is enabled.
  • vertex.cache_type / edge.cache_type: The cache implementation, allowed values are l1 and l2. The default is l2.
  • backend: The storage backend. Version 1.7.0 supports memory, rocksdb, hstore, and hbase.
  • serializer: The serializer used when writing schemas, vertices, and edges to the backend. RocksDB uses binary.
  • store: The storage name used by the graph in the backend.
  • task.schedule_period, task.retry, task.wait_timeout: Scheduling period (in seconds), retry count, and wait timeout (in seconds) for asynchronous tasks. The scheduler itself is picked from the backend, hstore uses the distributed scheduler and every other backend uses the local one. The old task.scheduler_type key is ignored.
  • search.text_analyzer / search.text_analyzer_mode: The analyzer used for full-text indexes and its mode. Available analyzers are ansj, hanlp, smartcn, jieba, jcseg, mmseg4j, and ikanalyzer, and each one accepts its own set of modes.
  • rocksdb.data_path: This item is only meaningful when the backend is set to rocksdb. It specifies the data directory for RocksDB, and defaults to rocksdb-data/data.
  • rocksdb.wal_path: This item is only meaningful when the backend is set to rocksdb. It specifies the log directory for RocksDB, and defaults to rocksdb-data/wal.

5. Multi-Graph Configuration

A Server can load multiple graphs, with a separate properties file for each graph. The following example creates a RocksDB graph named hugegraph_rocksdb and an in-memory graph named hugegraph_memory.

[Optional]: Modify rest-server.properties

You can modify the graph profile directory in the graphs option of rest-server.properties. The default configuration is graphs=./conf/graphs, if you want to change it to another directory then adjust the graphs option, e.g. adjust it to graphs=/etc/hugegraph/graphs, example is as follows:

graphs=./conf/graphs
graph.load_from_local_config=true

Under conf/graphs, create hugegraph_memory.properties and hugegraph_rocksdb.properties based on hugegraph.properties.

Configure hugegraph_memory.properties as follows:

backend=memory
serializer=text
store=hugegraph_memory

Configure hugegraph_rocksdb.properties as follows:

backend=rocksdb
serializer=binary

store=hugegraph_rocksdb

Stop the server, execute init-store.sh (to create a new database for the new graph), and restart the server.

$ ./bin/stop-hugegraph.sh
$ ./bin/init-store.sh

Initializing HugeGraph Store...
2023-06-11 14:16:14 [main] [INFO] o.a.h.u.ConfigUtil - Scanning option 'graphs' directory './conf/graphs'
2023-06-11 14:16:14 [main] [INFO] o.a.h.c.InitStore - Init graph with config file: ./conf/graphs/hugegraph_rocksdb.properties
...
2023-06-11 14:16:15 [main] [INFO] o.a.h.StandardHugeGraph - Graph 'hugegraph_rocksdb' has been initialized
2023-06-11 14:16:15 [main] [INFO] o.a.h.c.InitStore - Init graph with config file: ./conf/graphs/hugegraph_memory.properties
...
2023-06-11 14:16:16 [main] [INFO] o.a.h.StandardHugeGraph - Graph 'hugegraph_memory' has been initialized
2023-06-11 14:16:16 [main] [INFO] o.a.h.StandardHugeGraph - Close graph standardhugegraph[hugegraph_rocksdb]
...
2023-06-11 14:16:16 [main] [INFO] o.a.h.HugeFactory - HugeFactory shutdown
2023-06-11 14:16:16 [hugegraph-shutdown] [INFO] o.a.h.HugeFactory - HugeGraph is shutting down
Initialization finished.
$ ./bin/start-hugegraph.sh

Starting HugeGraphServer in daemon mode...
Connecting to HugeGraphServer (http://127.0.0.1:8080/graphs)...OK
Started [pid 21614]

Check out created graphs:

curl http://127.0.0.1:8080/graphspaces/DEFAULT/graphs

{"graphs":["hugegraph_rocksdb","hugegraph_memory"]}

Get details of a graph:

curl http://127.0.0.1:8080/graphspaces/DEFAULT/graphs/hugegraph_memory

{"name":"hugegraph_memory","backend":"memory"}
curl http://127.0.0.1:8080/graphspaces/DEFAULT/graphs/hugegraph_rocksdb

{"name":"hugegraph_rocksdb","backend":"rocksdb"}

2 - Server Complete Configuration Manual

Gremlin Server Config Options

Corresponding configuration file gremlin-server.yaml

config optiondefault valuedescription
host127.0.0.1The host or ip of Gremlin Server.
port8182The listening port of Gremlin Server.
graphs{}Graphs are loaded dynamically by the Server; do not configure them here.
evaluationTimeout30000Gremlin script evaluation timeout in milliseconds.
channelizerorg.apache.tinkerpop.gremlin.server.channel.WsAndHttpChannelizerHandles both WebSocket and HTTP requests.
maxContentLength65536Maximum size in bytes of a request that the server accepts.
maxChunkSize8192Maximum chunk size in bytes of an HTTP request.
maxHeaderSize8192Maximum size in bytes of the HTTP request headers.
resultIterationBatchSize64Number of results returned per batch when streaming a result set.
ssl.enabledfalseWhether Gremlin Server serves over TLS.
authenticationNot configuredWhen enabling authentication, configure the authenticator, handler, and path to rest-server.properties.

Rest Server & API Config Options

Corresponding configuration file rest-server.properties

config optiondefault valuedescription
graphs./conf/graphsDirectory containing graph configuration properties files.
graph.load_from_local_configfalseWhether to read the graphs directory when the Server starts; set to true when using local graph configuration.
graphs.enable_dynamic_create_droptrueWhether to enable create or drop graph dynamically.
init_store.enabledtrueWhether init-store initializes the local backend stores and the built-in admin account. Set false in distributed deployments (PD/HStore) where the storage side already owns the metadata.
server.idEmpty stringThe optional legacy id of hugegraph-server.
server.rolemasterThe role of nodes in the cluster, available types are [master, worker, computer]
server.role_electionfalseWhether to enable role election, if enabled, the server will elect a master node in the cluster.
server.node_idnode-id1The node id of the server.
server.node_roleworkerThe node role of the server.
server.graphspaceDEFAULTThe graph space of the server.
server.service_idDEFAULTThe service id of the server.
server.path_graphspaceDEFAULTThe default path graph space of the server.
server.start_ignore_single_graph_errortrueWhether to start ignore single graph error.
server.event_hub_threads1The event hub threads of server.
restserver.urlhttp://127.0.0.1:8080The url for listening of graph server.
ssl.keystore_fileconf/hugegraph-server.keystoreThe path of server keystore file used when https protocol is enabled.
ssl.keystore_passwordhugegraphThe password of the server keystore file when the https protocol is enabled.
white_ip.statusdisableThe status of whether enable white ip.
restserver.max_worker_threads2 * CPUsThe maximum worker threads of rest server.
restserver.task_threadsmax(4, CPUs / 2)The task threads of rest server.
restserver.min_free_memory64The minimum free memory(MB) of rest server, requests will be rejected when the available memory of system is lower than this value.
restserver.request_timeout30The time in seconds within which a request must complete, -1 means no timeout.
restserver.connection_idle_timeout30The time in seconds to keep an inactive connection alive, -1 means no timeout.
restserver.connection_max_requests256The max number of HTTP requests allowed to be processed on one keep-alive connection, -1 means unlimited.
gremlinserver.urlhttp://127.0.0.1:8182The url of gremlin server.
gremlinserver.max_route2 * CPUsThe max route number for gremlin server.
gremlinserver.timeout30The timeout in seconds of waiting for gremlin server.
batch.max_edges_per_batch2500The maximum number of edges submitted per batch.
batch.max_vertices_per_batch2500The maximum number of vertices submitted per batch.
batch.max_write_ratio70The maximum thread ratio for batch writing, only take effect if the batch.max_write_threads is 0.
batch.max_write_threads0The maximum threads for batch writing, if the value is 0, the actual value will be set to batch.max_write_ratio * restserver.max_worker_threads.
raft.group_peers127.0.0.1:8090The rpc address of raft group initial peers.
auth.authenticatorThe class path of authenticator implementation. e.g., org.apache.hugegraph.auth.StandardAuthenticator, or a custom implementation.
auth.graph_storehugegraphThe name of graph used to store authentication information, like users, only for org.apache.hugegraph.auth.StandardAuthenticator.
auth.admin_papaThe default password for built-in admin account, takes effect on first startup. It must be changed before deployment.
auth.audit_log_rate1000.0The max rate of audit log output per user, default value is 1000 records per second.
auth.cache_capacity10240The max cache capacity of each auth cache item.
auth.cache_expire600The expiration time in seconds of auth cache in auth client and auth server.
auth.remote_urlIf the address is empty, it provide auth service, otherwise it is auth client and also provide auth service through rpc forwarding. The remote url can be set to multiple addresses, which are concat by ‘,’.
auth.token_expire86400The expiration time in seconds after token created
auth.token_secretRandomly generated at startupHS256 secret; configure it explicitly if existing tokens must remain valid across restarts.
exception.allow_tracetrueWhether to allow exception trace stack.
memory_monitor.threshold0.85Threshold for JVM memory usage monitoring, 1 means disabling the memory monitoring task.
memory_monitor.period2000The period in ms of JVM memory usage monitoring, in each period we will detect the jvm memory usage and take corresponding actions.
log.slow_query_threshold1000The threshold time(ms) of logging slow query, 0 means logging slow query is disabled.
log.slow_query_body_limit512The max bytes of request body recorded in the slow query log, 0 means the body is not recorded. The recorded prefix is written as-is and may contain sensitive Gremlin or Cypher literals.
Role Election Config Options (Optional)

Corresponding configuration file rest-server.properties, only used when server.role_election=true.

config optiondefault valuedescription
server.role.node_external_urlhttp://127.0.0.1:8080The url of external accessibility.
server.role.base_timeout500The role state machine candidate state base timeout time, in ms.
server.role.random_timeout1000The random timeout in ms that be used when candidate node request to become master state to reduce competitive voting.
server.role.heartbeat_interval2The role state machine heartbeat interval second time.
server.role.fail_count5When the node failed count of update or query heartbeat is reaches this threshold, the node will become abdication state to guardsafe property.
server.role.master_dead_times10When the worker node detects that the number of times the master node fails to update heartbeat reaches this threshold, the worker node will become to a candidate node.

PD/Meta Config Options (Distributed Mode)

Corresponding configuration file rest-server.properties

config optiondefault valuedescription
usePDfalseWhether use pd.
pd.peers127.0.0.1:8686The pd server peers, separated with commas.
clusterhg-testThe cluster name.
metrics.data_to_pdtrueWhether to report metrics data to pd.
meta.endpointshttp://127.0.0.1:2379The URL of meta endpoints. No code reads this option, so setting it has no effect; the meta connection is built from pd.peers.
meta.use_cafalseWhether to use ca to meta server.
meta.caThe ca file of meta server.
meta.client_caThe client ca file of meta server.
meta.client_keyThe client key file of meta server.

The HStore backend also reads two options from the graph configuration file {graph-name}.properties. Both default to 0, which means the value is decided by PD:

config optiondefault valuedescription
hstore.partition_count0Number of partitions, which PD controls partitions based on.
hstore.shard_count0Number of copies, which PD controls partition copies based on.

Basic Config Options

Basic Config Options and Backend Config Options correspond to configuration files:{graph-name}.properties, such as hugegraph.properties

config optiondefault valuedescription
gremlin.graphorg.apache.hugegraph.HugeFactoryGremlin entrance to create graph.
backendmemoryThe data store type. For version 1.7.0+ the allowed values are [memory, rocksdb, hstore, hbase]; the shipped conf/graphs/hugegraph.properties sets rocksdb and conf/graphs/hstore.properties.template sets hstore. Note: cassandra, scylladb, mysql, postgresql were removed in 1.7.0 (use <= 1.5.x for legacy backends).
serializertextThe serializer for backend store, built-in values are [text, binary, binaryscatter]; a backend may register its own, like hbase. The shipped graph templates set binary.
serializer.buffer_max_capacity134217728The process-wide max capacity of one serialization buffer in bytes.
storehugegraphThe backend database namespace.
store.connection_detect_interval600The interval in seconds for detecting connections, if the idle time of a connection exceeds this value, detect it and reconnect if needed before using, value 0 means detecting every time.
store.graphgThe graph table name, which store vertex, edge and property.
graphspaceDEFAULTThe graph space name.
alias.graph.idThe graph alias id.
graph.read_modeOLTP_ONLYThe graph read mode, which could be ALL | OLTP_ONLY | OLAP_ONLY.
pd.peers127.0.0.1:8686The addresses of pd nodes, separated with commas. Only used by the hstore backend.
schema.illegal_name_regex.\s+$|~.The regex specified the illegal format for schema name.
schema.cache_capacity10000The max cache size(items) of schema cache.
schema.init_templateThe template schema used to init graph.
schema.index_rebuild_using_pushdowntrueWhether to use pushdown when to create/rebuild index.
vertex.cache_typel2The type of vertex cache, allowed values are [l1, l2].
vertex.cache_capacity10000000The max cache size(items) of vertex cache.
vertex.cache_expire600The expiration time in seconds of vertex cache.
vertex.check_customized_id_existfalseWhether to check the vertices exist for those using customized id strategy.
vertex.default_labelvertexThe default vertex label.
vertex.tx_capacity10000The max size(items) of vertices(uncommitted) in transaction.
vertex.check_adjacent_vertex_existfalseWhether to check the adjacent vertices of edges exist.
vertex.lazy_load_adjacent_vertextrueWhether to lazy load adjacent vertices of edges.
vertex.part_edge_commit_size5000Whether to enable the mode to commit part of edges of vertex, enabled if commit size > 0, 0 means disabled.
vertex.encode_primary_key_numbertrueWhether to encode number value of primary key in vertex id.
vertex.remove_left_index_at_overwritefalseWhether remove left index at overwrite.
edge.cache_typel2The type of edge cache, allowed values are [l1, l2].
edge.cache_capacity1000000The max cache size(items) of edge cache.
edge.cache_expire600The expiration time in seconds of edge cache.
edge.tx_capacity10000The max size(items) of edges(uncommitted) in transaction.
query.page_size500The size of each page when querying by paging.
query.batch_size1000The size of each batch when querying by batch.
query.ignore_invalid_datatrueWhether to ignore invalid data of vertex or edge.
query.index_intersect_threshold1000The maximum number of intermediate results to intersect indexes when querying by multiple single index properties.
query.max_indexes_available1The upper limit of the number of indexes that can be used to query.
query.dedup_optionlimitThe way to dedup data, allowed values are [limit, global].
query.trust_indexfalseWhether to trust index.
query.ramtable_edges_capacity20000000The maximum number of edges in ramtable, include OUT and IN edges.
query.ramtable_enablefalseWhether to enable ramtable for query of adjacent edges.
query.ramtable_vertices_capacity10000000The maximum number of vertices in ramtable, generally the largest vertex id is used as capacity.
query.optimize_aggregate_by_indexfalseWhether to optimize aggregate query(like count) by index.
oltp.concurrent_depth10The min depth to enable concurrent oltp algorithm.
oltp.concurrent_threadsmax(10, CPUs / 2)Thread number to concurrently execute oltp algorithm.
oltp.collection_typeECThe implementation type of collections used in oltp algorithm, allowed values are [JCF, EC, FU].
oltp.query_batch_size10000The size of each batch when executing oltp algorithm.
oltp.query_batch_avg_degree_ratio0.95The ratio of exponential approximation for average degree of iterator when executing oltp algorithm.
oltp.query_batch_expect_degree100000000The expect sum of degree in each batch when executing oltp algorithm.
rate_limit.read0The max rate(times/s) to execute query of vertices/edges.
rate_limit.write0The max rate(items/s) to add/update/delete vertices/edges.
task.schedule_period10Period time in seconds when scheduler to schedule task.
task.wait_timeout10Timeout in seconds for waiting for the task to complete, such as when truncating or clearing the backend.
task.retry0Task retry times, allowed range is [0, 3].
task.input_size_limit16777216The job input size limit in bytes.
task.result_size_limit16777216The job result size limit in bytes.
task.sync_deletionfalseWhether to delete schema or expired data synchronously.
task.ttl_delete_batch1The batch size used to delete expired data.
computer.config./conf/computer.yamlThe config file path of computer job.
k8s.operator_template./conf/operator-template.yamlThe path of operator container template.
k8s.quota_template./conf/resource-quota-template.yamlThe path of resource quota template.
search.text_analyzerikanalyzerChoose a text analyzer for searching the vertex/edge properties, available type are [ansj, hanlp, smartcn, jieba, jcseg, mmseg4j, ikanalyzer]. The shipped graph templates set jieba. If use ‘ikanalyzer’, need download jar from ‘https://github.com/apache/hugegraph-doc/raw/ik_binary/dist/server/ikanalyzer-2012_u6.jar' to lib directory
search.text_analyzer_modesmartSpecify the mode for the text analyzer, the available mode of analyzer are {ansj: [BaseAnalysis, IndexAnalysis, ToAnalysis, NlpAnalysis], hanlp: [standard, nlp, index, nShort, shortest, speed], smartcn: [], jieba: [SEARCH, INDEX], jcseg: [Simple, Complex], mmseg4j: [Simple, Complex, MaxWord], ikanalyzer: [smart, max_word]}.
snowflake.datacenter_id0The datacenter id of snowflake id generator.
snowflake.force_stringfalseWhether to force the snowflake long id to be a string.
snowflake.worker_id0The worker id of snowflake id generator.
memory.modeoff-heapThe memory mode used for query in HugeGraph.
memory.max_capacity1073741824The maximum memory capacity in bytes that can be managed for all queries in HugeGraph.
memory.one_query_max_capacity104857600The maximum memory capacity in bytes that can be managed for a query in HugeGraph.
memory.alignment8The alignment used for round memory size.
Raft Config Options (Deprecated)

The shipped graph configuration templates mark these options as deprecated. They only take effect when raft.mode=true, and raft.group_peers is read from rest-server.properties instead of the graph file.

config optiondefault valuedescription
raft.modefalseWhether the backend storage works in raft mode.
raft.safe_readfalseWhether to use linearly consistent read.
raft.path./raftlogThe log path of current raft node.
raft.use_replicator_pipelinetrueWhether to use replicator line, when turned on it multiple logs can be sent in parallel, and the next log doesn’t have to wait for the ack message of the current log to be sent.
raft.election_timeout10000Timeout in milliseconds to launch a round of election.
raft.snapshot_interval3600The interval in seconds to trigger snapshot save.
raft.snapshot_threads4The thread number used to do snapshot.
raft.snapshot_parallel_compressfalseWhether to enable parallel compress.
raft.snapshot_compress_threads4The thread number used to do snapshot compress.
raft.snapshot_decompress_threads4The thread number used to do snapshot decompress.
raft.backend_threadsCPUsThe thread number used to apply task to backend.
raft.read_index_threads8The thread number used to execute reading index.
raft.read_strategyReadOnlyLeaseBasedThe linearizability of read strategy, allowed values are [ReadOnlyLeaseBased, ReadOnlySafe].
raft.apply_batch1The apply batch size to trigger disruptor event handler.
raft.queue_size16384The disruptor buffers size for jraft RaftNode, StateMachine and LogManager.
raft.queue_publish_timeout60The timeout in second when publish event into disruptor.
raft.rpc_threadsmax(CPUs * 2, 80)The rpc threads for jraft RPC layer.
raft.rpc_connect_timeout5000The rpc connect timeout in milliseconds for jraft rpc.
raft.rpc_timeout60The general rpc timeout in seconds for jraft rpc.
raft.install_snapshot_rpc_timeout36000The install snapshot rpc timeout in seconds for jraft rpc.
raft.rpc_buf_low_water_mark10485760The ChannelOutboundBuffer’s low water mark of netty, when buffer size less than this size, the method ChannelOutboundBuffer.isWritable() will return true, it means that low downstream pressure or good network.
raft.rpc_buf_high_water_mark20971520The ChannelOutboundBuffer’s high water mark of netty, only when buffer size exceed this size, the method ChannelOutboundBuffer.isWritable() will return false, it means that the downstream pressure is too great to process the request or network is very congestion, upstream needs to limit rate at this time.

RocksDB Backend Config Options

config optiondefault valuedescription
backendMust be set to rocksdb.
serializerMust be set to binary.
rocksdb.data_pathrocksdb-data/dataThe path for storing data of RocksDB.
rocksdb.wal_pathrocksdb-data/walThe path for storing WAL of RocksDB.
rocksdb.sst_pathThe path for ingesting SST file into RocksDB.
rocksdb.data_disks[]The optimized disks for storing data of RocksDB. The format of each element: STORE/TABLE: /path/disk.Allowed keys are [g/vertex, g/edge_out, g/edge_in, g/vertex_label_index, g/edge_label_index, g/range_int_index, g/range_float_index, g/range_long_index, g/range_double_index, g/secondary_index, g/search_index, g/shard_index, g/unique_index, g/olap]
rocksdb.log_levelINFOThe info log level of RocksDB.
rocksdb.num_levels7Set the number of levels for this database.
rocksdb.compaction_styleLEVELSet compaction style for RocksDB: LEVEL/UNIVERSAL/FIFO.
rocksdb.optimize_modetrueOptimize for heavy workloads and big datasets.
rocksdb.bulkload_modefalseSwitch to the mode to bulk load data into RocksDB.
rocksdb.compression_per_level[none, none, snappy, snappy, snappy, snappy, snappy]The compression algorithms for different levels of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd.
rocksdb.bottommost_compressionnoneThe compression algorithm for the bottommost level of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd.
rocksdb.compressionsnappyThe compression algorithm for compressing blocks of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd.
rocksdb.max_background_jobs8Maximum number of concurrent background jobs, including flushes and compactions.
rocksdb.max_subcompactions4The value represents the maximum number of threads per compaction job.
rocksdb.delayed_write_rate16777216The rate limit in bytes/s of user write requests when need to slow down if the compaction gets behind.
rocksdb.max_open_files-1The maximum number of open files that can be cached by RocksDB, -1 means no limit.
rocksdb.max_manifest_file_size104857600The max size of manifest file in bytes.
rocksdb.skip_stats_update_on_db_openfalseWhether to skip statistics update when opening the database, setting this flag true allows us to not update statistics.
rocksdb.skip_check_sst_size_on_db_openfalseWhether to skip checking sizes of all sst files when opening the database.
rocksdb.max_file_opening_threads16The max number of threads used to open files.
rocksdb.max_total_wal_size0Total size of WAL files in bytes. Once WALs exceed this size, we will start forcing the flush of column families related, 0 means no limit.
rocksdb.bytes_per_sync0Allows OS to incrementally sync SST files to disk while they are being written, asynchronously in the background. Issue one request for every bytes_per_sync written. 0 turns it off.
rocksdb.wal_bytes_per_sync0Allows OS to incrementally sync WAL files to disk while they are being written, asynchronously in the background. Issue one request for every bytes_per_sync written. 0 turns it off.
rocksdb.strict_bytes_per_syncfalseWhen true, guarantees SST/WAL files have at most bytes_per_sync/wal_bytes_per_sync bytes submitted for writeback at any given time. This can be used to handle cases where processing speed exceeds I/O speed.
rocksdb.db_write_buffer_size0Total size of write buffers in bytes across all column families, 0 means no limit.
rocksdb.log_readahead_size0The number of bytes to prefetch when reading the log. 0 means the prefetching is disabled.
rocksdb.compaction_readahead_size0The number of bytes to perform bigger reads when doing compaction. If running RocksDB on spinning disks, you should set this to at least 2MB. 0 means the prefetching is disabled.
rocksdb.row_cache_capacity0The capacity in bytes of global cache for table-level rows. 0 means the row_cache is disabled.
rocksdb.delete_obsolete_files_period21600The periodicity in seconds when obsolete files get deleted, 0 means always do full purge.
rocksdb.write_buffer_size134217728Amount of data in bytes to build up in memory.
rocksdb.max_write_buffer_number6The maximum number of write buffers that are built up in memory.
rocksdb.min_write_buffer_number_to_merge2The minimum number of write buffers that will be merged together.
rocksdb.max_write_buffer_number_to_maintain0The total maximum number of write buffers to maintain in memory for conflict checking when transactions are used.
rocksdb.memtable_bloom_size_ratio0.0If prefix-extractor is set and memtable_bloom_size_ratio is not 0, or if memtable_whole_key_filtering is set true, create bloom filter for memtable with the size of write_buffer_size * memtable_bloom_size_ratio. If it is larger than 0.25, it is santinized to 0.25.
rocksdb.memtable_whole_key_filteringfalseEnable whole key bloom filter in memtable, it can potentially reduce CPU usage for point-look-ups. Note this will only take effect if memtable_bloom_size_ratio > 0.
rocksdb.memtable_huge_page_size0The page size for huge page TLB for bloom in memtable. If <= 0, not allocate from huge page TLB but from malloc.
rocksdb.inplace_update_supportfalseAllows thread-safe inplace updates if a put key exists in current memtable and sizeof new value is smaller.
rocksdb.level_compaction_dynamic_level_bytesfalseWhether to enable level_compaction_dynamic_level_bytes, if it’s enabled we give max_bytes_for_level_multiplier a priority against max_bytes_for_level_base, the bytes of base level is dynamic for a more predictable LSM tree, it is useful to limit worse case space amplification. Turning this feature on/off for an existing DB can cause unexpected LSM tree structure so it’s not recommended.
rocksdb.max_bytes_for_level_base536870912The upper-bound of the total size of level-1 files in bytes.
rocksdb.max_bytes_for_level_multiplier10.0The ratio between the total size of level (L+1) files and the total size of level L files for all L.
rocksdb.target_file_size_base67108864The target file size for compaction in bytes.
rocksdb.target_file_size_multiplier1The size ratio between a level L file and a level (L+1) file.
rocksdb.level0_file_num_compaction_trigger2Number of files to trigger level-0 compaction.
rocksdb.level0_slowdown_writes_trigger20Soft limit on number of level-0 files for slowing down writes.
rocksdb.level0_stop_writes_trigger36Hard limit on number of level-0 files for stopping writes.
rocksdb.soft_pending_compaction_bytes_limit68719476736The soft limit to impose on pending compaction in bytes.
rocksdb.hard_pending_compaction_bytes_limit274877906944The hard limit to impose on pending compaction in bytes.
rocksdb.allow_mmap_writesfalseAllow the OS to mmap file for writing.
rocksdb.allow_mmap_readsfalseAllow the OS to mmap file for reading sst tables.
rocksdb.use_direct_readsfalseEnable the OS to use direct I/O for reading sst tables.
rocksdb.use_direct_io_for_flush_and_compactionfalseEnable the OS to use direct read/writes in flush and compaction.
rocksdb.use_fsyncfalseIf true, then every store to stable storage will issue a fsync.
rocksdb.atomic_flushfalseIf true, flushing multiple column families and committing their results atomically to MANIFEST. Note that it’s not necessary to set atomic_flush=true if WAL is always enabled.
rocksdb.format_version5The format version of BlockBasedTable, allowed values are 0~5.
rocksdb.index_typekBinarySearchThe index type used to lookup between data blocks with the sst table, allowed values are [kBinarySearch,kHashSearch,kTwoLevelIndexSearch,kBinarySearchWithFirstKey].
rocksdb.data_block_index_typekDataBlockBinarySearchThe search type used to point lookup in data block with the sst table, allowed values are [kDataBlockBinarySearch,kDataBlockBinaryAndHash].
rocksdb.data_block_hash_table_util_ratio0.75The hash table utilization ratio value of entries/buckets. It is valid only when data_block_index_type=kDataBlockBinaryAndHash.
rocksdb.block_size4096Approximate size of user data packed per block, Note that it corresponds to uncompressed data.
rocksdb.block_size_deviation10The percentage of free space used to close a block.
rocksdb.block_restart_interval16The block restart interval for delta encoding in blocks.
rocksdb.block_cache_capacity8388608The amount of block cache in bytes that will be used by RocksDB, 0 means no block cache.
rocksdb.cache_index_and_filter_blockstrueSet this option true if we’d put index/filter blocks to the block cache.
rocksdb.pin_l0_filter_and_index_blocks_in_cachetrueSet this option true if we’d pin L0 index/filter blocks to the block cache.
rocksdb.bloom_filter_bits_per_key-1The bits per key in bloom filter, a good value is 10, which yields a filter with ~ 1% false positive rate. Set bloom_filter_bits_per_key > 0 to enable bloom filter, -1 means no bloom filter (0~0.5 round down to no filter).
rocksdb.bloom_filter_block_based_modefalseIf bloom filter is enabled, set this option true to use block based filter rather than full filter.
rocksdb.bloom_filter_whole_key_filteringtrueIf bloom filter is enabled, set this option true to place whole keys in the bloom filter, else place the prefix of keys when prefix-extractor is set.
rocksdb.optimize_filters_for_hitstrueIf bloom filter is enabled, this flag allows us to not store filters for the last level. set this option true to optimize the filters mainly for cases where keys are found rather than also optimize for keys missed.
rocksdb.partition_filters_and_indexesfalseIf bloom filter is enabled, set this option true to use partitioned full filters and indexes for each sst file. This option is incompatible with block-based filters.
rocksdb.pin_top_level_index_and_filtertrueIf partition_filters_and_indexes is set true, set this option true if we’d pin top-level index of partitioned filter and index blocks to the block cache.
rocksdb.prefix_extractor_n_bytes0The prefix-extractor uses the first N bytes of a key as its prefix, it will use the full key when a key is shorter than the N. 0 means unset prefix-extractor.
K8s Config Options (Optional)

Corresponding configuration file rest-server.properties

config optiondefault valuedescription
server.use_k8sfalseWhether to use k8s to support multiple tenancy.
server.deploy_in_k8sfalseWhether to deploy server in k8s.
server.urls_to_pdhttp://0.0.0.0:8080Used as the server address reserved for PD and provided to clients, only used when starting the server in k8s.
server.k8s_urlhttps://127.0.0.1:8888The url of k8s.
server.k8s_use_cafalseWhether to use ca to k8s api server.
server.k8s_caThe ca file of k8s api server.
server.k8s_client_caThe client ca file of k8s api server.
server.k8s_client_keyThe client key file of k8s api server.
k8s.apifalseThe k8s api start status when the computer service is enabled.
k8s.namespacehugegraph-computer-systemThe namespace used for k8s work when the computer service is enabled.
k8s.kubeconfigThe k8s kube config file when the computer service is enabled.
k8s.hugegraph_urlThe hugegraph url for k8s work when the computer service is enabled.
k8s.enable_internal_algorithmtrueWhether to open k8s internal algorithm.
service.access_pd_namehgService name for server to access pd service.
service.access_pd_tokenService token for server to access pd service.
server.k8s_oltp_image127.0.0.1/kgs_bd/hugegraphserver:3.0.0The oltp server image of k8s.
server.k8s_olap_imagehugegraph/hugegraph-server:v1The olap server image of k8s.
server.k8s_storage_imagehugegraph/hugegraph-server:v1The storage server image of k8s.
server.default_oltp_k8s_namespacehugegraph-serverThe default oltp namespace for HugeGraph default graph space.
server.default_olap_k8s_namespacehugegraph-computer-systemThe default olap namespace for HugeGraph default graph space.
k8s.internal_algorithm[page-rank, degree-centrality, wcc, triangle-count, rings, rings-with-filter, betweenness-centrality, closeness-centrality, lpa, links, kcore, louvain, clustering-coefficient, ppr, subgraph-match]The names of the built-in k8s algorithms.
k8s.algorithmsSee ServerOptions.K8S_ALGORITHMSThe name:paramsClass mapping of the built-in k8s algorithms.
Arthas Diagnostic Config Options (Optional)

Corresponding configuration file rest-server.properties

config optiondefault valuedescription
arthas.telnetPort8562Arthas telnet port.
arthas.httpPort8561Arthas HTTP port.
arthas.ip0.0.0.0Arthas bind IP.
arthas.disabledCommandsjadDisabled Arthas commands, separated by commas.
RPC Server Config Options

Corresponding configuration file rest-server.properties

config optiondefault valuedescription
rpc.server_hostThe hosts/ips bound by rpc server to provide services, empty value means not enabled.
rpc.server_port8090The port bound by rpc server to provide services.
rpc.server_adaptive_portfalseWhether the bound port is adaptive, if it’s enabled, when the port is in use, automatically +1 to detect the next available port. Note that this process is not atomic, so there may still be port conflicts.
rpc.server_timeout30The timeout(in seconds) of rpc server execution.
rpc.remote_urlThe remote urls of rpc peers, it can be set to multiple addresses, which are concat by ‘,’, empty value means not enabled.
rpc.client_connect_timeout20The timeout(in seconds) of rpc client connect to rpc server.
rpc.client_reconnect_period10The period(in seconds) of rpc client reconnect to rpc server.
rpc.client_read_timeout40The timeout(in seconds) of rpc client read from rpc server.
rpc.client_retries3Failed retry number of rpc client calls to rpc server.
rpc.client_load_balancerconsistentHashThe rpc client uses a load-balancing algorithm to access multiple rpc servers in one cluster. Default value is ‘consistentHash’, means forwarding by request parameters.
rpc.protocolboltRpc communication protocol, client and server need to be specified the same value.
rpc.serializationhessian2Rpc serialization type, client and server must set the same value. Note: If you choose ‘protobuf’, you need to add the relative IDL file. (Could refer PD/Store *.proto)
rpc.config_order999Sofa-RPC configuration file loading order, the larger the more later loading.
rpc.logger_implcom.alipay.sofa.rpc.log.SLF4JLoggerImplSofa-RPC log implementation class.
HBase Backend Config Options
config optiondefault valuedescription
backendMust be set to hbase.
serializerMust be set to hbase.
hbase.hostslocalhostThe hostnames or ip addresses of HBase zookeeper, separated with commas.
hbase.port2181The port address of HBase zookeeper.
hbase.threads_max64The max threads num of hbase connections.
hbase.znode_parent/hbaseThe znode parent path of HBase zookeeper.
hbase.zk_retry3The recovery retry times of HBase zookeeper.
hbase.truncate_timeout30The timeout in seconds of waiting for store truncate.
hbase.aggregation_timeout43200The timeout in seconds of waiting for aggregation.
hbase.kerberos_enablefalseIs Kerberos authentication enabled for HBase.
hbase.kerberos_keytabThe HBase’s key tab file for kerberos authentication.
hbase.kerberos_principalThe HBase’s principal for kerberos authentication.
hbase.krb5_conf/etc/krb5.confKerberos configuration file, including KDC IP, default realm, etc.
hbase.hbase_site/etc/hbase/conf/hbase-site.xmlThe HBase’s configuration file
hbase.enable_partitiontrueIs pre-split partitions enabled for HBase.
hbase.vertex_partitions10The number of partitions of the HBase vertex table.
hbase.edge_partitions30The number of partitions of the HBase edge table.

≤ 1.5 Version Config (Legacy)

The following backend stores are no longer supported in version 1.7.0+ and are only available in version 1.5.x and earlier:

Cassandra Backend Config Options
config optiondefault valuedescription
backendMust be set to cassandra.
serializerMust be set to cassandra.
cassandra.hostlocalhostThe seeds hostname or ip address of cassandra cluster.
cassandra.port9042The seeds port address of cassandra cluster.
cassandra.connect_timeout5The cassandra driver connect server timeout(seconds).
cassandra.read_timeout20The cassandra driver read from server timeout(seconds).
cassandra.keyspace.strategySimpleStrategyThe replication strategy of keyspace, valid value is SimpleStrategy or NetworkTopologyStrategy.
cassandra.keyspace.replication[3]The keyspace replication factor of SimpleStrategy, like ‘[3]’.Or replicas in each datacenter of NetworkTopologyStrategy, like ‘[dc1:2,dc2:1]’.
cassandra.usernameThe username to use to login to cassandra cluster.
cassandra.passwordThe password corresponding to cassandra.username.
cassandra.compression_typenoneThe compression algorithm of cassandra transport: none/snappy/lz4.
cassandra.jmx_port=71997199The port of JMX API service for cassandra.
cassandra.aggregation_timeout43200The timeout in seconds of waiting for aggregation.
ScyllaDB Backend Config Options
config optiondefault valuedescription
backendMust be set to scylladb.
serializerMust be set to scylladb.

Other options are consistent with the Cassandra backend.

MySQL & PostgreSQL Backend Config Options
config optiondefault valuedescription
backendMust be set to mysql.
serializerMust be set to mysql.
jdbc.drivercom.mysql.jdbc.DriverThe JDBC driver class to connect database.
jdbc.urljdbc:mysql://127.0.0.1:3306The url of database in JDBC format.
jdbc.usernamerootThe username to login database.
jdbc.password******The password corresponding to jdbc.username.
jdbc.ssl_modefalseThe SSL mode of connections with database.
jdbc.reconnect_interval3The interval(seconds) between reconnections when the database connection fails.
jdbc.reconnect_max_times3The reconnect times when the database connection fails.
jdbc.storage_engineInnoDBThe storage engine of backend store database, like InnoDB/MyISAM/RocksDB for MySQL.
jdbc.postgresql.connect_databasetemplate1The database used to connect when init store, drop store or check store exist.
PostgreSQL Backend Config Options
config optiondefault valuedescription
backendMust be set to postgresql.
serializerMust be set to postgresql.

Other options are consistent with the MySQL backend.

The driver and url of the PostgreSQL backend should be set to:

  • jdbc.driver=org.postgresql.Driver
  • jdbc.url=jdbc:postgresql://localhost:5432/

3 - Built-in User Authentication and Authorization Configuration and Usage in HugeGraph

Overview

To facilitate authentication usage in different user scenarios, HugeGraph currently provides built-in authorization StandardAuthenticator mode, which supports multi-user authentication and fine-grained access control. It adopts a 4-layer design based on “User-UserGroup-Operation-Resource” to flexibly control user roles and permissions (supports multiple GraphServers).

Some key designs of the StandardAuthenticator mode include:

  • During initialization, a super administrator (admin) user is created. Subsequently, other users can be created by the super administrator. Once newly created users are assigned sufficient permissions, they can create or manage more users.
  • It supports dynamic creation of users, user groups, and resources, as well as dynamic allocation or revocation of permissions.
  • Users can belong to one or multiple user groups. Each user group can have permissions to operate on any number of resources. The types of operations include read, write, delete, execute, and others.
  • “Resource” describes the data in the graph database, such as vertices that meet certain criteria. Each resource consists of three elements: type, label, and properties. There are 18 types in total, with the ability to combine any label and properties. The internal condition of a resource is an AND relationship, while the condition between multiple resources is an OR relationship.

Here is an example to illustrate:

// Scenario: A user only has data read permission for the Beijing area
user(name=xx) -belong-> group(name=xx) -access(read)-> target(graph=graph1, resource={label: person, city: Beijing})

Configure User Authentication

By default, HugeGraph does not enable user authentication, and it needs to be enabled by modifying the configuration file.

Because the flexibility of graph query languages can introduce potential system security risks, do not expose Gremlin, Cypher, or other query endpoints directly to the public network. In production, enable authentication, an IP allowlist, and audit logging, and isolate the Server process with Docker or Kubernetes.

You need to modify the configuration file to enable this feature. HugeGraph provides built-in authentication mode: StandardAuthenticator. This mode supports multi-user authentication and fine-grained permission control. Additionally, developers can implement their own HugeAuthenticator interface to integrate with their existing authentication systems.

HugeGraph uses HTTP Basic Authentication. The value after Basic is the Base64 encoding of username:password. With curl, pass the credentials directly through -u:

curl -u 'admin:<password>' \
  http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/schema/vertexlabels

Warning: Versions of HugeGraph-Server prior to 1.5.0 have a JWT-related security vulnerability in the Auth mode. Users are advised to update to a newer version or manually set the JWT token’s secretKey. It can be set in the rest-server.properties file by setting the auth.token_secret information:

auth.token_secret=XXXX   # should be a 32-chars string, consist of A-Z, a-z and 0-9

You can also generate it with the following command:

RANDOM_STRING=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 32)
echo "auth.token_secret=${RANDOM_STRING}" >> rest-server.properties

Since 1.5.0 the option defaults to a key generated randomly at startup, so it does not have to be configured. Set it explicitly when tokens have to survive a restart, or when more than one server must accept the same token. Tokens expire after auth.token_expire seconds (default 86400).

StandardAuthenticator Mode

The StandardAuthenticator mode supports user authentication and permission control by storing user information in the database backend. This implementation authenticates users based on their names and passwords (encrypted) stored in the database and controls user permissions based on their roles. Below is the specific configuration process (requires service restart):

Configure the authenticator and its rest-server file path in the gremlin-server.yaml configuration file:

authentication: {
  authenticator: org.apache.hugegraph.auth.StandardAuthenticator,
  authenticationHandler: org.apache.hugegraph.auth.WsAndHttpBasicAuthHandler,
  config: {tokens: conf/rest-server.properties}
}

Configure the authenticator and the graph that stores authorization data in rest-server.properties:

auth.authenticator=org.apache.hugegraph.auth.StandardAuthenticator
auth.graph_store=hugegraph
# The password of the built-in admin account, default is pa, it takes effect on the first startup
#auth.admin_pa=<your-admin-password>

# Auth Client Config
# If GraphServer and AuthServer are deployed separately, you also need to specify the following configuration. Fill in the IP:RPC port of AuthServer.
# auth.remote_url=127.0.0.1:8899,127.0.0.1:8898,127.0.0.1:8897

In the above configuration, the graph_store option specifies which graph to use for storing user information. If there are multiple graphs, you can choose any of them.

In the hugegraph{n}.properties configuration file, configure the gremlin.graph information:

gremlin.graph=org.apache.hugegraph.auth.HugeFactoryAuthProxy

For authorization API usage, see the Authentication API documentation.

Custom User Authentication System

If you need to support a more flexible user system, you can customize the authenticator for extension. Simply implement the org.apache.hugegraph.auth.HugeAuthenticator interface with your custom authenticator, and then modify the authenticator configuration item in the configuration file to point to your implementation.

Switching authentication mode

When init-store.sh is run for the first time and the admin user does not yet exist, the command prompts for the administrator password. For an initialized persistent backend, init-store.sh adds the system metadata required for authentication without deleting existing graph data.

# stop the hugeGraph firstly
bin/stop-hugegraph.sh

# Initialize authentication system metadata; existing backend data is preserved
bin/init-store.sh

# start hugeGraph again
bin/start-hugegraph.sh

Use docker to enable authentication mode

For versions of the hugegraph/hugegraph image equal to or greater than 1.2.0, you can enable authentication mode while starting the Docker image.

The steps are as follows:

1. Use docker run

To enable authentication mode, add the environment variable PASSWORD=xxx (you can freely set the password) in the docker run command:

docker run -itd -e PASSWORD=xxx --name=server -p 8080:8080 hugegraph/hugegraph:1.7.0

2. Use docker-compose

Use docker-compose and set the environment variable PASSWORD=xxx:

version: '3'
services:
  server:
    image: hugegraph/hugegraph:1.7.0
    container_name: server
    ports:
      - 8080:8080
    environment:
      - PASSWORD=xxx

3. Enter the container to enable authentication mode

Enter the container first:

docker exec -it server bash
# Modify the config quickly, the modified file are save in the conf-bak folder
bin/enable-auth.sh

Then follow Switching authentication mode

4 - Configuring HugeGraphServer to Use HTTPS Protocol

Overview

By default, HugeGraphServer uses the HTTP protocol. However, if you have security requirements for your requests, you can configure it to use HTTPS.

Server Configuration

Modify the conf/rest-server.properties configuration file and change the schema part of restserver.url to https.

# Set the protocol to HTTPS
restserver.url=https://127.0.0.1:8080
# Server keystore file path. This default value is automatically effective when using HTTPS, and you can modify it as needed.
ssl.keystore_file=conf/hugegraph-server.keystore
# Server keystore file password. This default value is automatically effective when using HTTPS, and you can modify it as needed.
ssl.keystore_password=******

The keystore file is not shipped inside the distribution, because it carries no license declaration. When restserver.url starts with https and conf/hugegraph-server.keystore is missing, bin/start-hugegraph.sh downloads it from the binary-1.5 branch of the hugegraph-doc repository before starting the server. The password of that file is hugegraph. Both values are the defaults of ssl.keystore_file and ssl.keystore_password; users can generate their own keystore file and password and then change the two options.

Client Configuration

Using HTTPS in HugeGraph-Client

When constructing a HugeClient, pass the HTTPS-related configurations. Here’s an example in Java:

String url = "https://localhost:8080";
String graphName = "hugegraph";
HugeClientBuilder builder = HugeClient.builder(url, graphName);
// Client keystore file path
String trustStoreFilePath = "hugegraph.truststore";
// Client keystore password
String trustStorePassword = "******";
builder.configSSL(trustStoreFilePath, trustStorePassword);
HugeClient hugeClient = builder.build();

Note: Before version 1.9.0, HugeGraph-Client was created directly using the new keyword and did not support the HTTPS protocol. Starting from version 1.9.0, it changed to use the builder pattern and supports configuring the HTTPS protocol.

Using HTTPS in HugeGraph-Loader

When starting an import task, add the following options in the command line:

# HTTPS
--protocol https
# Client certificate file path. When specifying --protocol as https, the default value conf/hugegraph.truststore is automatically used, and you can modify it as needed.
--trust-store-file {file}
# Client certificate file password. When specifying --protocol as https, the default value hugegraph is automatically used, and you can modify it as needed.
--trust-store-password {password}

Under the conf directory of hugegraph-loader, there is already a default client certificate file named hugegraph.truststore, and its password is hugegraph.

Using HTTPS in HugeGraph-Tools

When executing commands, add the following options in the command line:

# Client certificate file path. When using the HTTPS protocol in the URL, the default value conf/hugegraph.truststore is automatically used, and you can modify it as needed.
--trust-store-file {file}
# Client certificate file password. When using the HTTPS protocol in the URL, the default value hugegraph is automatically used, and you can modify it as needed.
--trust-store-password {password}
# When executing migration commands and using the --target-url with the HTTPS protocol, the default value conf/hugegraph.truststore is automatically used, and you can modify it as needed.
--target-trust-store-file {target-file}
# When executing migration commands and using the --target-url with the HTTPS protocol, the default value hugegraph is automatically used, and you can modify it as needed.
--target-trust-store-password {target-password}

Under the conf directory of hugegraph-tools, there is already a default client certificate file named hugegraph.truststore, and its password is hugegraph.

How to Generate Certificate Files

This section provides an example of generating certificates. If the default certificate is sufficient or if you already know how to generate certificates, you can skip this section.

Server

  1. Generate the server’s private key and import it into the server’s keystore file. The server.keystore is for the server’s use and contains its private key.
keytool -genkey -alias serverkey -keyalg RSA -keystore server.keystore

During the process, fill in the description information according to your requirements. The description information for the default certificate is as follows:

First and Last Name: hugegraph
Organizational Unit Name: hugegraph
Organization Name: hugegraph
City or Locality Name: BJ
State or Province Name: BJ
Country Code: CN
  1. Export the server certificate based on the server’s private key.
keytool -export -alias serverkey -keystore server.keystore -file server.crt

server.crt is the server’s certificate.

Client

keytool -import -alias serverkey -file server.crt -keystore client.truststore

client.truststore is for the client’s use and contains the trusted certificate.

5 - Configuring the RocksDB Backend

Overview

RocksDB is an embedded LSM-tree key-value store. With the rocksdb backend, HugeGraph-Server keeps all graph data in RocksDB instances that live inside the server process, so there is no separate storage service to deploy. This is the backend used by the shipped conf/graphs/hugegraph.properties.

Since version 1.7.0 the server accepts only memory, rocksdb, hbase and hstore as the backend. The rocksdb backend stores data on the local disks of one server: it does not support shared storage, so a graph cannot be served by several servers over the same data directory. For a distributed deployment use the hstore backend with PD and Store.

The RocksDB JNI library is pinned to version 8.10.2 by hugegraph-rocksdb/pom.xml, so the on-disk format and the option semantics are those of RocksDB 8.10.

The backend driver version reported by this store is 1.11, and it is written into the meta table of the system store when the graph is initialized.

Selecting the backend

Set the backend and the serializer in the graph properties file (conf/graphs/<graph>.properties):

gremlin.graph=org.apache.hugegraph.HugeFactory

backend=rocksdb
serializer=binary

store=hugegraph

# rocksdb backend config
#rocksdb.data_path=/path/to/disk
#rocksdb.wal_path=/path/to/disk
  • backend=rocksdb selects the RocksDB store provider.
  • serializer=binary is the serializer the shipped template uses for this backend. The built-in serializers are binary, binaryscatter and text.
  • store is the database namespace of the graph, and it is also part of the graph name that the provider passes down to the store.

Run bin/init-store.sh once before the first start to create the stores, then start the server. Both bin/init-store.sh and bin/hugegraph-server.sh load the RocksDB library, so the data directories are created on the machine that runs them.

The distribution registers the option space and the store provider for each backend listed in the packaged backend.properties, whose value comes from the hugegraph.backends build property. A default build registers rocksdb, hbase, hstore; building with -Drocksdb-only activates the rocksdb-only profile and produces a distribution that registers only rocksdb. A backend that is not registered fails at startup with Not exists BackendStoreProvider.

The provider registration also adds a second name, rocksdbsst, for the store that writes SST files instead of a live database. That name is not in the list of allowed backends, so backend=rocksdbsst is rejected with backend is illegal: rocksdbsst. To load SST files into a normal rocksdb graph, use rocksdb.sst_path as described below.

Data directory layout

Two directories matter: rocksdb.data_path (default rocksdb-data/data) and rocksdb.wal_path (default rocksdb-data/wal). Relative paths resolve against the working directory of the server, which is the installation directory.

Each graph opens three stores: m for schema, g for graph data, and s for the system store. The store name is appended to both configured paths, so a default single-graph installation looks like this:

rocksdb-data/
  data/
    m/    # schema store: property keys, vertex/edge/index labels, counters
    g/    # graph store: vertices, edges, index tables, olap tables
    s/    # system store: tasks, server info, backend meta (driver version)
  wal/
    m/
    g/
    s/

Every backend table becomes a RocksDB column family inside the store it belongs to, named <database>+<table>, where the database is derived from the graph name. Column families of existing data directories are always reopened, so tables created by an older version stay readable.

Other points to keep in mind:

  • Two graphs must not share a data path. When a graph is created by cloning an existing configuration through the API, the provider appends _<newGraph> to both rocksdb.data_path and rocksdb.wal_path. Deleting such a graph deletes both directories.
  • Snapshots are created beside the data directory: the last two segments of the data path are rewritten with a prefix, so with the default paths the snapshot of the graph store goes to rocksdb-data/<prefix>_data/g. Resuming a snapshot closes the instance, deletes the data directory and moves the snapshot into its place.
  • With rocksdb.data_disks set, the tables named there are opened as separate RocksDB instances under the given paths instead of under rocksdb.data_path. The server opens up to 8 instances in parallel, waits at most 600 seconds for the open to finish and 30 seconds for sessions to close.

Path and log options

config optiondefault valuedescription
rocksdb.data_pathrocksdb-data/dataThe path for storing data of RocksDB. Must not be empty.
rocksdb.data_disks[]The optimized disks for storing data of RocksDB. The format of each element: STORE/TABLE: /path/disk. Allowed keys are [g/vertex, g/edge_out, g/edge_in, g/vertex_label_index, g/edge_label_index, g/range_int_index, g/range_float_index, g/range_long_index, g/range_double_index, g/secondary_index, g/search_index, g/shard_index, g/unique_index, g/olap]. A disk path must differ from rocksdb.data_path.
rocksdb.wal_pathrocksdb-data/walThe path for storing WAL of RocksDB. Must not be empty.
rocksdb.sst_path(empty)The path for ingesting SST file into RocksDB. Empty disables ingestion.
rocksdb.log_levelINFOThe info log level of RocksDB. Allowed values: DEBUG, INFO, WARN, ERROR, FATAL, HEADER.

Compaction and compression options

config optiondefault valuedescription
rocksdb.num_levels7Set the number of levels for this database. Range: 1 to 2^31-1.
rocksdb.compaction_styleLEVELSet compaction style for RocksDB: LEVEL/UNIVERSAL/FIFO.
rocksdb.optimize_modetrueOptimize for heavy workloads and big datasets. See “How the options are applied” below.
rocksdb.bulkload_modefalseSwitch to the mode to bulk load data into RocksDB.
rocksdb.compression_per_level[none, none, snappy, snappy, snappy, snappy, snappy]The compression algorithms for different levels of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd. The list must be empty or hold exactly rocksdb.num_levels elements.
rocksdb.bottommost_compressionnoneThe compression algorithm for the bottommost level of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd.
rocksdb.compressionsnappyThe compression algorithm for compressing blocks of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd.

Database level options

config optiondefault valuedescription
rocksdb.max_background_jobs8Maximum number of concurrent background jobs, including flushes and compactions. Range: 1 to 2^31-1.
rocksdb.max_subcompactions4The value represents the maximum number of threads per compaction job. Range: 1 to 2^31-1.
rocksdb.delayed_write_rate16777216 (16 MB/s)The rate limit in bytes/s of user write requests when need to slow down if the compaction gets behind.
rocksdb.max_open_files-1The maximum number of open files that can be cached by RocksDB, -1 means no limit.
rocksdb.max_manifest_file_size104857600 (100 MB)The max size of manifest file in bytes.
rocksdb.skip_stats_update_on_db_openfalseWhether to skip statistics update when opening the database, setting this flag true allows us to not update statistics.
rocksdb.skip_check_sst_size_on_db_openfalseWhether to skip checking sizes of all sst files when opening the database.
rocksdb.max_file_opening_threads16The max number of threads used to open files. Range: 1 to 2^31-1.
rocksdb.max_total_wal_size0Total size of WAL files in bytes. Once WALs exceed this size, we will start forcing the flush of column families related, 0 means no limit.
rocksdb.bytes_per_sync0Allows OS to incrementally sync SST files to disk while they are being written, asynchronously in the background. Issue one request for every bytes_per_sync written. 0 turns it off.
rocksdb.wal_bytes_per_sync0Same as above for WAL files. 0 turns it off.
rocksdb.strict_bytes_per_syncfalseWhen true, guarantees SST/WAL files have at most bytes_per_sync/wal_bytes_per_sync bytes submitted for writeback at any given time. This can be used to handle cases where processing speed exceeds I/O speed.
rocksdb.db_write_buffer_size0Total size of write buffers in bytes across all column families, 0 means no limit.
rocksdb.log_readahead_size0The number of bytes to prefetch when reading the log. 0 means the prefetching is disabled.
rocksdb.compaction_readahead_size0The number of bytes to perform bigger reads when doing compaction. If running RocksDB on spinning disks, you should set this to at least 2MB. 0 means the prefetching is disabled.
rocksdb.row_cache_capacity0The capacity in bytes of global cache for table-level rows. 0 means the row_cache is disabled.
rocksdb.delete_obsolete_files_period21600 (6 hours)The periodicity in seconds when obsolete files get deleted, 0 means always do full purge. The value is converted to microseconds before it reaches RocksDB.

Memtable options

config optiondefault valuedescription
rocksdb.write_buffer_size134217728 (128 MB)Amount of data in bytes to build up in memory. Minimum 1 MB. This is per column family.
rocksdb.max_write_buffer_number6The maximum number of write buffers that are built up in memory. Range: 1 to 2^31-1.
rocksdb.min_write_buffer_number_to_merge2The minimum number of write buffers that will be merged together. Range: 1 to 2^31-1.
rocksdb.max_write_buffer_number_to_maintain0The total maximum number of write buffers to maintain in memory for conflict checking when transactions are used.
rocksdb.memtable_bloom_size_ratio0.0If prefix-extractor is set and memtable_bloom_size_ratio is not 0, or if memtable_whole_key_filtering is set true, create bloom filter for memtable with the size of write_buffer_size * memtable_bloom_size_ratio. A value larger than 0.25 is reduced to 0.25. Range: 0.0 to 1.0.
rocksdb.memtable_whole_key_filteringfalseEnable whole key bloom filter in memtable, it can potentially reduce CPU usage for point-look-ups. Note this will only take effect if memtable_bloom_size_ratio > 0.
rocksdb.memtable_huge_page_size0The page size for huge page TLB for bloom in memtable. If <= 0, not allocate from huge page TLB but from malloc.
rocksdb.inplace_update_supportfalseAllows thread-safe inplace updates if a put key exists in current memtable and sizeof new value is smaller.

Level sizing and write stall options

config optiondefault valuedescription
rocksdb.level_compaction_dynamic_level_bytesfalseWhether to enable level_compaction_dynamic_level_bytes, if it’s enabled we give max_bytes_for_level_multiplier a priority against max_bytes_for_level_base, the bytes of base level is dynamic for a more predictable LSM tree, it is useful to limit worse case space amplification. Turning this feature on/off for an existing DB can cause unexpected LSM tree structure so it’s not recommended.
rocksdb.max_bytes_for_level_base536870912 (512 MB)The upper-bound of the total size of level-1 files in bytes. Minimum 1 MB.
rocksdb.max_bytes_for_level_multiplier10.0The ratio between the total size of level (L+1) files and the total size of level L files for all L. Minimum 1.0.
rocksdb.target_file_size_base67108864 (64 MB)The target file size for compaction in bytes. Minimum 1 MB.
rocksdb.target_file_size_multiplier1The size ratio between a level L file and a level (L+1) file.
rocksdb.level0_file_num_compaction_trigger2Number of files to trigger level-0 compaction.
rocksdb.level0_slowdown_writes_trigger20Soft limit on number of level-0 files for slowing down writes.
rocksdb.level0_stop_writes_trigger36Hard limit on number of level-0 files for stopping writes.
rocksdb.soft_pending_compaction_bytes_limit68719476736 (64 GB)The soft limit to impose on pending compaction in bytes. Minimum 1 GB.
rocksdb.hard_pending_compaction_bytes_limit274877906944 (256 GB)The hard limit to impose on pending compaction in bytes. Minimum 1 GB.

File I/O options

config optiondefault valuedescription
rocksdb.allow_mmap_writesfalseAllow the OS to mmap file for writing.
rocksdb.allow_mmap_readsfalseAllow the OS to mmap file for reading sst tables.
rocksdb.use_direct_readsfalseEnable the OS to use direct I/O for reading sst tables.
rocksdb.use_direct_io_for_flush_and_compactionfalseEnable the OS to use direct read/writes in flush and compaction.
rocksdb.use_fsyncfalseIf true, then every store to stable storage will issue a fsync.
rocksdb.atomic_flushfalseIf true, flushing multiple column families and committing their results atomically to MANIFEST. Note that it’s not necessary to set atomic_flush=true if WAL is always enabled.

SST table format and block cache options

config optiondefault valuedescription
rocksdb.format_version5The format version of BlockBasedTable, allowed values are 0~5.
rocksdb.index_typekBinarySearchThe index type used to lookup between data blocks with the sst table, allowed values are [kBinarySearch, kHashSearch, kTwoLevelIndexSearch, kBinarySearchWithFirstKey].
rocksdb.data_block_index_typekDataBlockBinarySearchThe search type used to point lookup in data block with the sst table, allowed values are [kDataBlockBinarySearch, kDataBlockBinaryAndHash].
rocksdb.data_block_hash_table_util_ratio0.75The hash table utilization ratio value of entries/buckets. It is valid only when data_block_index_type=kDataBlockBinaryAndHash. Range: 0.0 to 1.0.
rocksdb.block_size4096 (4 KB)Approximate size of user data packed per block, Note that it corresponds to uncompressed data.
rocksdb.block_size_deviation10The percentage of free space used to close a block. Range: 0 to 100.
rocksdb.block_restart_interval16The block restart interval for delta encoding in blocks.
rocksdb.block_cache_capacity8388608 (8 MB)The amount of block cache in bytes that will be used by RocksDB, 0 means no block cache. A separate cache of this size is created for each column family.

Bloom filter options

The options in this group are read only when rocksdb.bloom_filter_bits_per_key is 0 or greater. With the default value of -1 there is no bloom filter and none of the other options in this table take effect, including the index and filter block caching ones.

config optiondefault valuedescription
rocksdb.bloom_filter_bits_per_key-1The bits per key in bloom filter, a good value is 10, which yields a filter with ~ 1% false positive rate. Set bloom_filter_bits_per_key > 0 to enable bloom filter, -1 means no bloom filter (0~0.5 round down to no filter).
rocksdb.bloom_filter_block_based_modefalseIf bloom filter is enabled, set this option true to use block based filter rather than full filter.
rocksdb.bloom_filter_whole_key_filteringtrueIf bloom filter is enabled, set this option true to place whole keys in the bloom filter, else place the prefix of keys when prefix-extractor is set.
rocksdb.cache_index_and_filter_blockstrueSet this option true if we’d put index/filter blocks to the block cache.
rocksdb.pin_l0_filter_and_index_blocks_in_cachetrueSet this option true if we’d pin L0 index/filter blocks to the block cache.
rocksdb.optimize_filters_for_hitstrueIf bloom filter is enabled, this flag allows us to not store filters for the last level. set this option true to optimize the filters mainly for cases where keys are found rather than also optimize for keys missed. This one is applied even when the filter is disabled.
rocksdb.partition_filters_and_indexesfalseIf bloom filter is enabled, set this option true to use partitioned full filters and indexes for each sst file. This option is incompatible with block-based filters. Enabling it also forces the index type to kTwoLevelIndexSearch and sets the metadata block size to rocksdb.block_size.
rocksdb.pin_top_level_index_and_filtertrueIf partition_filters_and_indexes is set true, set this option true if we’d pin top-level index of partitioned filter and index blocks to the block cache.
rocksdb.prefix_extractor_n_bytes0The prefix-extractor uses the first N bytes of a key as its prefix, it will use the full key when a key is shorter than the N. 0 means unset prefix-extractor.

How the options are applied

The server builds the RocksDB option objects once per store and per column family, so a change to any of the options above takes effect on the next server start.

  • rocksdb.optimize_mode=true applies presets before the values in the tables above: at the database level it raises parallelism to half of the available processors (at least one), allows concurrent memtable writes and enables the write thread adaptive yield; at the column family level it calls the RocksDB level-style and universal-style compaction presets. The explicit options are applied afterwards, so any value you set in the properties file wins over the preset.
  • rocksdb.bulkload_mode=true disables automatic compaction, raises the three level-0 triggers to the maximum integer and the two pending compaction limits to the maximum long value. Turn it off and restart after the load, otherwise compaction never runs.
  • rocksdb.block_cache_capacity=0 turns the block cache off completely rather than making it unbounded.
  • rocksdb.prefix_extractor_n_bytes greater than 0 installs a capped prefix extractor of that length.
  • Every column family uses the uint64add merge operator, which is what the counter table relies on.
  • The database is created if it is missing, and avoid_unnecessary_blocking_io and write_dbid_to_manifest are always on.

Memory notes

The caches and write buffers of RocksDB are native allocations, so they are not part of the JVM heap sizing in bin/hugegraph-server.sh. The GET /metrics/backend endpoint reports what the store uses: the memory number is the sum of the block cache usage, the pinned block cache usage, the estimated table reader memory (index and filter blocks) and the size of all memtables, taken from the RocksDB properties of every open column family.

Two option values multiply with the number of column families:

  • rocksdb.block_cache_capacity creates one cache instance per column family, so the total block cache of a server is roughly this value times the number of open tables across the m, g and s stores of every graph, plus the instances opened for rocksdb.data_disks.
  • rocksdb.write_buffer_size times rocksdb.max_write_buffer_number bounds the memtable memory of one column family. rocksdb.db_write_buffer_size caps the total across all column families of one store, and its default of 0 means there is no such cap.

rocksdb.row_cache_capacity is different: it is one cache per store, and 0 disables it.

Ingesting SST files

Setting rocksdb.sst_path turns on ingestion. When a store is opened, and again whenever tables are created, the server walks <sst_path>/<column family>/, collects every non-empty *.sst file below it and ingests those files into the matching column family. The files are moved rather than copied, so the source directory is consumed by the ingestion.

Raft mode

The RocksDB backend can still run behind the raft state machine: with raft.mode=true the store provider of any local backend is wrapped by the raft provider. The wrapper rejects backends with shared storage, so rocksdb is accepted while hbase is not. Under raft mode a RocksDB session writes with the WAL disabled and without sync, because the state machine can restore from a snapshot plus the raft log, and snapshots are supported by this backend.

Notes for anyone using it:

  • bin/init-store.sh forces raft.mode=false while it initializes the backend, so initialization never goes through raft.
  • The shipped conf/graphs/hugegraph.properties marks the raft options as deprecated. Distributed deployments of 1.7.0 and later use the hstore backend with PD and Store instead.
  • The raft peer endpoints are served under graphspaces/{graphspace}/graphs/{graph}/raft/, with list_peers, get_leader, set_leader, transfer_leader, add_peer and remove_peer. bin/raft-tools.sh wraps the same operations, but it still builds URLs without the graphspace segment, so the path has to be adjusted for a 1.7.0 server.
  • The remaining raft.* options are listed in the Server Complete Configuration Manual.

Backend capabilities

The feature flags of this backend affect what the server can push down to the store:

  • Scans by key prefix and by key range, paged queries, range conditions and order-by are supported.
  • There is no index inside RocksDB, so querying schema by name, querying by label and deleting edges by label are done by the server instead of the store.
  • Transactions are supported through RocksDB write batches.
  • Snapshots are supported, which is what raft mode and backup rely on.
  • Shared storage is not supported, so one data directory belongs to one server.
  • Olap properties are supported, and their tables are created as extra column families.
  • The store does not expire data by itself, so the server filters out elements whose TTL has passed when it reads them.
  • in, contains and contains_key conditions, aggregate properties and vertex or edge property updates in place are not supported at the store level.

Platform note for riscv64

On Linux riscv64 the RocksDB JNI library needs libatomic.so.1. bin/util.sh looks for it and adds it to LD_PRELOAD before bin/hugegraph-server.sh, bin/init-store.sh and bin/dump-store.sh start the JVM. If it is missing, those scripts stop with RISC-V RocksDB requires libatomic.so.1; install libatomic1, and installing the libatomic1 package fixes it.

6 - Configuring the HStore Distributed Backend

1 Overview

hstore is the distributed storage backend of HugeGraph. When a graph uses it, HugeGraph-Server keeps no graph data on its own disk. Two other processes do that work:

  • HugeGraph-PD (Placement Driver) owns the cluster metadata: the registered store list, the partition layout of every graph, the partition to store mapping, the graph schema and the schema id counters.
  • HugeGraph-Store owns the key value data itself, replicated across store nodes with Raft.

Server links a PD client and a Store client into its own process. For every read and write it asks PD which partition owns the key and which store node currently leads that partition, then sends the request directly to that store node.

The server side adapter is the hugegraph-hstore module. It registers under the backend name hstore and reports driver version 1.13.

Selecting hstore changes more than where the bytes are written. Server switches these behaviors on the backend type:

AreaWith hstoreWith a local backend
Schema storageSchema is read and written through the PD meta driverSchema lives in the m store
Schema idsAllocated by PD through the PD clientAllocated by the schema store
System storeNone, system data goes to the graph storeSeparate s store
Task schedulerdistributedlocal
Auth managerStandardAuthManagerV2StandardAuthManager
Backend version checkReads the graph storeReads the system store
init-store.shSkips the graph, PD and Store already own the metadataCreates the local store

2 Prerequisites

hstore is not self contained. A PD cluster and at least one Store node must be running before Server opens an hstore graph, and they have to be started in this order:

  1. PD, so that it can form its Raft group.
  2. Store, which registers itself with PD over gRPC. A store whose gRPC address is listed in PD’s own pd.initial-store-list goes to state Up right away. A store that is not in that list, and that PD has never seen Up or Offline before, registers as Pending and has to be activated before it serves data.
  3. Server, which then reads the store list back out of PD.

Default ports the Server side needs to know about:

ProcessgRPC portREST port
PD86868620
Store85008520

pd.peers on the Server side points at the PD gRPC port, not the REST port.

For installing and configuring the other two processes, see Install/Build HugeGraph-PD and Install/Build HugeGraph-Store.

3 Selecting the hstore backend

3.1 Graph configuration file

Set the backend in the graph properties file, for example conf/graphs/hugegraph.properties:

backend=hstore
serializer=binary
store=hugegraph
pd.peers=127.0.0.1:8686

Notes on those four keys:

  • backend=hstore selects the adapter. Since 1.7.0 the allowed values are memory, rocksdb, hbase and hstore. The value is compared case insensitively where the distribution checks it.
  • serializer=binary is required. Registering the hstore backend adds a config space and a store provider but no serializer of its own, and the adapter is written against the binary serializer. The built-in default of serializer is text, so this value has to be written out.
  • store=hugegraph is the namespace part of the name PD sees. Server opens the provider with <graphspace>/<store> and each backing store appends its own suffix, so PD ends up with one graph entry per store: DEFAULT/hugegraph/g for graph data and DEFAULT/hugegraph/m for the schema store slot. graphspace defaults to DEFAULT, while g and m are fixed.
  • pd.peers is the comma separated list of PD gRPC addresses. The adapter reads it from the graph config, not from rest-server.properties, and the graph level metadata connection uses the same value.

If the graph file does not contain pd.peers, Server copies the value from rest-server.properties into the graph config while loading the graph, provided that usePD is true or the backend is hstore. Writing the key explicitly in the graph file is still the clearer option.

3.2 rest-server.properties

# use pd
usePD=true
pd.peers=127.0.0.1:8686

usePD=true makes the Server load its metadata from PD at startup. On that path it connects the meta manager to PD, creates the built-in admin account and the default graph space, loads the graph spaces and services, creates the internal system graph (always with backend=hstore), and loads the graph configs that PD holds.

It is a separate switch from the graph level backend=hstore: a graph can use hstore with usePD left at its default of false, and Server then never opens the PD backed metadata path. The distribution’s own test startup script sets it whenever the backend is hstore.

3.3 The shipped template

The distribution ships a ready made graph file for this backend at conf/graphs/hstore.properties.template. It matches hugegraph.properties except that it sets backend=hstore, leaves pd.peers=127.0.0.1:8686 uncommented, and carries no memory management block.

The hstore Docker image applies that template for you: it deletes conf/graphs/hugegraph.properties and renames the template over it, so a container starts with the hstore backend already selected.

A locally built distribution has the hstore provider compiled in by default. The rocksdb-only Maven profile narrows the compiled backend list to rocksdb, and a distribution built that way rejects backend=hstore with Unsupported backend type.

4 hstore config options

These are the only keys in the hstore config space. They belong in the graph properties file.

config optiondefault valuedescription
hstore.partition_count0Number of partitions, which PD controls partitions based on.
hstore.shard_count0Number of copies, which PD controls partition copies based on.

4.1 hstore.partition_count

Server sends this number to PD once per graph store, the first time the store is opened, together with the graph name. A negative value is rejected at that point with The value of hstore.partition_count cannot be less than 0.

How PD reads the number:

  • 0, the default, means let PD decide. For a graph data store PD uses its own cluster wide partition total, which it derives from the number of entries in pd.initial-store-list, partition.store-max-shard-count and partition.default-shard-count. For the /m and /s stores it uses a fixed count of 1.
  • A value between 1 and that total is used as is.
  • A value above that total is clamped down to it.

The number is applied when the store is first registered with PD, so changing it later in the properties file does not repartition an existing graph.

4.2 hstore.shard_count

hstore.shard_count is declared in the hstore config space and is accepted in the properties file, but no code on the Server side reads it in this release: hstore.partition_count is the only one of the two the adapter reads. The replica count in effect is the one PD is configured with, partition.default-shard-count in PD’s application.yml.

5 Other options that only apply in hstore mode

These keys live in the shared rest-server.properties and graph properties files, but only take effect, or only change behavior, when PD and the hstore backend are in use. The source column gives the file and line on the HugeGraph master branch where the option is declared.

config optionfiledefaultwhy it matters with hstoresource
pd.peersrest-server.properties127.0.0.1:8686PD addresses used for metadata, service discovery and the system graphServerOptions.java:195-201
pd.peers{graph}.properties127.0.0.1:8686PD addresses used by the backend adapter itselfCoreOptions.java:649-654
usePDrest-server.propertiesfalseWhether Server loads its metadata from PD at startupServerOptions.java:390-396
clusterrest-server.propertieshg-testCluster name used as the prefix of every PD metadata keyServerOptions.java:187-193
init_store.enabledrest-server.propertiestrueSet it to false in a PD/Store deployment, where the storage side already owns the metadataServerOptions.java:371-380
graph.load_from_local_configrest-server.propertiesfalseWhether conf/graphs is scanned at startup in addition to the graph configs held in PDServerOptions.java:355-361
auth.graph_storerest-server.propertieshugegraphThe graph that holds auth data, checked against the hstore backend when init-store is offServerOptions.java:591-598
graphspace{graph}.propertiesDEFAULTFirst segment of the graph name PD seesCoreOptions.java:679-685

init-store.sh never initializes an hstore graph. On the enabled path it scans conf/graphs and skips every graph whose backend is hstore. If you turn the whole step off with init_store.enabled=false, it validates instead that the admin account can still be created on the PD startup path: usePD has to be true, the auth graph has to exist locally with backend hstore, and auth.admin_pa has to be set to an explicit non-empty value. Otherwise startup fails rather than handing out the public default password.

6 How the Server finds the stores

The adapter builds its clients once per process, on the first hstore graph it opens:

  1. A PD client config from pd.peers, with the PD authority credentials and the client side partition cache enabled.
  2. The process wide PD client.
  3. The process wide store client, created from that PD client.

Creating the store client installs a PD backed partitioner as the node provider, partitioner and notifier of the store client’s node manager. That partitioner is the whole of the routing logic:

  • Point and prefix requests ask PD for the partition that owns the key, take the leader shard of that partition and send the request to that store id.
  • Code range scans walk the partitions by code until the range is covered, producing one target store per partition.
  • Whole graph scans ask PD for the active stores of the graph and fan out to every one of them.
  • Store address lookup resolves a store id to a host and port through PD.
  • Cache invalidation: when a store answers that a partition leader moved, the notifier updates the partition leader in PD’s client cache and invalidates the stale partition entry, so later requests follow the new leader.

Because the store list comes from PD rather than from configuration, a store node is added or removed by starting or stopping it against the same PD cluster. No Server side config change is needed.

7 Backend capabilities

hstore does not support every query form the local backends do. The differences visible to a user:

FeatureSupported
Scan by key prefixyes
Scan by key rangeyes
Query with range conditionyes
Query with order byyes
Query by pageyes
OLAP propertiesyes
Task and server vertexyes
Scan tokenno
Query schema by nameno
Query by labelno
Query with in conditionno
Query with containsno
Query with contains keyno
Sort results by input idsno
Delete edge by labelno
Update vertex propertyno
Update edge propertyno
Transactionno
Number typeno
Aggregate propertyno
TTLno

Sorting by input ids is off because multi node batch scans group the input keys by store and lose the global order. Vertex and edge property updates are off because the properties are stored in a single cell.

8 Verification

Once the Server is up, the backend metrics endpoint reports the number of stores that PD currently considers active:

curl http://localhost:8080/metrics/backend

The nodes value in the response is the count of active stores PD returns. A nodes value of 0 means the Server reached PD but PD has no store in state Up, which usually means the Store nodes have not registered yet, or registered as Pending because they are not in PD’s pd.initial-store-list.

7 - Configuring the HBase Backend

Overview

The HBase backend stores graph data in Apache HBase tables. HugeGraph acts as an HBase client only: it connects through the HBase ZooKeeper quorum, creates one HBase namespace per graph, and creates that graph’s schema, data and index tables inside it. Counting queries are answered by the HBase AggregateImplementation coprocessor, which HugeGraph attaches to every table it creates.

Note: the HBase backend is deprecated and is planned for removal in HugeGraph 2.0. New deployments should use hstore (distributed) or rocksdb (embedded, the default), and existing HBase deployments should plan a migration.

Since 1.7.0 the only backends shipped in the distribution are hstore, rocksdb, hbase and memory. The backend driver version reported by the HBase provider is 1.12.

Supported HBase Versions

The client jars are pinned to HBase 2.6.5 (hbase-endpoint plus hbase-shaded-client). HBase 2.x is required on the server side: when the detected HBase version is older than 2.0 the scan path rewrites an inclusive stop row into an exclusive one plus a trailing 0 byte, because inclusive stop rows do not work before that release. The CI job and the local Docker image both use HBase 2.6.5, so that is the version the backend is tested against.

Selecting the Backend

Edit conf/graphs/hugegraph.properties of the graph that should use HBase:

backend=hbase
serializer=hbase

# the namespace name is derived from this value
store=hugegraph

hbase.hosts=localhost
hbase.port=2181
hbase.znode_parent=/hbase

Note: serializer must be set to hbase, not to binary. The HBase serializer is a BinarySerializer subclass that drops the id prefix from row keys and writes the pre-split partition prefix that the pre-split vertex and edge tables expect. With serializer=binary neither of these applies.

Then initialize the store and start the server:

./bin/init-store.sh
./bin/start-hugegraph.sh

The default distribution is built with the backends rocksdb, hbase, hstore, so no extra jar is needed. A distribution built with the rocksdb-only Maven profile does not contain the HBase backend, and backend=hbase then fails to open with Not exists BackendStoreProvider: hbase.

All options below live in the graph properties file (conf/graphs/hugegraph.properties), not in rest-server.properties. They are registered only when the hbase backend is part of the distribution.

Connection Options

OptionDefaultDescription
hbase.hostslocalhostThe hostnames or ip addresses of HBase zookeeper, separated with commas. Must not be empty. Maps to hbase.zookeeper.quorum.
hbase.port2181The port address of HBase zookeeper, in the range 1 to 65535. Maps to hbase.zookeeper.property.clientPort.
hbase.znode_parent/hbaseThe znode parent path of HBase zookeeper. Must not be empty. Maps to zookeeper.znode.parent.
hbase.zk_retry3The recovery retry times of HBase zookeeper, in the range 0 to 1000. Maps to zookeeper.recovery.retry.
hbase.threads_max64The max threads num of hbase connections, in the range 1 to 1000. Maps to hbase.hconnection.threads.max, which HBase itself defaults to 256; the lower value is used to avoid running out of memory.

Timeout Options

OptionDefaultDescription
hbase.truncate_timeout30The timeout in seconds of waiting for store truncate. Must be positive. It applies per store, and a graph has three stores, so a truncate can take up to three times this value.
hbase.aggregation_timeout43200 (12 hours)The timeout in seconds of waiting for aggregation. Must be positive. Sets hbase.rpc.timeout on the aggregation client used by count queries.

Kerberos and HBase Site Options

OptionDefaultDescription
hbase.kerberos_enablefalseIs Kerberos authentication enabled for HBase.
hbase.krb5_conf/etc/krb5.confKerberos configuration file, including KDC IP, default realm, etc. Applied as the java.security.krb5.conf system property.
hbase.hbase_site/etc/hbase/conf/hbase-site.xmlThe HBase’s configuration file. It is added as a configuration resource on every connection, whether or not Kerberos is enabled.
hbase.kerberos_principal(empty)The HBase’s principal for kerberos authentication.
hbase.kerberos_keytab(empty)The HBase’s key tab file for kerberos authentication.

When hbase.kerberos_enable=true, HugeGraph sets hadoop.security.authentication and hbase.security.authentication to kerberos on the connection, then logs in from the keytab with the configured principal before opening the connection. A Kerberos setup therefore needs all four of hbase.krb5_conf, hbase.hbase_site, hbase.kerberos_principal and hbase.kerberos_keytab to be valid:

hbase.kerberos_enable=true
hbase.krb5_conf=/etc/krb5.conf
hbase.hbase_site=/etc/hbase/conf/hbase-site.xml
hbase.kerberos_principal=hugegraph/host@EXAMPLE.COM
hbase.kerberos_keytab=/etc/security/keytabs/hugegraph.keytab

hbase.hbase_site is read even with Kerberos disabled, so a path that does not exist is simply an empty resource. Point it at the cluster’s own hbase-site.xml when HBase settings beyond the options above are needed.

Pre-split Partition Options

OptionDefaultDescription
hbase.enable_partitiontrueIs pre-split partitions enabled for HBase. Also decides whether the backend reports support for key-prefix and key-range scans.
hbase.vertex_partitions10The number of partitions of the HBase vertex table. Must not be negative.
hbase.edge_partitions30The number of partitions of the HBase edge table. Must not be negative.

With pre-split enabled, the vertex table is created with hbase.vertex_partitions regions and each of the two edge tables with hbase.edge_partitions regions, and the serializer prefixes row keys with the partition the id hashes to.

Note: set the partition counts to match the actual data volume and the number of region servers before the store is initialized. They change the load speed considerably, and they are only applied at table creation time.

Turning hbase.enable_partition off restores plain, unprefixed row keys. In exchange the backend then reports support for key-prefix scans and key-range scans, which pre-split row keys cannot serve.

Namespace and Table Layout

Each graph maps to one HBase namespace named <graphspace>/<store>, lowercased, with / replaced by _ because an HBase namespace name may only contain alphanumeric characters and the _ character. With the defaults graphspace=DEFAULT and store=hugegraph, the namespace is default_hugegraph.

Inside that namespace a graph keeps three stores, the schema store m, the graph store g and the system store s:

StoreTables
schema (m)VL, EL, PK, IL, C, m_si
graph (g)g_v, g_oe, g_ie, g_si, g_vi, g_ei, g_ii, g_fi, g_li, g_di, g_ai, g_hi, g_ui
system (s)s_v, s_oe, s_ie, s_si, s_vi, s_ei, s_ii, s_fi, s_li, s_di, s_ai, s_hi, s_ui, M

g_v is the vertex table, g_oe and g_ie are the out-edge and in-edge tables, and the remaining g_* tables are the secondary, vertex-label, edge-label, range (int, float, long, double), search, shard and unique index tables. Every table has a single column family named f, and every table is created with the org.apache.hadoop.hbase.coprocessor.AggregateImplementation coprocessor attached. Only g_v, g_oe and g_ie are pre-split; the system store’s copies of those tables are created with a single region.

The M table in the system store holds the backend version written by init-store.sh. It is excluded when a graph is truncated, because losing it makes the version check fail on the next startup. Clearing a graph drops the tables; clearing it with the storage space included drops the whole namespace.

GET /metrics/backend reports the HBase cluster state: cluster_id, master_name, average_load, hbase_version, region_count, leaving_servers, nodes, region_servers, and a servers map with heap, disk, request and per region details for each region server. PUT /graphspaces/{graphspace}/graphs/{name}/compact asks HBase to compact every table of the graph.

Local Testing with Docker

docker/hbase in the server repository builds a standalone HBase 2.6.5 image (hugegraph/hbase:2.6.5, container name hg-hbase-test) for local development and tests. Run these from the repository root.

Start HBase for a HugeGraph server running on the host:

docker compose -p hg-hbase -f docker/hbase/docker-compose.hbase.yml build --no-cache hbase
HBASE_MASTER_HOSTNAME=localhost HBASE_REGIONSERVER_HOSTNAME=localhost \
docker compose -p hg-hbase -f docker/hbase/docker-compose.hbase.yml up -d
until docker exec hg-hbase-test nc -z localhost 2181 >/dev/null 2>&1; do sleep 2; done

Start HBase for a HugeGraph server running in a container on the same Docker network:

HBASE_HOSTNAME=hbase docker compose -p hg-hbase -f docker/hbase/docker-compose.hbase.yml up -d

The advertised hostnames matter: the container writes HBASE_MASTER_HOSTNAME and HBASE_REGIONSERVER_HOSTNAME into its hbase-site.xml on startup, falling back to HBASE_HOSTNAME (default hbase). A client that cannot resolve the advertised name fails with UnknownHostException: hbase:16000 even though ZooKeeper answers.

Ports published to the host:

PortService
2181ZooKeeper, matches the hbase.port default
16000HBase Master RPC
16010HBase Master web UI, http://localhost:16010
16020HBase RegionServer RPC
16030HBase RegionServer web UI, http://localhost:16030

Run the backend test suite against it:

mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,hbase

Stop it and remove its volumes:

docker compose -p hg-hbase -f docker/hbase/docker-compose.hbase.yml down -v

The image starts ZooKeeper, the master and the region server as separate daemons and waits for the master to report a live server before it starts tailing the logs, so the first startup can take a while. Give Docker at least 4 GB of memory. The compose health check has a 90 second start period for the same reason.

Limitations

The HBase backend does not support these features:

  • Transactions. A rollback only discards the batch that has not been committed yet, and a commit writes one table at a time, so it is not atomic across tables.
  • Updating a single vertex or edge property in place, and merging vertex properties. Properties are stored in one cell, so the whole property column is rewritten.
  • Querying schema by name, and querying vertices or edges by label alone. Both would need an HBase secondary index.
  • Deleting edges by label.
  • Queries with an in condition, a contains condition or a contains_key condition.
  • Aggregate properties and OLAP properties.
  • Native number types (the supportsNumberType backend feature is off).
  • Scan tokens.
  • Key-prefix scans and key-range scans while hbase.enable_partition is true.
  • Aggregation other than count. Any other aggregate function is rejected.
  • Snapshots. Creating or resuming a backend snapshot throws UnsupportedOperationException.

Supported features include TTL on vertices and edges, paged queries, order-by queries, range conditions, and sorting by input ids.