This is the multi-page printable view of this section. .
HugeGraph-Server Configuration
- 1: Server Startup Guide
- 2: Server Complete Configuration Manual
- 3: Built-in User Authentication and Authorization Configuration and Usage in HugeGraph
- 4: Configuring HugeGraphServer to Use HTTPS Protocol
- 5: Configuring the RocksDB Backend
- 6: Configuring the HStore Distributed Backend
- 7: Configuring the HBase Backend
This section covers HugeGraph-Server configuration files, available options, authentication, and HTTPS settings.
1 - Server Startup Guide
1 Overview
The directory for the configuration files is hugegraph-release/conf, and all the configurations related to the service and the graph itself are located in this directory.
The main configuration files include gremlin-server.yaml, rest-server.properties, and hugegraph.properties.
The HugeGraphServer integrates the GremlinServer and RestServer internally, and gremlin-server.yaml and rest-server.properties are used to configure these two servers.
- GremlinServer: GremlinServer accepts Gremlin requests and invokes the graph engine.
- RestServer: It provides a RESTful API that, based on different HTTP requests, calls the corresponding Core API. If the user’s request body is a Gremlin statement, it will be forwarded to GremlinServer to perform operations on the graph data.
Now let’s introduce these three configuration files one by one.
2. gremlin-server.yaml
The main structure of gremlin-server.yaml is shown below. Some imports are omitted from this example; refer to the file included in the release package for the complete content.
In most cases, you only need to pay attention to channelizer, host, and port. Graphs are not loaded from the Gremlin Server graphs section. Whether local graph configurations are loaded is controlled by graph.load_from_local_config in rest-server.properties.
channelizer: The defaultWsAndHttpChannelizersupports both WebSocket and HTTP. Gremlin Console uses WebSocket, while HugeGraph Client, Loader, and Hubble use HTTP.
By default, the GremlinServer serves at 127.0.0.1:8182. If you need to modify it, configure the host and port settings.
host: The hostname or IP address of the machine where the GremlinServer is deployed. GremlinServer is not directly exposed to users, the RestServer forwards Gremlin requests to it.port: The port number of the machine where the GremlinServer is deployed.
Additionally, you need to add the corresponding configuration gremlinserver.url=http://host:port in rest-server.properties.
3. rest-server.properties
The following is an example of the available rest-server.properties options. The current upstream release template does not include graph.load_from_local_config, whose source-code default is false; set it explicitly to true when using local graph configurations under conf/graphs.
restserver.url: The URL at which the RestServer provides its services. Modify it according to the actual environment. If you can’t connet to server from other IP address, try to modify it as specific IP; or modify it ashttp://0.0.0.0to listen all network interfaces as a convenient solution, but need to take care of the network area that might access.graphs: The directory containing graph configuration files. The default is./conf/graphs.init-storescans this directory; the Server loads its properties files only whengraph.load_from_local_config=true.graph.load_from_local_config: Whether the Server reads local graph configurations at startup. Its default value in the source code isfalse.
The current upstream template still uses
arthas.telnet_port,arthas.http_port, andarthas.disabled_commands, butServerOptionsreads the camelCase names shown in the example above. Custom configurations should usearthas.telnetPort,arthas.httpPort, andarthas.disabledCommands.
The
gremlinserver.urlconfiguration option is the URL at which the GremlinServer provides services to the RestServer. By default, it is set tohttp://127.0.0.1:8182. If you need to modify it, it should match thehostandportsettings ingremlin-server.yaml. The value may omit the scheme, as the template does, becausehttp://is prepended when it is missing.
4. hugegraph.properties
hugegraph.properties is a type of file. If the system has multiple graphs, there will be multiple similar files. This file is used to configure parameters related to graph storage and querying. The default content of the file is as follows:
Pay attention to the following uncommented items:
gremlin.graph: The entry point for GremlinServer startup. Users should not modify this item, except to switch it toorg.apache.hugegraph.auth.HugeFactoryAuthProxywhen authentication is enabled.vertex.cache_type/edge.cache_type: The cache implementation, allowed values arel1andl2. The default isl2.backend: The storage backend. Version 1.7.0 supportsmemory,rocksdb,hstore, andhbase.serializer: The serializer used when writing schemas, vertices, and edges to the backend. RocksDB usesbinary.store: The storage name used by the graph in the backend.task.schedule_period,task.retry,task.wait_timeout: Scheduling period (in seconds), retry count, and wait timeout (in seconds) for asynchronous tasks. The scheduler itself is picked from the backend,hstoreuses the distributed scheduler and every other backend uses the local one. The oldtask.scheduler_typekey is ignored.search.text_analyzer/search.text_analyzer_mode: The analyzer used for full-text indexes and its mode. Available analyzers areansj,hanlp,smartcn,jieba,jcseg,mmseg4j, andikanalyzer, and each one accepts its own set of modes.rocksdb.data_path: This item is only meaningful when the backend is set torocksdb. It specifies the data directory for RocksDB, and defaults torocksdb-data/data.rocksdb.wal_path: This item is only meaningful when the backend is set torocksdb. It specifies the log directory for RocksDB, and defaults torocksdb-data/wal.
5. Multi-Graph Configuration
A Server can load multiple graphs, with a separate properties file for each graph. The following example creates a RocksDB graph named hugegraph_rocksdb and an in-memory graph named hugegraph_memory.
[Optional]: Modify rest-server.properties
You can modify the graph profile directory in the graphs option of rest-server.properties. The default configuration is graphs=./conf/graphs, if you want to change it to another directory then adjust the graphs option, e.g. adjust it to graphs=/etc/hugegraph/graphs, example is as follows:
Under conf/graphs, create hugegraph_memory.properties and hugegraph_rocksdb.properties based on hugegraph.properties.
Configure hugegraph_memory.properties as follows:
Configure hugegraph_rocksdb.properties as follows:
Stop the server, execute init-store.sh (to create a new database for the new graph), and restart the server.
Check out created graphs:
Get details of a graph:
2 - Server Complete Configuration Manual
Gremlin Server Config Options
Corresponding configuration file gremlin-server.yaml
| config option | default value | description |
|---|---|---|
| host | 127.0.0.1 | The host or ip of Gremlin Server. |
| port | 8182 | The listening port of Gremlin Server. |
| graphs | {} | Graphs are loaded dynamically by the Server; do not configure them here. |
| evaluationTimeout | 30000 | Gremlin script evaluation timeout in milliseconds. |
| channelizer | org.apache.tinkerpop.gremlin.server.channel.WsAndHttpChannelizer | Handles both WebSocket and HTTP requests. |
| maxContentLength | 65536 | Maximum size in bytes of a request that the server accepts. |
| maxChunkSize | 8192 | Maximum chunk size in bytes of an HTTP request. |
| maxHeaderSize | 8192 | Maximum size in bytes of the HTTP request headers. |
| resultIterationBatchSize | 64 | Number of results returned per batch when streaming a result set. |
| ssl.enabled | false | Whether Gremlin Server serves over TLS. |
| authentication | Not configured | When enabling authentication, configure the authenticator, handler, and path to rest-server.properties. |
Rest Server & API Config Options
Corresponding configuration file rest-server.properties
| config option | default value | description |
|---|---|---|
| graphs | ./conf/graphs | Directory containing graph configuration properties files. |
| graph.load_from_local_config | false | Whether to read the graphs directory when the Server starts; set to true when using local graph configuration. |
| graphs.enable_dynamic_create_drop | true | Whether to enable create or drop graph dynamically. |
| init_store.enabled | true | Whether init-store initializes the local backend stores and the built-in admin account. Set false in distributed deployments (PD/HStore) where the storage side already owns the metadata. |
| server.id | Empty string | The optional legacy id of hugegraph-server. |
| server.role | master | The role of nodes in the cluster, available types are [master, worker, computer] |
| server.role_election | false | Whether to enable role election, if enabled, the server will elect a master node in the cluster. |
| server.node_id | node-id1 | The node id of the server. |
| server.node_role | worker | The node role of the server. |
| server.graphspace | DEFAULT | The graph space of the server. |
| server.service_id | DEFAULT | The service id of the server. |
| server.path_graphspace | DEFAULT | The default path graph space of the server. |
| server.start_ignore_single_graph_error | true | Whether to start ignore single graph error. |
| server.event_hub_threads | 1 | The event hub threads of server. |
| restserver.url | http://127.0.0.1:8080 | The url for listening of graph server. |
| ssl.keystore_file | conf/hugegraph-server.keystore | The path of server keystore file used when https protocol is enabled. |
| ssl.keystore_password | hugegraph | The password of the server keystore file when the https protocol is enabled. |
| white_ip.status | disable | The status of whether enable white ip. |
| restserver.max_worker_threads | 2 * CPUs | The maximum worker threads of rest server. |
| restserver.task_threads | max(4, CPUs / 2) | The task threads of rest server. |
| restserver.min_free_memory | 64 | The minimum free memory(MB) of rest server, requests will be rejected when the available memory of system is lower than this value. |
| restserver.request_timeout | 30 | The time in seconds within which a request must complete, -1 means no timeout. |
| restserver.connection_idle_timeout | 30 | The time in seconds to keep an inactive connection alive, -1 means no timeout. |
| restserver.connection_max_requests | 256 | The max number of HTTP requests allowed to be processed on one keep-alive connection, -1 means unlimited. |
| gremlinserver.url | http://127.0.0.1:8182 | The url of gremlin server. |
| gremlinserver.max_route | 2 * CPUs | The max route number for gremlin server. |
| gremlinserver.timeout | 30 | The timeout in seconds of waiting for gremlin server. |
| batch.max_edges_per_batch | 2500 | The maximum number of edges submitted per batch. |
| batch.max_vertices_per_batch | 2500 | The maximum number of vertices submitted per batch. |
| batch.max_write_ratio | 70 | The maximum thread ratio for batch writing, only take effect if the batch.max_write_threads is 0. |
| batch.max_write_threads | 0 | The maximum threads for batch writing, if the value is 0, the actual value will be set to batch.max_write_ratio * restserver.max_worker_threads. |
| raft.group_peers | 127.0.0.1:8090 | The rpc address of raft group initial peers. |
| auth.authenticator | The class path of authenticator implementation. e.g., org.apache.hugegraph.auth.StandardAuthenticator, or a custom implementation. | |
| auth.graph_store | hugegraph | The name of graph used to store authentication information, like users, only for org.apache.hugegraph.auth.StandardAuthenticator. |
| auth.admin_pa | pa | The default password for built-in admin account, takes effect on first startup. It must be changed before deployment. |
| auth.audit_log_rate | 1000.0 | The max rate of audit log output per user, default value is 1000 records per second. |
| auth.cache_capacity | 10240 | The max cache capacity of each auth cache item. |
| auth.cache_expire | 600 | The expiration time in seconds of auth cache in auth client and auth server. |
| auth.remote_url | If the address is empty, it provide auth service, otherwise it is auth client and also provide auth service through rpc forwarding. The remote url can be set to multiple addresses, which are concat by ‘,’. | |
| auth.token_expire | 86400 | The expiration time in seconds after token created |
| auth.token_secret | Randomly generated at startup | HS256 secret; configure it explicitly if existing tokens must remain valid across restarts. |
| exception.allow_trace | true | Whether to allow exception trace stack. |
| memory_monitor.threshold | 0.85 | Threshold for JVM memory usage monitoring, 1 means disabling the memory monitoring task. |
| memory_monitor.period | 2000 | The period in ms of JVM memory usage monitoring, in each period we will detect the jvm memory usage and take corresponding actions. |
| log.slow_query_threshold | 1000 | The threshold time(ms) of logging slow query, 0 means logging slow query is disabled. |
| log.slow_query_body_limit | 512 | The max bytes of request body recorded in the slow query log, 0 means the body is not recorded. The recorded prefix is written as-is and may contain sensitive Gremlin or Cypher literals. |
Corresponding configuration file rest-server.properties, only used when server.role_election=true.
| config option | default value | description |
|---|---|---|
| server.role.node_external_url | http://127.0.0.1:8080 | The url of external accessibility. |
| server.role.base_timeout | 500 | The role state machine candidate state base timeout time, in ms. |
| server.role.random_timeout | 1000 | The random timeout in ms that be used when candidate node request to become master state to reduce competitive voting. |
| server.role.heartbeat_interval | 2 | The role state machine heartbeat interval second time. |
| server.role.fail_count | 5 | When the node failed count of update or query heartbeat is reaches this threshold, the node will become abdication state to guardsafe property. |
| server.role.master_dead_times | 10 | When the worker node detects that the number of times the master node fails to update heartbeat reaches this threshold, the worker node will become to a candidate node. |
PD/Meta Config Options (Distributed Mode)
Corresponding configuration file rest-server.properties
| config option | default value | description |
|---|---|---|
| usePD | false | Whether use pd. |
| pd.peers | 127.0.0.1:8686 | The pd server peers, separated with commas. |
| cluster | hg-test | The cluster name. |
| metrics.data_to_pd | true | Whether to report metrics data to pd. |
| meta.endpoints | http://127.0.0.1:2379 | The URL of meta endpoints. No code reads this option, so setting it has no effect; the meta connection is built from pd.peers. |
| meta.use_ca | false | Whether to use ca to meta server. |
| meta.ca | The ca file of meta server. | |
| meta.client_ca | The client ca file of meta server. | |
| meta.client_key | The client key file of meta server. |
The HStore backend also reads two options from the graph configuration file {graph-name}.properties. Both default to
0, which means the value is decided by PD:
| config option | default value | description |
|---|---|---|
| hstore.partition_count | 0 | Number of partitions, which PD controls partitions based on. |
| hstore.shard_count | 0 | Number of copies, which PD controls partition copies based on. |
Basic Config Options
Basic Config Options and Backend Config Options correspond to configuration files:{graph-name}.properties, such as hugegraph.properties
| config option | default value | description |
|---|---|---|
| gremlin.graph | org.apache.hugegraph.HugeFactory | Gremlin entrance to create graph. |
| backend | memory | The data store type. For version 1.7.0+ the allowed values are [memory, rocksdb, hstore, hbase]; the shipped conf/graphs/hugegraph.properties sets rocksdb and conf/graphs/hstore.properties.template sets hstore. Note: cassandra, scylladb, mysql, postgresql were removed in 1.7.0 (use <= 1.5.x for legacy backends). |
| serializer | text | The serializer for backend store, built-in values are [text, binary, binaryscatter]; a backend may register its own, like hbase. The shipped graph templates set binary. |
| serializer.buffer_max_capacity | 134217728 | The process-wide max capacity of one serialization buffer in bytes. |
| store | hugegraph | The backend database namespace. |
| store.connection_detect_interval | 600 | The interval in seconds for detecting connections, if the idle time of a connection exceeds this value, detect it and reconnect if needed before using, value 0 means detecting every time. |
| store.graph | g | The graph table name, which store vertex, edge and property. |
| graphspace | DEFAULT | The graph space name. |
| alias.graph.id | The graph alias id. | |
| graph.read_mode | OLTP_ONLY | The graph read mode, which could be ALL | OLTP_ONLY | OLAP_ONLY. |
| pd.peers | 127.0.0.1:8686 | The addresses of pd nodes, separated with commas. Only used by the hstore backend. |
| schema.illegal_name_regex | .\s+$|~. | The regex specified the illegal format for schema name. |
| schema.cache_capacity | 10000 | The max cache size(items) of schema cache. |
| schema.init_template | The template schema used to init graph. | |
| schema.index_rebuild_using_pushdown | true | Whether to use pushdown when to create/rebuild index. |
| vertex.cache_type | l2 | The type of vertex cache, allowed values are [l1, l2]. |
| vertex.cache_capacity | 10000000 | The max cache size(items) of vertex cache. |
| vertex.cache_expire | 600 | The expiration time in seconds of vertex cache. |
| vertex.check_customized_id_exist | false | Whether to check the vertices exist for those using customized id strategy. |
| vertex.default_label | vertex | The default vertex label. |
| vertex.tx_capacity | 10000 | The max size(items) of vertices(uncommitted) in transaction. |
| vertex.check_adjacent_vertex_exist | false | Whether to check the adjacent vertices of edges exist. |
| vertex.lazy_load_adjacent_vertex | true | Whether to lazy load adjacent vertices of edges. |
| vertex.part_edge_commit_size | 5000 | Whether to enable the mode to commit part of edges of vertex, enabled if commit size > 0, 0 means disabled. |
| vertex.encode_primary_key_number | true | Whether to encode number value of primary key in vertex id. |
| vertex.remove_left_index_at_overwrite | false | Whether remove left index at overwrite. |
| edge.cache_type | l2 | The type of edge cache, allowed values are [l1, l2]. |
| edge.cache_capacity | 1000000 | The max cache size(items) of edge cache. |
| edge.cache_expire | 600 | The expiration time in seconds of edge cache. |
| edge.tx_capacity | 10000 | The max size(items) of edges(uncommitted) in transaction. |
| query.page_size | 500 | The size of each page when querying by paging. |
| query.batch_size | 1000 | The size of each batch when querying by batch. |
| query.ignore_invalid_data | true | Whether to ignore invalid data of vertex or edge. |
| query.index_intersect_threshold | 1000 | The maximum number of intermediate results to intersect indexes when querying by multiple single index properties. |
| query.max_indexes_available | 1 | The upper limit of the number of indexes that can be used to query. |
| query.dedup_option | limit | The way to dedup data, allowed values are [limit, global]. |
| query.trust_index | false | Whether to trust index. |
| query.ramtable_edges_capacity | 20000000 | The maximum number of edges in ramtable, include OUT and IN edges. |
| query.ramtable_enable | false | Whether to enable ramtable for query of adjacent edges. |
| query.ramtable_vertices_capacity | 10000000 | The maximum number of vertices in ramtable, generally the largest vertex id is used as capacity. |
| query.optimize_aggregate_by_index | false | Whether to optimize aggregate query(like count) by index. |
| oltp.concurrent_depth | 10 | The min depth to enable concurrent oltp algorithm. |
| oltp.concurrent_threads | max(10, CPUs / 2) | Thread number to concurrently execute oltp algorithm. |
| oltp.collection_type | EC | The implementation type of collections used in oltp algorithm, allowed values are [JCF, EC, FU]. |
| oltp.query_batch_size | 10000 | The size of each batch when executing oltp algorithm. |
| oltp.query_batch_avg_degree_ratio | 0.95 | The ratio of exponential approximation for average degree of iterator when executing oltp algorithm. |
| oltp.query_batch_expect_degree | 100000000 | The expect sum of degree in each batch when executing oltp algorithm. |
| rate_limit.read | 0 | The max rate(times/s) to execute query of vertices/edges. |
| rate_limit.write | 0 | The max rate(items/s) to add/update/delete vertices/edges. |
| task.schedule_period | 10 | Period time in seconds when scheduler to schedule task. |
| task.wait_timeout | 10 | Timeout in seconds for waiting for the task to complete, such as when truncating or clearing the backend. |
| task.retry | 0 | Task retry times, allowed range is [0, 3]. |
| task.input_size_limit | 16777216 | The job input size limit in bytes. |
| task.result_size_limit | 16777216 | The job result size limit in bytes. |
| task.sync_deletion | false | Whether to delete schema or expired data synchronously. |
| task.ttl_delete_batch | 1 | The batch size used to delete expired data. |
| computer.config | ./conf/computer.yaml | The config file path of computer job. |
| k8s.operator_template | ./conf/operator-template.yaml | The path of operator container template. |
| k8s.quota_template | ./conf/resource-quota-template.yaml | The path of resource quota template. |
| search.text_analyzer | ikanalyzer | Choose a text analyzer for searching the vertex/edge properties, available type are [ansj, hanlp, smartcn, jieba, jcseg, mmseg4j, ikanalyzer]. The shipped graph templates set jieba. If use ‘ikanalyzer’, need download jar from ‘https://github.com/apache/hugegraph-doc/raw/ik_binary/dist/server/ikanalyzer-2012_u6.jar' to lib directory |
| search.text_analyzer_mode | smart | Specify the mode for the text analyzer, the available mode of analyzer are {ansj: [BaseAnalysis, IndexAnalysis, ToAnalysis, NlpAnalysis], hanlp: [standard, nlp, index, nShort, shortest, speed], smartcn: [], jieba: [SEARCH, INDEX], jcseg: [Simple, Complex], mmseg4j: [Simple, Complex, MaxWord], ikanalyzer: [smart, max_word]}. |
| snowflake.datacenter_id | 0 | The datacenter id of snowflake id generator. |
| snowflake.force_string | false | Whether to force the snowflake long id to be a string. |
| snowflake.worker_id | 0 | The worker id of snowflake id generator. |
| memory.mode | off-heap | The memory mode used for query in HugeGraph. |
| memory.max_capacity | 1073741824 | The maximum memory capacity in bytes that can be managed for all queries in HugeGraph. |
| memory.one_query_max_capacity | 104857600 | The maximum memory capacity in bytes that can be managed for a query in HugeGraph. |
| memory.alignment | 8 | The alignment used for round memory size. |
The shipped graph configuration templates mark these options as deprecated. They only take effect when
raft.mode=true, and raft.group_peers is read from rest-server.properties instead of the graph file.
| config option | default value | description |
|---|---|---|
| raft.mode | false | Whether the backend storage works in raft mode. |
| raft.safe_read | false | Whether to use linearly consistent read. |
| raft.path | ./raftlog | The log path of current raft node. |
| raft.use_replicator_pipeline | true | Whether to use replicator line, when turned on it multiple logs can be sent in parallel, and the next log doesn’t have to wait for the ack message of the current log to be sent. |
| raft.election_timeout | 10000 | Timeout in milliseconds to launch a round of election. |
| raft.snapshot_interval | 3600 | The interval in seconds to trigger snapshot save. |
| raft.snapshot_threads | 4 | The thread number used to do snapshot. |
| raft.snapshot_parallel_compress | false | Whether to enable parallel compress. |
| raft.snapshot_compress_threads | 4 | The thread number used to do snapshot compress. |
| raft.snapshot_decompress_threads | 4 | The thread number used to do snapshot decompress. |
| raft.backend_threads | CPUs | The thread number used to apply task to backend. |
| raft.read_index_threads | 8 | The thread number used to execute reading index. |
| raft.read_strategy | ReadOnlyLeaseBased | The linearizability of read strategy, allowed values are [ReadOnlyLeaseBased, ReadOnlySafe]. |
| raft.apply_batch | 1 | The apply batch size to trigger disruptor event handler. |
| raft.queue_size | 16384 | The disruptor buffers size for jraft RaftNode, StateMachine and LogManager. |
| raft.queue_publish_timeout | 60 | The timeout in second when publish event into disruptor. |
| raft.rpc_threads | max(CPUs * 2, 80) | The rpc threads for jraft RPC layer. |
| raft.rpc_connect_timeout | 5000 | The rpc connect timeout in milliseconds for jraft rpc. |
| raft.rpc_timeout | 60 | The general rpc timeout in seconds for jraft rpc. |
| raft.install_snapshot_rpc_timeout | 36000 | The install snapshot rpc timeout in seconds for jraft rpc. |
| raft.rpc_buf_low_water_mark | 10485760 | The ChannelOutboundBuffer’s low water mark of netty, when buffer size less than this size, the method ChannelOutboundBuffer.isWritable() will return true, it means that low downstream pressure or good network. |
| raft.rpc_buf_high_water_mark | 20971520 | The ChannelOutboundBuffer’s high water mark of netty, only when buffer size exceed this size, the method ChannelOutboundBuffer.isWritable() will return false, it means that the downstream pressure is too great to process the request or network is very congestion, upstream needs to limit rate at this time. |
RocksDB Backend Config Options
| config option | default value | description |
|---|---|---|
| backend | Must be set to rocksdb. | |
| serializer | Must be set to binary. | |
| rocksdb.data_path | rocksdb-data/data | The path for storing data of RocksDB. |
| rocksdb.wal_path | rocksdb-data/wal | The path for storing WAL of RocksDB. |
| rocksdb.sst_path | The path for ingesting SST file into RocksDB. | |
| rocksdb.data_disks | [] | The optimized disks for storing data of RocksDB. The format of each element: STORE/TABLE: /path/disk.Allowed keys are [g/vertex, g/edge_out, g/edge_in, g/vertex_label_index, g/edge_label_index, g/range_int_index, g/range_float_index, g/range_long_index, g/range_double_index, g/secondary_index, g/search_index, g/shard_index, g/unique_index, g/olap] |
| rocksdb.log_level | INFO | The info log level of RocksDB. |
| rocksdb.num_levels | 7 | Set the number of levels for this database. |
| rocksdb.compaction_style | LEVEL | Set compaction style for RocksDB: LEVEL/UNIVERSAL/FIFO. |
| rocksdb.optimize_mode | true | Optimize for heavy workloads and big datasets. |
| rocksdb.bulkload_mode | false | Switch to the mode to bulk load data into RocksDB. |
| rocksdb.compression_per_level | [none, none, snappy, snappy, snappy, snappy, snappy] | The compression algorithms for different levels of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd. |
| rocksdb.bottommost_compression | none | The compression algorithm for the bottommost level of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd. |
| rocksdb.compression | snappy | The compression algorithm for compressing blocks of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd. |
| rocksdb.max_background_jobs | 8 | Maximum number of concurrent background jobs, including flushes and compactions. |
| rocksdb.max_subcompactions | 4 | The value represents the maximum number of threads per compaction job. |
| rocksdb.delayed_write_rate | 16777216 | The rate limit in bytes/s of user write requests when need to slow down if the compaction gets behind. |
| rocksdb.max_open_files | -1 | The maximum number of open files that can be cached by RocksDB, -1 means no limit. |
| rocksdb.max_manifest_file_size | 104857600 | The max size of manifest file in bytes. |
| rocksdb.skip_stats_update_on_db_open | false | Whether to skip statistics update when opening the database, setting this flag true allows us to not update statistics. |
| rocksdb.skip_check_sst_size_on_db_open | false | Whether to skip checking sizes of all sst files when opening the database. |
| rocksdb.max_file_opening_threads | 16 | The max number of threads used to open files. |
| rocksdb.max_total_wal_size | 0 | Total size of WAL files in bytes. Once WALs exceed this size, we will start forcing the flush of column families related, 0 means no limit. |
| rocksdb.bytes_per_sync | 0 | Allows OS to incrementally sync SST files to disk while they are being written, asynchronously in the background. Issue one request for every bytes_per_sync written. 0 turns it off. |
| rocksdb.wal_bytes_per_sync | 0 | Allows OS to incrementally sync WAL files to disk while they are being written, asynchronously in the background. Issue one request for every bytes_per_sync written. 0 turns it off. |
| rocksdb.strict_bytes_per_sync | false | When true, guarantees SST/WAL files have at most bytes_per_sync/wal_bytes_per_sync bytes submitted for writeback at any given time. This can be used to handle cases where processing speed exceeds I/O speed. |
| rocksdb.db_write_buffer_size | 0 | Total size of write buffers in bytes across all column families, 0 means no limit. |
| rocksdb.log_readahead_size | 0 | The number of bytes to prefetch when reading the log. 0 means the prefetching is disabled. |
| rocksdb.compaction_readahead_size | 0 | The number of bytes to perform bigger reads when doing compaction. If running RocksDB on spinning disks, you should set this to at least 2MB. 0 means the prefetching is disabled. |
| rocksdb.row_cache_capacity | 0 | The capacity in bytes of global cache for table-level rows. 0 means the row_cache is disabled. |
| rocksdb.delete_obsolete_files_period | 21600 | The periodicity in seconds when obsolete files get deleted, 0 means always do full purge. |
| rocksdb.write_buffer_size | 134217728 | Amount of data in bytes to build up in memory. |
| rocksdb.max_write_buffer_number | 6 | The maximum number of write buffers that are built up in memory. |
| rocksdb.min_write_buffer_number_to_merge | 2 | The minimum number of write buffers that will be merged together. |
| rocksdb.max_write_buffer_number_to_maintain | 0 | The total maximum number of write buffers to maintain in memory for conflict checking when transactions are used. |
| rocksdb.memtable_bloom_size_ratio | 0.0 | If prefix-extractor is set and memtable_bloom_size_ratio is not 0, or if memtable_whole_key_filtering is set true, create bloom filter for memtable with the size of write_buffer_size * memtable_bloom_size_ratio. If it is larger than 0.25, it is santinized to 0.25. |
| rocksdb.memtable_whole_key_filtering | false | Enable whole key bloom filter in memtable, it can potentially reduce CPU usage for point-look-ups. Note this will only take effect if memtable_bloom_size_ratio > 0. |
| rocksdb.memtable_huge_page_size | 0 | The page size for huge page TLB for bloom in memtable. If <= 0, not allocate from huge page TLB but from malloc. |
| rocksdb.inplace_update_support | false | Allows thread-safe inplace updates if a put key exists in current memtable and sizeof new value is smaller. |
| rocksdb.level_compaction_dynamic_level_bytes | false | Whether to enable level_compaction_dynamic_level_bytes, if it’s enabled we give max_bytes_for_level_multiplier a priority against max_bytes_for_level_base, the bytes of base level is dynamic for a more predictable LSM tree, it is useful to limit worse case space amplification. Turning this feature on/off for an existing DB can cause unexpected LSM tree structure so it’s not recommended. |
| rocksdb.max_bytes_for_level_base | 536870912 | The upper-bound of the total size of level-1 files in bytes. |
| rocksdb.max_bytes_for_level_multiplier | 10.0 | The ratio between the total size of level (L+1) files and the total size of level L files for all L. |
| rocksdb.target_file_size_base | 67108864 | The target file size for compaction in bytes. |
| rocksdb.target_file_size_multiplier | 1 | The size ratio between a level L file and a level (L+1) file. |
| rocksdb.level0_file_num_compaction_trigger | 2 | Number of files to trigger level-0 compaction. |
| rocksdb.level0_slowdown_writes_trigger | 20 | Soft limit on number of level-0 files for slowing down writes. |
| rocksdb.level0_stop_writes_trigger | 36 | Hard limit on number of level-0 files for stopping writes. |
| rocksdb.soft_pending_compaction_bytes_limit | 68719476736 | The soft limit to impose on pending compaction in bytes. |
| rocksdb.hard_pending_compaction_bytes_limit | 274877906944 | The hard limit to impose on pending compaction in bytes. |
| rocksdb.allow_mmap_writes | false | Allow the OS to mmap file for writing. |
| rocksdb.allow_mmap_reads | false | Allow the OS to mmap file for reading sst tables. |
| rocksdb.use_direct_reads | false | Enable the OS to use direct I/O for reading sst tables. |
| rocksdb.use_direct_io_for_flush_and_compaction | false | Enable the OS to use direct read/writes in flush and compaction. |
| rocksdb.use_fsync | false | If true, then every store to stable storage will issue a fsync. |
| rocksdb.atomic_flush | false | If true, flushing multiple column families and committing their results atomically to MANIFEST. Note that it’s not necessary to set atomic_flush=true if WAL is always enabled. |
| rocksdb.format_version | 5 | The format version of BlockBasedTable, allowed values are 0~5. |
| rocksdb.index_type | kBinarySearch | The index type used to lookup between data blocks with the sst table, allowed values are [kBinarySearch,kHashSearch,kTwoLevelIndexSearch,kBinarySearchWithFirstKey]. |
| rocksdb.data_block_index_type | kDataBlockBinarySearch | The search type used to point lookup in data block with the sst table, allowed values are [kDataBlockBinarySearch,kDataBlockBinaryAndHash]. |
| rocksdb.data_block_hash_table_util_ratio | 0.75 | The hash table utilization ratio value of entries/buckets. It is valid only when data_block_index_type=kDataBlockBinaryAndHash. |
| rocksdb.block_size | 4096 | Approximate size of user data packed per block, Note that it corresponds to uncompressed data. |
| rocksdb.block_size_deviation | 10 | The percentage of free space used to close a block. |
| rocksdb.block_restart_interval | 16 | The block restart interval for delta encoding in blocks. |
| rocksdb.block_cache_capacity | 8388608 | The amount of block cache in bytes that will be used by RocksDB, 0 means no block cache. |
| rocksdb.cache_index_and_filter_blocks | true | Set this option true if we’d put index/filter blocks to the block cache. |
| rocksdb.pin_l0_filter_and_index_blocks_in_cache | true | Set this option true if we’d pin L0 index/filter blocks to the block cache. |
| rocksdb.bloom_filter_bits_per_key | -1 | The bits per key in bloom filter, a good value is 10, which yields a filter with ~ 1% false positive rate. Set bloom_filter_bits_per_key > 0 to enable bloom filter, -1 means no bloom filter (0~0.5 round down to no filter). |
| rocksdb.bloom_filter_block_based_mode | false | If bloom filter is enabled, set this option true to use block based filter rather than full filter. |
| rocksdb.bloom_filter_whole_key_filtering | true | If bloom filter is enabled, set this option true to place whole keys in the bloom filter, else place the prefix of keys when prefix-extractor is set. |
| rocksdb.optimize_filters_for_hits | true | If bloom filter is enabled, this flag allows us to not store filters for the last level. set this option true to optimize the filters mainly for cases where keys are found rather than also optimize for keys missed. |
| rocksdb.partition_filters_and_indexes | false | If bloom filter is enabled, set this option true to use partitioned full filters and indexes for each sst file. This option is incompatible with block-based filters. |
| rocksdb.pin_top_level_index_and_filter | true | If partition_filters_and_indexes is set true, set this option true if we’d pin top-level index of partitioned filter and index blocks to the block cache. |
| rocksdb.prefix_extractor_n_bytes | 0 | The prefix-extractor uses the first N bytes of a key as its prefix, it will use the full key when a key is shorter than the N. 0 means unset prefix-extractor. |
Corresponding configuration file rest-server.properties
| config option | default value | description |
|---|---|---|
| server.use_k8s | false | Whether to use k8s to support multiple tenancy. |
| server.deploy_in_k8s | false | Whether to deploy server in k8s. |
| server.urls_to_pd | http://0.0.0.0:8080 | Used as the server address reserved for PD and provided to clients, only used when starting the server in k8s. |
| server.k8s_url | https://127.0.0.1:8888 | The url of k8s. |
| server.k8s_use_ca | false | Whether to use ca to k8s api server. |
| server.k8s_ca | The ca file of k8s api server. | |
| server.k8s_client_ca | The client ca file of k8s api server. | |
| server.k8s_client_key | The client key file of k8s api server. | |
| k8s.api | false | The k8s api start status when the computer service is enabled. |
| k8s.namespace | hugegraph-computer-system | The namespace used for k8s work when the computer service is enabled. |
| k8s.kubeconfig | The k8s kube config file when the computer service is enabled. | |
| k8s.hugegraph_url | The hugegraph url for k8s work when the computer service is enabled. | |
| k8s.enable_internal_algorithm | true | Whether to open k8s internal algorithm. |
| service.access_pd_name | hg | Service name for server to access pd service. |
| service.access_pd_token | Service token for server to access pd service. | |
| server.k8s_oltp_image | 127.0.0.1/kgs_bd/hugegraphserver:3.0.0 | The oltp server image of k8s. |
| server.k8s_olap_image | hugegraph/hugegraph-server:v1 | The olap server image of k8s. |
| server.k8s_storage_image | hugegraph/hugegraph-server:v1 | The storage server image of k8s. |
| server.default_oltp_k8s_namespace | hugegraph-server | The default oltp namespace for HugeGraph default graph space. |
| server.default_olap_k8s_namespace | hugegraph-computer-system | The default olap namespace for HugeGraph default graph space. |
| k8s.internal_algorithm | [page-rank, degree-centrality, wcc, triangle-count, rings, rings-with-filter, betweenness-centrality, closeness-centrality, lpa, links, kcore, louvain, clustering-coefficient, ppr, subgraph-match] | The names of the built-in k8s algorithms. |
| k8s.algorithms | See ServerOptions.K8S_ALGORITHMS | The name:paramsClass mapping of the built-in k8s algorithms. |
Corresponding configuration file rest-server.properties
| config option | default value | description |
|---|---|---|
| arthas.telnetPort | 8562 | Arthas telnet port. |
| arthas.httpPort | 8561 | Arthas HTTP port. |
| arthas.ip | 0.0.0.0 | Arthas bind IP. |
| arthas.disabledCommands | jad | Disabled Arthas commands, separated by commas. |
Corresponding configuration file rest-server.properties
| config option | default value | description |
|---|---|---|
| rpc.server_host | The hosts/ips bound by rpc server to provide services, empty value means not enabled. | |
| rpc.server_port | 8090 | The port bound by rpc server to provide services. |
| rpc.server_adaptive_port | false | Whether the bound port is adaptive, if it’s enabled, when the port is in use, automatically +1 to detect the next available port. Note that this process is not atomic, so there may still be port conflicts. |
| rpc.server_timeout | 30 | The timeout(in seconds) of rpc server execution. |
| rpc.remote_url | The remote urls of rpc peers, it can be set to multiple addresses, which are concat by ‘,’, empty value means not enabled. | |
| rpc.client_connect_timeout | 20 | The timeout(in seconds) of rpc client connect to rpc server. |
| rpc.client_reconnect_period | 10 | The period(in seconds) of rpc client reconnect to rpc server. |
| rpc.client_read_timeout | 40 | The timeout(in seconds) of rpc client read from rpc server. |
| rpc.client_retries | 3 | Failed retry number of rpc client calls to rpc server. |
| rpc.client_load_balancer | consistentHash | The rpc client uses a load-balancing algorithm to access multiple rpc servers in one cluster. Default value is ‘consistentHash’, means forwarding by request parameters. |
| rpc.protocol | bolt | Rpc communication protocol, client and server need to be specified the same value. |
| rpc.serialization | hessian2 | Rpc serialization type, client and server must set the same value. Note: If you choose ‘protobuf’, you need to add the relative IDL file. (Could refer PD/Store *.proto) |
| rpc.config_order | 999 | Sofa-RPC configuration file loading order, the larger the more later loading. |
| rpc.logger_impl | com.alipay.sofa.rpc.log.SLF4JLoggerImpl | Sofa-RPC log implementation class. |
| config option | default value | description |
|---|---|---|
| backend | Must be set to hbase. | |
| serializer | Must be set to hbase. | |
| hbase.hosts | localhost | The hostnames or ip addresses of HBase zookeeper, separated with commas. |
| hbase.port | 2181 | The port address of HBase zookeeper. |
| hbase.threads_max | 64 | The max threads num of hbase connections. |
| hbase.znode_parent | /hbase | The znode parent path of HBase zookeeper. |
| hbase.zk_retry | 3 | The recovery retry times of HBase zookeeper. |
| hbase.truncate_timeout | 30 | The timeout in seconds of waiting for store truncate. |
| hbase.aggregation_timeout | 43200 | The timeout in seconds of waiting for aggregation. |
| hbase.kerberos_enable | false | Is Kerberos authentication enabled for HBase. |
| hbase.kerberos_keytab | The HBase’s key tab file for kerberos authentication. | |
| hbase.kerberos_principal | The HBase’s principal for kerberos authentication. | |
| hbase.krb5_conf | /etc/krb5.conf | Kerberos configuration file, including KDC IP, default realm, etc. |
| hbase.hbase_site | /etc/hbase/conf/hbase-site.xml | The HBase’s configuration file |
| hbase.enable_partition | true | Is pre-split partitions enabled for HBase. |
| hbase.vertex_partitions | 10 | The number of partitions of the HBase vertex table. |
| hbase.edge_partitions | 30 | The number of partitions of the HBase edge table. |
≤ 1.5 Version Config (Legacy)
The following backend stores are no longer supported in version 1.7.0+ and are only available in version 1.5.x and earlier:
| config option | default value | description |
|---|---|---|
| backend | Must be set to cassandra. | |
| serializer | Must be set to cassandra. | |
| cassandra.host | localhost | The seeds hostname or ip address of cassandra cluster. |
| cassandra.port | 9042 | The seeds port address of cassandra cluster. |
| cassandra.connect_timeout | 5 | The cassandra driver connect server timeout(seconds). |
| cassandra.read_timeout | 20 | The cassandra driver read from server timeout(seconds). |
| cassandra.keyspace.strategy | SimpleStrategy | The replication strategy of keyspace, valid value is SimpleStrategy or NetworkTopologyStrategy. |
| cassandra.keyspace.replication | [3] | The keyspace replication factor of SimpleStrategy, like ‘[3]’.Or replicas in each datacenter of NetworkTopologyStrategy, like ‘[dc1:2,dc2:1]’. |
| cassandra.username | The username to use to login to cassandra cluster. | |
| cassandra.password | The password corresponding to cassandra.username. | |
| cassandra.compression_type | none | The compression algorithm of cassandra transport: none/snappy/lz4. |
| cassandra.jmx_port=7199 | 7199 | The port of JMX API service for cassandra. |
| cassandra.aggregation_timeout | 43200 | The timeout in seconds of waiting for aggregation. |
| config option | default value | description |
|---|---|---|
| backend | Must be set to scylladb. | |
| serializer | Must be set to scylladb. |
Other options are consistent with the Cassandra backend.
| config option | default value | description |
|---|---|---|
| backend | Must be set to mysql. | |
| serializer | Must be set to mysql. | |
| jdbc.driver | com.mysql.jdbc.Driver | The JDBC driver class to connect database. |
| jdbc.url | jdbc:mysql://127.0.0.1:3306 | The url of database in JDBC format. |
| jdbc.username | root | The username to login database. |
| jdbc.password | ****** | The password corresponding to jdbc.username. |
| jdbc.ssl_mode | false | The SSL mode of connections with database. |
| jdbc.reconnect_interval | 3 | The interval(seconds) between reconnections when the database connection fails. |
| jdbc.reconnect_max_times | 3 | The reconnect times when the database connection fails. |
| jdbc.storage_engine | InnoDB | The storage engine of backend store database, like InnoDB/MyISAM/RocksDB for MySQL. |
| jdbc.postgresql.connect_database | template1 | The database used to connect when init store, drop store or check store exist. |
| config option | default value | description |
|---|---|---|
| backend | Must be set to postgresql. | |
| serializer | Must be set to postgresql. |
Other options are consistent with the MySQL backend.
The driver and url of the PostgreSQL backend should be set to:
jdbc.driver=org.postgresql.Driverjdbc.url=jdbc:postgresql://localhost:5432/
3 - Built-in User Authentication and Authorization Configuration and Usage in HugeGraph
Overview
To facilitate authentication usage in different user scenarios, HugeGraph currently provides built-in authorization StandardAuthenticator mode,
which supports multi-user authentication and fine-grained access control. It adopts a 4-layer design based on “User-UserGroup-Operation-Resource” to
flexibly control user roles and permissions (supports multiple GraphServers).
Some key designs of the StandardAuthenticator mode include:
- During initialization, a super administrator (
admin) user is created. Subsequently, other users can be created by the super administrator. Once newly created users are assigned sufficient permissions, they can create or manage more users. - It supports dynamic creation of users, user groups, and resources, as well as dynamic allocation or revocation of permissions.
- Users can belong to one or multiple user groups. Each user group can have permissions to operate on any number of resources. The types of operations include read, write, delete, execute, and others.
- “Resource” describes the data in the graph database, such as vertices that meet certain criteria. Each resource consists of three elements:
type,label, andproperties. There are 18 types in total, with the ability to combine any label and properties. The internal condition of a resource is an AND relationship, while the condition between multiple resources is an OR relationship.
Here is an example to illustrate:
Configure User Authentication
By default, HugeGraph does not enable user authentication, and it needs to be enabled by modifying the configuration file.
Because the flexibility of graph query languages can introduce potential system security risks, do not expose Gremlin, Cypher, or other query endpoints directly to the public network. In production, enable authentication, an IP allowlist, and audit logging, and isolate the Server process with Docker or Kubernetes.
You need to modify the configuration file to enable this feature. HugeGraph provides built-in authentication mode: StandardAuthenticator. This mode supports multi-user authentication and fine-grained permission control. Additionally, developers can implement their own HugeAuthenticator interface to integrate with their existing authentication systems.
HugeGraph uses HTTP Basic Authentication. The value after
Basic is the Base64 encoding of username:password. With curl, pass the credentials directly through -u:
Warning: Versions of HugeGraph-Server prior to 1.5.0 have a JWT-related security vulnerability in the Auth mode.
Users are advised to update to a newer version or manually set the JWT token’s secretKey. It can be set in the rest-server.properties file by setting the auth.token_secret information:
You can also generate it with the following command:
Since 1.5.0 the option defaults to a key generated randomly at startup, so it does not have to be configured. Set it
explicitly when tokens have to survive a restart, or when more than one server must accept the same token. Tokens expire
after auth.token_expire seconds (default 86400).
StandardAuthenticator Mode
The StandardAuthenticator mode supports user authentication and permission control by storing user information in the database backend. This
implementation authenticates users based on their names and passwords (encrypted) stored in the database and controls user permissions based on their
roles. Below is the specific configuration process (requires service restart):
Configure the authenticator and its rest-server file path in the gremlin-server.yaml configuration file:
Configure the authenticator and the graph that stores authorization data in rest-server.properties:
In the above configuration, the graph_store option specifies which graph to use for storing user information. If there are multiple graphs, you can choose any of them.
In the hugegraph{n}.properties configuration file, configure the gremlin.graph information:
For authorization API usage, see the Authentication API documentation.
Custom User Authentication System
If you need to support a more flexible user system, you can customize the authenticator for extension.
Simply implement the org.apache.hugegraph.auth.HugeAuthenticator interface with your custom authenticator,
and then modify the authenticator configuration item in the configuration file to point to your implementation.
Switching authentication mode
When init-store.sh is run for the first time and the admin user does not yet exist, the command prompts for the
administrator password. For an initialized persistent backend, init-store.sh adds the system metadata required for
authentication without deleting existing graph data.
Use docker to enable authentication mode
For versions of the hugegraph/hugegraph image equal to or greater than 1.2.0, you can enable authentication mode while starting the Docker image.
The steps are as follows:
1. Use docker run
To enable authentication mode, add the environment variable PASSWORD=xxx (you can freely set the password) in the docker run command:
2. Use docker-compose
Use docker-compose and set the environment variable PASSWORD=xxx:
3. Enter the container to enable authentication mode
Enter the container first:
Then follow Switching authentication mode
4 - Configuring HugeGraphServer to Use HTTPS Protocol
Overview
By default, HugeGraphServer uses the HTTP protocol. However, if you have security requirements for your requests, you can configure it to use HTTPS.
Server Configuration
Modify the conf/rest-server.properties configuration file and change the schema part of restserver.url to https.
The keystore file is not shipped inside the distribution, because it carries no license declaration. When restserver.url
starts with https and conf/hugegraph-server.keystore is missing, bin/start-hugegraph.sh downloads it from the
binary-1.5 branch of the hugegraph-doc repository before starting the server. The password of that file is hugegraph.
Both values are the defaults of ssl.keystore_file and ssl.keystore_password; users can generate their own keystore
file and password and then change the two options.
Client Configuration
Using HTTPS in HugeGraph-Client
When constructing a HugeClient, pass the HTTPS-related configurations. Here’s an example in Java:
Note: Before version 1.9.0, HugeGraph-Client was created directly using the
newkeyword and did not support the HTTPS protocol. Starting from version 1.9.0, it changed to use the builder pattern and supports configuring the HTTPS protocol.
Using HTTPS in HugeGraph-Loader
When starting an import task, add the following options in the command line:
Under the conf directory of hugegraph-loader, there is already a default client certificate file named hugegraph.truststore, and its password is hugegraph.
Using HTTPS in HugeGraph-Tools
When executing commands, add the following options in the command line:
Under the conf directory of hugegraph-tools, there is already a default client certificate file named hugegraph.truststore, and its password is hugegraph.
How to Generate Certificate Files
This section provides an example of generating certificates. If the default certificate is sufficient or if you already know how to generate certificates, you can skip this section.
Server
- Generate the server’s private key and import it into the server’s keystore file. The
server.keystoreis for the server’s use and contains its private key.
During the process, fill in the description information according to your requirements. The description information for the default certificate is as follows:
- Export the server certificate based on the server’s private key.
server.crt is the server’s certificate.
Client
client.truststore is for the client’s use and contains the trusted certificate.
5 - Configuring the RocksDB Backend
Overview
RocksDB is an embedded LSM-tree key-value store. With the rocksdb backend, HugeGraph-Server keeps all
graph data in RocksDB instances that live inside the server process, so there is no separate storage
service to deploy. This is the backend used by the shipped conf/graphs/hugegraph.properties.
Since version 1.7.0 the server accepts only memory, rocksdb, hbase and hstore as the backend.
The rocksdb backend stores data on the local disks of one server: it does not support shared storage,
so a graph cannot be served by several servers over the same data directory. For a distributed
deployment use the hstore backend with PD and Store.
The RocksDB JNI library is pinned to version 8.10.2 by hugegraph-rocksdb/pom.xml, so the on-disk
format and the option semantics are those of RocksDB 8.10.
The backend driver version reported by this store is 1.11, and it is written into the meta table of
the system store when the graph is initialized.
Selecting the backend
Set the backend and the serializer in the graph properties file (conf/graphs/<graph>.properties):
backend=rocksdbselects the RocksDB store provider.serializer=binaryis the serializer the shipped template uses for this backend. The built-in serializers arebinary,binaryscatterandtext.storeis the database namespace of the graph, and it is also part of the graph name that the provider passes down to the store.
Run bin/init-store.sh once before the first start to create the stores, then start the server. Both
bin/init-store.sh and bin/hugegraph-server.sh load the RocksDB library, so the data directories are
created on the machine that runs them.
The distribution registers the option space and the store provider for each backend listed in the
packaged backend.properties, whose value comes from the hugegraph.backends build property. A default
build registers rocksdb, hbase, hstore; building with -Drocksdb-only activates the rocksdb-only
profile and produces a distribution that registers only rocksdb. A backend that is not registered
fails at startup with Not exists BackendStoreProvider.
The provider registration also adds a second name, rocksdbsst, for the store that writes SST files
instead of a live database. That name is not in the list of allowed backends, so backend=rocksdbsst
is rejected with backend is illegal: rocksdbsst. To load SST files into a normal rocksdb graph, use
rocksdb.sst_path as described below.
Data directory layout
Two directories matter: rocksdb.data_path (default rocksdb-data/data) and rocksdb.wal_path
(default rocksdb-data/wal). Relative paths resolve against the working directory of the server, which
is the installation directory.
Each graph opens three stores: m for schema, g for graph data, and s for the system store. The
store name is appended to both configured paths, so a default single-graph installation looks like this:
Every backend table becomes a RocksDB column family inside the store it belongs to, named
<database>+<table>, where the database is derived from the graph name. Column families of existing
data directories are always reopened, so tables created by an older version stay readable.
Other points to keep in mind:
- Two graphs must not share a data path. When a graph is created by cloning an existing configuration
through the API, the provider appends
_<newGraph>to bothrocksdb.data_pathandrocksdb.wal_path. Deleting such a graph deletes both directories. - Snapshots are created beside the data directory: the last two segments of the data path are rewritten
with a prefix, so with the default paths the snapshot of the graph store goes to
rocksdb-data/<prefix>_data/g. Resuming a snapshot closes the instance, deletes the data directory and moves the snapshot into its place. - With
rocksdb.data_disksset, the tables named there are opened as separate RocksDB instances under the given paths instead of underrocksdb.data_path. The server opens up to 8 instances in parallel, waits at most 600 seconds for the open to finish and 30 seconds for sessions to close.
Path and log options
| config option | default value | description |
|---|---|---|
| rocksdb.data_path | rocksdb-data/data | The path for storing data of RocksDB. Must not be empty. |
| rocksdb.data_disks | [] | The optimized disks for storing data of RocksDB. The format of each element: STORE/TABLE: /path/disk. Allowed keys are [g/vertex, g/edge_out, g/edge_in, g/vertex_label_index, g/edge_label_index, g/range_int_index, g/range_float_index, g/range_long_index, g/range_double_index, g/secondary_index, g/search_index, g/shard_index, g/unique_index, g/olap]. A disk path must differ from rocksdb.data_path. |
| rocksdb.wal_path | rocksdb-data/wal | The path for storing WAL of RocksDB. Must not be empty. |
| rocksdb.sst_path | (empty) | The path for ingesting SST file into RocksDB. Empty disables ingestion. |
| rocksdb.log_level | INFO | The info log level of RocksDB. Allowed values: DEBUG, INFO, WARN, ERROR, FATAL, HEADER. |
Compaction and compression options
| config option | default value | description |
|---|---|---|
| rocksdb.num_levels | 7 | Set the number of levels for this database. Range: 1 to 2^31-1. |
| rocksdb.compaction_style | LEVEL | Set compaction style for RocksDB: LEVEL/UNIVERSAL/FIFO. |
| rocksdb.optimize_mode | true | Optimize for heavy workloads and big datasets. See “How the options are applied” below. |
| rocksdb.bulkload_mode | false | Switch to the mode to bulk load data into RocksDB. |
| rocksdb.compression_per_level | [none, none, snappy, snappy, snappy, snappy, snappy] | The compression algorithms for different levels of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd. The list must be empty or hold exactly rocksdb.num_levels elements. |
| rocksdb.bottommost_compression | none | The compression algorithm for the bottommost level of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd. |
| rocksdb.compression | snappy | The compression algorithm for compressing blocks of RocksDB, allowed values are none/snappy/z/bzip2/lz4/lz4hc/xpress/zstd. |
Database level options
| config option | default value | description |
|---|---|---|
| rocksdb.max_background_jobs | 8 | Maximum number of concurrent background jobs, including flushes and compactions. Range: 1 to 2^31-1. |
| rocksdb.max_subcompactions | 4 | The value represents the maximum number of threads per compaction job. Range: 1 to 2^31-1. |
| rocksdb.delayed_write_rate | 16777216 (16 MB/s) | The rate limit in bytes/s of user write requests when need to slow down if the compaction gets behind. |
| rocksdb.max_open_files | -1 | The maximum number of open files that can be cached by RocksDB, -1 means no limit. |
| rocksdb.max_manifest_file_size | 104857600 (100 MB) | The max size of manifest file in bytes. |
| rocksdb.skip_stats_update_on_db_open | false | Whether to skip statistics update when opening the database, setting this flag true allows us to not update statistics. |
| rocksdb.skip_check_sst_size_on_db_open | false | Whether to skip checking sizes of all sst files when opening the database. |
| rocksdb.max_file_opening_threads | 16 | The max number of threads used to open files. Range: 1 to 2^31-1. |
| rocksdb.max_total_wal_size | 0 | Total size of WAL files in bytes. Once WALs exceed this size, we will start forcing the flush of column families related, 0 means no limit. |
| rocksdb.bytes_per_sync | 0 | Allows OS to incrementally sync SST files to disk while they are being written, asynchronously in the background. Issue one request for every bytes_per_sync written. 0 turns it off. |
| rocksdb.wal_bytes_per_sync | 0 | Same as above for WAL files. 0 turns it off. |
| rocksdb.strict_bytes_per_sync | false | When true, guarantees SST/WAL files have at most bytes_per_sync/wal_bytes_per_sync bytes submitted for writeback at any given time. This can be used to handle cases where processing speed exceeds I/O speed. |
| rocksdb.db_write_buffer_size | 0 | Total size of write buffers in bytes across all column families, 0 means no limit. |
| rocksdb.log_readahead_size | 0 | The number of bytes to prefetch when reading the log. 0 means the prefetching is disabled. |
| rocksdb.compaction_readahead_size | 0 | The number of bytes to perform bigger reads when doing compaction. If running RocksDB on spinning disks, you should set this to at least 2MB. 0 means the prefetching is disabled. |
| rocksdb.row_cache_capacity | 0 | The capacity in bytes of global cache for table-level rows. 0 means the row_cache is disabled. |
| rocksdb.delete_obsolete_files_period | 21600 (6 hours) | The periodicity in seconds when obsolete files get deleted, 0 means always do full purge. The value is converted to microseconds before it reaches RocksDB. |
Memtable options
| config option | default value | description |
|---|---|---|
| rocksdb.write_buffer_size | 134217728 (128 MB) | Amount of data in bytes to build up in memory. Minimum 1 MB. This is per column family. |
| rocksdb.max_write_buffer_number | 6 | The maximum number of write buffers that are built up in memory. Range: 1 to 2^31-1. |
| rocksdb.min_write_buffer_number_to_merge | 2 | The minimum number of write buffers that will be merged together. Range: 1 to 2^31-1. |
| rocksdb.max_write_buffer_number_to_maintain | 0 | The total maximum number of write buffers to maintain in memory for conflict checking when transactions are used. |
| rocksdb.memtable_bloom_size_ratio | 0.0 | If prefix-extractor is set and memtable_bloom_size_ratio is not 0, or if memtable_whole_key_filtering is set true, create bloom filter for memtable with the size of write_buffer_size * memtable_bloom_size_ratio. A value larger than 0.25 is reduced to 0.25. Range: 0.0 to 1.0. |
| rocksdb.memtable_whole_key_filtering | false | Enable whole key bloom filter in memtable, it can potentially reduce CPU usage for point-look-ups. Note this will only take effect if memtable_bloom_size_ratio > 0. |
| rocksdb.memtable_huge_page_size | 0 | The page size for huge page TLB for bloom in memtable. If <= 0, not allocate from huge page TLB but from malloc. |
| rocksdb.inplace_update_support | false | Allows thread-safe inplace updates if a put key exists in current memtable and sizeof new value is smaller. |
Level sizing and write stall options
| config option | default value | description |
|---|---|---|
| rocksdb.level_compaction_dynamic_level_bytes | false | Whether to enable level_compaction_dynamic_level_bytes, if it’s enabled we give max_bytes_for_level_multiplier a priority against max_bytes_for_level_base, the bytes of base level is dynamic for a more predictable LSM tree, it is useful to limit worse case space amplification. Turning this feature on/off for an existing DB can cause unexpected LSM tree structure so it’s not recommended. |
| rocksdb.max_bytes_for_level_base | 536870912 (512 MB) | The upper-bound of the total size of level-1 files in bytes. Minimum 1 MB. |
| rocksdb.max_bytes_for_level_multiplier | 10.0 | The ratio between the total size of level (L+1) files and the total size of level L files for all L. Minimum 1.0. |
| rocksdb.target_file_size_base | 67108864 (64 MB) | The target file size for compaction in bytes. Minimum 1 MB. |
| rocksdb.target_file_size_multiplier | 1 | The size ratio between a level L file and a level (L+1) file. |
| rocksdb.level0_file_num_compaction_trigger | 2 | Number of files to trigger level-0 compaction. |
| rocksdb.level0_slowdown_writes_trigger | 20 | Soft limit on number of level-0 files for slowing down writes. |
| rocksdb.level0_stop_writes_trigger | 36 | Hard limit on number of level-0 files for stopping writes. |
| rocksdb.soft_pending_compaction_bytes_limit | 68719476736 (64 GB) | The soft limit to impose on pending compaction in bytes. Minimum 1 GB. |
| rocksdb.hard_pending_compaction_bytes_limit | 274877906944 (256 GB) | The hard limit to impose on pending compaction in bytes. Minimum 1 GB. |
File I/O options
| config option | default value | description |
|---|---|---|
| rocksdb.allow_mmap_writes | false | Allow the OS to mmap file for writing. |
| rocksdb.allow_mmap_reads | false | Allow the OS to mmap file for reading sst tables. |
| rocksdb.use_direct_reads | false | Enable the OS to use direct I/O for reading sst tables. |
| rocksdb.use_direct_io_for_flush_and_compaction | false | Enable the OS to use direct read/writes in flush and compaction. |
| rocksdb.use_fsync | false | If true, then every store to stable storage will issue a fsync. |
| rocksdb.atomic_flush | false | If true, flushing multiple column families and committing their results atomically to MANIFEST. Note that it’s not necessary to set atomic_flush=true if WAL is always enabled. |
SST table format and block cache options
| config option | default value | description |
|---|---|---|
| rocksdb.format_version | 5 | The format version of BlockBasedTable, allowed values are 0~5. |
| rocksdb.index_type | kBinarySearch | The index type used to lookup between data blocks with the sst table, allowed values are [kBinarySearch, kHashSearch, kTwoLevelIndexSearch, kBinarySearchWithFirstKey]. |
| rocksdb.data_block_index_type | kDataBlockBinarySearch | The search type used to point lookup in data block with the sst table, allowed values are [kDataBlockBinarySearch, kDataBlockBinaryAndHash]. |
| rocksdb.data_block_hash_table_util_ratio | 0.75 | The hash table utilization ratio value of entries/buckets. It is valid only when data_block_index_type=kDataBlockBinaryAndHash. Range: 0.0 to 1.0. |
| rocksdb.block_size | 4096 (4 KB) | Approximate size of user data packed per block, Note that it corresponds to uncompressed data. |
| rocksdb.block_size_deviation | 10 | The percentage of free space used to close a block. Range: 0 to 100. |
| rocksdb.block_restart_interval | 16 | The block restart interval for delta encoding in blocks. |
| rocksdb.block_cache_capacity | 8388608 (8 MB) | The amount of block cache in bytes that will be used by RocksDB, 0 means no block cache. A separate cache of this size is created for each column family. |
Bloom filter options
The options in this group are read only when rocksdb.bloom_filter_bits_per_key is 0 or greater. With
the default value of -1 there is no bloom filter and none of the other options in this table take
effect, including the index and filter block caching ones.
| config option | default value | description |
|---|---|---|
| rocksdb.bloom_filter_bits_per_key | -1 | The bits per key in bloom filter, a good value is 10, which yields a filter with ~ 1% false positive rate. Set bloom_filter_bits_per_key > 0 to enable bloom filter, -1 means no bloom filter (0~0.5 round down to no filter). |
| rocksdb.bloom_filter_block_based_mode | false | If bloom filter is enabled, set this option true to use block based filter rather than full filter. |
| rocksdb.bloom_filter_whole_key_filtering | true | If bloom filter is enabled, set this option true to place whole keys in the bloom filter, else place the prefix of keys when prefix-extractor is set. |
| rocksdb.cache_index_and_filter_blocks | true | Set this option true if we’d put index/filter blocks to the block cache. |
| rocksdb.pin_l0_filter_and_index_blocks_in_cache | true | Set this option true if we’d pin L0 index/filter blocks to the block cache. |
| rocksdb.optimize_filters_for_hits | true | If bloom filter is enabled, this flag allows us to not store filters for the last level. set this option true to optimize the filters mainly for cases where keys are found rather than also optimize for keys missed. This one is applied even when the filter is disabled. |
| rocksdb.partition_filters_and_indexes | false | If bloom filter is enabled, set this option true to use partitioned full filters and indexes for each sst file. This option is incompatible with block-based filters. Enabling it also forces the index type to kTwoLevelIndexSearch and sets the metadata block size to rocksdb.block_size. |
| rocksdb.pin_top_level_index_and_filter | true | If partition_filters_and_indexes is set true, set this option true if we’d pin top-level index of partitioned filter and index blocks to the block cache. |
| rocksdb.prefix_extractor_n_bytes | 0 | The prefix-extractor uses the first N bytes of a key as its prefix, it will use the full key when a key is shorter than the N. 0 means unset prefix-extractor. |
How the options are applied
The server builds the RocksDB option objects once per store and per column family, so a change to any of the options above takes effect on the next server start.
rocksdb.optimize_mode=trueapplies presets before the values in the tables above: at the database level it raises parallelism to half of the available processors (at least one), allows concurrent memtable writes and enables the write thread adaptive yield; at the column family level it calls the RocksDB level-style and universal-style compaction presets. The explicit options are applied afterwards, so any value you set in the properties file wins over the preset.rocksdb.bulkload_mode=truedisables automatic compaction, raises the three level-0 triggers to the maximum integer and the two pending compaction limits to the maximum long value. Turn it off and restart after the load, otherwise compaction never runs.rocksdb.block_cache_capacity=0turns the block cache off completely rather than making it unbounded.rocksdb.prefix_extractor_n_bytesgreater than 0 installs a capped prefix extractor of that length.- Every column family uses the
uint64addmerge operator, which is what the counter table relies on. - The database is created if it is missing, and
avoid_unnecessary_blocking_ioandwrite_dbid_to_manifestare always on.
Memory notes
The caches and write buffers of RocksDB are native allocations, so they are not part of the JVM heap
sizing in bin/hugegraph-server.sh. The GET /metrics/backend endpoint reports what the store uses:
the memory number is the sum of the block cache usage, the pinned block cache usage, the estimated
table reader memory (index and filter blocks) and the size of all memtables, taken from the RocksDB
properties of every open column family.
Two option values multiply with the number of column families:
rocksdb.block_cache_capacitycreates one cache instance per column family, so the total block cache of a server is roughly this value times the number of open tables across them,gandsstores of every graph, plus the instances opened forrocksdb.data_disks.rocksdb.write_buffer_sizetimesrocksdb.max_write_buffer_numberbounds the memtable memory of one column family.rocksdb.db_write_buffer_sizecaps the total across all column families of one store, and its default of 0 means there is no such cap.
rocksdb.row_cache_capacity is different: it is one cache per store, and 0 disables it.
Ingesting SST files
Setting rocksdb.sst_path turns on ingestion. When a store is opened, and again whenever tables are
created, the server walks <sst_path>/<column family>/, collects every non-empty *.sst file below it
and ingests those files into the matching column family. The files are moved rather than copied, so the
source directory is consumed by the ingestion.
Raft mode
The RocksDB backend can still run behind the raft state machine: with raft.mode=true the store
provider of any local backend is wrapped by the raft provider. The wrapper rejects backends with shared
storage, so rocksdb is accepted while hbase is not. Under raft mode a RocksDB session writes with
the WAL disabled and without sync, because the state machine can restore from a snapshot plus the raft
log, and snapshots are supported by this backend.
Notes for anyone using it:
bin/init-store.shforcesraft.mode=falsewhile it initializes the backend, so initialization never goes through raft.- The shipped
conf/graphs/hugegraph.propertiesmarks the raft options as deprecated. Distributed deployments of 1.7.0 and later use thehstorebackend with PD and Store instead. - The raft peer endpoints are served under
graphspaces/{graphspace}/graphs/{graph}/raft/, withlist_peers,get_leader,set_leader,transfer_leader,add_peerandremove_peer.bin/raft-tools.shwraps the same operations, but it still builds URLs without the graphspace segment, so the path has to be adjusted for a 1.7.0 server. - The remaining
raft.*options are listed in the Server Complete Configuration Manual.
Backend capabilities
The feature flags of this backend affect what the server can push down to the store:
- Scans by key prefix and by key range, paged queries, range conditions and order-by are supported.
- There is no index inside RocksDB, so querying schema by name, querying by label and deleting edges by label are done by the server instead of the store.
- Transactions are supported through RocksDB write batches.
- Snapshots are supported, which is what raft mode and backup rely on.
- Shared storage is not supported, so one data directory belongs to one server.
- Olap properties are supported, and their tables are created as extra column families.
- The store does not expire data by itself, so the server filters out elements whose TTL has passed when it reads them.
in,containsandcontains_keyconditions, aggregate properties and vertex or edge property updates in place are not supported at the store level.
Platform note for riscv64
On Linux riscv64 the RocksDB JNI library needs libatomic.so.1. bin/util.sh looks for it and adds it
to LD_PRELOAD before bin/hugegraph-server.sh, bin/init-store.sh and bin/dump-store.sh start the
JVM. If it is missing, those scripts stop with
RISC-V RocksDB requires libatomic.so.1; install libatomic1, and installing the libatomic1 package
fixes it.
6 - Configuring the HStore Distributed Backend
1 Overview
hstore is the distributed storage backend of HugeGraph. When a graph uses it, HugeGraph-Server keeps no
graph data on its own disk. Two other processes do that work:
- HugeGraph-PD (Placement Driver) owns the cluster metadata: the registered store list, the partition layout of every graph, the partition to store mapping, the graph schema and the schema id counters.
- HugeGraph-Store owns the key value data itself, replicated across store nodes with Raft.
Server links a PD client and a Store client into its own process. For every read and write it asks PD which partition owns the key and which store node currently leads that partition, then sends the request directly to that store node.
The server side adapter is the hugegraph-hstore module. It registers under the backend name hstore and
reports driver version 1.13.
Selecting hstore changes more than where the bytes are written. Server switches these behaviors on the
backend type:
| Area | With hstore | With a local backend |
|---|---|---|
| Schema storage | Schema is read and written through the PD meta driver | Schema lives in the m store |
| Schema ids | Allocated by PD through the PD client | Allocated by the schema store |
| System store | None, system data goes to the graph store | Separate s store |
| Task scheduler | distributed | local |
| Auth manager | StandardAuthManagerV2 | StandardAuthManager |
| Backend version check | Reads the graph store | Reads the system store |
init-store.sh | Skips the graph, PD and Store already own the metadata | Creates the local store |
2 Prerequisites
hstore is not self contained. A PD cluster and at least one Store node must be running before Server opens
an hstore graph, and they have to be started in this order:
- PD, so that it can form its Raft group.
- Store, which registers itself with PD over gRPC. A store whose gRPC address is listed in PD’s own
pd.initial-store-listgoes to stateUpright away. A store that is not in that list, and that PD has never seenUporOfflinebefore, registers asPendingand has to be activated before it serves data. - Server, which then reads the store list back out of PD.
Default ports the Server side needs to know about:
| Process | gRPC port | REST port |
|---|---|---|
| PD | 8686 | 8620 |
| Store | 8500 | 8520 |
pd.peers on the Server side points at the PD gRPC port, not the REST port.
For installing and configuring the other two processes, see Install/Build HugeGraph-PD and Install/Build HugeGraph-Store.
3 Selecting the hstore backend
3.1 Graph configuration file
Set the backend in the graph properties file, for example conf/graphs/hugegraph.properties:
Notes on those four keys:
backend=hstoreselects the adapter. Since 1.7.0 the allowed values arememory,rocksdb,hbaseandhstore. The value is compared case insensitively where the distribution checks it.serializer=binaryis required. Registering thehstorebackend adds a config space and a store provider but no serializer of its own, and the adapter is written against the binary serializer. The built-in default ofserializeristext, so this value has to be written out.store=hugegraphis the namespace part of the name PD sees. Server opens the provider with<graphspace>/<store>and each backing store appends its own suffix, so PD ends up with one graph entry per store:DEFAULT/hugegraph/gfor graph data andDEFAULT/hugegraph/mfor the schema store slot.graphspacedefaults toDEFAULT, whilegandmare fixed.pd.peersis the comma separated list of PD gRPC addresses. The adapter reads it from the graph config, not fromrest-server.properties, and the graph level metadata connection uses the same value.
If the graph file does not contain pd.peers, Server copies the value from rest-server.properties into the
graph config while loading the graph, provided that usePD is true or the backend is hstore. Writing the
key explicitly in the graph file is still the clearer option.
3.2 rest-server.properties
usePD=true makes the Server load its metadata from PD at startup. On that path it connects the meta manager
to PD, creates the built-in admin account and the default graph space, loads the graph spaces and services,
creates the internal system graph (always with backend=hstore), and loads the graph configs that PD holds.
It is a separate switch from the graph level backend=hstore: a graph can use hstore with usePD left at
its default of false, and Server then never opens the PD backed metadata path. The distribution’s own test
startup script sets it whenever the backend is hstore.
3.3 The shipped template
The distribution ships a ready made graph file for this backend at
conf/graphs/hstore.properties.template. It matches hugegraph.properties except that it sets
backend=hstore, leaves pd.peers=127.0.0.1:8686 uncommented, and carries no memory management block.
The hstore Docker image applies that template for you: it deletes conf/graphs/hugegraph.properties and
renames the template over it, so a container starts with the hstore backend already selected.
A locally built distribution has the hstore provider compiled in by default. The rocksdb-only Maven
profile narrows the compiled backend list to rocksdb, and a distribution built that way rejects
backend=hstore with Unsupported backend type.
4 hstore config options
These are the only keys in the hstore config space. They belong in the graph properties file.
| config option | default value | description |
|---|---|---|
| hstore.partition_count | 0 | Number of partitions, which PD controls partitions based on. |
| hstore.shard_count | 0 | Number of copies, which PD controls partition copies based on. |
4.1 hstore.partition_count
Server sends this number to PD once per graph store, the first time the store is opened, together with the
graph name. A negative value is rejected at that point with
The value of hstore.partition_count cannot be less than 0.
How PD reads the number:
0, the default, means let PD decide. For a graph data store PD uses its own cluster wide partition total, which it derives from the number of entries inpd.initial-store-list,partition.store-max-shard-countandpartition.default-shard-count. For the/mand/sstores it uses a fixed count of1.- A value between
1and that total is used as is. - A value above that total is clamped down to it.
The number is applied when the store is first registered with PD, so changing it later in the properties file does not repartition an existing graph.
4.2 hstore.shard_count
hstore.shard_count is declared in the hstore config space and is accepted in the properties file, but no
code on the Server side reads it in this release: hstore.partition_count is the only one of the two the
adapter reads. The replica count in effect is the one PD is configured with,
partition.default-shard-count in PD’s application.yml.
5 Other options that only apply in hstore mode
These keys live in the shared rest-server.properties and graph properties files, but only take effect, or
only change behavior, when PD and the hstore backend are in use. The source column gives the file and line
on the HugeGraph master branch where the option is declared.
| config option | file | default | why it matters with hstore | source |
|---|---|---|---|---|
| pd.peers | rest-server.properties | 127.0.0.1:8686 | PD addresses used for metadata, service discovery and the system graph | ServerOptions.java:195-201 |
| pd.peers | {graph}.properties | 127.0.0.1:8686 | PD addresses used by the backend adapter itself | CoreOptions.java:649-654 |
| usePD | rest-server.properties | false | Whether Server loads its metadata from PD at startup | ServerOptions.java:390-396 |
| cluster | rest-server.properties | hg-test | Cluster name used as the prefix of every PD metadata key | ServerOptions.java:187-193 |
| init_store.enabled | rest-server.properties | true | Set it to false in a PD/Store deployment, where the storage side already owns the metadata | ServerOptions.java:371-380 |
| graph.load_from_local_config | rest-server.properties | false | Whether conf/graphs is scanned at startup in addition to the graph configs held in PD | ServerOptions.java:355-361 |
| auth.graph_store | rest-server.properties | hugegraph | The graph that holds auth data, checked against the hstore backend when init-store is off | ServerOptions.java:591-598 |
| graphspace | {graph}.properties | DEFAULT | First segment of the graph name PD sees | CoreOptions.java:679-685 |
init-store.sh never initializes an hstore graph. On the enabled path it scans conf/graphs and skips
every graph whose backend is hstore. If you turn the whole step off with init_store.enabled=false, it
validates instead that the admin account can still be created on the PD startup path: usePD has to be true,
the auth graph has to exist locally with backend hstore, and auth.admin_pa has to be set to an explicit
non-empty value. Otherwise startup fails rather than handing out the public default password.
6 How the Server finds the stores
The adapter builds its clients once per process, on the first hstore graph it opens:
- A PD client config from
pd.peers, with the PD authority credentials and the client side partition cache enabled. - The process wide PD client.
- The process wide store client, created from that PD client.
Creating the store client installs a PD backed partitioner as the node provider, partitioner and notifier of the store client’s node manager. That partitioner is the whole of the routing logic:
- Point and prefix requests ask PD for the partition that owns the key, take the leader shard of that partition and send the request to that store id.
- Code range scans walk the partitions by code until the range is covered, producing one target store per partition.
- Whole graph scans ask PD for the active stores of the graph and fan out to every one of them.
- Store address lookup resolves a store id to a host and port through PD.
- Cache invalidation: when a store answers that a partition leader moved, the notifier updates the partition leader in PD’s client cache and invalidates the stale partition entry, so later requests follow the new leader.
Because the store list comes from PD rather than from configuration, a store node is added or removed by starting or stopping it against the same PD cluster. No Server side config change is needed.
7 Backend capabilities
hstore does not support every query form the local backends do. The differences visible to a user:
| Feature | Supported |
|---|---|
| Scan by key prefix | yes |
| Scan by key range | yes |
| Query with range condition | yes |
| Query with order by | yes |
| Query by page | yes |
| OLAP properties | yes |
| Task and server vertex | yes |
| Scan token | no |
| Query schema by name | no |
| Query by label | no |
Query with in condition | no |
Query with contains | no |
Query with contains key | no |
| Sort results by input ids | no |
| Delete edge by label | no |
| Update vertex property | no |
| Update edge property | no |
| Transaction | no |
| Number type | no |
| Aggregate property | no |
| TTL | no |
Sorting by input ids is off because multi node batch scans group the input keys by store and lose the global order. Vertex and edge property updates are off because the properties are stored in a single cell.
8 Verification
Once the Server is up, the backend metrics endpoint reports the number of stores that PD currently considers active:
The nodes value in the response is the count of active stores PD returns. A nodes value of 0 means the
Server reached PD but PD has no store in state Up, which usually means the Store nodes have not registered
yet, or registered as Pending because they are not in PD’s pd.initial-store-list.
7 - Configuring the HBase Backend
Overview
The HBase backend stores graph data in Apache HBase tables. HugeGraph acts as an HBase client only: it connects through the HBase ZooKeeper quorum, creates one HBase namespace per graph, and creates that graph’s schema, data and index tables inside it. Counting queries are answered by the HBase AggregateImplementation coprocessor, which HugeGraph attaches to every table it creates.
Note: the HBase backend is deprecated and is planned for removal in HugeGraph 2.0. New deployments should use
hstore(distributed) orrocksdb(embedded, the default), and existing HBase deployments should plan a migration.
Since 1.7.0 the only backends shipped in the distribution are hstore, rocksdb, hbase and memory. The backend driver version reported by the HBase provider is 1.12.
Supported HBase Versions
The client jars are pinned to HBase 2.6.5 (hbase-endpoint plus hbase-shaded-client). HBase 2.x is required on the server side: when the detected HBase version is older than 2.0 the scan path rewrites an inclusive stop row into an exclusive one plus a trailing 0 byte, because inclusive stop rows do not work before that release. The CI job and the local Docker image both use HBase 2.6.5, so that is the version the backend is tested against.
Selecting the Backend
Edit conf/graphs/hugegraph.properties of the graph that should use HBase:
Note:
serializermust be set tohbase, not tobinary. The HBase serializer is aBinarySerializersubclass that drops the id prefix from row keys and writes the pre-split partition prefix that the pre-split vertex and edge tables expect. Withserializer=binaryneither of these applies.
Then initialize the store and start the server:
The default distribution is built with the backends rocksdb, hbase, hstore, so no extra jar is needed. A distribution built with the rocksdb-only Maven profile does not contain the HBase backend, and backend=hbase then fails to open with Not exists BackendStoreProvider: hbase.
All options below live in the graph properties file (conf/graphs/hugegraph.properties), not in rest-server.properties. They are registered only when the hbase backend is part of the distribution.
Connection Options
| Option | Default | Description |
|---|---|---|
| hbase.hosts | localhost | The hostnames or ip addresses of HBase zookeeper, separated with commas. Must not be empty. Maps to hbase.zookeeper.quorum. |
| hbase.port | 2181 | The port address of HBase zookeeper, in the range 1 to 65535. Maps to hbase.zookeeper.property.clientPort. |
| hbase.znode_parent | /hbase | The znode parent path of HBase zookeeper. Must not be empty. Maps to zookeeper.znode.parent. |
| hbase.zk_retry | 3 | The recovery retry times of HBase zookeeper, in the range 0 to 1000. Maps to zookeeper.recovery.retry. |
| hbase.threads_max | 64 | The max threads num of hbase connections, in the range 1 to 1000. Maps to hbase.hconnection.threads.max, which HBase itself defaults to 256; the lower value is used to avoid running out of memory. |
Timeout Options
| Option | Default | Description |
|---|---|---|
| hbase.truncate_timeout | 30 | The timeout in seconds of waiting for store truncate. Must be positive. It applies per store, and a graph has three stores, so a truncate can take up to three times this value. |
| hbase.aggregation_timeout | 43200 (12 hours) | The timeout in seconds of waiting for aggregation. Must be positive. Sets hbase.rpc.timeout on the aggregation client used by count queries. |
Kerberos and HBase Site Options
| Option | Default | Description |
|---|---|---|
| hbase.kerberos_enable | false | Is Kerberos authentication enabled for HBase. |
| hbase.krb5_conf | /etc/krb5.conf | Kerberos configuration file, including KDC IP, default realm, etc. Applied as the java.security.krb5.conf system property. |
| hbase.hbase_site | /etc/hbase/conf/hbase-site.xml | The HBase’s configuration file. It is added as a configuration resource on every connection, whether or not Kerberos is enabled. |
| hbase.kerberos_principal | (empty) | The HBase’s principal for kerberos authentication. |
| hbase.kerberos_keytab | (empty) | The HBase’s key tab file for kerberos authentication. |
When hbase.kerberos_enable=true, HugeGraph sets hadoop.security.authentication and hbase.security.authentication to kerberos on the connection, then logs in from the keytab with the configured principal before opening the connection. A Kerberos setup therefore needs all four of hbase.krb5_conf, hbase.hbase_site, hbase.kerberos_principal and hbase.kerberos_keytab to be valid:
hbase.hbase_site is read even with Kerberos disabled, so a path that does not exist is simply an empty resource. Point it at the cluster’s own hbase-site.xml when HBase settings beyond the options above are needed.
Pre-split Partition Options
| Option | Default | Description |
|---|---|---|
| hbase.enable_partition | true | Is pre-split partitions enabled for HBase. Also decides whether the backend reports support for key-prefix and key-range scans. |
| hbase.vertex_partitions | 10 | The number of partitions of the HBase vertex table. Must not be negative. |
| hbase.edge_partitions | 30 | The number of partitions of the HBase edge table. Must not be negative. |
With pre-split enabled, the vertex table is created with hbase.vertex_partitions regions and each of the two edge tables with hbase.edge_partitions regions, and the serializer prefixes row keys with the partition the id hashes to.
Note: set the partition counts to match the actual data volume and the number of region servers before the store is initialized. They change the load speed considerably, and they are only applied at table creation time.
Turning hbase.enable_partition off restores plain, unprefixed row keys. In exchange the backend then reports support for key-prefix scans and key-range scans, which pre-split row keys cannot serve.
Namespace and Table Layout
Each graph maps to one HBase namespace named <graphspace>/<store>, lowercased, with / replaced by _ because an HBase namespace name may only contain alphanumeric characters and the _ character. With the defaults graphspace=DEFAULT and store=hugegraph, the namespace is default_hugegraph.
Inside that namespace a graph keeps three stores, the schema store m, the graph store g and the system store s:
| Store | Tables |
|---|---|
schema (m) | VL, EL, PK, IL, C, m_si |
graph (g) | g_v, g_oe, g_ie, g_si, g_vi, g_ei, g_ii, g_fi, g_li, g_di, g_ai, g_hi, g_ui |
system (s) | s_v, s_oe, s_ie, s_si, s_vi, s_ei, s_ii, s_fi, s_li, s_di, s_ai, s_hi, s_ui, M |
g_v is the vertex table, g_oe and g_ie are the out-edge and in-edge tables, and the remaining g_* tables are the secondary, vertex-label, edge-label, range (int, float, long, double), search, shard and unique index tables. Every table has a single column family named f, and every table is created with the org.apache.hadoop.hbase.coprocessor.AggregateImplementation coprocessor attached. Only g_v, g_oe and g_ie are pre-split; the system store’s copies of those tables are created with a single region.
The M table in the system store holds the backend version written by init-store.sh. It is excluded when a graph is truncated, because losing it makes the version check fail on the next startup. Clearing a graph drops the tables; clearing it with the storage space included drops the whole namespace.
GET /metrics/backend reports the HBase cluster state: cluster_id, master_name, average_load, hbase_version, region_count, leaving_servers, nodes, region_servers, and a servers map with heap, disk, request and per region details for each region server. PUT /graphspaces/{graphspace}/graphs/{name}/compact asks HBase to compact every table of the graph.
Local Testing with Docker
docker/hbase in the server repository builds a standalone HBase 2.6.5 image (hugegraph/hbase:2.6.5, container name hg-hbase-test) for local development and tests. Run these from the repository root.
Start HBase for a HugeGraph server running on the host:
Start HBase for a HugeGraph server running in a container on the same Docker network:
The advertised hostnames matter: the container writes HBASE_MASTER_HOSTNAME and HBASE_REGIONSERVER_HOSTNAME into its hbase-site.xml on startup, falling back to HBASE_HOSTNAME (default hbase). A client that cannot resolve the advertised name fails with UnknownHostException: hbase:16000 even though ZooKeeper answers.
Ports published to the host:
| Port | Service |
|---|---|
| 2181 | ZooKeeper, matches the hbase.port default |
| 16000 | HBase Master RPC |
| 16010 | HBase Master web UI, http://localhost:16010 |
| 16020 | HBase RegionServer RPC |
| 16030 | HBase RegionServer web UI, http://localhost:16030 |
Run the backend test suite against it:
Stop it and remove its volumes:
The image starts ZooKeeper, the master and the region server as separate daemons and waits for the master to report a live server before it starts tailing the logs, so the first startup can take a while. Give Docker at least 4 GB of memory. The compose health check has a 90 second start period for the same reason.
Limitations
The HBase backend does not support these features:
- Transactions. A rollback only discards the batch that has not been committed yet, and a commit writes one table at a time, so it is not atomic across tables.
- Updating a single vertex or edge property in place, and merging vertex properties. Properties are stored in one cell, so the whole property column is rewritten.
- Querying schema by name, and querying vertices or edges by label alone. Both would need an HBase secondary index.
- Deleting edges by label.
- Queries with an
incondition, acontainscondition or acontains_keycondition. - Aggregate properties and OLAP properties.
- Native number types (the
supportsNumberTypebackend feature is off). - Scan tokens.
- Key-prefix scans and key-range scans while
hbase.enable_partitionistrue. - Aggregation other than
count. Any other aggregate function is rejected. - Snapshots. Creating or resuming a backend snapshot throws
UnsupportedOperationException.
Supported features include TTL on vertices and edges, paged queries, order-by queries, range conditions, and sorting by input ids.