Config on Apache HugeGraph

HugeGraph configuration

Mon, 01 Jan 0001 00:00:00 +0000

1 Overview

The directory for the configuration files is hugegraph-release/conf, and all the configurations related to the service and the graph itself are located in this directory.

The main configuration files include gremlin-server.yaml, rest-server.properties, and hugegraph.properties.

The HugeGraphServer integrates the GremlinServer and RestServer internally, and gremlin-server.yaml and rest-server.properties are used to configure these two servers.

GremlinServer: GremlinServer accepts Gremlin statements from users, parses them, and then invokes the Core code.
RestServer: It provides a RESTful API that, based on different HTTP requests, calls the corresponding Core API. If the user’s request body is a Gremlin statement, it will be forwarded to GremlinServer to perform operations on the graph data.

Now let’s introduce these three configuration files one by one.

HugeGraph Config Options

Mon, 01 Jan 0001 00:00:00 +0000

Gremlin Server Config Options

Corresponding configuration file gremlin-server.yaml

config option	default value	description
host	127.0.0.1	The host or ip of Gremlin Server.
port	8182	The listening port of Gremlin Server.
graphs	hugegraph: conf/hugegraph.properties	The map of graphs with name and config file path.
scriptEvaluationTimeout	30000	The timeout for gremlin script execution(millisecond).
channelizer	org.apache.tinkerpop.gremlin.server.channel.HttpChannelizer	Indicates the protocol which the Gremlin Server provides service.
authentication	authenticator: org.apache.hugegraph.auth.StandardAuthenticator, config: {tokens: conf/rest-server.properties}	The authenticator and config(contains tokens path) of authentication mechanism.

Rest Server & API Config Options

Corresponding configuration file rest-server.properties

Built-in User Authentication and Authorization Configuration and Usage in HugeGraph

Mon, 01 Jan 0001 00:00:00 +0000

Overview

To facilitate authentication usage in different user scenarios, HugeGraph currently provides built-in authorization StandardAuthenticator mode, which supports multi-user authentication and fine-grained access control. It adopts a 4-layer design based on “User-UserGroup-Operation-Resource” to flexibly control user roles and permissions (supports multiple GraphServers).

Some key designs of the StandardAuthenticator mode include:

During initialization, a super administrator (admin) user is created. Subsequently, other users can be created by the super administrator. Once newly created users are assigned sufficient permissions, they can create or manage more users.
It supports dynamic creation of users, user groups, and resources, as well as dynamic allocation or revocation of permissions.
Users can belong to one or multiple user groups. Each user group can have permissions to operate on any number of resources. The types of operations include read, write, delete, execute, and others.
“Resource” describes the data in the graph database, such as vertices that meet certain criteria. Each resource consists of three elements: type, label, and properties. There are 18 types in total, with the ability to combine any label and properties. The internal condition of a resource is an AND relationship, while the condition between multiple resources is an OR relationship.

Here is an example to illustrate:

Configuring HugeGraphServer to Use HTTPS Protocol

Mon, 01 Jan 0001 00:00:00 +0000

Overview

By default, HugeGraphServer uses the HTTP protocol. However, if you have security requirements for your requests, you can configure it to use HTTPS.

Server Configuration

Modify the conf/rest-server.properties configuration file and change the schema part of restserver.url to https.

# Set the protocol to HTTPS
restserver.url=https://127.0.0.1:8080
# Server keystore file path. This default value is automatically effective when using HTTPS, and you can modify it as needed.
ssl.keystore_file=conf/hugegraph-server.keystore
# Server keystore file password. This default value is automatically effective when using HTTPS, and you can modify it as needed.
ssl.keystore_password=******

The server’s conf directory already includes a keystore file named hugegraph-server.keystore, and the password for this file is hugegraph. These are the default values when enabling the HTTPS protocol. Users can generate their own keystore file and password, and then modify the values of ssl.keystore_file and ssl.keystore_password.

HugeGraph-Computer Config

Mon, 01 Jan 0001 00:00:00 +0000

Computer Config Options

config option	default value	description
algorithm.message_class	org.apache.hugegraph.computer.core.config.Null	The class of message passed when compute vertex.
algorithm.params_class	org.apache.hugegraph.computer.core.config.Null	The class used to transfer algorithms’ parameters before algorithm been run.
algorithm.result_class	org.apache.hugegraph.computer.core.config.Null	The class of vertex’s value, the instance is used to store computation result for the vertex.
allocator.max_vertices_per_thread	10000	Maximum number of vertices per thread processed in each memory allocator
bsp.etcd_endpoints	http://localhost:2379	The end points to access etcd.
bsp.log_interval	30000	The log interval(in ms) to print the log while waiting bsp event.
bsp.max_super_step	10	The max super step of the algorithm.
bsp.register_timeout	300000	The max timeout to wait for master and works to register.
bsp.wait_master_timeout	86400000	The max timeout(in ms) to wait for master bsp event.
bsp.wait_workers_timeout	86400000	The max timeout to wait for workers bsp event.
hgkv.max_data_block_size	65536	The max byte size of hgkv-file data block.
hgkv.max_file_size	2147483648	The max number of bytes in each hgkv-file.
hgkv.max_merge_files	10	The max number of files to merge at one time.
hgkv.temp_file_dir	/tmp/hgkv	This folder is used to store temporary files, temporary files will be generated during the file merging process.
hugegraph.name	hugegraph	The graph name to load data and write results back.
hugegraph.url	http://127.0.0.1:8080	The hugegraph url to load data and write results back.
input.edge_direction	OUT	The data of the edge in which direction is loaded, when the value is BOTH, the edges in both OUT and IN direction will be loaded.
input.edge_freq	MULTIPLE	The frequency of edges can exist between a pair of vertices, allowed values: [SINGLE, SINGLE_PER_LABEL, MULTIPLE]. SINGLE means that only one edge can exist between a pair of vertices, use sourceId + targetId to identify it; SINGLE_PER_LABEL means that each edge label can exist one edge between a pair of vertices, use sourceId + edgelabel + targetId to identify it; MULTIPLE means that many edge can exist between a pair of vertices, use sourceId + edgelabel + sortValues + targetId to identify it.
input.filter_class	org.apache.hugegraph.computer.core.input.filter.DefaultInputFilter	The class to create input-filter object, input-filter is used to Filter vertex edges according to user needs.
input.loader_schema_path		The schema path of loader input, only takes effect when the input.source_type=loader is enabled
input.loader_struct_path		The struct path of loader input, only takes effect when the input.source_type=loader is enabled
input.max_edges_in_one_vertex	200	The maximum number of adjacent edges allowed to be attached to a vertex, the adjacent edges will be stored and transferred together as a batch unit.
input.source_type	hugegraph-server	The source type to load input data, allowed values: [‘hugegraph-server’, ‘hugegraph-loader’], the ‘hugegraph-loader’ means use hugegraph-loader load data from HDFS or file, if use ‘hugegraph-loader’ load data then please config ‘input.loader_struct_path’ and ‘input.loader_schema_path’.
input.split_fetch_timeout	300	The timeout in seconds to fetch input splits
input.split_max_splits	10000000	The maximum number of input splits
input.split_page_size	500	The page size for streamed load input split data
input.split_size	1048576	The input split size in bytes
job.id	local_0001	The job id on Yarn cluster or K8s cluster.
job.partitions_count	1	The partitions count for computing one graph algorithm job.
job.partitions_thread_nums	4	The number of threads for partition parallel compute.
job.workers_count	1	The workers count for computing one graph algorithm job.
master.computation_class	org.apache.hugegraph.computer.core.master.DefaultMasterComputation	Master-computation is computation that can determine whether to continue next superstep. It runs at the end of each superstep on master.
output.batch_size	500	The batch size of output
output.batch_threads	1	The threads number used to batch output
output.hdfs_core_site_path		The hdfs core site path.
output.hdfs_delimiter	,	The delimiter of hdfs output.
output.hdfs_kerberos_enable	false	Is Kerberos authentication enabled for Hdfs.
output.hdfs_kerberos_keytab		The Hdfs’s key tab file for kerberos authentication.
output.hdfs_kerberos_principal		The Hdfs’s principal for kerberos authentication.
output.hdfs_krb5_conf	/etc/krb5.conf	Kerberos configuration file.
output.hdfs_merge_partitions	true	Whether merge output files of multiple partitions.
output.hdfs_path_prefix	/hugegraph-computer/results	The directory of hdfs output result.
output.hdfs_replication	3	The replication number of hdfs.
output.hdfs_site_path		The hdfs site path.
output.hdfs_url	hdfs://127.0.0.1:9000	The hdfs url of output.
output.hdfs_user	hadoop	The hdfs user of output.
output.output_class	org.apache.hugegraph.computer.core.output.LogOutput	The class to output the computation result of each vertex. Be called after iteration computation.
output.result_name	value	The value is assigned dynamically by #name() of instance created by WORKER_COMPUTATION_CLASS.
output.result_write_type	OLAP_COMMON	The result write-type to output to hugegraph, allowed values are: [OLAP_COMMON, OLAP_SECONDARY, OLAP_RANGE].
output.retry_interval	10	The retry interval when output failed
output.retry_times	3	The retry times when output failed
output.single_threads	1	The threads number used to single output
output.thread_pool_shutdown_timeout	60	The timeout seconds of output threads pool shutdown
output.with_adjacent_edges	false	Output the adjacent edges of the vertex or not
output.with_edge_properties	false	Output the properties of the edge or not
output.with_vertex_properties	false	Output the properties of the vertex or not
sort.thread_nums	4	The number of threads performing internal sorting.
transport.client_connect_timeout	3000	The timeout(in ms) of client connect to server.
transport.client_threads	4	The number of transport threads for client.
transport.close_timeout	10000	The timeout(in ms) of close server or close client.
transport.finish_session_timeout	0	The timeout(in ms) to finish session, 0 means using (transport.sync_request_timeout * transport.max_pending_requests).
transport.heartbeat_interval	20000	The minimum interval(in ms) between heartbeats on client side.
transport.io_mode	AUTO	The network IO Mode, either ‘NIO’, ‘EPOLL’, ‘AUTO’, the ‘AUTO’ means selecting the property mode automatically.
transport.max_pending_requests	8	The max number of client unreceived ack, it will trigger the sending unavailable if the number of unreceived ack >= max_pending_requests.
transport.max_syn_backlog	511	The capacity of SYN queue on server side, 0 means using system default value.
transport.max_timeout_heartbeat_count	120	The maximum times of timeout heartbeat on client side, if the number of timeouts waiting for heartbeat response continuously > max_heartbeat_timeouts the channel will be closed from client side.
transport.min_ack_interval	200	The minimum interval(in ms) of server reply ack.
transport.min_pending_requests	6	The minimum number of client unreceived ack, it will trigger the sending available if the number of unreceived ack < min_pending_requests.
transport.network_retries	3	The number of retry attempts for network communication,if network unstable.
transport.provider_class	org.apache.hugegraph.computer.core.network.netty.NettyTransportProvider	The transport provider, currently only supports Netty.
transport.receive_buffer_size	0	The size of socket receive-buffer in bytes, 0 means using system default value.
transport.recv_file_mode	true	Whether enable receive buffer-file mode, it will receive buffer write file from socket by zero-copy if enable.
transport.send_buffer_size	0	The size of socket send-buffer in bytes, 0 means using system default value.
transport.server_host	127.0.0.1	The server hostname or ip to listen on to transfer data.
transport.server_idle_timeout	360000	The max timeout(in ms) of server idle.
transport.server_port	0	The server port to listen on to transfer data. The system will assign a random port if it’s set to 0.
transport.server_threads	4	The number of transport threads for server.
transport.sync_request_timeout	10000	The timeout(in ms) to wait response after sending sync-request.
transport.tcp_keep_alive	true	Whether enable TCP keep-alive.
transport.transport_epoll_lt	false	Whether enable EPOLL level-trigger.
transport.write_buffer_high_mark	67108864	The high water mark for write buffer in bytes, it will trigger the sending unavailable if the number of queued bytes > write_buffer_high_mark.
transport.write_buffer_low_mark	33554432	The low water mark for write buffer in bytes, it will trigger the sending available if the number of queued bytes < write_buffer_low_mark.org.apache.hugegraph.config.OptionChecker$$Lambda$97/0x00000008001c8440@776a6d9b
transport.write_socket_timeout	3000	The timeout(in ms) to write data to socket buffer.
valuefile.max_segment_size	1073741824	The max number of bytes in each segment of value-file.
worker.combiner_class	org.apache.hugegraph.computer.core.config.Null	Combiner can combine messages into one value for a vertex, for example page-rank algorithm can combine messages of a vertex to a sum value.
worker.computation_class	org.apache.hugegraph.computer.core.config.Null	The class to create worker-computation object, worker-computation is used to compute each vertex in each superstep.
worker.data_dirs	[jobs]	The directories separated by ‘,’ that received vertices and messages can persist into.
worker.edge_properties_combiner_class	org.apache.hugegraph.computer.core.combiner.OverwritePropertiesCombiner	The combiner can combine several properties of the same edge into one properties at inputstep.
worker.partitioner	org.apache.hugegraph.computer.core.graph.partition.HashPartitioner	The partitioner that decides which partition a vertex should be in, and which worker a partition should be in.
worker.received_buffers_bytes_limit	104857600	The limit bytes of buffers of received data, the total size of all buffers can’t excess this limit. If received buffers reach this limit, they will be merged into a file.
worker.vertex_properties_combiner_class	org.apache.hugegraph.computer.core.combiner.OverwritePropertiesCombiner	The combiner can combine several properties of the same vertex into one properties at inputstep.
worker.wait_finish_messages_timeout	86400000	The max timeout(in ms) message-handler wait for finish-message of all workers.
worker.wait_sort_timeout	600000	The max timeout(in ms) message-handler wait for sort-thread to sort one batch of buffers.
worker.write_buffer_capacity	52428800	The initial size of write buffer that used to store vertex or message.
worker.write_buffer_threshold	52428800	The threshold of write buffer, exceeding it will trigger sorting, the write buffer is used to store vertex or message.

K8s Operator Config Options

NOTE: Option needs to be converted through environment variable settings, e.g. k8s.internal_etcd_url => INTERNAL_ETCD_URL