Skip to content

1 - HugeGraph (OLTP)

This section covers the HugeGraph Server standalone quick start and the PD and Store guides needed for distributed deployments.

See the DeepWiki project overview. For installation, configuration, and runtime behavior, use the documentation and code for the relevant HugeGraph version.

Start with the Server standalone quick start, which distinguishes ASF 1.7.0 release packages from builds of master. HStore deployments also require PD and Store.

GitHub Access: https://github.com/apache/hugegraph

1.1 - HugeGraph Server Quick Start

1 HugeGraph Server Overview

apache/hugegraph is the main repository for the HugeGraph graph database. Its top-level modules include hugegraph-server, hugegraph-pd, and hugegraph-store. This page describes the hugegraph-server module and the service it runs.

The hugegraph-server module contains hugegraph-core, hugegraph-api, hugegraph-dist, and storage adapters. Core implements the property graph model, transactions, and TinkerPop interfaces. API provides the HTTP service and delegates client requests to Core. Graph data is stored in RocksDB (the default standalone backend), HStore (distributed), or HBase.

Warning

Version scope: This guide distinguishes between official releases and the development master branch. For official installations, use official release archives (such as 1.7.0 or the upcoming 1.8.0). Source builds, default configurations, and startup behaviors follow the apache/hugegraph master branch (currently targeting 1.8.0). A master build is not an ASF release artifact, even if its version number matches a release. Supported storage backends include RocksDB, HStore, and HBase; see 1.5.x documentation for deprecated legacy backends.

Naming: HugeGraph means the overall project or main repository, hugegraph-server is its Server module, and HugeGraphServer is the Java service class. Server below means the running graph database service.

flowchart TD
    A{Choose installation source}
    A -->|Release| R["HugeGraph 1.7.0 archive"]
    R --> V[Verify SHA-512]
    V --> X[Extract and use bundled configuration]
    A -->|Development| M["master source"]
    M --> B["mvn package -DskipTests"]
    B --> C[Check RocksDB configuration in conf/graphs]
    C --> I[bin/init-store.sh]
    I --> S["bin/start-hugegraph.sh -p true"]
    S --> Q["Request /versions and graph vertices"]

The map uses short labels for the repository modules described above. server, core, api, dist, pd, and store refer to the corresponding hugegraph- modules; dist packages the distribution.

---
markmap:
  spacingHorizontal: 40
---
# HugeGraph
## server
- core
- api
- dist
## pd
## store

2 Dependency for Building/Running

2.1 Install Java 17 (JDK 17)

Starting with HugeGraph 1.8.0, use Java 17; 1.7.0 is the last release supporting Java 11. Build all modules with JDK 17 and Maven 3.6.3 or later, and run Server, PD and Store with JDK 17. Server security check (-s true) requires Java 17 to 23 and a readable conf/java-security.properties; on Java 24 or later, start Server with -s false (JDK 24+ dropped Security Manager via JEP 486).

Before continuing, run java -version to confirm your JDK version.

Note

TinkerPop 3.8.1 Compatibility & Key Changes

HugeGraph 1.8.0 upgrades to TinkerPop 3.8.1 and Groovy 4.0.25. Use the Gremlin configuration files bundled with Server.

  • Serializer Packages: Driver serializer classes moved from org.apache.tinkerpop.gremlin.driver.ser to org.apache.tinkerpop.gremlin.util.ser. Keep HugeGraph registries.
  • Graph.Features: supportsUniformListValues() returns false for properties. Schema valueList()/LIST properties remain supported without rewriting existing data.
  • Traversal Multiplicity: Mid-traversal scans retain input multiplicity (e.g. for 3 vertices, g.V().V().count() returns 9L).
  • Date & Time Types: DATE properties accept OffsetDateTime for writes and indexed equality/range queries (instant normalized to milliseconds).
  • Filtering Behavior: P.typeOf() runs as a local filter; count filtering applies range optimizations only when predicate branches are provably safe.
GraphBinary (Recommended)
1# Bundled Driver and Console default to GraphBinary V1
2# - Schema results are standard maps
3# - Blob values are binary (including inside returned elements)
4serializer:
5  className: org.apache.tinkerpop.gremlin.util.ser.GraphBinaryMessageSerializerV1
6  config:
7    builder: org.apache.hugegraph.io.HugeGraphTypeSerializerRegistryBuilder
8    ioRegistries: [org.apache.hugegraph.io.HugeGraphIoRegistry]
GraphSON (HTTP / Legacy)
1# Default HTTP JSON uses untyped GraphSON V1
2# - Preserves the legacy Tree key/value array structure
3# - Typed GraphSON V1 is disabled for security (typed V2/V3 remain available for simple values and IDs)
4# - graph.schema() returns property keys, vertex labels, edge labels, and index labels
Tip

For complete Gremlin syntax rewrites, compatibility requirements, and reference tables, see the 1.8 Gremlin & TinkerPop Migration Guide. Additional runtime details are in the server runtime documentation.

3 Deploy

There are four ways to deploy the Server service:

Warning

Production requires authentication and restricted network access

HugeGraph disables user authentication by default. Enable authentication and authorization, maintain an IP allowlist, and grant minimum permissions. Do not expose Gremlin, Cypher, or other query APIs directly to the public Internet. Retain Server audit-*.log and restrict read access. auth.audit_log_rate limits per-user output rate rather than turning audit logging on or off. See the security guide.

3.1 Use Docker container (Convenient for Test/Dev)

For complete container setup instructions, refer to the Docker deployment guide.

Run a standalone Server container with the RocksDB backend:

Quick Start (Default)
1docker run -itd --name=server -p 8080:8080 hugegraph/hugegraph:1.7.0
With Sample Graph
1# Preload the built-in sample graph at startup
2docker run -itd --name=server -p 8080:8080 -e PRELOAD="true" hugegraph/hugegraph:1.7.0
With Authentication
1# Enable authentication mode and set the admin password
2docker run -itd --name=server -p 8080:8080 -e PASSWORD="replace-with-your-password" hugegraph/hugegraph:1.7.0

Common container inspection operations:

  • Inspect container shell: docker exec -it server bash
  • Check startup logs: docker logs -f server
  • Verify service health: curl -fsS http://127.0.0.1:8080/versions

If you use Docker Desktop, you can set the options as follows:

Docker Desktop settings for a HugeGraph container
Note

The Docker Compose files use bridge networking (hg-net) and work on Linux and Mac (Docker Desktop). For the 3-node distributed cluster on Mac (Docker Desktop), allocate at least 12 GB of memory (Settings → Resources → Memory). On Linux, Docker uses host memory directly.

Use docker compose to manage multiple HugeGraph services from one configuration. Four Compose files are available in the docker/ directory:

TopologyCompose fileServices
Standalone (start here)docker-compose.yml1 RocksDB Server + 1 Hubble
Minimal HStore (current master)docker-compose-hstore.yml1 PD + 1 Store + 1 Server + 1 Hubble; locally built master images required
HA reference (current master)docker-compose-3pd-3store-3server.yml3 PD + 3 Store + 3 Server + 1 Hubble; locally built master images required
Source build override for the minimal HStore topologydocker-compose.dev.yml(used together with docker-compose-hstore.yml)
1cd hugegraph/docker
2# Keep the version aligned with the latest release, for example 1.x.0
3HUGEGRAPH_VERSION=1.7.0 docker compose -f docker-compose.yml up -d --wait

The standalone topology publishes the Server on port 8080 and Hubble on 127.0.0.1:8088. HUGEGRAPH_VERSION selects the Server, PD, and Store image tags; Hubble is selected separately with HUBBLE_IMAGE.

The standalone docker-compose.yml example uses hugegraph/hugegraph:1.7.0; its docker/conf/hubble/standalone.properties is bundled in source. Compose reads the administrator password from HUGEGRAPH_ADMIN_PASSWORD and JWT secret from HUGEGRAPH_AUTH_TOKEN_SECRET, normally in docker/.env. A nonempty password enables authentication, automatically detected by Hubble. With docker run, use -e PASSWORD=xxx. Master HStore Compose files cannot be combined with 1.7.0 PD/Store/Server release images.

JWT version differences: Master-built Server maps HUGEGRAPH_AUTH_TOKEN_SECRET to HG_SERVER_AUTH_TOKEN_SECRET and writes startup configuration; preserve the secret across recreation. The 1.7.0 entrypoint handles older variables such as PASSWORD, ignoring HG_SERVER_AUTH_TOKEN_SECRET. To keep a stable JWT secret in 1.7.0, persist auth.token_secret in that version’s conf/graphs/hugegraph.properties. With a randomly generated default, do not assume issued JWTs survive Server restarts.

See docker/README.md for the full setup guide.

Server service identity with PD

For a PD-backed Server, the Helm chart exposes server.serviceId and server.graphSpace. Both default to DEFAULT, which preserves the existing server.service_id and server.graphspace values in conf/rest-server.properties. A non-default chart value injects HG_SERVER_SERVICE_ID or HG_SERVER_GRAPHSPACE into the Server container. These service identity chart settings (server.serviceId, server.graphSpace, and server.testGraphAlias) apply to development 1.8.0+ builds and charts; older versions such as 1.7.0 do not support these options. Do not use this example with older chart or image versions lacking these features. Non-empty service IDs supplied through HG_SERVER_SERVICE_ID, SERVICE_ID, or server.serviceId must use printable ASCII; spaces and backslashes are supported, while deployment checks reject non-ASCII and control characters. This entrypoint restriction does not change Core Service naming or existing manually supplied properties. Helm preserves dollar signs in server.serviceId literally, including $(POD_IP); this field does not expand Kubernetes environment references. The full service ID, including / and -, is retained when service metadata is reloaded or its graph space is cleared. Set the identity through these chart values; Server extraEnv reserves both the new and legacy identity variable names. HG_SERVER_GRAPHSPACE, the effective GRAPH_SPACE fallback, and server.graphSpace accept DEFAULT or an existing space name of up to 48 lowercase letters, digits, and underscores, starting with a letter:

1helm upgrade --install hg ./helm/hugegraph \
2  --set-string server.serviceId=analytics \
3  --set-string server.graphSpace=team_a

Before deploying Servers assigned to team_a, create that graph space through an existing Server in DEFAULT, using an account with permission to create graph spaces. The chart and these environment variables do not create graph spaces; all replicas in the Server Deployment use the same identity.

To check a graph in that space with helm test, first create the target graph and set --set-string server.testGraphAlias=team_a-demo to its actual Gremlin binding. Creating the graph space does not create a hugegraph graph.

For container deployments, HG_SERVER_SERVICE_ID and HG_SERVER_GRAPHSPACE write the same two properties. The legacy SERVICE_ID and GRAPH_SPACE names remain accepted: a non-empty new variable wins over its legacy counterpart. If both names for a property are unset or empty, its existing value is preserved. Compose does not automatically forward these host variables; add explicit Server environment: entries in a Compose override. See the Helm chart documentation for details.

These settings select the Server’s PD service metadata. They do not move existing graphs or change server.path_graphspace for REST paths that omit the graph space. Use /graphspaces/team_a/... paths and select the matching graph space and service in Hubble. When authentication is enabled, explicit paths still require authorization.

server.testGraphAlias defaults to DEFAULT-hugegraph. The hook checks this existing local binding on every Server Pod; it does not derive the alias from server.graphSpace. A local binding may exist independently of the Server’s assigned PD graph space, so a successful default hook does not validate graphs in team_a. If Could not rebind appears, first check that the requested graph exists and that its alias is correct. A Pod restart cannot create a missing graph or correct an alias. Match the binding’s case: a local Demo.properties can bind DEFAULT-Demo, while REST-created graph names are lowercase.

Configuration rendering and entrypoint checks alone do not establish non-default Linux discovery, authentication, or isolation between graph spaces.

Note
  1. HugeGraph Docker images are provided as a convenient way to start HugeGraph quickly, but they are not official ASF distribution artifacts. You can find more details in the ASF Release Distribution Policy.

  2. We recommend using a release tag (such as 1.7.0 or 1.x.0) for stable deployments. Use the latest tag only if you want the newest features still under development.

3.2 Download a Released Archive

This example uses HugeGraph 1.7.0. For other versions, confirm the release and filenames in the ASF download directory. A master source build is not a released binary.

1cd /path/to/downloads
2wget https://downloads.apache.org/hugegraph/1.7.0/apache-hugegraph-incubating-1.7.0.tar.gz
3wget https://downloads.apache.org/hugegraph/1.7.0/apache-hugegraph-incubating-1.7.0.tar.gz.sha512
4sha512sum -c apache-hugegraph-incubating-1.7.0.tar.gz.sha512
5tar zxf apache-hugegraph-incubating-1.7.0.tar.gz

Extract only after SHA-512 verification succeeds. You can also verify the PGP signature following Apache release procedures:

1wget https://downloads.apache.org/hugegraph/1.7.0/apache-hugegraph-incubating-1.7.0.tar.gz.asc
2wget https://downloads.apache.org/hugegraph/KEYS
3gpg --import KEYS
4gpg --verify apache-hugegraph-incubating-1.7.0.tar.gz.asc apache-hugegraph-incubating-1.7.0.tar.gz

Trust publisher keys only after verifying them through Apache project channels. Signature verification can complement SHA-512 checks.

3.3 Build from master Source

These commands build the development branch, not the 1.7.0 release.

Install wget or curl before building.

Download HugeGraph source:

1git clone --branch master https://github.com/apache/hugegraph.git

Compile and package:

1cd hugegraph
2# (Optional) use "-P stage" param if you build failed with the latest code(during pre-release period)
3mvn package -DskipTests -ntp

A successful build logs:

1[INFO] BUILD SUCCESS

The build creates an aggregate master archive under repository-root target/ (the current source version property is 1.8.0):

1target/apache-hugegraph-1.8.0.tar.gz

The filename version comes from source revision; it does not make the build an ASF release artifact. The build also creates repository-root apache-hugegraph-1.8.0/, containing a Server directory named by that module’s final.name, currently apache-hugegraph-server-1.8.0. If retaining only the archive, first run tar zxf target/apache-hugegraph-1.8.0.tar.gz, then enter the Server directory.

Default builds bundle rocksdb, hbase, and hstore modules, listed in backends within backend.properties in the hugegraph-dist JAR. For a RocksDB-only distribution, add -Drocksdb-only:

1mvn package -DskipTests -ntp -Drocksdb-only
Outdated tools

3.4 One-click deployment (Outdated)

Outdated This status applies only to this one-click deployment entry.

HugeGraph-Tools provides a one-click deployment command that downloads, extracts, configures, and starts the Server service and HugeGraph-Hubble. These tools are included in the HugeGraph-Toolchain distribution.

Of course, you should download the tarball of HugeGraph-Toolchain first.

1# download toolchain binary package, it includes loader + tool + hubble
2# please check the latest version (e.g. here is 1.7.0)
3wget https://downloads.apache.org/hugegraph/1.7.0/apache-hugegraph-toolchain-incubating-1.7.0.tar.gz
4tar zxf *hugegraph-*.tar.gz
5
6# enter the tool's package
7cd *hugegraph*/*tool* 

note: ${version} is the version, The latest version can refer to Download Page, or click the link to download directly from the Download page

The general entry script for HugeGraph-Tools is bin/hugegraph, Users can use the help command to view its usage, here only the commands for one-click deployment are introduced.

1bin/hugegraph deploy -v {hugegraph-version} -p {install-path} [-u {download-path-prefix}]

{hugegraph-version} is the Server service and HugeGraphStudio version; see conf/version-mapping.yaml for supported mappings. {install-path} is the installation directory, while {download-path-prefix} optionally overrides the tarball download location. For example, deploy version 0.6 with bin/hugegraph deploy -v 0.6 -p services.

4 Configuration

Server archives include conf/rest-server.properties, conf/gremlin-server.yaml, and conf/graphs/hugegraph.properties; source builds assemble them from hugegraph-server/hugegraph-dist/src/assembly/static/conf/. Edit them inside the extracted Server directory without a separate generation step. Defaults vary by release; the following defaults and startup behavior describe master.

For standalone RocksDB, confirm backend=rocksdb and serializer=binary in conf/graphs/hugegraph.properties. Normal Server initialization scans conf/graphs/ and loads local graphs without requiring graph.load_from_local_config=true. This option defaults to false and controls constructor preloading. Restart Server to apply graph configuration changes; do not use hot reload for maintenance.

See the configuration guide and option reference for details.

5 Startup

5.1 Use a startup script to startup

Startup is divided into “first startup” and “non-first startup”. On the first startup, you need to initialize the backend database before starting the service.

If the service was stopped manually, or needs to be started again for any other reason, you can usually start it directly because the backend database is persistent.

When HugeGraphServer starts, it connects to the backend storage and checks its version information. If the backend has not been initialized, or if it was initialized with an incompatible version (for example, old-version data), HugeGraphServer will fail to start and report an error.

If you need to access HugeGraphServer externally, modify the restserver.url configuration item in rest-server.properties (the default is http://127.0.0.1:8080) and change it to the machine name or IP address.

Since the configuration (hugegraph.properties) and startup steps required by various backends are slightly different, the following will introduce the configuration and startup of each backend one by one.

Warning

Configure authentication before production startup

Enable Server authentication and authorization with a strong, nondefault administrator password, an IP allowlist, and minimum business-user permissions.

5.1.1 Distributed Storage (HStore)

First deploy PD and Store; see the PD Quick Start and Store Quick Start.

Warning

umask 077 protects the new .env, but set-hubble-pd-password.sh sets generated .local.properties permissions to 0644, making the PD secret readable by other local users. This example assumes trusted local users. On shared hosts, set ownership and read permissions for the Hubble runtime user so Hubble can read the files while unauthorized users cannot. Keeping files untracked does not replace file permission controls.

Distributed storage configuration and startup

Distributed storage, introduced after HugeGraph 1.5.0, uses HugeGraph-PD and HugeGraph-Store for distributed data storage and computing.

After PD and Store are running:

  1. Edit Server hugegraph.properties:
1backend=hstore
2serializer=binary
3
4# PD RPC addresses, separated by commas
5pd.peers=127.0.0.1:8686,127.0.0.1:8687,127.0.0.1:8688
6pd.cluster=hg
 1# Simple example with authentication
 2gremlin.graph=org.apache.hugegraph.auth.HugeFactoryAuthProxy
 3
 4# Required HStore backend
 5backend=hstore
 6serializer=binary
 7store=hugegraph
 8
 9# pd config
10pd.peers=127.0.0.1:8686
11pd.cluster=hg

The distribution includes conf/graphs/hstore.properties.template; copy it over conf/graphs/hugegraph.properties and adjust pd.peers.

This example uses cluster=hg in rest-server.properties and pd.cluster=hg in graph configuration, matching the master Compose cluster below. These are separate keys for the Server process and graph respectively, not one setting. Select an appropriate cluster name. Likewise, pd.peers in each file has its own configuration scope; this example supplies the same PD RPC addresses to both.

The backend selects the task scheduler: HStore uses distributed scheduling; others use local scheduling. task.scheduler_type is unnecessary and is ignored with a warning when retained for compatibility.

  1. Edit Server rest-server.properties:
1usePD=true
2# Use cluster hg, matching pd.cluster in graph configuration
3cluster=hg
4# Server process PD addresses; replace with actual PD RPC addresses
5pd.peers=127.0.0.1:8686,127.0.0.1:8687,127.0.0.1:8688
6
7# Optional in local tests; authentication is mandatory in production
8# auth.authenticator=org.apache.hugegraph.auth.StandardAuthenticator

These addresses support multiple Server processes on one machine; 127.0.0.1 is local only. Across hosts, use routable IPs or DNS for restserver.url, gremlinserver.url, pd.peers, Gremlin host, and rpc.server_host. Advertised RPC addresses and ports must be mutually reachable.

For multiple Servers, edit each node’s rest-server.properties:

Node 1 (master):

 1usePD=true
 2restserver.url=http://127.0.0.1:8081
 3gremlinserver.url=http://127.0.0.1:8181
 4pd.peers=127.0.0.1:8686
 5
 6rpc.server_host=127.0.0.1
 7rpc.server_port=8091
 8
 9server.id=server-1
10server.role=master

Node 2 (worker):

 1usePD=true
 2restserver.url=http://127.0.0.1:8082
 3gremlinserver.url=http://127.0.0.1:8182
 4pd.peers=127.0.0.1:8686
 5
 6rpc.server_host=127.0.0.1
 7rpc.server_port=8092
 8
 9server.id=server-2
10server.role=worker

Also configure each node’s gremlin-server.yaml ports:

Node 1:

1host: 127.0.0.1
2port: 8181

Node 2:

1host: 127.0.0.1
2port: 8182

Start Server:

1bin/start-hugegraph.sh

Distributed startup order:

  1. Start HugeGraph-PD
  2. Start HugeGraph-Store
  3. Start Server

PD and Store manage HStore metadata and storage, so init-store skips that backend. With authentication enabled, init-store still creates the built-in admin account. If storage already holds this account, set init_store.enabled=false in rest-server.properties to skip the entire step, as Docker HStore topologies do.

Verify startup:

1curl http://localhost:8081/graphspaces/DEFAULT/graphs
2# Expected: {"graphs":["hugegraph"]}

Stop in reverse order:

  1. Stop Server
  2. Stop HugeGraph-Store
  3. Stop HugeGraph-PD
1bin/stop-hugegraph.sh
Docker HStore HA Cluster (Current Master)

docker-compose-3pd-3store-3server.yml uses master HG_PD_*, HG_STORE_*, and HG_SERVER_* variables unsupported by 1.7.0 release images. Build PD, Store, and Server from the same master source, all tagged local:

1# Run from the HugeGraph repository root
2docker build -f hugegraph-pd/Dockerfile -t hugegraph/pd:local .
3docker build -f hugegraph-store/Dockerfile -t hugegraph/store:local .
4docker build -f hugegraph-server/Dockerfile-hstore -t hugegraph/server:local .
5
6cd docker

Create the authentication environment in docker/ and generate HStore Hubble configuration. Replace the example administrator password; because .env uses single quotes, the password must contain neither single quotes nor newlines. The script generates random JWT and PD secrets and writes the PD secret to two untracked .local.properties files mounted by HStore Compose:

 1(
 2  set -eu
 3  command -v openssl >/dev/null
 4  jwt_secret="$(openssl rand -hex 32)"
 5  pd_secret="$(openssl rand -hex 24)"
 6  umask 077
 7  test ! -e .env || {
 8    echo ".env already exists; edit it instead of overwriting it" >&2
 9    exit 1
10  }
11  printf "HUGEGRAPH_ADMIN_PASSWORD='%s'\nHUGEGRAPH_AUTH_TOKEN_SECRET='%s'\nHG_PD_AUTH_SECRET_KEY='%s'\n" \
12    'replace-with-your-password' "${jwt_secret}" "${pd_secret}" > .env
13  HG_PD_AUTH_SECRET_KEY="${pd_secret}" ./set-hubble-pd-password.sh hstore
14  HG_PD_AUTH_SECRET_KEY="${pd_secret}" ./set-hubble-pd-password.sh hstore-ha
15)

Start HA from docker/. HUGEGRAPH_VERSION=local selects the locally built PD, Store, and Server images; Compose pull_policy: missing uses existing local tags first. HUBBLE_IMAGE still selects Hubble independently:

1set -a
2. ./.env
3set +a
4HUGEGRAPH_VERSION=local docker compose \
5  -f docker-compose-3pd-3store-3server.yml \
6  up -d --wait pd0 pd1 pd2 store0 store1 store2 server0 server1 server2 hubble

Generate HG_PD_AUTH_SECRET_KEY once for new data directories and reuse it on restart or when retaining data. Do not commit .env or generated conf/hubble/*.local.properties. Services communicate through container hostnames on the hg-net bridge. Server replicas need the shared JWT secret as well as the administrator password; set HUGEGRAPH_AUTH_TOKEN_SECRET. See the Docker cluster guide and docker/README.md for all variables.

Verify the cluster:

1curl -fsS http://localhost:8080/versions
2curl -fsS -u "hg:${HG_PD_AUTH_SECRET_KEY:?Load .env first}" \
3  http://localhost:8620/v1/stores

Master PD /v1/stores requires Basic authentication; this uses the generated, loaded HG_PD_AUTH_SECRET_KEY as the hg password.

Inspect runtime logs with docker logs <container-name>, such as docker logs hg-pd0.

See docker/README.md for variables, ports, and troubleshooting.

5.1.2 RocksDB / ToplingDB

The minimal standalone flow below uses a master build and preloads sample data to verify graph reads and writes.

  1. Check Java against the master build requirements.
  2. Build master source; the commands below use its nested Server directory.
  3. Confirm RocksDB configuration, then enter the build output’s nested Server directory.
  4. Initialize storage, then start the service with sample data using the commands below.
  5. Request /versions and sample graph vertices, and check the actual response contents.
Note

RocksDB is embedded and needs no separate deployment. GCC ≥ 4.3.0 (GLIBCXX_3.4.10) is required; upgrade first if needed.

Master conf/graphs/hugegraph.properties already sets backend=rocksdb and serializer=binary. Keep the defaults if using default data directories; otherwise check your backend and paths before initialization.

Initialize storage on first startup or after adding graph configuration under conf/graphs/:

1# Enter the extracted or built Server directory
2cd apache-hugegraph-*/apache-hugegraph-server-*
3bin/init-store.sh

Start Server and preload the built-in sample graph:

1bin/start-hugegraph.sh -p true

The startup script polls /graphs at configured restserver.url and exits nonzero on timeout or process failure. Check service version, then read sample vertices:

1curl -fsS http://127.0.0.1:8080/versions
2curl --compressed -fsS \
3  http://127.0.0.1:8080/graphspaces/DEFAULT/graphs/hugegraph/graph/vertices

The first response should contain versions; the second should contain vertices with sample names such as marko and lop. A process check or HTTP status alone does not prove backend and business request health.

ToplingDB (Beta): A high-performance RocksDB alternative; see the ToplingDB Quick Start.

5.1.3 HBase

Note

users need to install HBase by themselves, requiring version 2.0 or above,download link

HBase configuration and startup methods

Update hugegraph.properties

 1backend=hbase
 2serializer=hbase
 3
 4# hbase backend config
 5hbase.hosts=localhost
 6hbase.port=2181
 7# Note: recommend to modify the HBase partition number by the actual/env data amount & RS amount before init store
 8# it may influence the loading speed a lot
 9#hbase.enable_partition=true
10#hbase.vertex_partitions=10
11#hbase.edge_partitions=30

Initialize the database (required on the first startup, or a new configuration was manually added under ‘conf/graphs/’)

1cd *hugegraph-${version}
2bin/init-store.sh

Start server

1bin/start-hugegraph.sh
2Starting HugeGraphServer in daemon mode...
3Connecting to HugeGraphServer (http://127.0.0.1:8080/graphs)....OK
4Started [pid 21614]

5.1.4 Create an example graph when startup

Pass the -p true argument when starting the script to enable preload, which creates a sample graph.

1bin/start-hugegraph.sh -p true
2Starting HugeGraphServer in daemon mode...
3Connecting to HugeGraphServer (http://127.0.0.1:8080/graphs)......OK

And use the RESTful API to request HugeGraphServer and get the following result:

1> curl "http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/graph/vertices" | gunzip
2
3{"vertices":[{"id":"2:lop","label":"software","type":"vertex","properties":{"name":"lop","lang":"java","price":328}},{"id":"1:josh","label":"person","type":"vertex","properties":{"name":"josh","age":32,"city":"Beijing"}},{"id":"1:marko","label":"person","type":"vertex","properties":{"name":"marko","age":29,"city":"Beijing"}},{"id":"1:peter","label":"person","type":"vertex","properties":{"name":"peter","age":35,"city":"Shanghai"}},{"id":"1:vadas","label":"person","type":"vertex","properties":{"name":"vadas","age":27,"city":"Hongkong"}},{"id":"2:ripple","label":"software","type":"vertex","properties":{"name":"ripple","lang":"java","price":199}}]}

This indicates the successful creation of the sample graph.

5.1.5 Startup script options

bin/start-hugegraph.sh accepts the following options. Every one of them takes a value, so write -d false, not a bare -d.

OptionValuesDefaultPurpose
-dtrue, falsetrueDaemon mode. With -d false the script stays in the foreground and forwards SIGTERM/SIGINT to the server.
-gg1, G1, zgc, ZGCJVM defaultGarbage collector to use (g1/G1 sets -XX:+UseG1GC; zgc/ZGC sets -XX:+UseZGC). Omitting -g uses the JVM default collector.
-mtrue, falsefalseInstall the cron-based monitor task (bin/start-monitor.sh). For VM and bare-metal deployments only.
-ptrue, falsefalsePreload the sample graph, as in 5.1.4.
-strue, falsetrueEnable HugeSecurityManager; requires Java 17 to 23 and readable conf/java-security.properties. On Java 24+, start with -s false.
-jJVM optionsemptyExtra JVM options appended to the server command line.
-tseconds30How long to wait for the service to answer before reporting a failed startup.
-ytrue, falsefalseEnable the OpenTelemetry agent for traces.

bin/stop-hugegraph.sh accepts -m true|false (default true), which controls whether the cron monitor task is removed along with the service.

bin/hugegraph-server.sh puts -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath and -XX:ErrorFile first in JAVA_TOOL_OPTIONS, so, whether or not JAVA_OPTIONS is set and with the exceptions below, an out-of-memory heap dump (heapdump_<host>_<launch time>/java_pid<pid>.hprof) and a JVM crash log (hs_err_pid<pid>_<host>_<launch time>.log) land in logs/, where <launch time> is the local time as YYYYMMDD-HHMMSS. Each start gets its own heapdump_* directory, so the server and the JVMs it starts, such as computer jobs, write separate dumps (HotSpot names each dump and crash log after the JVM’s PID, so a JVM that reuses an earlier PID from the same start cannot write its dump, and its crash log replaces the earlier one’s). The host name keeps servers that share one log directory from picking the same name, and a counter (<launch time>-1, -2, …) is added when the same host already used that second. If the directory cannot be created, for example on a full disk, the server still starts and dumps go to logs/ itself, as a best effort: a full disk may have no room for the dump either. A heap dump can be as large as the heap. The launcher never deletes dumps or heapdump_* directories: each start that reaches Java leaves one, empty unless something ran out of memory, and the launcher keeps it because a computer-job JVM can go on using it after the server exits. Move or delete old ones once no HugeGraph JVM from that launch is running, but never the newest one of a running server; without it, the next dump is written as a file with that name. The JVM reads your own JAVA_TOOL_OPTIONS, JDK_JAVA_OPTIONS (including @argfiles), the command line (JAVA_OPTIONS, or -j when JAVA_OPTIONS is unset or empty) and _JAVA_OPTIONS after these defaults, so a value for one of these flags in any of them overrides the default. Only the environment variables reach the JVMs the server starts: to turn dumps off for all of them, set JAVA_TOOL_OPTIONS=-XX:-HeapDumpOnOutOfMemoryError, while -j or JAVA_OPTIONS turns them off for the server JVM only. Either way each start that reaches Java still creates its empty heapdump_* directory, unless creating it failed and dumps fell back to logs/. If the logs path contains a double quote or %, the launcher warns and heap dumps go to logs/ itself, with no per-launch directory and no crash log default; the JAVA_TOOL_OPTIONS opt-out still applies. A path with both quote characters gets no heap dump default at all. Because of the defaults, the JVM prints Picked up JAVA_TOOL_OPTIONS: ... on stderr at startup.

Version scope: this applies to current development/master builds. In 1.7.0 the heap dump flags are added only when JAVA_OPTIONS is unset, heap dumps go directly to logs/java_pid<pid>.hprof, and JVM crash logs go to the install directory.

5.2 Use Docker to startup

In 3.1 Use Docker container, we introduced how to deploy hugegraph-server with Docker. You can also switch storage backends or preload a sample graph by setting the corresponding parameters.

5.2.1 Create an example graph when starting a server

Set the environment variable PRELOAD=true when starting Docker so that sample data is loaded during startup.

  1. Use docker run

    Use docker run -itd --name=server -p 8080:8080 -e PRELOAD=true hugegraph/hugegraph:1.7.0

  2. Use docker-compose

    Create a docker-compose.yml file like the following and set PRELOAD=true in the environment. example.groovy is a predefined script used to preload sample data. If needed, you can mount a new example.groovy script to change the preload data.

     1version: '3'
     2services:
     3  server:
     4    image: hugegraph/hugegraph:1.7.0
     5    container_name: server
     6    environment:
     7      - PRELOAD=true
     8      - PASSWORD=xxx
     9    volumes:
    10      - /path/to/yourscript:/hugegraph-server/scripts/example.groovy
    11    ports:
    12      - 8080:8080

    Server images built from current master run as the non-root UID 10001, so a mounted script must be readable by that user.

    Use docker compose up -d to start the container.

And use the RESTful API to request HugeGraphServer and get the following result:

1> curl "http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/graph/vertices" | gunzip
2
3{"vertices":[{"id":"2:lop","label":"software","type":"vertex","properties":{"name":"lop","lang":"java","price":328}},{"id":"1:josh","label":"person","type":"vertex","properties":{"name":"josh","age":32,"city":"Beijing"}},{"id":"1:marko","label":"person","type":"vertex","properties":{"name":"marko","age":29,"city":"Beijing"}},{"id":"1:peter","label":"person","type":"vertex","properties":{"name":"peter","age":35,"city":"Shanghai"}},{"id":"1:vadas","label":"person","type":"vertex","properties":{"name":"vadas","age":27,"city":"Hongkong"}},{"id":"2:ripple","label":"software","type":"vertex","properties":{"name":"ripple","lang":"java","price":199}}]}

This indicates that the sample graph was created successfully.

6. Access server

6.1 Service startup status check

Use jps to see a service process

1jps
26475 HugeGraphServer

curl request RESTfulAPI

1curl -fsS http://127.0.0.1:8080/versions

HTTP 2xx with a JSON versions field confirms REST responsiveness. HugeGraphServer in jps confirms only process existence; verify backend health by preloading the sample graph and requesting vertices as above.

6.2 Request Server

The RESTful API of HugeGraphServer includes various types of resources, typically including graph, schema, gremlin, traverser and task.

  • graph contains vertices、edges
  • schema contains vertexlabels、 propertykeys、 edgelabels、indexlabels
  • gremlin contains various Gremlin statements, such as g.v(), which can be executed synchronously or asynchronously
  • traverser contains various advanced queries including shortest paths, intersections, N-step reachable neighbors, etc.
  • task contains query and delete with asynchronous tasks
1curl http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/graph/vertices

explanation

  1. Server can compress vertex, edge, and other list responses. Use curl --compressed for automatic decompression:

    1curl --compressed -fsS "http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/graph/vertices"
  2. The default listener is 127.0.0.1, inaccessible directly from other machines.

    1vim conf/rest-server.properties
    2
    3restserver.url=http://0.0.0.0:8080
Warning

Changing the Server listener

Setting 0.0.0.0 listens on every interface. In production, enable authentication and authorization, an IP allowlist, and minimum permissions; retain and protect audit logs. Do not expose Gremlin or Cypher directly to the public Internet.

response body:

 1{
 2    "vertices": [
 3        {
 4            "id": "2lop",
 5            "label": "software",
 6            "type": "vertex",
 7            "properties": {
 8                "price": [
 9                    {
10                        "id": "price",
11                        "value": 328
12                    }
13                ],
14                "name": [
15                    {
16                        "id": "name",
17                        "value": "lop"
18                    }
19                ],
20                "lang": [
21                    {
22                        "id": "lang",
23                        "value": "java"
24                    }
25                ]
26            }
27        },
28        {
29            "id": "1josh",
30            "label": "person",
31            "type": "vertex",
32            "properties": {
33                "name": [
34                    {
35                        "id": "name",
36                        "value": "josh"
37                    }
38                ],
39                "age": [
40                    {
41                        "id": "age",
42                        "value": 32
43                    }
44                ]
45            }
46        },
47        ...
48    ]
49}

For the detailed API, please refer to RESTful-API

You can also visit localhost:8080/swagger-ui/index.html to check the API.

HugeGraph RESTful API endpoints in Swagger UI

When using Swagger UI to debug the API provided by HugeGraph, if HugeGraph Server turns on authentication mode, you can enter authentication information on the Swagger page.

Authorize button in the HugeGraph Swagger UI

Currently, HugeGraph supports setting authentication information in two forms: Basic and Bearer.

Basic and Bearer credential fields in the Swagger UI authorization dialog

7 Stop Server

1# From within the Server directory
2bin/stop-hugegraph.sh

8 Debug Server with IntelliJ IDEA

Please refer to Setup Server in IDEA

1.2 - HugeGraph-PD Quick Start

1 HugeGraph-PD Overview

HugeGraph-PD (Placement Driver) is the metadata management component of HugeGraph’s distributed version, responsible for managing the distribution of graph data and coordinating storage nodes. It plays a central role in distributed HugeGraph, maintaining cluster status and coordinating HugeGraph-Store storage nodes.

PD keeps cluster metadata in an embedded RocksDB store under pd.data-path and replicates it across PD nodes with Raft, so a 3-node or 5-node PD cluster keeps serving while a minority of nodes is down. On top of that it registers and activates Store nodes, allocates and rebalances partitions, tracks Store heartbeats, and answers service discovery queries from Store and Server.

PD listens on three ports:

PortDefaultConfigured byUsed by
gRPC8686grpc.portStore and Server clients
REST8620server.portManagement, health checks, metrics
Raft8610raft.addressThe other PD nodes only

2 Prerequisites

2.1 Requirements

  • Operating System: Linux or macOS (Windows has not been fully tested)
  • Java version: Java 17 for HugeGraph 1.8.0+ (the Maven build accepts only JDK 17; the start script rejects older runtimes; 1.7.0 is the last release supporting Java 11)
  • Maven version: ≥ 3.6.3 (for building from source)

3 Deployment

Choose a deployment method for the HugeGraph-PD component:

3.1 Download the tar package

Apache downloads currently provide the complete 1.7.0 binary bundle containing PD, Store, and Server, without a separate PD binary. Verify version, signatures, and SHA512 on the official download page. This historical incubating release retains incubating in archive and extracted directory names:

1# Historical release 1.7.0 retains incubating in filenames and directories
2curl -fLO --connect-timeout 10 --max-time 1800 https://downloads.apache.org/hugegraph/1.7.0/apache-hugegraph-incubating-1.7.0.tar.gz
3tar zxf apache-hugegraph-incubating-1.7.0.tar.gz
4cd apache-hugegraph-incubating-1.7.0/apache-hugegraph-pd-incubating-1.7.0

3.2 Compile from source

  1. Clone the source code:

    1git clone https://github.com/apache/hugegraph.git
    2cd hugegraph
  2. Build the PD module:

    1# Build only the PD distribution and its dependencies
    2mvn clean package -pl hugegraph-pd/hg-pd-dist -am -DskipTests
  3. Locate the distribution output:

    1cd hugegraph-pd/apache-hugegraph-pd-*/
    2ls -la bin conf lib

The unpacked distribution contains just three directories: bin (start and stop scripts), conf (application.yml, application.yml.template, log4j2.xml, verify-license.json) and lib (the hg-pd-service jar).

3.3 Docker Deployment

HG_PD_* mappings and /v1/ready are features of current master; the Server 1.7.0 source tag lacks these Docker mappings and readiness API. The following examples require images built from current master and tagged local. For release 1.7.0 assets, use bundled configuration and version-specific instructions:

1# Run from the HugeGraph repository root
2docker build -f hugegraph-pd/Dockerfile -t hugegraph/pd:local .

To run master-built PD alone, first set a deployment-specific secret and replace example addresses with addresses reachable by the container:

The mkdir -p and nonrecursive chown below are for a new empty directory. Before reusing data written by a root image, stop PD and run sudo chown -R 10001:10001 /path/to/data once, then start the UID 10001 image.

 1export HG_PD_AUTH_SECRET_KEY="$(openssl rand -hex 24)"
 2# Master-built images run as UID 10001; the host data directory must be writable by it
 3mkdir -p /path/to/data && sudo chown 10001:10001 /path/to/data
 4docker run -d \
 5  -p 8620:8620 -p 8686:8686 -p 8610:8610 \
 6  -e HG_PD_GRPC_HOST=192.168.1.10 \
 7  -e HG_PD_RAFT_ADDRESS=192.168.1.10:8610 \
 8  -e HG_PD_RAFT_PEERS_LIST=192.168.1.10:8610 \
 9  -e HG_PD_INITIAL_STORE_LIST=192.168.1.20:8500 \
10  -e HG_PD_AUTH_SECRET_KEY="$HG_PD_AUTH_SECRET_KEY" \
11  -v /path/to/data:/hugegraph-pd/pd_data \
12  --name hugegraph-pd \
13  hugegraph/pd:local
VariableRequiredDefaultConfiguration keyDescription
HG_PD_GRPC_HOSTYesNonegrpc.hostAdvertised gRPC hostname/IP; use container hostnames within a container network.
HG_PD_RAFT_ADDRESSYesNoneraft.addressThis PD node’s Raft address.
HG_PD_RAFT_PEERS_LISTYesNoneraft.peers-listRaft addresses of every PD node, including this node.
HG_PD_INITIAL_STORE_LISTYesNonepd.initial-store-listExpected Store gRPC addresses.
HG_PD_AUTH_SECRET_KEYYesNoneauth.secret-keyREST Basic password shared by all PD REST clients.
HG_PD_GRPC_PORTNo8686grpc.portgRPC service port.
HG_PD_REST_PORTNo8620server.portREST API port.
HG_PD_DATA_PATHNo/hugegraph-pd/pd_datapd.data-pathMetadata path.
HG_PD_INITIAL_STORE_COUNTNo1pd.initial-store-countMinimum Store count needed for cluster availability.
HG_PD_ACTUATOR_EXPOSURENohealth,metrics,prometheusmanagement.endpoints.web.exposure.includePublic Actuator endpoint allowlist; * is forbidden.

The master entrypoint requires all five mandatory variables, builds SPRING_APPLICATION_JSON overrides, then starts PD. Other settings come from image conf/application.yml; JAVA_OPTS is passed to the JVM. Deprecated GRPC_HOST, RAFT_ADDRESS, RAFT_PEERS, and PD_INITIAL_STORE_LIST remain supported with warnings.

Docker HEALTHCHECK polls /v1/health every 15 seconds, confirming only REST listener liveness, not a Raft quorum. Compose uses the same liveness check; verify /v1/ready separately before starting Store. Containers run Java in the foreground and exit when Java exits; automatic restart requires a Docker restart policy. Inspect logs with docker logs <container-name>.

See the Store Docker section and docker/README.md for master Compose and the minimal topology. When starting the full topology with Hubble, first generate the untracked Hubble configuration described in that README.

4 Configuration

PD startup reads installation conf/application.yml. Match this file to the installed version: 1.7.0 configuration or master configuration. Master leaves auth.secret-key empty, causing protected REST calls to be rejected until configured. Startup does not read application.yml.template.

4.1 Configuration Reference

These tables list master configuration values and code defaults. See the PDConfig implementation for Raft timeouts. Use the original tagged configuration with 1.7.0 binaries and master configuration with master builds. Differences from 1.7.0 are noted below.

gRPC and REST

KeyMaster distribution valueBuilt-in defaultDescription
grpc.host127.0.0.1none, requiredAddress this PD advertises for gRPC. Store and Server connect here, so set it to a reachable IPv4 address or hostname, never 127.0.0.1 or 0.0.0.0, in a distributed deployment.
grpc.port8686none, requiredgRPC port.
server.port8620none, requiredREST API port. Also the port reported in Raft member information.

application.yml.template also carries grpc.netty-server.max-inbound-message-size: 100MB, but PD sets the gRPC server’s inbound message limit to 1 GB in code, so that key has no effect.

Raft

KeyMaster distribution valueBuilt-in defaultDescription
raft.address127.0.0.1:8610none, requiredRaft address of this node as host:port. Must be unique per node and must appear in raft.peers-list.
raft.peers-list127.0.0.1:8610none, requiredComma separated Raft addresses of every PD node, including this one. Must be identical on all nodes.
raft.enablenot settrueWhen true, metadata writes go through the Raft state machine. When false, PD writes straight to its local store with no replication.
raft.ip-whitelist.enablednot settrueWhen true, the Raft RPC port accepts connections only from the addresses resolved from raft.peers-list; other clients are dropped and logged as Blocked connection from <ip>. The allowlist is re-resolved when the peer list changes, but a peer that keeps its hostname and changes IP (a restarted container, for example) needs a PD restart.
raft.snapshotIntervalnot set300Seconds between Raft snapshots.
raft.rpc-timeoutnot set10000Timeout of a Raft RPC between PD nodes, in milliseconds. Also the deadline of the ReadIndex wait that PD runs before it reads an id counter; id allocation fails when the wait takes longer.
raft.rpc-connect-timeoutnot set1000Timeout of the Ping that opens a Raft connection to a peer, in milliseconds. A candidate connects to its peers one at a time, so an election waits this long for each peer that accepts the TCP connection but does not answer. Requests on an open connection use raft.rpc-timeout.
raft.rpc-install-snapshot-timeoutnot set0Timeout for sending a snapshot to a follower that fell behind the log, in milliseconds. 0 uses raft.rpc-timeout. Raise it if large snapshots time out.

raft.rpc-connect-timeout and raft.rpc-install-snapshot-timeout exist only in versions after 1.7.0. In 1.7.0, raft.rpc-timeout is also the connect and install-snapshot timeout, and id allocation runs no ReadIndex wait.

PD core

KeyMaster distribution valueBuilt-in defaultDescription
pd.data-path./pd_datanone, requiredMetadata directory. Holds the RocksDB store in rocksdb/ and the Raft log, metadata and snapshots in pd_raft/.
pd.patrol-interval1800300Seconds between patrol runs, which check partition health across stores and rebalance partition counts.
pd.initial-store-count13Minimum number of active Store nodes. Below this the cluster state becomes Cluster_Not_Ready and the cluster is treated as unavailable. Set it to the number of stores you deploy.
pd.min-store-countnot set0Minimum active Store count required during Store drain/retirement checks. If remaining active stores fall below this value, the transition to Exiting is rejected.
pd.initial-store-list127.0.0.1:8500emptyComma separated Store gRPC addresses (ip:port) that are activated automatically when they register. An entry may also carry a group id as store_address/group_id.
pd.cluster_idnot set1Cluster id, used to keep separate PD clusters apart.

Store management

KeyMaster distribution valueBuilt-in defaultDescription
store.keepAlive-timeoutnot set300Seconds without a heartbeat after which a Store is treated as temporarily unavailable and its partition leaders move to other replicas.
store.max-down-time1728001800Seconds after which a Store is treated as permanently unavailable and its replicas are reallocated to other machines.
store.monitor_data_enabledtruefalseWhether to persist Store monitoring samples.
store.monitor_data_interval1 minute1 minuteSampling interval, written as <number> <unit> with unit one of second, minute, hour, day, month, year. The number defaults to 1 when omitted.
store.monitor_data_retention1 day1 dayHow long monitoring samples are kept, same format as above.

Partitions

KeyMaster distribution valueBuilt-in defaultDescription
partition.default-shard-count13Number of replicas per partition. Use 3 for a production cluster.
partition.store-max-shard-count1224Maximum number of partition replicas one Store holds.

The initial partition count is derived from these two values and the size of pd.initial-store-list:

1initial partitions = store count * partition.store-max-shard-count / partition.default-shard-count

Discovery, license and metrics

KeyMaster distribution valueBuilt-in defaultDescription
discovery.heartbeat-try-countnot set3Number of missed heartbeats after which a registered client’s discovery entry is deleted.
license.verify-path./conf/verify-license.jsonNone; required configuration keyThe PD distribution includes this JSON file; current master has no runtime read of this configuration option.
license.license-path./conf/hugegraph.licenseNone; required configuration keyNo license file is bundled at this path. Internal gRPC putLicense writes uploads here. Master REST GET /v1/license returns an empty object; a missing file does not block startup.
auth.secret-keyEmptyEmptyMaster PD REST Basic password for usernames hg, store, hubble, or vermeer. Docker requires it; bare-metal PD can start with an empty secret, but rejects protected REST calls.
management.metrics.export.prometheus.enabledtrueSpring Boot defaultExposes /actuator/prometheus.
management.endpoints.web.exposure.includehealth,metrics,prometheusSpring Boot exposes only health by defaultActuator allowlist; these endpoints bypass the PD REST Basic interceptor.
logging.configfile:./conf/log4j2.xmlnoneLog4j2 configuration. Writes logs/hugegraph-pd.log, logs/hugegraph-pd_raft.log and logs/audit-hugegraph-pd.log.

Thread pools

KeyBuilt-in defaultDescription
thread.pool.grpc.core600Core size of the pool that serves gRPC calls.
thread.pool.grpc.max1000Maximum size of that pool.
thread.pool.grpc.queueunboundedQueue capacity of that pool.
job.uninterruptibleThreadPool.core0Core size of the background metadata job pool. A value of 0 or less means half the available processors.
job.uninterruptibleThreadPool.max256Maximum size of that pool.
job.uninterruptibleThreadPool.queueunboundedQueue capacity of that pool.

4.2 Single-node configuration

For master-built distributions, start with conf/application.yml, adjust per-node gRPC/Raft addresses and data paths, and set auth.secret-key. The following values are for development and testing. One PD node provides no majority fault tolerance; partition.default-shard-count: 1 means one replica per partition.

 1grpc:
 2  host: 127.0.0.1
 3  port: 8686
 4server:
 5  port: 8620
 6raft:
 7  address: 127.0.0.1:8610
 8  peers-list: 127.0.0.1:8610
 9pd:
10  data-path: ./pd_data
11  initial-store-count: 1
12  initial-store-list: 127.0.0.1:8500
13partition:
14  default-shard-count: 1
15auth:
16  secret-key: replace-with-deployment-secret

Generate a deployment secret with openssl rand -hex 24; Docker supplies it through HG_PD_AUTH_SECRET_KEY. See REST API authentication for the different 1.7.0 behavior.

4.3 Three-node cluster configuration

For a production cluster run 3 or 5 PD nodes, an odd number so Raft always has a quorum. A 3-node cluster tolerates one node failure. raft.peers-list must list every node and must be byte-for-byte identical on all of them, while grpc.host and raft.address differ per node.

Node 1 (192.168.1.10):

 1grpc:
 2  host: 192.168.1.10
 3  port: 8686
 4server:
 5  port: 8620
 6raft:
 7  address: 192.168.1.10:8610
 8  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610
 9pd:
10  data-path: /data/pd
11  initial-store-count: 3
12  initial-store-list: 192.168.1.20:8500,192.168.1.21:8500,192.168.1.22:8500
13partition:
14  default-shard-count: 3

Node 2 (192.168.1.11) and node 3 (192.168.1.12) use the same file with grpc.host and raft.address changed to their own address:

 1# Node 2
 2grpc:
 3  host: 192.168.1.11
 4raft:
 5  address: 192.168.1.11:8610
 6  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610
 7
 8# Node 3
 9grpc:
10  host: 192.168.1.12
11raft:
12  address: 192.168.1.12:8610
13  peers-list: 192.168.1.10:8610,192.168.1.11:8610,192.168.1.12:8610

To put all three PD nodes on one machine for testing, give each node its own pd.data-path and its own ports, for example raft 8610/8611/8612, gRPC 8686/8687/8688 and REST 8620/8621/8622.

In Docker bridge networking the same configuration comes from environment variables and uses container hostnames instead of IP addresses:

 1# Share the same secret across PD nodes; inject it from private environment variables
 2HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY}
 3
 4# pd0
 5HG_PD_GRPC_HOST: pd0
 6HG_PD_RAFT_ADDRESS: pd0:8610
 7HG_PD_RAFT_PEERS_LIST: pd0:8610,pd1:8610,pd2:8610
 8HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500
 9HG_PD_INITIAL_STORE_COUNT: 3
10
11# pd1
12HG_PD_GRPC_HOST: pd1
13HG_PD_RAFT_ADDRESS: pd1:8610
14HG_PD_RAFT_PEERS_LIST: pd0:8610,pd1:8610,pd2:8610
15
16# pd2
17HG_PD_GRPC_HOST: pd2
18HG_PD_RAFT_ADDRESS: pd2:8610
19HG_PD_RAFT_PEERS_LIST: pd0:8610,pd1:8610,pd2:8610

5 Start and Stop

5.1 Start PD

In the PD installation directory, execute:

1./bin/start-hugegraph-pd.sh

The script requires Java 17 or later on PATH or in JAVA_HOME (Java ≥ 11 on 1.7.0). It exits without doing anything if it finds a Java process already using this installation’s conf directory.

The lifecycle details below describe current master builds. The downloaded 1.7.0 launcher does not include those repairs. Supported flags, with GC version limits noted below:

FlagValuesDefaultDescription
-dtrue, falsetrueDaemon mode. See the note below.
-gSee version limits belownot setSelect the garbage collector. Unsupported values abort startup.
-jJVM optionsemptyExtra JVM options, for example -j "-Xmx8g -Xms8g".
-ytrue, falsefalseAttach the OpenTelemetry Java agent. The agent is downloaded into plugins/ on first use, its MD5 is verified, and traces are exported over gRPC to http://127.0.0.1:4317.

Master builds accept g1, G1, zgc and ZGC. The downloaded 1.7.0 launcher accepts only zgc or ZGC as explicit choices; -g g1, -g G1 and other values abort startup. Omit -g for the default path. For rejected values, inspect logs/hugegraph-pd-stdout.log.

The -d flag controls daemon mode:

  • -d true (default): run as a background daemon; the script returns immediately.
  • -d false: run in foreground. The script execs Java, so the container or supervisor process IS Java. Use this when running under Docker or a process supervisor (systemd, supervisord) so crashes are detected and the service is restarted automatically.

Each flag also has an environment variable equivalent: DAEMON, GC_OPTION, USER_OPTION and OPEN_TELEMETRY. Setting JAVA_OPTIONS replaces the computed heap settings entirely; otherwise the script sizes the heap between 512 MB and 32 GB from available memory. Setting STDOUT_MODE=true leaves the JVM output on stdout instead of redirecting it to logs/hugegraph-pd-stdout.log, which is what the Docker image does.

After successful startup, you can see logs similar to the following in logs/hugegraph-pd-stdout.log:

1YYYY-mm-dd xx:xx:xx [main] [INFO] o.a.h.p.b.HugePDServer - Started HugePDServer in x.xxx seconds (JVM running for x.xxx)

The process id is written to bin/pid.

5.2 Stop PD

In the PD installation directory, execute:

1./bin/stop-hugegraph-pd.sh

The script reads bin/pid, sends the process a termination signal, waits up to 30 seconds for it to exit, and removes the pid file. If bin/pid is missing it reports that and exits successfully.

6 Startup Order in a Distributed Cluster

Start the components in this order:

  1. All PD nodes. Form the Raft group and elect a leader. Check GET /v1/ready on each node; start Store only after HTTP 200 with ready:true. /v1/health checks REST listener liveness only.
  2. All Store nodes. Register with PD; wait for /v1/stores to show every target Store as Up.
  3. All Server nodes. Start after PD and Store are ready.
sequenceDiagram
    participant Operator
    participant PD
    participant Store
    participant Server
    Operator->>PD: GET /v1/ready (check each PD)
    PD-->>Operator: HTTP 200 and ready=true
    Operator->>Store: Start Store
    Store->>PD: gRPC registration and heartbeat
    Operator->>PD: GET /v1/stores (Basic authentication)
    PD-->>Operator: Every target Store is Up
    Operator->>Server: Start Server
    Server->>PD: gRPC partitions and discovery
    Server->>Store: gRPC graph data access

Master Compose waits on PD /v1/health through Store depends_on: condition: service_healthy, and on Store REST liveness before Server. Server also polls PD /v1/stores for an Up Store before starting. Therefore, docker compose up --wait does not replace PD Raft readiness and Store registration checks.

PD is also the last component to stop: shut down Server, then Store, then PD.

7 Verification

7.1 REST API Authentication

Current master requires HTTP Basic Authorization for all PD REST paths except /actuator/*, /v1/health, /v1/ready, and /v1/prom/targets/*. Username must be hg, store, hubble, or vermeer, and password must equal auth.secret-key. A missing secret or wrong password returns HTTP 401. Query Stores using the same secret configured at PD startup:

1{"status": -1, "error": "Unauthorized"}
1curl -u "store:${HG_PD_AUTH_SECRET_KEY:?Set the PD deployment secret first}" \
2  http://localhost:8620/v1/stores

bin/wait-storage.sh uses the same credentials through PD_AUTH_USER and PD_AUTH_PASSWORD. Release 1.7.0 checks only internal service usernames and does not compare passwords; do not apply this older behavior to master builds.

Warning

Protect Server and PD ports separately in production

Enable Server authentication and authorization, an IP allowlist, and minimum graph API permissions; retain and protect Server audit-*.log. These controls do not protect PD. Master PD REST uses auth.secret-key, whereas 1.7.0 checks usernames only. Keep raft.ip-whitelist.enabled enabled for configured peers and restrict PD REST/gRPC to trusted networks. Unauthenticated /v1/health, /v1/ready, and Actuator probes require network restrictions too. Protect master PD audit logs at logs/audit-hugegraph-pd.log.

7.2 Health Checks

Unauthenticated GET /v1/health returns 200 with an empty body, confirming only PD REST listener startup:

1curl -i http://localhost:8620/v1/health

Spring Boot Actuator also provides a readable health response:

1curl http://localhost:8620/actuator/health

Actuator {"status":"UP"} does not confirm an available PD Raft leader. Master GET /v1/ready returns HTTP 200 with ready:true when the PD Raft node is active and sees a leader; otherwise it returns HTTP 503 with ready:false:

1curl -i http://localhost:8620/v1/ready

7.3 Cluster and member status

Check the PD members and which node is the Raft leader:

1curl -u "store:${HG_PD_AUTH_SECRET_KEY:?Set the PD deployment secret first}" \
2  http://localhost:8620/v1/members

The response carries pdList, the elected pdLeader, numOfService, numOfNormalService and a stateCountMap. In a healthy 3-node PD cluster numOfService and numOfNormalService are both 3 and exactly one member has role: "Leader".

GET /v1/cluster returns the same member list together with the Store list, graph list and overall cluster state, and GET / returns a short summary (leader address, cluster state, member count, store count, graph count, partition count).

7.4 Store status

You can also verify Store node status through the PD API:

1curl -u "store:${HG_PD_AUTH_SECRET_KEY:?Set the PD deployment secret first}" \
2  http://localhost:8620/v1/stores

If the response shows state as Up, the corresponding Store node is running normally. The example below shows a single Store node. In a healthy 3-node deployment, the storeId list should contain three IDs, and stateCountMap.Up, numOfService, and numOfNormalService should all be 3.

 1{
 2  "message": "OK",
 3  "data": {
 4    "stores": [
 5      {
 6        "storeId": 8319292642220586694,
 7        "address": "127.0.0.1:8500",
 8        "raftAddress": "127.0.0.1:8510",
 9        "version": "",
10        "state": "Up",
11        "deployPath": "/Users/{your_user_name}/hugegraph/apache-hugegraph-incubating-1.7.0/apache-hugegraph-store-incubating-1.7.0/lib/hg-store-node-1.7.0.jar",
12        "dataPath": "./storage",
13        "startTimeStamp": 1754027127969,
14        "registedTimeStamp": 1754027127969,
15        "lastHeartBeat": 1754027909444,
16        "capacity": 494384795648,
17        "available": 346535829504,
18        "partitionCount": 0,
19        "graphSize": 0,
20        "keyCount": 0,
21        "leaderCount": 0,
22        "serviceName": "127.0.0.1:8500-store",
23        "serviceVersion": "",
24        "serviceCreatedTimeStamp": 1754027127000,
25        "partitions": []
26      }
27    ],
28    "stateCountMap": {
29      "Up": 1
30    },
31    "numOfService": 1,
32    "numOfNormalService": 1
33  },
34  "status": 0
35}

7.5 Other REST endpoints

All paths below are relative to http://<pd-host>:8620 and need the Basic header from section 7.1 unless noted.

Method and pathDescription
GET /Brief cluster statistics: leader, state, member count, store count, graph count, partition count
GET /v1/healthLiveness only; HTTP 200 does not imply Raft readiness; no authentication
GET /v1/readyMaster readiness: HTTP 200 with ready:true means an active node that sees a leader; no authentication
GET /v1/clusterFull cluster statistics: PD members, stores, graphs, partitions
GET /v1/membersPD member list with roles and the elected leader
POST /v1/members/changeChange the Raft peer list, body {"peerList": "..."}
GET /v1/storesRegistered Store nodes with state and per-store statistics
GET /v1/store/{storeId}One Store node
POST /v1/store/{storeId}Update a Store’s state, body {"storeState": "..."}
DELETE /v1/store/{storeId}Remove a Store from the cluster
POST /v1/store/logStore state change log, body {"startTime": "...", "endTime": "..."}
GET /v1/storesAndStatsRaw Store metadata, for debugging
GET /v1/store_monitor/{storeId}Store monitoring samples as text
GET /v1/store_monitor/json/{storeId}Store monitoring samples as JSON
GET /v1/shardsEvery shard of every partition, with store id, role, state and progress
GET /v1/shardGroupsShard groups
GET /v1/shardGroupsCacheShard groups from PD’s in-memory cache
GET /v1/shardLeadersPartition leaders grouped by Store raft address
GET /v1/balanceLeadersRebalance partition leaders across Stores
GET /v1/partitionsPartition list with state and statistics
GET /v1/highLevelPartitionsPartitions with per-graph key counts and data sizes
GET /v1/partitionsAndStatsRaw partition metadata, for debugging
POST /v1/partitions/logPartition change log, body {"startTime": "...", "endTime": "..."}
GET /v1/resetPartitionStateReset the state of every partition
GET /v1/graphsGraph list
GET /v1/graph/**One graph by name
POST /v1/graph/**Update a graph’s partition count, body {"partitionCount": N}
GET /v1/graph/partitionSizeRangeMinimum and maximum partition count the cluster accepts
GET /v1/graph-spacesGraph space list
GET /v1/graph-spaces/**One graph space
POST /v1/graph-spaces/**Update a graph space
POST /v1/registryRegister a service instance for discovery
POST /v1/registryInfoQuery registered instances
GET /v1/allInfoAll registered instances
GET /v1/licenseLegacy endpoint; current master returns an empty object
GET /v1/license/machineInfoIP and MAC addresses seen by the license check
GET /v1/task/patrolStoresRun the store patrol task now
GET /v1/task/patrolPartitionsRun the partition patrol task now
GET /v1/task/balancePartitionsRebalance partitions across Stores
GET /v1/task/splitPartitionsRun automatic partition splitting now
GET /v1/task/balanceLeadersRebalance partition leaders
GET /v1/task/compactInstruct Store nodes to compact the RocksDB files of their partitions
GET /v1/prom/targets/{appName}Prometheus service discovery targets, no authentication required
GET /v1/prom/targets-allPrometheus targets for all app types
GET /v1/prom/sd_configPrometheus HTTP service discovery config
GET /actuator/healthSpring Boot health, no authentication required
GET /actuator/metricsSpring Boot metrics, no authentication required
GET /actuator/prometheusPrometheus scrape endpoint, no authentication required

The two log endpoints take a time range as {"startTime": "...", "endTime": "..."}; yyyy-MM-dd HH:mm:ss and yyyy-MM-dd are among the accepted formats.

PD registers its own meters under the hg prefix, so /actuator/prometheus exposes hg_up, hg_graphs, hg_stores and hg_terms alongside the standard JVM metrics, plus per-graph partition and size meters once graphs exist.

It also exposes Raft membership gauges for monitoring and alerting on quorum loss:

  • hg_raft_leader: 1 if this PD node is the elected Raft leader, 0 otherwise.
  • hg_raft_has_leader: 1 if this PD node observes an active Raft leader (i.e. is part of a quorum), 0 when quorum is lost.
  • hg_raft_alive_peers: Number of active Raft peers (including itself) that the leader has heard from within the leader lease timeout; reported as NaN on non-leader follower nodes.
Note

Quorum Loss Alerting: Cluster quorum is lost when sum(hg_raft_leader) == 0 or hg_raft_has_leader == 0 across all members. Alert rules should configure a for: window longer than the Raft election timeout (e.g. for: 30s) to avoid false positives during expected leader transitions. To alert on peer degradation from Prometheus, query hg_raft_alive_peers and on(instance) (hg_raft_leader == 1).

In versions after 1.7.0, PD also exports, for each registered store, hg_store_state{store="<id>",state="<state>"}, which is 1 for the store’s current state (Unknown, Pending, Up, Offline, Exiting or Tombstone) and 0 for the others, and hg_store_leader_count{store="<id>"}, the number of shard groups whose leader is on that store. A rebuilt Store reports 0 leaders until leadership moves back to it. Every PD member reads the same replicated metadata, so select one instance before aggregating these series.

7.6 Store Decommissioning and Drain Lifecycle

Decommissioning a Store node follows a state machine to safely drain partition replicas without service interruption:

  1. State Machine Overview:

    • Up: The Store is online, healthy, and accepting new shard allocations and leader roles.
    • Exiting: The Store is actively draining. While in Exiting, the Store continues renewing its heartbeat lease, but PD excludes it from new replica allocation and partition migration targets.
    • Partition Migration: PD schedules background tasks to migrate all shard replicas off the draining store to other eligible active Stores.
    • Tombstone: The Store is permanently retired. Once marked Tombstone, management updates cannot return the Store to an active state.
  2. Draining Invocation: Initiate store draining via:

    1curl -X POST http://<pd-host>:8620/v1/store/{storeId} \
    2  -H "Authorization: Basic <credentials>" \
    3  -H "Content-Type: application/json" \
    4  -d '{"storeState": "Exiting"}'
  3. Capacity Constraints & Retirement Checks:

    • Capacity Check: Draining checks that remaining eligible active stores satisfy minStoreCount (pd.min-store-count). If remaining capacity is insufficient, PD blocks the transition and rejects the request before state changes. In this case, add eligible Store capacity and resubmit the request.
    • Retirement Criteria: PD keeps the Store in Exiting until its latest heartbeat reports 0 local partition engines AND PD has no remaining shard groups referencing that Store ID.
    • Pre-Deletion Verification: Before deleting the Store Pod or persistent volume, operators must verify remaining shard groups have full Raft membership, complete replicas, and fresh heartbeats.
    • Re-registration Intent Preservation: Re-registering an Exiting Store (e.g. after process restart) preserves the Exiting intent even if its address is in pd.initial-store-list.

1.3 - HugeGraph-Store Quick Start

1 HugeGraph-Store Overview

HugeGraph-Store is the storage node component of HugeGraph’s distributed version, responsible for actually storing and managing graph data. It works in conjunction with HugeGraph-PD to form HugeGraph’s distributed storage engine, providing high availability and horizontal scalability.

Each Store node keeps graph data in RocksDB and replicates it with Raft (JRaft): every partition is a separate Raft group, so a partition survives the loss of a minority of its replicas. Store nodes do not know about each other directly. They register with PD, receive their partition assignment from PD, and report state back over a heartbeat. HugeGraph-Server reaches Store over gRPC after looking up partition locations in PD.

2 Prerequisites

2.1 Requirements

  • Operating System: Linux or macOS (Windows has not been fully tested)
  • Java version: Java 17 for HugeGraph 1.8.0+ (the Maven build accepts only JDK 17; the start script rejects older runtimes; 1.7.0 is the last release supporting Java 11)
  • Maven version: ≥ 3.6.3 (for building from source)
  • Deploy HugeGraph-PD first for multi-node deployment

3 Deployment

Choose a deployment method for the HugeGraph-Store component:

3.1 Download the tar package

Apache downloads currently provide the complete 1.7.0 binary bundle containing PD, Store, and Server, without a separate Store binary. Verify version, signatures, and SHA512 on the official download page. This historical incubating release retains incubating in archive and directory names:

1# Historical release 1.7.0 retains incubating in filenames and directories
2curl -fLO --connect-timeout 10 --max-time 1800 https://downloads.apache.org/hugegraph/1.7.0/apache-hugegraph-incubating-1.7.0.tar.gz
3tar zxf apache-hugegraph-incubating-1.7.0.tar.gz
4cd apache-hugegraph-incubating-1.7.0/apache-hugegraph-store-incubating-1.7.0

3.2 Compile from source

  1. Clone the source code:

    1git clone https://github.com/apache/hugegraph.git
    2cd hugegraph
  2. Build dependent struct and Store module:

    1# Build hugegraph-struct dependency and Store distribution
    2mvn install -pl hugegraph-struct -am -DskipTests
    3mvn clean package -pl hugegraph-store/hg-store-dist -am -DskipTests
  3. Locate the distribution output:

    1cd hugegraph-store/apache-hugegraph-store-*
    2ls -la bin conf lib

The assembled directory contains only bin/, conf/ and lib/hg-store-node-{version}.jar.

3.3 Docker Deployment

HG_STORE_* mappings are current master features, absent from Server tag 1.7.0. These Docker examples require PD, Store, and HStore Server images built from the same current master source with matching local tags:

1# Run from the HugeGraph repository root
2docker build -f hugegraph-pd/Dockerfile -t hugegraph/pd:local .
3docker build -f hugegraph-store/Dockerfile -t hugegraph/store:local .
4docker build -f hugegraph-server/Dockerfile-hstore -t hugegraph/server:local .

Before first startup, create .env in docker/ following the authentication environment instructions. The procedure uses umask 077 and refuses to overwrite existing files. Keep an existing .env and add missing entries. Never regenerate HG_PD_AUTH_SECRET_KEY for initialized data directories. Although .gitignore excludes this file, do not commit it. In every new shell, load the same file from docker/ before Compose or PD REST commands:

1# Run from the HugeGraph repository root
2cd docker
3set -a; . ./.env; set +a
4: "${HG_PD_AUTH_SECRET_KEY:?Set HG_PD_AUTH_SECRET_KEY in .env}"
5# Start PD, Store, and Server without optional Hubble
6HUGEGRAPH_VERSION=local HUGEGRAPH_PULL_POLICY=never \
7  docker compose -f docker-compose-hstore.yml \
8  up -d --wait pd store server

Load the same secret before subsequent restarts or PD REST calls; do not regenerate it for initialized directories. Explicitly include HUGEGRAPH_VERSION=local and HUGEGRAPH_PULL_POLICY=never in every Compose lifecycle command.

To start Hubble, use the topology-specific untracked configuration: minimal docker-compose-hstore.yml mounts conf/hubble/hstore.local.properties; HA docker-compose-3pd-3store-3server.yml mounts conf/hubble/hstore-ha.local.properties. The README initialization creates both files with the shared PD secret. If missing or requiring updates, load existing .env from docker/ and run:

1set -a; . ./.env; set +a
2./set-hubble-pd-password.sh hstore      # Minimal topology
3./set-hubble-pd-password.sh hstore-ha  # HA topology

Multi-node service names are pd0–pd2, store0–store2, and server0–server2. This master Compose file also requires matching locally built images; do not mix HUGEGRAPH_VERSION=1.7.0 release images with master Compose.

To run master-built Store alone, replace example addresses with actual routable Store and PD addresses:

The mkdir -p and nonrecursive chown below are for a new empty directory. Before reusing data written by a root image, stop Store and run sudo chown -R 10001:10001 /path/to/storage once, then start the UID 10001 image.

 1# Master-built images run as UID 10001; the host data directory must be writable by it
 2mkdir -p /path/to/storage && sudo chown 10001:10001 /path/to/storage
 3docker run -d \
 4  -p 8520:8520 -p 8500:8500 -p 8510:8510 \
 5  -e HG_STORE_PD_ADDRESS=192.168.1.10:8686 \
 6  -e HG_STORE_GRPC_HOST=192.168.1.20 \
 7  -e HG_STORE_RAFT_ADDRESS=192.168.1.20:8510 \
 8  -v /path/to/storage:/hugegraph-store/storage \
 9  --name hugegraph-store \
10  hugegraph/store:local

Current Master Docker Environment Variables:

VariableRequiredDefaultConfiguration keyDescription
HG_STORE_PD_ADDRESSYesNonepdserver.addressComma-separated PD gRPC addresses.
HG_STORE_GRPC_HOSTYesNonegrpc.hostAdvertised hostname/IP; use a container hostname within a container network.
HG_STORE_RAFT_ADDRESSYesNoneraft.addressThis node’s Raft address.
HG_STORE_GRPC_PORTNo8500grpc.portgRPC service port.
HG_STORE_REST_PORTNo8520server.portREST API port.
HG_STORE_DATA_PATHNo/hugegraph-store/storageapp.data-pathData path.
HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZENoNonerocksdb.total_memory_sizeRocksDB memory budget: auto or a number of bytes. An empty value counts as unset; any other value makes the entrypoint exit with code 2. When unset, the value in application-pd.yml or auto-sizing applies.

The master entrypoint writes these values into SPRING_APPLICATION_JSON, overriding image conf/application.yml. Other options still come from that file and application-pd.yml; Java runs in the foreground. Deprecated PD_ADDRESS, GRPC_HOST, and RAFT_ADDRESS remain supported with warnings.

PD and Store Docker HEALTHCHECK use /v1/health, confirming only local REST responses. Compose depends_on uses the same startup gate; separately verify PD /v1/ready and Up Store state in /v1/stores. Store defaults to STDOUT_MODE=true; its Dockerfile declares only EXPOSE 8520, so publish gRPC 8500 and Raft 8510 separately if needed outside the Docker network.

4 Configuration

Store reads two files under conf/. Release 1.7.0 sources: application.yml, application-pd.yml. Master sources: application.yml, application-pd.yml. Master places RocksDB settings in the second file, included through spring.profiles.include: pd; startup selects application.yml.

  • application.yml: Main PD address, ports, Raft, and data path settings.
  • application-pd.yml: Included through spring.profiles.include: pd; RocksDB memory and Actuator exposure.

4.1 application.yml

The following tables were checked against distribution values and Java defaults at Server master. Use tagged configuration for release 1.7.0 without replacing it with another version’s files.

4.2 application-pd.yml

This file contains only rocksdb and Actuator exposure settings.

Warning

Protect Server and Store ports separately in production

Enable Server authentication and authorization, an IP allowlist, and minimum graph API permissions; retain and protect Server audit-*.log. These settings do not protect Store: master exposes all Actuator endpoints without PD-style Basic authentication. Restrict Store REST, gRPC, and Raft to cluster nodes and trusted operations networks.

4.3 Configuration reference

“Master distribution value” comes from the two master files above. “Code default” is the fallback when a key is absent; rely on code defaults for options omitted from the distribution files.

Core

KeyMaster distribution valueCode defaultMeaning
pdserver.addresslocalhost:8686requiredPD gRPC endpoints, comma separated. Store registers itself here and receives its partition assignment. Must be PD’s grpc.port, not its REST port.
grpc.host127.0.0.1requiredAddress this node advertises for its own gRPC service. Set it to a routable IP or hostname, 127.0.0.1 is only usable for a single-machine setup.
grpc.port8500requiredgRPC port. Server and the Store client connect here.
grpc.netty-server.max-inbound-message-size1000MBgRPC defaultMaximum size of a single inbound gRPC message. Bound by the grpc-spring-boot-starter Netty server.
grpc.server.wait-timenot set3600Seconds a scan stream waits for the client to consume a page before the server aborts it.
server.port8520requiredREST and Actuator port. Also reported to PD as the rest.port label.

Raft

KeyMaster distribution valueCode defaultMeaning
raft.address127.0.0.1:8510requiredRaft service address of this node, host:port. Must be reachable from every other Store node. There is no peer list to configure: PD tells each node which peers belong to a partition’s Raft group.
raft.disruptorBufferSize10240Raft task queue size. 0 derives it from rocksdb.total_memory_size, by rounding that size in GB (at least 1) to the nearest power of two and multiplying by 32.
raft.max-log-file-size60000000000050000000000Maximum byte size of Raft logs.
raft.snapshotInterval1800300Seconds between Raft snapshots.
raft.snapshotLogIndexMarginnot set0Minimum applied-index distance since the last snapshot before a snapshot is actually written. 0 disables the distance check.
raft.rpc-timeoutnot set10000Raft RPC timeout in milliseconds.
raft.metricsnot settrueCollect JRaft node metrics, readable at /metrics/raft.
raft.useRocksDBSegmentLogStoragenot settrueStore Raft logs in the RocksDB segment log storage.
raft.maxSegmentFileSizenot set67108864Segment log file size in bytes (64 MB).
raft.maxReplicatorInflightMsgsnot set256Maximum in-flight replication requests per follower.
raft.maxEntriesSizenot set256Maximum number of entries in one AppendEntries request.
raft.maxBodySizenot set524288Maximum byte size of one AppendEntries request.
ave-logEntry-size-rationot set0.95Smoothing ratio used to estimate the average log entry size. Note that this key sits at the top level, not under raft.

Storage and labels

KeyMaster distribution valueCode defaultMeaning
app.data-path./storagestoreRocksDB data directory. Multiple paths separated by commas spread partitions over several disks.
app.raft-pathcommented outemptyDirectory for Raft logs and snapshots. Falls back to app.data-path when empty.
app.fake-pdnot setfalseBuilt-in PD mode for standalone testing. Do not use it in production.
app.placeholder-sizenot set10Size in GB of a placeholder file created in each data path at startup, so space can be freed in an emergency. 0 disables it.
app.label.<name>not setnoneArbitrary key/value labels sent to PD in the store heartbeat. The node adds rest.port on its own.

RocksDB

KeyMaster distribution valueCode defaultMeaning
rocksdb.total_memory_sizeautoAuto-sized, see noteOff-heap memory budget shared across all RocksDB instances on this node (auto or byte count). See below for derivation rules.
rocksdb.write_buffer_size32000000Derived, see MeaningMemtable size in bytes. When absent or 0, the node uses total_memory_size / 1000, at least 1048576 (1 MiB).
rocksdb.min_write_buffer_number_to_merge1616Number of memtables merged together before a flush.
rocksdb.write_buffer_rationot set0.66Share of total_memory_size given to the write cache. The rest becomes the block cache.

RocksDB memory budget sizing rules

Note

When rocksdb.total_memory_size is set to auto (or absent / 0), the node sizes the off-heap RocksDB budget automatically:

  • Calculation Formula: Takes physical memory (using the visible cgroup limit in containers) minus JVM maximum heap (-Xmx), direct memory (MaxDirectMemorySize), and a 1.5 GiB system reserve, then allocates half the remainder, clamped between 256 MiB and 32 GB (32000000000 bytes).
  • Write Cache & Block Cache: Budget is divided according to rocksdb.write_buffer_ratio (default 0.66). This is a soft capacity target; under heavy concurrent write loads, set rocksdb.write_buffer_allow_stall to true to hold the write cache to its share at the cost of write latency. The block cache is not a strict limit, so native memory can still exceed the budget and a container can still be OOM-killed.
  • Upgrading Existing Deployments: Deployments carrying legacy 32000000000 can enable auto-sizing by setting environment variable HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto or JVM property -Drocksdb.total_memory_size=auto.

Any other option defined in org/apache/hugegraph/rocksdb/access/RocksDBOptions.java can be added under the same rocksdb: block, for example rocksdb.max_background_jobs, rocksdb.level0_file_num_compaction_trigger or rocksdb.bloom_filter_bits_per_key.

Thread pools

KeyCode defaultMeaning
thread.pool.grpc.core600Core threads serving gRPC requests.
thread.pool.grpc.max1000Maximum gRPC threads.
thread.pool.grpc.queue2147483647gRPC task queue capacity.
thread.pool.scan.core128Core threads serving scans. 0 means 4 times the CPU count.
thread.pool.scan.max1000Maximum scan threads.
thread.pool.scan.queue0Scan task queue capacity.

gRPC callback queue (1.8.0)

Store uses an unbounded queue for gRPC callbacks so that a full queue does not discard cancellation or completion notifications and leave requests unfinished. You can keep thread.pool.grpc.queue in the configuration, but it no longer limits this queue.

thread.pool.grpc.core and thread.pool.grpc.max still configure the thread pool. Because callbacks can continue to queue, a backlog normally does not grow the pool from core to max; increasing max alone does not increase callback processing capacity.

Sustained overload can still grow the queue and increase memory usage. Limit client concurrency and size the scan and query worker pools for the node’s resources; do not rely on thread.pool.grpc.queue for load control. Version 1.7.0 still uses the configured gRPC queue capacity.

Query pushdown

KeyCode defaultMeaning
query.push-down.threads1500Thread pool size for pushed-down queries.
query.push-down.fetch_batch20000Rows fetched per request.
query.push-down.fetch_timeout300000Fetch timeout in milliseconds.
query.push-down.memory_limit_count50000Row limit for in-memory operations such as sorting.
query.push-down.index_size_limit_count50000Index sst file size limit in kB.

Background jobs

KeyCode defaultMeaning
job.interruptableThreadPool.core128Core threads of the TTL cleaner pool. 0 means the CPU count.
job.interruptableThreadPool.max256Maximum threads of the TTL cleaner pool. 0 means 4 times the CPU count.
job.interruptableThreadPool.queue2147483647Queue capacity of the TTL cleaner pool.
job.uninterruptibleThreadPool.core0Core threads of the engine’s uninterruptible job pool. 0 means the CPU count.
job.uninterruptibleThreadPool.max256Maximum threads of the uninterruptible job pool.
job.uninterruptibleThreadPool.queue2147483647Queue capacity of the uninterruptible job pool.
job.cleaner.batch.size10000Keys deleted per batch by the TTL cleaner.
job.start-time0Hour of day (0 to 23) at which the daily TTL cleanup runs. Values outside that range fall back to 19.

Built-in PD mode

Only for single-node development and debugging, activated by app.fake-pd: true. The node then plays PD’s role itself and ignores pdserver.address.

KeyCode defaultMeaning
fake-pd.store-list''gRPC addresses of the Store nodes in the fake cluster.
fake-pd.peers-list''Raft addresses of the same nodes.
fake-pd.partition-count3Number of partitions.
fake-pd.shard-count3Replicas per partition.

Diagnostics

Starting with HugeGraph 1.8.0, Store HTTP activation (GET /v1/arthasstart) is disabled and returns 403. Legacy configuration fields are kept only for compatibility. Run the bundled script as the service OS user on the host to diagnose the running JVM (official container images include JDK 17):

1bin/arthas.sh

For multiple processes on one host, specify --pid <PID> --telnet-port 3659. For command options, authentication, and security practices, see Start the Arthas agent.

4.4 Per-node changes

For multi-node deployment, you need to modify the following configurations for each Store node:

  1. grpc.host and grpc.port (the address other components dial)
  2. raft.address (Raft protocol address)
  3. server.port (REST port)
  4. app.data-path (data storage path)

pdserver.address is the same on every node, it lists the whole PD cluster.

5 Start and Stop

5.1 Start Store

Ensure that the PD service is already started, then in the Store installation directory, execute:

1./bin/start-hugegraph-store.sh

The lifecycle and timeout details below describe master builds that include launcher script repairs. The downloaded 1.7.0 launcher does not include those repairs. The script accepts four flags, with GC version limits noted below:

FlagValuesDefaultDescription
-dtrue, falsetrueDaemon mode. See below.
-gSee version limits belownot setSelect the garbage collector. Unsupported values abort startup.
-jJVM options stringemptyExtra JVM options, for example -j "-Xmx16g -Xms8g".
-ytrue, falsefalseAttach the OpenTelemetry Java agent, downloading it into plugins/ on first use, and export traces to 127.0.0.1:4317.

Master builds accept g1, G1, zgc and ZGC. The downloaded 1.7.0 launcher accepts only zgc or ZGC as explicit choices; -g g1, -g G1 and other values abort startup. Omit -g for the default path. For rejected values, inspect logs/hugegraph-store-server.log.

Daemon mode:

  • -d true (default): run as a background daemon. The script returns immediately and writes the Java pid to bin/pid.
  • -d false: run in the foreground. The script execs Java, so the container or supervisor process is Java itself. Use this under Docker or a process supervisor (systemd, supervisord) so crashes are detected and the service is restarted automatically.

JVM memory, unless you set JAVA_OPTIONS yourself: -Xms512m, and -Xmx set to half the free memory, clamped to the 512 MB to 2048 MB range. The script also adds -XX:MaxDirectMemorySize equal to that heap, -XX:MetaspaceSize=256M, a heap dump on out-of-memory into logs/, and a rolling GC log at logs/gc.log. Production nodes normally need a much larger heap, so pass one explicitly together with a direct-memory cap, for example -j "-Xmx32g -Xms32g -XX:MaxDirectMemorySize=2g". Without the cap the JVM allows as much direct memory as the 32 GB heap, the RocksDB budget counts it, and on a host with less than about 66 GB it falls to the 256 MiB floor.

The script refuses to start if ulimit -n or ulimit -u is below 1024, and it preloads jemalloc on x86_64 and arm64 when the shared object can be downloaded and verified.

In master builds, each optional jemalloc download allows 5 seconds to connect and 30 seconds in total. The 1.7.0 helper has no connection or total timeout. A timeout or failed verification follows the existing fallback: startup continues without preloading the downloaded library. Verified cached libraries can still be reused; partial downloads are removed.

After successful startup, you can see logs similar to the following in logs/hugegraph-store-server.log:

1YYYY-mm-dd xx:xx:xx [main] [INFO] o.a.h.s.n.StoreNodeApplication - Started StoreNodeApplication in x.xxx seconds (JVM running for x.xxx)

5.2 Stop Store

In the Store installation directory, execute:

1./bin/stop-hugegraph-store.sh

The script reads bin/pid, signals that process, and waits up to 30 seconds for it to exit before removing the pid file. If bin/pid is missing it exits without doing anything.

5.3 Restart Store

1./bin/restart-hugegraph-store.sh

It sources the stop script and then the start script, and forwards the flags from section 5.1.

5.4 Startup order

  1. All PD nodes first. Each Store’s grpc.host:grpc.port must appear in PD pd.initial-store-list, or the node remains Pending instead of Up, blocking partition allocation. On master, verify HTTP 200 with ready:true from every PD /v1/ready first; /v1/health is liveness only.
  2. Store next. Registration retries while PD is unreachable, logging store heartbeat error: PD UNREACHABLE.
  3. Server last. Every Store should report Up; partitions must be ready before opening or initializing graphs.

Master Compose uses depends_on: condition: service_healthy for local liveness, but PD and Store Docker checks only call /v1/health; they do not confirm Raft quorum or Store registration. Server waits for PD to report at least one Up Store. Operators should still check PD /v1/ready and /v1/stores.

6 Multi-Node Deployment Example

Below is a configuration example for a three-node deployment:

6.1 Three-Node Configuration Reference

  • 3 PD nodes
    • raft ports: 8610, 8611, 8612
    • rpc ports: 8686, 8687, 8688
    • rest ports: 8620, 8621, 8622
  • 3 Store nodes
    • raft ports: 8510, 8511, 8512
    • rpc ports: 8500, 8501, 8502
    • rest ports: 8520, 8521, 8522

6.2 Store Node Configuration

For the three Store nodes, the main configuration differences are as follows:

Node A:

1grpc:
2  port: 8500
3raft:
4  address: 127.0.0.1:8510
5server:
6  port: 8520
7app:
8  data-path: ./storage-a

Node B:

1grpc:
2  port: 8501
3raft:
4  address: 127.0.0.1:8511
5server:
6  port: 8521
7app:
8  data-path: ./storage-b

Node C:

1grpc:
2  port: 8502
3raft:
4  address: 127.0.0.1:8512
5server:
6  port: 8522
7app:
8  data-path: ./storage-c

All nodes should point to the same PD cluster:

1pdserver:
2  address: 127.0.0.1:8686,127.0.0.1:8687,127.0.0.1:8688

And every PD node should list all three Store gRPC addresses:

1pd:
2  initial-store-list: 127.0.0.1:8500,127.0.0.1:8501,127.0.0.1:8502

6.3 Docker Distributed Cluster Configuration

The distributed Store cluster definition is included in docker/docker-compose-3pd-3store-3server.yml. Each Store node gets its own hostname and environment variables:

 1# store0, published as 8500 (gRPC), 8510 (Raft), 8520 (REST)
 2HG_STORE_PD_ADDRESS: pd0:8686,pd1:8686,pd2:8686
 3HG_STORE_GRPC_HOST: store0
 4HG_STORE_GRPC_PORT: "8500"
 5HG_STORE_REST_PORT: "8520"
 6HG_STORE_RAFT_ADDRESS: store0:8510
 7HG_STORE_DATA_PATH: /hugegraph-store/storage
 8
 9# store1, published as 8501, 8511, 8521
10HG_STORE_GRPC_HOST: store1
11HG_STORE_RAFT_ADDRESS: store1:8510
12
13# store2, published as 8502, 8512, 8522
14HG_STORE_GRPC_HOST: store2
15HG_STORE_RAFT_ADDRESS: store2:8510

The container ports stay 8500/8510/8520 on every node, only the published host ports differ. The PD nodes set HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500 to match.

Store nodes start only after all PD nodes pass healthchecks (/v1/health), enforced via depends_on: condition: service_healthy.

To view runtime logs for a running Store container use docker logs <container-name> (e.g. docker logs hg-store0).

See docker/README.md for the full setup guide.

7 Verify Store Service

Confirm that the Store service is running properly:

1curl http://localhost:8520/actuator/health

Actuator {"status":"UP"} confirms local Store application health, without proving registration as Up in PD.

GET /v1/health is the lighter check used by the Docker image and the compose files. It answers HTTP 200 with an empty body, so use curl -fsS and check the exit code rather than the output:

1curl -fsS http://localhost:8520/v1/health && echo OK

7.1 Store REST endpoints

The Store node exposes these read-only endpoints on server.port:

MethodPathDescription
GET/v1/healthLiveness probe, HTTP 200 with an empty body
GET/actuator/healthSpring Boot Actuator health, {"status":"UP"}
GET/actuator/prometheusPrometheus scrape endpoint
GET/Node summary, leaderCount and partitionCount
GET/-/stateOperator-set STARTING, ONLINE, or STOPPING flag; distinct from PD registration state
GET/-/lifecycleProcess lifecycle state (available in 1.8.0+)
GET/-/echo?name=<text>Echo check
GET/-/scanState of the running scan streams
GET/v1/partitionsAll Raft groups on this node with per-partition metrics. Add ?flags=accurate for exact key counts, which is slower.
GET/v1/partition/{id}One Raft group by partition id, including role, leader, peers and committed index
GET/metrics/systemHost CPU and memory metrics
GET/metrics/driveDisk metrics for the data paths
GET/metrics/raftJRaft node metrics, needs raft.metrics: true

GET /-/lifecycle reports the process lifecycle status as JSON, such as {"status":0,"text":"STARTED"}. The text field has four possible values:

  • STARTING: Spring application context or gRPC service is still initializing;
  • STARTED: Spring application is ready (ApplicationReadyEvent) and gRPC service is running;
  • STOPPING: Process shutdown has initiated (ContextClosedEvent takes precedence);
  • UNAVAILABLE: gRPC service stopped or terminated unexpectedly before shutdown.
Warning

Process lifecycle does not equal cluster readiness: STARTED reflects only local JVM and gRPC process state, not storage or cluster data readiness. It does not prove that the node has registered with PD, that data shards have completed recovery, or that Raft replicas have caught up.

  • Before continuing a rolling update or replacing nodes, always check PD via GET /v1/stores to confirm the store status is Up, and inspect GET /v1/partitions to ensure all Raft groups are healthy. Never rely solely on STARTED as a rolling update gate.
  • Versions prior to 1.8.0 do not support this endpoint and return HTTP 404; a 404 must not be treated as STARTED.

Master Store configuration exposes all Actuator endpoints with management.endpoints.web.exposure.include: "*" and enables Prometheus. These endpoints lack PD REST Basic protection; restrict Store REST network access.

The node also serves maintenance endpoints that change state or run heavy work: PUT /-/state, GET /-/cleaner, GET /v1/partition/dump/{id}, GET /v1/partition/clean/{id}, POST /v1/compat?id=<partition>, POST /raft/options, and the /fix/* and /test/* groups. Use them only for troubleshooting, and keep the REST port off untrusted networks.

7.2 Check registration from PD

You can also check Store node status through the PD API:

1curl -u "store:${HG_PD_AUTH_SECRET_KEY:?Set the PD deployment secret first}" \
2  http://localhost:8620/v1/stores

For master builds, PD REST usernames must be hg, store, hubble, or vermeer, and the password must equal PD auth.secret-key. HG_PD_AUTH_SECRET_KEY configures PD and Server in Compose. /v1/health, /v1/ready, /actuator/*, and /v1/prom/targets/* require no authentication. Release 1.7.0 checks only usernames, without comparing passwords; do not use that old behavior for master builds.

If Store is configured successfully, the response should include status information for the current node, and state: "Up" means the node is running normally. A node stuck at Pending is usually missing from PD’s pd.initial-store-list.

The example below shows a single Store node. If all three nodes are configured correctly and running, the storeId list should contain three IDs, and stateCountMap.Up, numOfService, and numOfNormalService should all be 3.

 1{
 2  "message": "OK",
 3  "data": {
 4    "stores": [
 5      {
 6        "storeId": 8319292642220586694,
 7        "address": "127.0.0.1:8500",
 8        "raftAddress": "127.0.0.1:8510",
 9        "version": "",
10        "state": "Up",
11        "deployPath": "/Users/{your_user_name}/hugegraph/hugegraph-store/apache-hugegraph-store-{version}/lib/hg-store-node-{version}.jar",
12        "dataPath": "./storage",
13        "startTimeStamp": 1754027127969,
14        "registedTimeStamp": 1754027127969,
15        "lastHeartBeat": 1754027909444,
16        "capacity": 494384795648,
17        "available": 346535829504,
18        "partitionCount": 0,
19        "graphSize": 0,
20        "keyCount": 0,
21        "leaderCount": 0,
22        "serviceName": "127.0.0.1:8500-store",
23        "serviceVersion": "",
24        "serviceCreatedTimeStamp": 1754027127000,
25        "partitions": []
26      }
27    ],
28    "stateCountMap": {
29      "Up": 1
30    },
31    "numOfService": 1,
32    "numOfNormalService": 1
33  },
34  "status": 0
35}

7.3 Minimal Graph Read/Write Verification

After Store appears as Up in PD, start an HStore-backed Server connected to the same PD cluster. See the Server Quick Start. Once GET /versions responds and the default hugegraph graph is loaded, create a property key, vertex label, and vertex, then read it:

 1SERVER_URL=http://localhost:8080
 2GRAPH_URL="$SERVER_URL/graphspaces/DEFAULT/graphs/hugegraph"
 3
 4# Names must be unique within the graphspace; choose new names if they already exist
 5curl -fsS -X POST "$GRAPH_URL/schema/propertykeys" \
 6  -H 'Content-Type: application/json' \
 7  -d '{"name":"pd_store_demo_name","data_type":"TEXT","cardinality":"SINGLE","properties":[]}'
 8
 9curl -fsS -X POST "$GRAPH_URL/schema/vertexlabels" \
10  -H 'Content-Type: application/json' \
11  -d '{"name":"pd_store_demo_vertex","id_strategy":"CUSTOMIZE_STRING","properties":["pd_store_demo_name"],"primary_keys":[],"nullable_keys":[]}'
12
13curl -fsS -X POST "$GRAPH_URL/graph/vertices" \
14  -H 'Content-Type: application/json' \
15  -d '{"id":"pd-store-demo-1","label":"pd_store_demo_vertex","properties":{"pd_store_demo_name":"stored in HStore"}}'
16
17curl -fsS "$GRAPH_URL/graph/vertices/%22pd-store-demo-1%22"

The final response should contain vertex ID pd-store-demo-1 and property pd_store_demo_name. If Server authentication is enabled, add credentials to each request.

1.4 - Deploy HugeGraph with Helm

This guide covers HugeGraph 1.8.0 and later, using Helm 3.10+ with OCI support and Kubernetes (K8s) 1.23+. Install hugegraph-helm from GHCR or Docker Hub. Chart and HugeGraph versions are maintained separately: for example, a 0.4.x Chart can deploy a 1.8.x service.

Warning

The current development Chart uses latest images for the base HugeGraph components and is intended for testing. Optional Vermeer is pinned to a verified published image and digest; see optional engines. For production, choose a stable Chart once the corresponding HugeGraph service has been released, and review its image versions. Image tags can change; set each component’s image.digest to keep using the same image content.

Artifact Hub helps you find the Chart, browse versions and read installation instructions. The packages themselves are downloaded from GHCR or Docker Hub. See the chart README for settings and source-based development, and the operations reference for lifecycle procedures.

Find a Chart version on Artifact Hub, download from GHCR or Docker Hub, and install with Helm

1 Install

Note

Default Topology & Storage Prerequisites

  • Components: Deploys 3 PD + 3 Store + 3 Server and Hubble by default, with authentication and persistent storage enabled.
  • Volume Requirements: Requires a default dynamic StorageClass or matching pre-created static PVs, totaling 7 persistent volumes (3 × 10 GiB PD volumes, 3 × 50 GiB Store volumes, and 1 × 1 GiB Hubble volume).
  • Memory Sizing: Retains container JVM memory auto-calculation; configure resource requests/limits to allocate sufficient physical memory for the 9 JVMs and Hubble.

Choose either registry; GHCR is recommended for public CI. For local lightweight testing, use the single-node preset.

Tip

--devel selects the highest Chart version, including development and stable versions, without limiting the version series. For production or reproducible installations, replace --devel with --version <published-version> to pin your chosen version.

GHCR (recommended)
1CHART=oci://ghcr.io/hugegraph/hugegraph-helm
2kubectl config current-context
3kubectl get nodes
4kubectl get storageclass
5helm install hugegraph "$CHART" --devel -n hugegraph \
6  --create-namespace --wait --timeout 15m
Docker Hub
1CHART=oci://registry-1.docker.io/hugegraph/hugegraph-helm
2kubectl config current-context
3kubectl get nodes
4kubectl get storageclass
5helm install hugegraph "$CHART" --devel -n hugegraph \
6  --create-namespace --wait --timeout 15m
Single-node trial
1# For local testing (Minikube, Kind): install the Chart downloaded below
2CHART=oci://ghcr.io/hugegraph/hugegraph-helm
3helm pull "$CHART" --devel --untar
4helm install hugegraph ./hugegraph-helm -n hugegraph \
5  -f ./hugegraph-helm/values-single.yaml \
6  --create-namespace --wait --timeout 10m

The single-node preset selects 1 PD + 1 Store + 1 Server using HStore, plus Hubble, with relaxed scheduling constraints. It leaves SeaTunnel and Vermeer disabled.

2 Modes and entry points

The extracted package is rooted at hugegraph-helm/; the source directory remains helm/hugegraph/.

HugeGraph Helm Chart Layout

  • hugegraph-helm/
    • Chart.yamlChart metadata and version definitions
    • values.yamlDefault values (easy mode: 3 PD + 3 Store + 3 Server + 1 Hubble, NodePort)
    • values-single.yamlSingle-node preset (local Minikube/Kind testing with 1 replica and no anti-affinity)
    • values-cluster.yamlMulti-node cluster preset (required anti-affinity, Store OnDelete rollout)
    • values-strict.yamlHardened security preset (strict mode: ClusterIP only, strict NetworkPolicies)
    • values.schema.jsonSchema validation for values configuration
    • templates/Kubernetes workload templates
      • pd-statefulset.yamlPD StatefulSet (metadata and scheduling)
      • store-statefulset.yamlStore StatefulSet (storage engine)
      • server-deployment.yamlServer Deployment (injects serviceId and graphSpace)
      • hubble-deployment.yamlHubble UI Deployment
      • tests/Smoke validation hook (helm test, supports custom testGraphAlias)
    • examples/Deployment examples: local PVs, GitOps credentials, optional engines and Ingress
    • docs/operations.mdLifecycle operations and recovery guide

mode: easy is the default. Server and Hubble use automatic NodePorts; PD stays ClusterIP. HTTP works without a domain, certificate or acknowledgement. mode: strict defaults Server/Hubble to ClusterIP and enables full NetworkPolicy isolation. Enabled SeaTunnel and Vermeer native HTTP/UI default to NodePort in easy mode and ClusterIP in strict mode; both require login. Internal RPC, Worker HTTP and SeaTunnel member ports remain private. Both modes keep authentication and persistence. Explicit component parameters take priority over global parameters, which take priority over mode defaults; explicit false remains an override. The global parameters are persistence.enabled and global.imageRegistry. Strict mode is also available through values-strict.yaml.

EntryAccessRequirements
NodePorthttp://<reachable-node-ip>:<assigned-node-port>Node connectivity and firewall access; no additional controller
ClusterIPInternal Service IP/DNS, for example hugegraph-server.hugegraph.svc:8080External clients use port-forward or an existing gateway
LoadBalancerInfrastructure-assigned addressCompatible load balancer controller
IngressHTTP routing through a controllerInstalled controller; hostname and TLS optional
1kubectl -n hugegraph get svc hugegraph-server hugegraph-hubble
2kubectl get nodes -o wide
3kubectl -n hugegraph get secret hugegraph-admin \
4  -o jsonpath='{.data.password}' | base64 --decode

Open Hubble on its assigned NodePort and log in as admin with this password. A node need not host the Pod with default externalTrafficPolicy: Cluster. The internal ClusterIP is a virtual Service address, not a node IP; a laptop normally cannot reach it directly. Use kubectl -n hugegraph port-forward svc/hugegraph-hubble 8088:8088 for a local browser at http://127.0.0.1:8088.

The base PD/Server/Hubble client Services support externalTrafficPolicy, loadBalancerClass, loadBalancerIP, loadBalancerSourceRanges, externalIPs and allocateLoadBalancerNodePorts, subject to the cluster/controller’s capabilities. Use LoadBalancer fields with that Service type. NodePort omission or 0 means automatic allocation. Fixed ports are validated against the cluster’s configured range by Kubernetes; the chart does not force 30000–32767. Legacy HTTP acknowledgement fields remain accepted but no longer gate rendering.

Optional engines: disabled by default

By default, the Chart provisions only the core graph services (PD + Store + Server + Hubble). For data migration and distributed memory graph computing, the Chart offers SeaTunnel and Vermeer as native optional engines, both disabled by default:

EngineDefaultPurposeTopologies (*.mode)Access & Security
HubbleEnabled (true)Graph visualization, multi-graphspace management & metricsSingle-replica DeploymentFollows global mode (NodePort or ClusterIP)
SeaTunnelDisabled (false)Heterogeneous data ingestion & full graph migrationstandalone / clusterNodePort in easy, ClusterIP in strict; HTTP Basic auth
VermeerDisabled (false)Memory-first high-performance distributed graph algorithmsstandalone / clusterNodePort in easy, ClusterIP in strict; Token auth, auto-wired to Hubble

Enable either engine by setting its enabled key to true in values. Their administrator passwords are generated independently from Server. For seatunnel.service.type and vermeer.service.type, null inherits the mode default: NodePort in easy mode, ClusterIP in strict mode. An explicit ClusterIP stays private; an explicit NodePort, including in strict mode, exposes authenticated HTTP/UI. Clear an older explicit ClusterIP override to null to inherit easy mode. With service.nodePort omitted or null, Kubernetes assigns the port. The engines’ standalone / cluster topologies are independent of these access settings.

SeaTunnel: On-demand Data Import, Export, and Migration
1helm install hugegraph ./hugegraph-helm -n hugegraph --create-namespace \
2  -f ./hugegraph-helm/values-single.yaml \
3  --set seatunnel.enabled=true --set seatunnel.mode=standalone \
4  --wait --timeout 30m
  • Topologies: standalone runs 1 Hybrid node for lightweight testing; cluster runs multiple homogeneous nodes for high-throughput batch jobs.

  • Access & Authentication: In easy mode, open http://<reachable-node-ip>:<NodePort>/#/overview using the assigned HTTP NodePort:

    1kubectl -n hugegraph get svc hugegraph-seatunnel \
    2  -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}{"\n"}'
    3kubectl get nodes -o wide

    Log in with HTTP Basic username admin and the password from hugegraph-seatunnel-auth Secret. Strict mode defaults to ClusterIP; use kubectl -n hugegraph port-forward svc/hugegraph-seatunnel 8081:8080 for local access. Port 5801 remains internal.

  • See the SeaTunnel Deployment Section or SeaTunnel Guide for full job configs and migration steps.

Vermeer: In-Memory Graph Computing Integrated with Hubble
1helm install hugegraph ./hugegraph-helm -n hugegraph --create-namespace \
2  -f ./hugegraph-helm/values-single.yaml \
3  --set vermeer.enabled=true --set vermeer.mode=standalone \
4  --wait --timeout 20m
1hugegraph-helm/
2  examples/
3    values-vermeer-standalone.yaml # Standalone preset (Master and Worker combined in one Pod)
4    values-vermeer-cluster.yaml    # Cluster preset (1 dedicated Master Pod + 3 distributed Worker Pods)
5  docs/
6    vermeer.md                     # Advanced algorithm parameters, API mapping and operations
  • Topologies: standalone runs Master and Worker in one Pod; cluster provides a separate Master coordinator Pod and scalable Worker computing Pods (default 3 replicas).

  • Hubble Integration: Hubble automatically acquires the internal Master Service URL and service tokens. Operators can trigger 15+ graph algorithms directly from Hubble without manual token configuration.

  • Native Dashboard Access: In easy mode, open http://<reachable-node-ip>:<NodePort>/ using the Master HTTP NodePort:

    1kubectl -n hugegraph get svc hugegraph-vermeer-master \
    2  -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}{"\n"}'
    3kubectl get nodes -o wide

    Sign in with the password from Secret <release>-vermeer-admin under Admin credential. Strict mode defaults to ClusterIP; use kubectl -n hugegraph port-forward svc/hugegraph-vermeer-master 6688:6688 for local access. Master/Worker RPC and Worker HTTP remain internal.

  • For image pinning and credentials management, see the Vermeer Deployment Guide.

PD metadata namespace

The top-level metadataCluster parameter (default hg-test) specifies the PD metadata cluster prefix. The Chart injects this value into Server HG_SERVER_CLUSTER and Store PD_CLUSTER, ensuring Store correctly locates the Schema initialized by Server when processing Vermeer graph scans.

Important

metadataCluster is a metadata namespace prefix, not a Raft group name. Modifying it does not automatically migrate existing graph data. Always preserve the original cluster prefix when enabling Vermeer; if graph load tasks fail with Schema missing or no such vertex label, check that the namespace prefix matches across all components. Refer to the Metadata Namespace Operations Guide for details. Standard older releases using hg-test need no namespace-adoption step. For a custom legacy prefix, follow metadata namespace adoption before upgrading.

Server service identity and graph space

Both server.serviceId and server.graphSpace default to DEFAULT. To register the Server with a specific service name or graph space in PD:

For configuration-only changes, keep the installed Chart version and values. Find the version with helm list and replace the placeholder below:

1helm list -n hugegraph
2CHART=oci://ghcr.io/hugegraph/hugegraph-helm
3CHART_VERSION='<installed-chart-version>'
4helm upgrade hugegraph "$CHART" --version "$CHART_VERSION" -n hugegraph --reuse-values \
5  --set-string server.serviceId=analytics \
6  --set-string server.graphSpace=team_a \
7  --set-string server.testGraphAlias=team_a-demo

A non-default graph space must be created through the PD API before starting its assigned Server. server.testGraphAlias specifies the Gremlin binding checked by helm test (case-sensitive) and does not auto-infer from graphSpace. Do not re-declare service ID or graph space in server.extraEnv.

3 Internal images and storage

Complete examples cover private registries, dynamic storage, seven local PVs, GitOps credentials, API-only deployment and a hostless HTTP Ingress.

Set global.imageRegistry to prefix unqualified image repositories; an explicit registry in a component repository takes precedence. The default mirror inventory is only hugegraph/pd, hugegraph/store, hugegraph/server and hugegraph/hubble, at the configured tags/digests. Store’s PD wait init container and the Helm test reuse Store and Server images respectively, which already contain curl. No separate curl image or runtime package installation is required. Explicit store.waitImage / server.waitImage overrides remain supported.

Enabled SeaTunnel additionally needs its configured apache/seatunnel image; enabled Vermeer uses the published build pinned above. Vermeer reuses that image for Master, Worker and service-token initialization. Mirror every enabled image and explicit helper override. For a custom Vermeer image, also update vermeer.image.digest, or explicitly clear it when selecting by tag; a nonempty digest takes priority.

Storage configuration

For storage, empty storageClassName inherits the cluster default, while "-" explicitly selects no StorageClass. Local PVs require prepared host directories, matching capacities, nodeAffinity and a kubernetes.io/no-provisioner StorageClass; WaitForFirstConsumer binds with the Pod’s scheduling constraints. See the complete seven-volume example before creating resources. For a new temporary installation, set persistence.enabled=false to use emptyDir; component storage flags override it. The chart never replaces a failed PVC with temporary storage and rejects PD/Store storage-mode changes on initialized releases.

Vermeer uses separate Master and Worker claims, even in its single-Pod mode: two additional volumes in standalone, or one Master volume plus one per Worker in cluster mode. With the default base topology this means 9 or 11 volumes respectively. The small HugeGraph preset plus standalone Vermeer needs 5. The seven-volume local example covers only the base installation. SeaTunnel checkpoints remain temporary local files.

Offline installation and internal repositories

Select the latest version, download that Chart and move the .tgz into your network. Install it directly or upload it to an existing OCI or HTTP repository. In air-gapped environments, override the image registry with --set global.imageRegistry=..., see values-internal-registry.yaml.

1CHART=oci://ghcr.io/hugegraph/hugegraph-helm
2CHART_VERSION=$(helm show chart "$CHART" --devel | sed -n 's/^version: //p')
3helm pull "$CHART" --version "$CHART_VERSION"
Install archive
1# Override the image registry in air-gapped clusters (or use -f values-internal-registry.yaml)
2helm install hugegraph "./hugegraph-helm-${CHART_VERSION}.tgz" -n hugegraph \
3  --set global.imageRegistry=registry.example.com \
4  --create-namespace --wait --timeout 15m
Internal OCI
1helm registry login registry.example.com
2helm push "./hugegraph-helm-${CHART_VERSION}.tgz" oci://registry.example.com/charts
3helm install hugegraph oci://registry.example.com/charts/hugegraph-helm \
4  --version "$CHART_VERSION" -n hugegraph \
5  --set global.imageRegistry=registry.example.com \
6  --create-namespace --wait --timeout 15m
Internal HTTP
1# Upload the same .tgz using your repository's upload mechanism and update index.yaml.
2# For a static HTTP repository, use helm repo index. Native helm push only supports OCI.
3helm repo add hugegraph-internal https://charts.example.com
4helm repo update hugegraph-internal
5helm install hugegraph hugegraph-internal/hugegraph-helm \
6  --version "$CHART_VERSION" -n hugegraph \
7  --set global.imageRegistry=registry.example.com \
8  --create-namespace --wait --timeout 15m

Replace the example registry addresses with your own. The archive contains no container images: mirror all configured container images separately, then override their addresses.

4 Authentication and secrets

Normal Helm installations generate and reuse three credentials: <release>-admin (password), <release>-auth-token (token_secret) and <release>-pd-auth (secret-key). PD REST requires the shared PD secret, supplied to the PD container as HG_PD_AUTH_SECRET_KEY; Server’s storage wait and Hubble read the same credential. It is not a separate user password that must be invented for each Pod.

For GitOps, use fixed encrypted values or pre-created Secrets through pd.auth.existingSecret, server.auth.admin.existingSecret and server.auth.token.existingSecret. existingSecret wins over inline values and is not managed by the chart. The admin password must be printable ASCII without spaces, colons or backslashes; the PD secret must be printable ASCII without surrounding whitespace or backslashes; the JWT signing key must be at least 32 bytes. Offline renderers cannot reuse live generated Secrets through lookup.

The admin Secret is an initial credential. Updating it does not rotate an existing account. Change the password through PUT /graphspaces/DEFAULT/auth/users/admin with {"user_password":"<new-password>"} using the existing credential, update the Secret/source to that same password, then restart the Server Deployment to clear every replica’s auth cache.

Hubble is enabled by default and its current login requires Server authentication. Set hubble.enabled=false for API-only access; optionally also set server.auth.enabled=false for an unauthenticated API. Inactive credential settings may remain in your values file.

Enabled optional engines always require native login, including in easy mode. Newly generated administrator passwords use 6 random alphanumeric characters in easy mode or 16 in strict mode. SeaTunnel uses HTTP Basic, while Vermeer uses Secret <release>-vermeer-admin and its Admin credential login. Upgrades retain existing passwords, including when changing mode; mode changes do not rotate them automatically. Vermeer password sources are vermeer.auth.admin.password or vermeer.auth.admin.existingSecret/key; only Master reads that password. Access native administration according to the final Service type: NodePort connects via the node IP and assigned port, while ClusterIP uses port-forwarding. Hubble service credentials are automatic.

Advanced: Vermeer deployment credentials and GitOps

For GitOps, preserve stable administrator credentials and the separate internal service identity. The service signing factor uses vermeer.auth.value or vermeer.auth.existingSecret/key; Hubble generates its service token automatically. Ordinary Hubble use and native password login do not require retrieving that token. Changing an external native administrator Secret needs a Master restart only. Service-factor rotation requires Master and Hubble, not Workers. Worker Schema reads reuse the deployment administrator unless vermeer.hugegraphCredentials.existingSecret/key supplies protected username/password JSON for an existing account with the required Schema permissions. No account is created automatically.

5 Upgrade and verification

Use native Helm value reuse to preserve an existing deployment’s UI, entry, storage and credentials:

1helm get values hugegraph -n hugegraph --all
2# Follow the latest version for testing; use --version with a reviewed stable version for production.
3helm upgrade hugegraph "$CHART" --devel -n hugegraph \
4  --reuse-values --wait --timeout 15m
5helm test hugegraph -n hugegraph --logs --timeout 5m

Existing explicit helper images also remain under --reuse-values; add --set-string store.waitImage= --set-string server.waitImage= to reuse component images. Adopt new defaults deliberately with --reset-values and a reviewed file retaining your image, credential, storage and topology settings. Do not treat PD membership, storage or credential changes as ordinary independent overrides; consult the lifecycle reference.

PD, Store and Server images built from current master run as UID/GID 10001. The chart gives the PD and Store pods fsGroup: 10001 (with fsGroupChangePolicy: OnRootMismatch) unless their podSecurityContext sets fsGroup, also when --reuse-values keeps an older release’s empty value, so they can write PVCs created by older images, which ran as root.

Switch PD and Store to images built from this version in the same rollout that first applies fsGroup, so the kubelet changes the volume ownership right before the 10001 image starts. Do not run images that still run as root with this chart first, for example a latest that has not moved to this version: files they create stay owned by root, and OnRootMismatch checks only the volume root, so it does not fix them later. Pin pd.image and store.image to a build from this version rather than relying on latest. A non-empty image.digest wins over image.tag, and --reuse-values keeps an existing digest, so a release pinned by digest must change the digest, not only the tag.

If a release already ran root images with fsGroup set, roll out the first 10001 image together with a one-time recursive ownership change. Set the images in the same command, so the recursive change runs for the 10001 image rather than for a pinned root image. The example clears any reused digest so the tag applies; a release pinned by digest sets pd.image.digest and store.image.digest to the digests of this version’s images instead:

1helm upgrade hugegraph "$CHART" --devel -n hugegraph --reuse-values \
2  --set pd.image.tag=<this-version> \
3  --set store.image.tag=<this-version> \
4  --set-string pd.image.digest= \
5  --set-string store.image.digest= \
6  --set pd.podSecurityContext.fsGroupChangePolicy=Always \
7  --set store.podSecurityContext.fsGroupChangePolicy=Always \
8  --wait --timeout 15m

Before switching the policy back, check that every PD and Store Pod runs the 10001 image you set, not an older root image:

1kubectl get pods -n hugegraph \
2  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'

--reuse-values keeps Always, which walks the whole volume on every Pod start, so set the policy back once those Pods are running:

1helm upgrade hugegraph "$CHART" --devel -n hugegraph --reuse-values \
2  --set pd.podSecurityContext.fsGroupChangePolicy=OnRootMismatch \
3  --set store.podSecurityContext.fsGroupChangePolicy=OnRootMismatch \
4  --wait --timeout 15m

Volumes whose CSI driver ignores fsGroup (for example many NFS drivers) and hostPath volumes need a one-time chown -R 10001:10001 instead.

server.storageWaitTimeoutSeconds defaults to 300 and is passed as WAIT_STORAGE_TIMEOUT_S. An unspecified startup probe threshold is derived from that wait plus 150 seconds; an explicit threshold is preserved and must cover the storage wait plus at least 120 seconds of Server startup. A contradictory budget fails rendering with the relevant parameter names. Storage wait timeout is not a JVM memory setting.

The default helm test hook checks connectivity and reads. After installation or recovery, also test a new UI login and your application’s read/write queries. For scheduling, NetworkPolicy, rolling updates and recovery, see Kubernetes operations.

6 Deploy and Use SeaTunnel (Optional Engine)

SeaTunnel is an optional data integration engine supported by HugeGraph, disabled by default. When enabled, Helm provisions a persistent SeaTunnel Zeta engine alongside a task Web dashboard within the same release. The official apache/seatunnel:3.0.0 image comes with built-in HugeGraph connectors (Source and Sink), requiring no custom image builds or plugin installation.

 1helm/hugegraph/
 2  examples/
 3    values-seatunnel-standalone.yaml # Standalone preset (1 Hybrid node, lightweight for local/dev use)
 4    values-seatunnel-cluster.yaml    # Cluster preset (multiple Hybrid nodes for distributed parallelism)
 5    seatunnel/
 6      hugegraph-sink.conf            # Import sample: batch inserts test vertices into HugeGraph
 7      hugegraph-source.conf          # Export sample: reads graph data from HugeGraph to stdout/logs
 8      clone/                         # Full graph migration configs
 9        copy-schema.py               # Automated Schema synchronization script
10        person.conf / knows.conf     # Per-label vertex and edge migration job definitions
flowchart TB
    A["External Data Source / Remote Graph"] --> ST
    subgraph Release["Same Helm Release"]
        ST["SeaTunnel Zeta<br/>Hybrid Node"] -->|Read / Write| HG["HugeGraph<br/>Server + PD + Store"]
    end
    U["Operators"] -->|Web UI / REST (HTTP Basic)| ST

6.1 Choose Standalone or Cluster Mode

SeaTunnel uses a Hybrid node architecture where each node handles both job scheduling and execution:

  • Standalone Mode: Starts 1 persistent node, minimizing memory footprint for development and lightweight jobs.
  • Cluster Mode: Starts multiple uniform nodes forming a cluster for distributed high availability and higher throughput.

SeaTunnel scaling is fully decoupled from the HugeGraph cluster topology. Install from the extracted chart directory or repository root by combining presets:

SeaTunnel Standalone
1helm install hugegraph ./hugegraph-helm -n hugegraph --create-namespace \
2  -f ./hugegraph-helm/values-single.yaml \
3  -f ./hugegraph-helm/examples/values-seatunnel-standalone.yaml \
4  --wait --timeout 30m
SeaTunnel Cluster
1helm install hugegraph ./hugegraph-helm -n hugegraph --create-namespace \
2  -f ./hugegraph-helm/values-single.yaml \
3  -f ./hugegraph-helm/examples/values-seatunnel-cluster.yaml \
4  --wait --timeout 30m
Tip

To add SeaTunnel to an existing HugeGraph deployment, simply set seatunnel.enabled: true with the chosen topology values and run helm upgrade. Existing graph services remain uninterrupted. Because the SeaTunnel image is large, mirror it to your private registry beforehand in air-gapped environments.

6.2 Access Web UI and Authentication

SeaTunnel inherits the global mode setting when seatunnel.service.type: null: easy defaults to NodePort, strict to ClusterIP. An explicit ClusterIP stays private; explicit NodePort, including in strict mode, exposes HTTP/UI with the same required login. Clear an older explicit ClusterIP override to null to inherit easy mode. Omit seatunnel.service.nodePort or use null for automatic allocation. The standalone / cluster topology does not change this access choice:

NodePort (easy mode)
1kubectl -n hugegraph get svc hugegraph-seatunnel \
2  -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}{"\n"}'
3kubectl get nodes -o wide
4# Open browser: http://<NodeIP>:<NodePort>/#/overview
Port-forward (General)
1kubectl -n hugegraph port-forward svc/hugegraph-seatunnel 8081:8080
2# Open locally: http://127.0.0.1:8081/#/overview

The Web dashboard and REST APIs require independent HTTP Basic authentication in both modes, distinct from HugeGraph Server credentials. The username is admin; newly generated passwords use 6 random alphanumeric characters in easy mode or 16 in strict mode. Upgrades and mode changes retain the existing password. Read it from the release Secret:

1kubectl -n hugegraph get secret hugegraph-seatunnel-auth \
2  -o jsonpath='{.data.password}' | base64 --decode
Warning

Port 5801 is reserved for internal Zeta communication and CLI commands; it does not use HTTP Basic auth and remains private. External access uses only the authenticated HTTP Service through its assigned NodePort; the application still listens on 8080.

6.3 Run Pre-configured Example Jobs

The Chart mounts sample jobs inside the SeaTunnel Pod, pre-configured with environment variables referencing the in-release Server address, graph name, and credentials:

  1. Prepare Graph: Ensure an empty target graph exists via Hubble or REST API (default DEFAULT/graph_a).

  2. Run Import Job: Ingest 10 sample vertices under the seatunnel_demo_person label:

    1kubectl -n hugegraph exec sts/hugegraph-seatunnel -- \
    2  /opt/seatunnel/bin/seatunnel.sh --master cluster \
    3  --config /opt/seatunnel/config/hugegraph-sink.conf
  3. Verify with Export Job: Run the Source sample job to read back the same vertices and print them to standard logs:

    1kubectl -n hugegraph exec sts/hugegraph-seatunnel -- \
    2  /opt/seatunnel/bin/seatunnel.sh --master cluster \
    3  --config /opt/seatunnel/config/hugegraph-source.conf

Inspect job status and DAG progress in real time on the Web dashboard. To use custom graphs or credentials, override environment variables under seatunnel.env or mount external configurations via seatunnel.config.

6.4 Full Migration and Topology Switches

  • Topology Switch: Switch between standalone and cluster modes by updating values and running helm upgrade. Since the engine process restarts, complete all running batch jobs before switching.
  • Full Graph Migration: Follow the sequence Schema metadata → Vertices → Edges. See examples/seatunnel/clone/ for automated migration scripts and configs. When migrating fields with DATE types, explicitly specify matching server timezones (e.g. timeZone = "UTC") in the Sink config to prevent timezone shift. See the SeaTunnel Source migration guide for details.

1.5 - Operate HugeGraph on Kubernetes

This guide accompanies Helm deployment. Detailed lifecycle checks and recovery notes are maintained in the operations reference; all parameters are commented in values.yaml. Commands assume release/namespace hugegraph; substitute the actual workload names if overridden.

1 Storage and private images

The base installation needs seven persistent volumes. Dynamic storage requires a working provisioner; static local storage requires prepared directories and matching PVs on the named nodes. Use the complete examples for all seven local PVs, WaitForFirstConsumer, registry mirrors and GitOps Secrets. An initially Pending PVC with WaitForFirstConsumer can be normal; persistent Pending needs PVC/Pod events and PV matching checks. A bound claim with a missing directory or permission mismatch can fail mounting. Do not replace failed PVCs with temporary data directories.

1kubectl -n hugegraph get pvc,pods
2kubectl -n hugegraph describe pvc <claim-name>
3kubectl -n hugegraph describe pod <pod-name>
4kubectl get pv,storageclass

For ImagePullBackOff, inspect Pod events and registry access/credentials. The default image inventory contains four component images; Store wait and Helm test reuse existing component images with curl already installed. Explicit helper-image overrides must also be mirrored. Enabled SeaTunnel adds its configured apache/seatunnel image. Enabled Vermeer uses hugegraph/vermeer:helm-30b67d93e7d0, pinned by digest and reused by Master, Worker and token initialization. Official publishing runs record the source and image digest; see the Vermeer deployment guide. Custom image overrides must also update vermeer.image.digest, or explicitly clear it to select by tag; digest takes priority. The base local-PV example omits Vermeer claims: standalone adds two, while cluster adds one Master claim plus one per Worker (four additional claims by default).

2 Ports and health

PD listens on REST 8620, gRPC 8686 and Raft 8610; Store on REST 8520, gRPC 8500 and Raft 8510; Server HTTP on 8080; Hubble HTTP on 8088. Client Services hide individual Pod addresses. NodePort is a separate allocated port on reachable nodes, not the application listener port. PD /v1/ready requires a Raft leader; /v1/health only confirms the listener. Store startup waits for a majority of ready PD peers. Server waits up to server.storageWaitTimeoutSeconds (default 300) before its application startup; adjust it for slow storage bootstrap. No local Helm rendering verifies live cluster reachability, volumes or JVM capacity.

Preserve the metadata namespace

metadataCluster: hg-test sets the PD metadata namespace prefix (injected into Server HG_SERVER_CLUSTER and Store PD_CLUSTER). It is not a Raft group name; changing it will not automatically migrate existing metadata. Never arbitrarily change this prefix on an initialized cluster to enable Vermeer. If Server REST queries succeed but Vermeer scanning Store fails with Schema missing or no such vertex label, check that the namespace prefix matches across all components.

If a legacy environment used a custom prefix, record it explicitly on the Server Deployment before upgrading:

1kubectl -n hugegraph annotate deployment hugegraph-server \
2  hugegraph.apache.org/metadata-cluster='<verified-existing-namespace>'

and ensure metadataCluster matches this value in your values file. For Store nodes using store.updateStrategy.type=OnDelete, follow the rolling upgrade procedure to recreate Pods one by one so the new settings take effect.

Optional Engines Operations: Probes, Credentials, and Topology Switching
  • Ports & Health: SeaTunnel HTTP/UI runs on 8080; Vermeer Master HTTP/gRPC listens on 6688/6689, and Worker on 6788/6789. Native HTTP/UI defaults to NodePort in easy mode and ClusterIP in strict mode; both require login. Internal RPC, Worker HTTP and SeaTunnel 5801 stay private. With service.type: null, the mode default applies; explicit ClusterIP stays private, while explicit NodePort exposes authenticated HTTP/UI. Use the assigned HTTP NodePort from Service hugegraph-seatunnel or hugegraph-vermeer-master with a reachable node IP. See native access commands; ClusterIP access can use port-forwarding.
  • Credentials & Sessions: Generated admin passwords use 6 characters in easy mode or 16 in strict mode, retained across upgrades and mode changes. In Vermeer native UI, enter the password from Secret <release>-vermeer-admin into Admin credential; Hubble internal service tokens are auto-wired.
  • Topology Changes: Ensure active computations and batch ingestion jobs are complete and in-memory graphs are unloaded before changing seatunnel.mode or vermeer.mode. Standalone and cluster use separate PVC claims, so mode changes do not migrate local files.
  • See the Vermeer Guide and SeaTunnel Guide for full recovery and configuration details.

Helm test coverage

helm test validates graph binding on every Server Pod (DEFAULT-hugegraph by default, or configured via server.testGraphAlias), and requires currently visible associated indexes to be in CREATED state. Run it after index creation/rebuild jobs finish, with no schema changes during the test. On HStore, reading an unfinished index can cache its temporary status on other Pods; retries do not refresh that state. Indexed queries on those Pods may keep failing after the job succeeds; replace affected Server Pods only after the jobs finish. The hook does not fix cross-Server schema cache consistency. Its 150-second internal retry window allows transient elections to settle; Helm’s --timeout does not change that window. It reads indexes by ID without refreshing the full index cache; it does not prove index query results or complete schema consistency. For existing data, use --set-file server.testGremlin=assertions.groovy during Helm upgrade to add read-only business assertions on every Pod. The script must use assert or throw on unexpected results; returning false alone does not fail the request. See business query assertions for an example and coverage limits. Without a script, business assertions are reported as skipped. The default hook creates no schema or data. Custom scripts use the hook’s Server credential (admin with auth enabled); read-only execution is not enforced, so supply trusted, read-only assertions.

Server API responses negotiate compression through Accept-Encoding. Gzip-capable clients can request gzip or use curl --compressed; identity or no compression request keeps the response uncompressed. Build a new Server image from the updated source to include this behavior; updating the chart alone does not update an existing image.

PD REST authentication

PD REST uses HTTP Basic auth: hg, store, hubble and vermeer share the PD REST Secret password. Server’s admin account is not a PD REST account and receives 401. For the default release/Secret and a reachable PD REST address:

1PD_SECRET=$(kubectl -n hugegraph get secret hugegraph-pd-auth \
2  -o jsonpath='{.data.secret-key}' | base64 --decode)
3curl --user "hg:${PD_SECRET}" "http://<pd-rest-address>:8620/v1/stores"

The HTTP example assumes a trusted internal network. For other access paths, use an HTTPS gateway or optionally forward the PD Service: kubectl -n hugegraph port-forward service/hugegraph-pd-client 8620:8620, then query http://127.0.0.1:8620.

With pd.auth.existingSecret or a custom pd.auth.key, read the actual Secret and key. Do not use the Server admin password. Variables avoid literal passwords in shell history, but expanded credentials can briefly appear in process arguments. On shared terminals, use curl --user hg to prompt for the password or a private curl config with permissions 600; the credentials guide also covers password-update payloads. Keep credential files out of logs and version control.

3 Scheduling

The default preferred anti-affinity permits scheduling with fewer nodes than replicas. values-cluster.yaml sets required PD/Store anti-affinity, so it needs at least three schedulable nodes. Explicit affinity takes priority over presets. Local PV node affinity must match the workload’s placement constraints; PV capacity declarations do not enforce filesystem quotas. Do not change a persisted PD/Store Raft hostname or port as an ordinary upgrade: those addresses are recorded identities.

4 Partition sharding

At first bootstrap, the chart seeds three shard replicas when there are at least three Stores, and one otherwise. PD persists the partition settings; changing chart values after initialization does not rewrite that metadata. Use the PD configuration API and a coordinated partition patrol for an existing cluster. An explicit shard count must be odd and no greater than Store replicas. See the topology constraints before changing a live cluster; the chart does not migrate existing shard metadata.

5 NetworkPolicy

Easy mode and the ordinary cluster preset leave full base-service isolation disabled by default; strict enables it. SeaTunnel and Vermeer always require login. Their native policies allow external HTTP only when the final Service type is NodePort, while internal RPC, Worker HTTP and SeaTunnel member ports retain their release-caller restrictions, even with networkPolicy.enabled=false. The cluster network plugin must enforce NetworkPolicy for these resources to have an effect. The generated policies admit component traffic and DNS. For base services, external clients and Ingress controllers need suitable networkPolicy.<component>.extraIngress rules; external discovery addresses may need matching extraEgress. For base-service rules, an empty list can block outside traffic and native Kubernetes rules include {} to allow all; there is no separate source-acknowledgement gate. Use your actual source addresses/selectors, considering NodePort SNAT and controller behavior.

 1networkPolicy:
 2  enabled: true
 3  server:
 4    extraIngress:
 5      - from:
 6          - ipBlock:
 7              cidr: 10.20.0.0/16
 8        ports:
 9          - protocol: TCP
10            port: 8080
11  hubble:
12    extraIngress:
13      - from:
14          - ipBlock:
15              cidr: 10.20.0.0/16
16        ports:
17          - protocol: TCP
18            port: 8088

Native-engine extraIngress accepts explicit internal Pod/namespace selectors only; IP blocks and unrestricted rules are rejected. NodePort HTTP access is admitted by the generated policy without an extraIngress rule. The base-service example above applies to Server. External data destinations can use engine extraEgress; strict mode additionally enforces egress isolation.

6 Rolling Store images safely

Use store.updateStrategy.type=OnDelete for a controlled maintenance rollout, as the cluster preset does. Delete one Store Pod at a time. Before replacing the next, verify the previous Pod is Ready, has a fresh Store heartbeat and each shard group has its full membership and exactly one leader, read from the PD leader. PD’s Up status alone does not prove completed shard recovery. PDBs limit voluntary eviction, not StatefulSet controller disruption. Stop on missing members or leaders. The operations reference summarizes these rollout conditions; it does not provide a complete per-group diagnostic or disk-loss recovery procedure.

A single Store replacement can cause request errors or timeouts while Raft leadership and client routes converge, even with full replicas. There is no zero-error guarantee. Observe application errors as well as shard recovery before continuing; do not blindly retry writes.

PD patrol and leader balance

Partition patrol repairs shard membership; leader balance is a separate manual task and is not scheduled automatically. Before either task, identify the PD leader through authenticated /v1/members (data.pdLeader.restUrl) and confirm its /v1/ready reports isLeader: true. The GET tasks /v1/task/patrolPartitions and /v1/task/balanceLeaders change cluster state; target that confirmed leader with PD REST credentials. Wait for membership recovery and split/move tasks to finish before balancing leaders; Stores must have fresh heartbeats and groups full, healthy membership. A busy task returns body status: 1001. A follower may return empty success, and leader transfers are asynchronous: HTTP 200 or a transfer plan does not prove completion. Recheck /v1/partitions, /v1/stores and application errors before further maintenance. Follow the task procedure for the separate patrol and balance steps; do not loop task requests while maintenance is busy.

7 Disaster recovery

Preserve existing volumes, claim identities and PD membership. Do not delete RocksDB lock files to recover a deployment, or replace persisted membership with freshly generated nodes. A new empty PVC does not restore the data from a lost Store disk. Local disks cannot migrate their files automatically when a node fails. Take backups outside these PVs and test recovery separately. The chart does not implement backup or automatic recovery. The recovery notes describe constraints, not a complete recovery sequence. Before destructive changes, establish and validate a recovery plan for the deployed version and remaining replicas or backups.

8 Scaling

Server replicas are routinely mutable; an HPA also needs CPU requests. Store growth is a coordinated storage/shard operation. PD growth/shrink and Store shrink cannot be handled as ordinary replica changes, because persistent Raft/shard membership must be updated first. Chart upgrade checks reject unsupported identity/membership changes on initialized StatefulSets; fresh installs do not query StatefulSets for upgrade checks. Scale through the documented lifecycle procedure rather than kubectl scale, which can leave Hubble’s Store metrics configuration stale.

9 Running Hubble outside the cluster

Direct mode only needs an externally reachable Server URL. PD discovery also needs externally reachable PD gRPC, and Server addresses registered with PD must be reachable from that external Hubble. Set server.advertiseUrl to a shared reachable Server URL when needed; it affects all discovery clients, including in-cluster Hubble. In-chart Hubble already uses internal service discovery and needs no public PD entry point. HTTP is supported directly; TLS termination remains optional. Hubble remains single-replica and its current login requires Server authentication.

10 When Gremlin fails with Could not rebind

A Server can become Ready and serve REST while Gremlin binding failed during a PD rollout. Run helm test hugegraph -n hugegraph --logs --timeout 5m; the hook checks authenticated graph-bound Gremlin on every Server Pod. Once PD is stable, replace the failing Server Pod and verify again. A test that only calls /versions does not prove graph binding; also check fresh Hubble login and a graph query.

For ERROR: missing required env 'HG_PD_AUTH_SECRET_KEY', inspect the PD Pod’s Secret reference and the corresponding key, not the Secret value in logs. The chart normally injects this shared PD REST credential automatically. Custom containers/manifests must provide the same value to PD and its clients. In GitOps, create the external Secret before workload reconciliation; with ordinary Helm, retain the chart-generated release Secret.

2 - HugeGraph ToolChain

HugeGraph Toolchain includes the Java and Go clients, Loader, Hubble, Tools, Spark Connector, and SeaTunnel Sink/Source. Choose an entry by the task you need to complete, then open the component guide for its configuration and commands.

TaskStart hereBest for
Visualize graphsHubbleViewing and managing graphs in a Web UI
Import graph dataLoader, SeaTunnel Sink, Spark ConnectorImporting data directly or connecting an existing pipeline
Export or migrate graph dataTools, SeaTunnel SourceBackup, export, cross-graph migration, and continuous reads

Testing Guide: For running toolchain tests locally, please refer to HugeGraph Toolchain Local Testing Guide

DeepWiki provides real-time updated project documentation with more comprehensive and accurate content, suitable for quickly understanding the latest project information.

📖 https://deepwiki.com/apache/hugegraph-toolchain

Source repository: apache/hugegraph-toolchain

2.1 - Graph visualization

Hubble provides a Web interface for HugeGraph data, schema, queries, and imports. Start with a standalone deployment without PD, then read the distributed differences when needed.

2.1.1 - Manage an HStore Cluster with Hubble

Connect HStore and PD to Hubble and understand GraphSpaces, Schema templates, cluster topology, and node metrics.

This guide highlights the key differences between HStore + PD distributed deployments and standalone RocksDB. For general graph modeling, data import, and query workflows, see the Hubble standalone guide. Content aligns with the Toolchain, Server, PD, and Store master branches (all currently 1.8.0).

The main repository’s docker/docker-compose-hstore.yml provides complete orchestration for PD, Store, Server, and Hubble. Follow the adjacent Docker README to prepare .env and generate the local Hubble configuration, then launch services from docker/ without maintaining separate Compose files. The guidance below focuses on connectivity and operations; ensure prerequisite credentials and service readiness from the README are met. Docker latest is a moving tag. Screenshots assist in UI navigation; verify deployed versions via image digests and build details. Metrics and permission APIs vary by version.

The screenshots show the HugeGraph Hubble interface in action; features and workflows are identical in both English and Chinese.

Connect to a distributed cluster

Hubble still manages graph data through the Server graph API. In distributed mode, it discovers Servers through PD and collects cluster information from PD and Store. Hubble does not read or write graph data directly in Store.

flowchart LR
    H[Hubble] -->|Graph API| S[Server]
    S -->|Graph reads and writes| T[Store]
    H -.->|Discovery and cluster information| P[PD]
    H -.->|Node metrics| T
  1. Generate Hubble configuration. Prepare .env in the main repository’s docker/ directory following the Docker README.

    1set -a
    2. ./.env
    3set +a
    4./set-hubble-pd-password.sh hstore

    The script reads HG_PD_AUTH_SECRET_KEY from .env and writes its value as operations.pd.password into the generated file docker/conf/hubble/hstore.local.properties. Edit this generated file to adjust connection or operations settings (do not overwrite the tracked .example template), ensuring operations.pd.password matches the PD secret. Compose mounts this file read-only into the container; do not edit inside the container. Both .env and the generated file contain secrets and should not be committed to Git. Re-running the script overwrites the generated file, requiring custom settings to be reapplied.

  2. Start the services. Use the same docker/ directory and confirm Server registration and Store readiness.

    1docker compose -f docker-compose-hstore.yml up -d --wait

For separately deployed services, follow the PD deployment guide and HStore deployment guide. A source or binary Hubble deployment instead uses the package’s conf/hugegraph-hubble.properties. These settings match the official minimal topology; use backend-reachable addresses for other deployments:

1pd.enabled=true
2cluster=hg
3pd.peers=pd:8686
4pd.server=pd:8620
SettingPurposeBundled value
pd.enabledExplicitly enable PD mode; server.direct_url is not used in this mode.false
clusterCluster name used for Server discovery; it must match the registration.hg
pd.peersPD gRPC addresses, separated by commas.127.0.0.1:8686
pd.serverPD REST address for cluster operations, not a list of gRPC peers.127.0.0.1:8620

Distinguish between the two ports: 8686 is for gRPC and 8620 is for REST. Do not interchange them, and use container service names or reachable IPs rather than 127.0.0.1 for cross-container traffic. The bundled package sets pd.enabled=false. If omitted, the fallback default is true, so declare this setting explicitly in both standalone and distributed setups. After editing configuration: restart the Hubble process for source or binary deployments; for Docker Compose, recreate the Hubble container from docker/ to reload the mounted configuration (retaining the project name and all -f arguments):

1docker compose -f docker-compose-hstore.yml up -d --force-recreate hubble

In the Hubble UI, you do not need to configure a Server host and port for each graph.

Organize graphs and permissions with GraphSpaces

A GraphSpace groups graphs, Schema templates, and access permissions. For example, create sales and research spaces so each team can work with its own graphs. Modeling, importing, and querying within a space follow the standalone guide.

Create and adjust a GraphSpace

With Server authentication enabled, creating, editing, and deleting GraphSpaces require super administrator access. Start with a globally unique GraphSpace name, an optional display alias, and the maximum graph count. For example, use research as the API identifier and “Research_graph” as the display alias. The name cannot change after creation. The alias and description can change; aliases do not participate in URLs or permission matching. Choose GraphSpace administrators from existing accounts.

“Advanced deployment and resource limits” includes CPU and memory limits for graph query/write services and asynchronous compute tasks, plus the storage capacity limit. These deployment quotas neither report current usage nor resize existing containers.

Advanced deployment defaults

Defaults are 100 graphs, 64 CPU cores and 128 GB of memory for each of the graph and compute services, and 1000000 GB of storage. Keep the defaults for a container trial. Configure Kubernetes namespaces, Operator images, and algorithm images only for the corresponding deployment or compute use.

Select a space before opening a graph. Check the current graph after switching spaces, especially when spaces contain identically named graphs. The list reflects account access. If a space is missing, check membership permissions before creating more graphs.

GraphSpace creation form and entry to advanced resource limits

Reuse Schema templates

User-defined Schema templates require PD mode and persist reusable Groovy Schema within a GraphSpace. When several business graphs share vertex labels, edge labels, and indexes, save the model as a template and select it when creating subsequent graphs. For example, a user template in research can be reused for new graphs in that space. After switching spaces, select a template belonging to the new space.

User templates differ from the built-in sample templates in the main guide: built-in templates help explore preset models, while user templates preserve your own models for future use. A template is not an import of data; loading sample data is a separate graph creation choice. With Server authentication enabled, creating a user template requires write access to its space. Updating or deleting it also requires ownership or the corresponding administrative permission. Anonymous mode does not enforce per-user permissions or template ownership checks. Standalone mode does not provide user template management.

Assign access to a GraphSpace

For account creation, login, and personal details, see the standalone guide. Distributed mode adds “Manage GraphSpace members” to assign permissions between existing accounts and selected spaces. Servers supporting permission presets provide these common choices:

PresetIntended userScope
GraphSpace read-onlyUsers who inspect and query graph dataSelected spaces.
GraphSpace read-writeUsers who model, import, and maintain graph dataSelected spaces.
GraphSpace administratorOwners managing a space’s members and graph resourcesSelected spaces; does not grant GraphSpace creation/editing or cluster operations access.
Super administratorOperators managing accounts, GraphSpaces, and the clusterGlobal; assign according to responsibilities.

An account may have different access in different spaces, such as read-write in research and read-only in sales. Accounts and space memberships are managed separately; removing a member does not delete the global account. After creating an account, continue directly to assigning space access. When changing an existing membership, check the selected space and preset and preserve permissions still needed in other spaces. Enabling pd.enabled grants no permissions; legacy custom roles on older Servers are not interchangeable with these presets. See Server authentication and authorization.

GraphSpace members and access presets

Locate problems through the cluster overview

The overview presents Server → PD → Store in one topology. Switch to the node list to filter by type, status, or name. PD Leader, online Store count, graphs, partitions, replicas, and data size help identify cluster scale and nodes requiring attention. Partitions and replicas describe data distribution. Data size is observed usage, distinct from a GraphSpace’s configured storage limit.

Start with cluster and source status, then inspect an affected node. UP indicates a successful collection from the source; DEGRADED indicates a cluster or partial source issue, and DOWN indicates that the corresponding probe failed. Graphs may remain queryable when topology is available but some metrics are missing. The UI distinguishes unsupported, unavailable, stale, and failed collections and includes observation times. An empty value is not zero, and an older value is not a current observation.

Read node details

NodeMain observationsUse
ServerAvailability, JVM/CPU/memory, and backend metricsCheck the query/write entrypoint and resource pressure.
PDLeader/Follower role, status, and runtime metrics supplied upstreamCheck the metadata/scheduling entrypoint; do not infer a role when Leader information is absent.
StorePartition and Leader partition counts, system/disk metrics, Raft group and enabled group countsInspect storage nodes and replica service and compare distribution across nodes.

The PD Leader and Store Leader partitions serve different purposes: PD coordinates cluster metadata, while a Store leads the respective data partition’s Raft group. Hubble displays upstream observations and does not replace a full Raft replica consistency check. Available metrics vary by component version.

With Server authentication enabled, cluster operations require a super administrator (ADMIN level). GraphSpace administrators and ordinary members do not inherit cluster access. Use container isolation, a trusted HTTPS entrypoint, Server authentication, and network allowlists; avoid publishing Hubble or component ports directly.

Configure operations access

Hubble’s backend uses the PD/Store operations credentials, separately from the Server account used to log in through the browser. For Compose, edit the generated host file docker/conf/hubble/hstore.local.properties; other deployments use the Hubble package configuration. The script-generated PD password must match the secret in .env. Configure the Store service account when Store authentication is enabled. Do not include passwords in documentation, screenshots, or committed configuration:

SettingBundled defaultConfiguration
operations.pd.username / operations.pd.passwordUsername hubble, empty passwordMatch PD operations REST authentication.
operations.store.username / operations.store.passwordUsername hubble, empty passwordSet the service account when upstream Store REST authentication is enabled.
operations.store.allowed_targets[http://127.0.0.1:8520,http://[::1]:8520]List trusted Store metric origins.

For example, when a containerized Store advertises store:8520, set:

1operations.store.allowed_targets=[http://store:8520]

For multi-node clusters, list every trusted target origin. Each entry must use http or https with an explicit port and no path, credentials, or wildcard, matching the Store metric target returned by PD. Adding an origin here only authorizes access—it does not register or discover nodes.

If topology renders but node metrics are missing, check Hubble backend connectivity and credentials to PD REST, verify the Store REST and metric addresses returned by PD, and ensure targets are in operations.store.allowed_targets. A partially available overview indicates metrics collection was blocked for specific sources, not that graph APIs are down.

2.2 - Graph Visualization with Hubble: Standalone Quick Start

Use Hubble and RocksDB Server to manage schema, import CSV, and query graph data with Gremlin, Cypher, or Text2GQL.

Hubble is the HugeGraph Web management and graph visualization interface. Use a single workspace to manage schema, import data, run queries, and switch smoothly between graph, table, and JSON results. This guide uses standalone RocksDB Server + Hubble, without requiring PD or Store.

For use with an HStore distributed cluster, read the foundational operations here first, then consult the distributed supplement. This guide aligns with the Toolchain and Server master branches (currently based on 1.8, unreleased). Docker latest is a moving tag; use released versions for production deployments.

The screenshots show the HugeGraph Hubble interface in action; features and workflows are identical in both English and Chinese.

Warning

Do not expose unauthenticated Hubble or Server instances directly to the public internet. In production, use HTTPS, container isolation, authentication and authorization, and network allowlists.

Jump directly to a task below. If this is your first visit, start with the standalone deployment:

Start the standalone pair

Use the main repository’s docker/docker-compose.yml instead of writing custom Compose files. It already integrates RocksDB Server and Hubble with networking, health checks, and data volumes; see the adjacent README for deployment details.

  1. Prepare the official Compose configuration.

    1git clone --branch master --single-branch --depth 1 https://github.com/apache/hugegraph.git
    2cd hugegraph/docker

    If you already have the repository locally, navigate directly to its docker/ directory. Compose automatically mounts conf/hubble/standalone.properties, which pre-configures pd.enabled=false and server.direct_url=http://server:8080. Because both services communicate over a shared Docker network, you do not need to configure Server endpoints per graph. Start Compose directly from docker/ so relative file paths resolve correctly.

    Hubble publishes on host loopback port 8088 by default (controlled by HUBBLE_PUBLISH_HOST, defaulting to 127.0.0.1), whereas Server publishes 8080 across all interfaces. For a local trial, set Server’s ports in Compose to 127.0.0.1:8080:8080 to prevent exposing unauthenticated ports beyond loopback (set HUBBLE_PUBLISH_HOST=0.0.0.0 if remote access to Hubble is required).

  2. Start and check the services.

    Choose an unused project name for this trial and keep these variables in the same terminal session:

    1export HUGEGRAPH_VERSION=latest
    2export HUBBLE_IMAGE=hugegraph/hubble:latest
    3export HUBBLE_DEMO_PROJECT="hubble-demo-$(date +%Y%m%d-%H%M%S)"
    4docker compose ls
    5docker compose -p "$HUBBLE_DEMO_PROJECT" -f docker-compose.yml pull
    6docker compose -p "$HUBBLE_DEMO_PROJECT" -f docker-compose.yml up -d --wait
    7docker compose -p "$HUBBLE_DEMO_PROJECT" -f docker-compose.yml ps
    8curl -fsS http://127.0.0.1:8080/versions
  3. Open Hubble.

    Once services are healthy, open http://127.0.0.1:8088. When HUGEGRAPH_ADMIN_PASSWORD is unset in .env, Server permits anonymous access and Hubble opens the homepage directly. If authentication is enabled, configure an admin password and JWT secret in .env as described in the Docker README, then sign in with a Server account (Hubble has no independent user database). Profile and account management options appear only when authentication is enabled and the account holds requisite permissions.

Use latest to explore recent features; pin a released image version or digest for production. Official release archives are on the download page.

Verify the deployed versions

Screenshots help locate controls. Verify actual versions from the deployed image digests and build information.

Compose’s server-data and hubble-data retain graph data and Hubble metadata respectively. For further persistence and production settings, see the Server deployment guide.

Home: find the right starting point

Home groups the workspace into graph overview, data preparation, and graph queries. Use it to understand the workflow, then return to any section through the sidebar. The top graph selector determines the target of queries, schema operations, and async tasks; check the current graph after changing pages. Standalone mode has only the DEFAULT GraphSpace and needs no PD configuration.

SectionPurpose
Graph Overview and detailsSelect a graph, load samples, inspect its size, then model or query it
Schema configurationDefine properties, vertex/edge labels, and indexes
GQL TraversalWrite queries and explore graph, table, and JSON results
Built-in AlgorithmsExplore neighbors, paths, and similarity with parameter forms
Async TasksTrack background queries, schema changes, and index operations
Data Source ManagementUpload files or configure external readers
Data ImportMap source fields to the graph model and run or schedule ingestion
Profile and Account ManagementUpdate personal details/passwords and manage accounts or space members according to permissions
OperationsInspect Server nodes; PD mode also includes cluster overview and PD/Store nodes

Interface at a glance

Compare the graph overview, schema, and query workspaces below. Click a screenshot to view the original.

Graph Overview and details: get to know a graph

Select the default graph hugegraph in Graph Overview. Click the graph card or details to view its schema structure and data statistics, or quickly jump to modeling, data import, or query pages.

Load the People & Software Demo Graph from the graph card’s More actions menu. The sample automatically provisions the required schema and imports sample records without clearing existing data. For your initial trial, load it into an empty graph to avoid name collisions. The remaining examples use this graph to explore connections between people and software.

To create a new graph, click the create button on the overview page and provide a graph name, optional alias, and schema template or sample data. Connection details are managed globally by the Hubble backend, so you do not re-enter Server host or credentials per graph. User-defined schema templates require PD mode; see the distributed supplement.

Hubble overview uses bounded scans. When the response contains approximate:true and scan_limit:100000, the vertex or edge count can stop at 100000 even with tens of millions of records. These values are scan observations, not exact totals. Field names depend on the endpoint (e.g. gremlin-query returns vertexcount / edgecount). For exact counts, read completed asynchronous task results from Server.

Schema modeling: define the shape of your data

Schema defines allowable properties, vertex labels, and edge labels, while determining vertex ID strategies and query index rules. Under Schema configuration, use the list view to manage definitions or the graph view to inspect vertex-edge topology visually.

DefinitionWhat to decide
PropertiesData type and cardinality; distinguish numbers from text
Vertex labelsProperties, nullable properties, ID strategy, and primary keys
Edge labelsSource/target labels, properties, frequency, and sort keys
Vertex / edge indexesIndex type and fields that match filtering and range queries

In the person/software sample: person derives its ID from the primary key name, with nullable age and city; software uses custom numeric IDs; the created edge label links people to software. This matches the Loader example.

When modeling your own graph, create elements in order: Properties → Vertex labels → Edge labels → Indexes. Deleting labels or creating/rebuilding indexes typically submits background asynchronous tasks; verify their completion in Async Tasks.

GQL workspace: query and explore relationships

Open GQL Traversal and confirm hugegraph is selected. The workspace puts the query editor and graph display together, with immediate or async execution, query favorites, and reusable execution history (see Graph Query Languages for syntax guides). Start with this Gremlin query:

1g.V().hasLabel('person').valueMap()

Inspect person properties in table or JSON view. To visualize the people-to-software relationships, run:

1g.V().hasLabel('person').outE('created').inV().path()

Graph results support 2D / 3D. Click a vertex or edge to inspect its ID, label, and properties; double-click a vertex to expand its neighbors. Layout, styling, and filtering help highlight relevant relationships, while export helps share results. Use New to create elements, or edit existing data when authorized. Layout adjustments, styling, and display bounds only affect the local canvas; adding or editing elements via the UI or executing Gremlin write statements modifies underlying Server graph data.

Press Ctrl / Command + Enter to execute queries quickly. Immediate execution is ideal for interactive exploration; submit long or resource-intensive queries as background async tasks. Cypher is available when supported by the backend Server. The keyboard shortcut acts on whichever input currently has focus: generating a query from the natural language prompt, or running statements in the code editor.

Text2GQL natural-language queries

Describe your query requirement in the Natural language input. Text2GQL combines the current graph schema to generate read-only Gremlin or Cypher and execute it automatically. Generated statements can be edited freely (read-only safety checks still apply before execution). In Async mode, both generated and handwritten queries can be submitted as background tasks. Inputs automatically restore the graph’s most recent query prompt without running it, and the execution history retains original natural-language questions.

The screenshots show an unconfigured model. The graph results come from handwritten Gremlin; connect a working model before generating natural-language queries.

Model sourceConfiguration and storage
Use built-in modelConfigure under System & Operations → Model settings; Keys are encrypted in H2 and never returned to the browser
Use temporary modelEnter the API type, Base URL, model name, and temporary API Key on the query page; the Key clears on refresh

Choose OpenAI compatible for OpenAI, DeepSeek, GLM, and similar services, or Ollama (local) for local Ollama; unauthenticated Ollama needs no Key. The Base URL must be reachable from Hubble’s backend; container localhost refers to that container. Public endpoints require HTTPS, while plain HTTP is accepted only for loopback or private-network models; when changing the Base URL in Model settings, supply a new Key or explicitly clear the saved Key. Administrators with model-settings management permission manage shared settings. When Server authentication is disabled, anyone who can access the deployment can manage them.

Under model settings, click Text2GQL Prompt in the header to customize Gremlin and Cypher generation rules and few-shot examples. Click Apply and then save model settings to persist templates to the H2 database for both built-in and temporary models. The current graph schema is appended automatically during generation.

Key storage and privacy notes

When backing up the Hubble H2 database, always back up the master-key file (data/text2gql-master.key by default); losing it prevents decrypting saved Keys. Query generation sends only the prompt, graph schema, and necessary error-correction context to the model provider—it never transmits graph query results. Browsers can remember frequently used Base URLs and model names, but temporary API Keys remain in memory only and clear on page refresh.

Input footers show model generation time and query request time separately. Async submission time excludes background execution.

Built-in algorithms: explore with parameter forms

Hubble cleanly separates algorithm execution into two dedicated workflows: OLTP online exploration and OLAP batch computing.

OLTP online algorithms

Use Built-in Algorithms when you prefer parameter forms to writing manual Gremlin traversals for real-time queries:

  • Neighbor exploration: answers what surrounds a vertex (e.g. K-neighbor, K-hop).
  • Paths: discovers connections between two vertices (e.g. shortest path, all paths).
  • Similarity & ranking: compares or ranks vertices within local subgraphs.

Forms provide parameter guidance and documentation links, and restore common parameters when navigating away and back. While the parameter form is focused, Ctrl / Command + Enter runs the current algorithm.

For example, select K-neighbor (GET) with source=1:marko, max_depth=1, and limit=20. After parameter validation, use the run button on the card or the form shortcut to explore one-hop neighbors.

Built-in neighbor algorithm parameters and graph result

Native Vermeer OLAP integration

For large-scale, whole-graph analytical algorithms, Hubble integrates natively with the high-performance Vermeer graph computing engine. Hubble prefers Vermeer for OLAP workloads by default, while Computer remains available as an opt-in alternative.

Configuration in hugegraph-hubble.properties

Enable direct Vermeer communication in conf/hugegraph-hubble.properties:

1# Direct Vermeer integration endpoint
2vermeer.url=http://vermeer-master:6688
3# Protected file containing the Vermeer service bearer token
4vermeer.token_file=/path/to/vermeer-service.token
Note

When vermeer.url is empty, Hubble falls back to the legacy Server proxy path. Direct Vermeer integration requires PD/Store cluster mode; it is disabled in standalone mode.

Supported OLAP algorithms and execution lifecycle

Vermeer executes batch algorithms over in-memory graph partitions:

  • Supported algorithms: PageRank, Personalized PageRank (PPR), Weakly Connected Components (WCC), Label Propagation Algorithm (LPA), Degree Centrality, Closeness Centrality, Betweenness Centrality, Triangle Count, K-Core, and Single-Source Shortest Path (SSSP).
  • Loading graph into memory: The workflow begins by loading or reloading graph data from PD/Store into Vermeer worker memory. Accepted loads retain their task ID per graph so queued operations stay in Loading state until ready.
  • Task monitoring & cancellation: Native tasks run asynchronously and can be filtered by state (RUNNING, SUCCESS, FAILED) or cancelled on demand.
  • Cursor-paged results: Analytical outputs (e.g., vertex scores, cluster IDs) are retrieved through cursor-based pagination. Results preserve values consistently across page boundaries and stop cleanly at an exactly full final page.

Async Tasks: confirm background results

Async Tasks lists background work for the current graph, including async queries and some schema/index operations. Filter by task type and status, inspect IDs, creation times, and execution states, open successful query results, or expand failure information. Completed task records can be deleted where the interface permits. These tasks are separate from import execution history.

For example, submit g.V().count() asynchronously, confirm success in the list, then inspect the returned count. A successful submission means the request was accepted, not that computation or indexing has finished. Use task errors and Server logs together when diagnosing failures.

Data Source Management: prepare the input

A data source defines where data comes from and how to parse it, and can be referenced by import tasks. Hubble supports FILE, HDFS, JDBC, and Kafka, each with its own path, connection, or subscription settings. FILE is an easy starting point: upload a file, configure its format, delimiter, encoding, and header, and check column names before mapping. Hubble configuration controls upload limits and permitted extensions.

Save this UTF-8 file as people.csv for the person example:

1name,age,city
2docs_alice,28,Beijing
3docs_bob,32,Shanghai

Create a FILE data source and upload it. Select CSV (comma separation and UTF-8 by default), with column names name,age,city. Header, delimiter, and encoding belong to the data source, not mapping settings. The source fields must match the file.

Data Import: turn fields into a queryable graph

Data Import converts source rows into vertices and edges. Its four configuration sections identify the target, select fields, map them to the graph, and choose execution timing. Use the preceding data source to create a person import into DEFAULT / hugegraph:

  1. Basic Information Select the target graph, new data source, and a task name.
  2. Source Fields Select name, age, and city, moving them to the selected field list.
  3. Mapping Fields Add a person vertex mapping; use Auto Match for same-name properties, then verify types.
  4. Schedule Choose one-time execution and confirm to submit the task.

person uses PRIMARY_KEY, so do not select a separate ID column. Custom ID strategies require an ID column; AUTOMATIC lets Server generate IDs, while PRIMARY_KEY derives them from mapped primary-key properties. Edge mappings need source/target fields that follow the corresponding vertex ID rules.

The task list manages configuration and execution entry points; execution history in task details shows each instance’s state, count, and errors. Periodic schedules and real-time Kafka tasks are also available; choose an execution mode compatible with the source. After completion, verify the result in the GQL workspace:

1g.V().hasLabel('person').has('name', within('docs_alice', 'docs_bob')).valueMap()

The result should include both new people. Import counts are not necessarily counts of newly created vertices: reruns may update existing elements, and header processing can affect reader counts. If the import fails, check source fields, numeric types, nullable properties, and target schema. Use Hubble for small trials and HugeGraph Loader for production bulk ingestion.

Profile and account permissions

Hubble uses Server authentication and accounts, with no separate user database. Anonymous mode hides Profile and Account Management. After authentication is enabled, Profile shows the current account’s details and permissions and allows changing your password. Editing details such as a nickname requires Server support for the personal-profile API. Changing a password ends the current session; sign in again with the new password.

Standalone account management

In standalone mode, the administrator can create, inspect, edit, delete, and batch-create accounts. Ordinary standalone accounts created through Hubble receive read, write, delete, and execute permissions across all graphs; they are not read-only or isolated to one graph. Configure finer resource permissions through Server authentication and authorization rather than relying on GraphSpace presets.

GraphSpace permissions in PD mode

With a Server supporting default-role APIs, global accounts and space access can be managed separately. The administrator manages global accounts; a space administrator manages members only within authorized spaces. Ordinary members do not receive account-management or operations entry points. The following presets are for PD mode, and are not universally editable on older or standalone Servers:

PresetScope and purpose
SUPER_ADMINGlobal account, GraphSpace, and operations management; grant or revoke super-administrator access
GS_ADMINManage authorized spaces and their members, without granting other-space or global super-administrator access
GS_READ_WRITERead and write graph data within authorized spaces, without managing global accounts
GS_READ_ONLYRead graph data within authorized spaces, without writes

An account may have different permissions in different spaces. Select a space before adding an existing account or changing member permissions. Before replacing custom permissions with a preset, inspect the grants that need to be retained; complex permissions may not match a single preset. After changes, refresh permission context or sign in again and verify menus and space selection. Server still validates every request. Older Servers may hide or disable unsupported operations; a visible button alone does not establish resource authorization. See the distributed supplement for space management.

Authenticated account list in standalone mode

Hubble HTTP APIs

Hubble exposes a set of HTTP REST endpoints for automation and integration, listening by default on http://localhost:8088.

Login and sessions

In authenticated mode, log in with snake_case field names and retain the session cookie:

1curl -c hubble.cookies -X POST 'http://localhost:8088/api/v1.3/auth/login' \
2  -H 'Content-Type: application/json' \
3  -d '{"user_name":"<username>","user_password":"<password>"}'

Camel-case userName and userPassword do not populate this login model. Repeated failed attempts can return HTTP 429; correct the fields and credentials before retrying. Anonymous mode does not use the login endpoint.

Integration Gremlin and Cypher queries

The integration Gremlin endpoint currently requires username and password in the body as well as the authenticated session:

1curl -b hubble.cookies -X POST \
2  'http://localhost:8088/api/v1.3/graphspaces/DEFAULT/graphs/hugegraph/gremlin' \
3  -H 'Content-Type: application/json' \
4  -d '{"query":"g.V().count()","username":"<username>","password":"<password>"}'

Use HTTPS outside a trusted local network: JSON does not encrypt these credentials. This integration body differs from Server’s gremlin field and from Hubble’s workspace gremlin-query contract. The session establishes the caller identity; body credentials do not replace the login cookie.

Hubble’s Cypher endpoint uses the workspace query model, not the integration Gremlin model:

1curl -b hubble.cookies -X POST \
2  'http://localhost:8088/api/v1.3/graphspaces/DEFAULT/graphs/hugegraph/cypher' \
3  -H 'Content-Type: application/json' \
4  -d '{"content":"MATCH (n) RETURN count(n)","preview":false}'

The query field is content; a query field or a nested gremlin-query object does not populate it. The session supplies authentication. Server must advertise Cypher support. Hubble UI routes such as /asyncTasks are browser navigation routes. Task API requests use the /api/v1.3/graphspaces/{space}/graphs/{graph}/async-tasks prefix.

Schema defaults

Hubble in version 1.8.0 and above supports optional field defaults. When using older versions without this feature, explicitly supply "open_label_index":true,"style":{"display_fields":["~id"]}:

1curl -b hubble.cookies -X POST \
2  'http://localhost:8088/api/v1.3/graphspaces/DEFAULT/graphs/hugegraph/schema/vertexlabels' \
3  -H 'Content-Type: application/json' \
4  -d '{"name":"http_demo_person","id_strategy":"AUTOMATIC","properties":[]}'

Missing or null style uses the standard style displaying ~id; missing or null open_label_index defaults to true, preserving explicit false. Style fields use snake_case (display_fields, join_symbols), not displayFields. PRIMARY_KEY types also require properties and primary_keys; create properties before labels. Server schema uses a different request model without Hubble’s visual style fields.

Import graph elements through Hubble

POST /api/v1.3/graphspaces/{space}/graphs/{graph}/import accepts multipart field file containing JSON graph elements. It does not accept raw CSV or a Loader job specification. Create the referenced schema first. For example, with a person vertex label using CUSTOMIZE_STRING IDs and a name property, and a knows edge label linking two person vertices, save this as graph.json:

1{
2  "vertices": [
3    {"id": "alice", "label": "person", "properties": {"name": "Alice"}},
4    {"id": "bob", "label": "person", "properties": {"name": "Bob"}}
5  ],
6  "edges": [
7    {"label": "knows", "source": "alice", "target": "bob", "properties": {}}
8  ]
9}
1curl -b hubble.cookies -X POST \
2  'http://localhost:8088/api/v1.3/graphspaces/DEFAULT/graphs/hugegraph/import' \
3  -F 'file=@graph.json;type=application/json'

Both arrays are required; empty arrays perform an empty import. Supply all non-nullable schema properties. Vertex IDs must match the schema ID strategy. Edge endpoints use source and target; a file key inside a vertex is not a data source. This endpoint returns a graph view. Use the separate data-source/Loader workflow for CSV ingestion.

Use GraphSpace permission presets

When Server disables custom GraphSpace role assignments, the legacy GraphSpace user POST is rejected by policy. That endpoint assigns roles to an existing account; it is not account registration. Use the permission presets exposed by Hubble, rather than inventing role IDs or disabling the Server policy. For an existing account, an authorized GraphSpace manager can assign a preset using:

1curl -b hubble.cookies -X PUT \
2  'http://localhost:8088/api/v1.3/graphspaces/DEFAULT/auth/users/<user_id>/preset' \
3  -H 'Content-Type: application/json' \
4  -d '{"user_id":"<user_id>","username":"<username>","permission_preset":"GS_READ_ONLY"}'

Supported presets are GS_READ_ONLY, GS_READ_WRITE and GS_ADMIN, subject to the caller’s grant permissions. The account identity in the path and body must match. Anonymous mode has no account-management capability.

Operations: inspect the standalone Server

Standalone Operations provides Node Information for Server only, with no PD/Store nodes or cluster overview. Search nodes, filter health status, and open node details to inspect available version, system, JVM, and Server-backend metrics. When metrics are unavailable or stale, use collection state and the last successful observation time; a missing value is neither zero nor proof of health.

Anonymous mode can read operations information. With authentication enabled, operations are available only to administrators with the capability. This is an observation and diagnosis interface, not a start/stop or scaling console. The distributed supplement covers cluster overview and the PD/Store node hierarchy.

Vermeer cluster health view

When Vermeer is deployed, Hubble provides an independent Vermeer compute cluster health view under Operations:

  • Configure telemetry in conf/hugegraph-hubble.properties:
    1operations.vermeer.enabled=true
    2operations.vermeer.address=http://vermeer-master:6688
    3operations.vermeer.token_file=/path/to/vermeer-service.token
  • Readiness & node topology: Inspects Vermeer Master readiness, registered Worker node health, and cluster topology.
  • Resource metrics: Visualizes active compute jobs, CPU utilization, and process RSS memory usage across workers.
  • Fault isolation: Service credentials remain backend-only. If telemetry times out or workers become unavailable, stale status is distinguished from zero usage or current readiness.

Keyboard shortcuts and graph interactions

Use the topbar shortcut-help button to see key bindings. Their scope differs: typing ? in an input does not trigger global help.

ActionKey or gestureScope
Open / close shortcut help?Outside inputs and editors
Execute queryCtrl with Enter / Command with EnterQuery editor
Run current algorithmCtrl with Enter / Command with EnterAlgorithm parameter form
Toggle graph fullscreenFClick to focus the graph canvas first; not a global binding
Inspect element detailsClick a vertex or edgeGraph result
Expand neighboring relationshipsDouble-click a vertexGraph result

Diagnose connection and result issues

SymptomCheck first
The Hubble page does not openCheck container status and docker compose -p "$HUBBLE_DEMO_PROJECT" -f docker-compose.yml logs hubble
The page opens but graphs are unavailableServer health, a server.direct_url reachable from Hubble, and a shared network
Login appears or write actions are missingServer authentication and account permissions; Hubble has no independent authentication switch
Expected data is missingCurrent graph, sample load result, matching labels/properties; distinguish canvas display from stored data
No cluster overviewThis example has no PD; see the distributed supplement

Configuration can affect displayed query size. gremlin.suffix_limit defaults to 250 and supplies .limit(N) appended to applicable Gremlin queries; it is not a universal hard limit. gremlin.vertex_degree_limit (100) and gremlin.edges_total_limit (500) constrain expansion. FILE uploads allow csv,txt by default, with 1 GB per file and 10 GB total. Override upload_file.* settings when needed.

Stop the trial or build from source

When finished, run this in the main repository’s docker/ directory:

1docker compose -p "${HUBBLE_DEMO_PROJECT:?}" -f docker-compose.yml down --volumes

This removes the project’s containers, network, named volumes, and anonymous volume, losing the sample data and Hubble import tasks. To retain data, omit --volumes when taking the deployment down and reuse the same project name when starting it again.

Building current master requires JDK 17, Maven 3.9+, Python 3.9+, and stable Node.js 24 (at least 24.22.0), with pnpm 12.14.0 selected through Corepack. Maven no longer installs Node/Yarn automatically.

  1. Follow the Java 17 migration instructions to install the Server SDK in an isolated Maven repository.

  2. Follow the Hubble source-build procedure to install Python dependencies and build Client, Loader, and Hubble with that same repository (skipping tests).

  3. Open the resulting hugegraph-hubble/apache-hugegraph-hubble-* directory, configure Server connection settings in conf/hugegraph-hubble.properties, then start:

    1bin/start-hubble.sh -d

Current master uses a new H2 metadata database. Retain the old Hubble distribution and its data, and use a separate data directory for the new version. Old metadata has no automatic migration and cannot be reused directly.

The bundled configuration binds to localhost:8088. bin/stop-hubble.sh requests graceful shutdown before forcing termination on timeout. For development and testing, see the Toolchain local test guide.

2.3 - Graph import

Choose an import tool when you need to write file, database, or message data into HugeGraph. Use Loader for a direct import, SeaTunnel Sink when an existing Source, Transform, and Sink pipeline should be reused, or Spark Connector from a Spark job.

2.3.1 - Import Graph Data with SeaTunnel Sink

SeaTunnel connects data sources such as databases and Kafka to HugeGraph. The connector has two parts: Source reads data and Sink writes data[1][2], with SeaTunnel transform components available between them. To export or migrate data from HugeGraph, see the SeaTunnel Source export and migration guide.

Version requirement: This guide targets SeaTunnel 3.0+. All examples use the mappings configuration available in SeaTunnel 3.0+.

With HugeGraph Helm, you can directly enable the built-in SeaTunnel engine. That path includes the pre-configured connector, runtime environment, and same-release credential references. This page covers job mappings for external data sources.

Loader imports data directly with graph mappings; SeaTunnel 3.0+ combines Source, Transform, and Sink, and both support JDBC, Kafka, and graph data

Click a diagram to view the original size.

1 Loader, Tools, and SeaTunnel

HugeGraph-Loader is suited to direct imports from common data sources. HugeGraph-Tools focuses on standalone graph management, backup, and export. SeaTunnel organizes a job as Source → Transform → Sink, so you can reuse existing connectors, transforms, and data pipelines.

Table legend

✅ Supported natively; ⚠️ conditional support or requires an extra component/external platform; ❌ not provided

ComparisonLoaderToolsSeaTunnel
Task coverage✅ Direct graph imports✅ Backup, restore, and export✅ Import, export, and migration with composable Source, Transform, and Sink stages
Job configurationJSON mapping file describing the source, vertices, and edgesCommand-line options and operationsHOCON job file[3] combining Source, Transform, and Sink
Default deployment✅ Standalone CLI; ⚠️ Spark Loader can extend it✅ Standalone CLI✅ Standalone; ✅ distributed
Execution engine⚠️ Mainly CLI; Spark Loader is a separate extension❌ Does not provide a Spark/Flink execution engine✅ HugeGraph Source and Sink support Zeta, Spark, and Flink[1][2][7][8][9][10]
Frontend and observability❌ No built-in frontend; inspect CLI logs❌ No built-in frontend; inspect CLI logs✅ Built-in Web UI job panel for task status and runtime information
Input and output⚠️ Focused on graph imports and common files, JDBC, Kafka, and similar sources⚠️ Focused on graph data and backup files in common storage✅ Dozens of connectors, including JDBC, Kafka, and SQL-CDC
Scheduling and resource management❌ No unified cross-task scheduling or resource allocation❌ No unified cross-task scheduling or resource allocation⚠️ Can integrate with DolphinScheduler for scheduling and task management
Simplicity✅ Focused and simple; a future binary CLI will make quick use easier✅ Direct commands for standalone operations⚠️ More runtime components, suited to long-lived data pipelines
High-throughput import✅ Supports bypass-server and other optimizations; measured peaks can reach 1-2 million records/s with specific backends and hardware, so benchmark the actual setup⚠️ Focuses on backup and export rather than bulk-import throughput✅ Scales throughput through parallelism, distributed engines, and connectors

SeaTunnel covers Loader’s graph-import and Tools’ export and migration scenarios in one expandable pipeline, and it also supports SQL-CDC and dozens of input and output types. Loader and Tools normally run on one machine, while SeaTunnel supports both standalone and distributed deployments and scales with data and task volume. Tools’ schedule-backup can create a crontab entry, but it does not provide unified workflow orchestration and resource management.

Existing Spark/Flink daily jobs

Both HugeGraph Source and Sink list SeaTunnel Engine (Zeta), Spark, and Flink as supported engines in SeaTunnel 3.0+. If you express the daily job as a SeaTunnel job and submit it to that engine, records can move directly from Source to Transform to Sink without an intermediate file. If you keep the existing Spark/Flink DAG, SeaTunnel does not automatically take over its in-memory DataFrame or stream. Adapt it into a SeaTunnel job or expose the data through a Source connector

Loader and Tools are focused, direct, and quick to start. Use Loader for a direct graph import; use Tools for backup, restore, export, or daily operations. If a SeaTunnel job already exists, adding HugeGraph to that pipeline is usually simpler. For higher import throughput, Loader’s bypass-server path and other import optimizations are a better fit; measured peaks of 1-2 million records/s require a specific backend, data set, and hardware configuration and are not a general performance guarantee. For new SeaTunnel jobs, use 3.0+ and mappings. Recheck the connector configuration when using another version.

2 Prepare the environment

2.1 Get SeaTunnel 3.0+

Use JDK 11 and set JAVA_HOME; the official deployment guide lists Java 8 and 11 as prerequisites. Choose a 3.0+ binary distribution from the official release page[4] and verify its checksum or signature using the files linked there. The commands below use 3.0.0 as an example; set SEATUNNEL_VERSION to the release you downloaded.

1export SEATUNNEL_VERSION=3.0.0
2tar -xzf "apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz"
3cd "apache-seatunnel-${SEATUNNEL_VERSION}"

The binary distribution does not include connector plugins. In config/plugin_config, select the connectors needed by the examples:

1--seatunnel-connectors--
2connector-hugegraph
3connector-jdbc
4connector-kafka
5--end--

Install the matching released plugins using the upstream plugin installation instructions[5]:

1sh bin/install-plugin.sh "${SEATUNNEL_VERSION}"

Run the remaining commands from this installation directory and keep the engine and connector plugins at the same version. This guide uses the bundled Zeta engine in local mode[6][7]. Check that connectors/ contains HugeGraph, JDBC, and Kafka as needed[11][12]. The JDBC examples also require the MySQL driver JAR in lib/, with driver class com.mysql.cj.jdbc.Driver.

2.2 Prepare HugeGraph and data sources

Start HugeGraph Server and create a graph for testing. The examples use the hugegraph graph in the DEFAULT graph space. Adjust these names to match the server configuration; graph space names are case-sensitive. If authentication is enabled, provide username and password in the HugeGraph Source and Sink configurations.

The following graph model is shared by the JDBC and Kafka examples. mappings creates missing PropertyKey, VertexLabel, and EdgeLabel definitions by default; existing schema definitions must be compatible.

Graph elementName and properties
Propertiesname is Text; age and since are Int
Vertexperson, primary key name, properties name and age
Edgeknows, from person to person, property since

The mysql, kafka, and hugegraph host names in the examples are placeholders. Replace them with addresses reachable from the SeaTunnel runtime. Inside a container, 127.0.0.1 points to that container; services on the same Docker network can use their service names. Set host to a host name or IP address, and set the port separately.

3 Import from a relational database (sql2graph)

Use two jobs for this import: write the person table as vertices first, then write the knows table as edges. Both edge endpoints will already exist when the edge job runs.

The person table creates marko and vadas vertices; the knows table creates a directed edge with since 2010 through endpoint fields

3.1 Import vertices

Prepare the sample data in the MySQL demo database and grant the configured account read access:

1CREATE TABLE person (
2  name VARCHAR(64) PRIMARY KEY,
3  age INT NOT NULL
4);
5INSERT INTO person VALUES ('marko', 29), ('vadas', 27);

Save the following as config/sql2graph-person.conf and replace the database user name and password:

 1env {
 2  job.mode = "BATCH"
 3}
 4
 5source {
 6  Jdbc {
 7    url = "jdbc:mysql://mysql:3306/demo?useSSL=false&serverTimezone=UTC"
 8    driver = "com.mysql.cj.jdbc.Driver"
 9    username = "seatunnel"
10    password = "change_me"
11    query = "SELECT name, age FROM person ORDER BY name"
12  }
13}
14
15sink {
16  HugeGraph {
17    host = "hugegraph"
18    port = 8080
19    graph_name = "hugegraph"
20    graph_space = "DEFAULT"
21    batch_failure_fallback = false
22    mappings = [
23      {
24        type = "VERTEX"
25        label = "person"
26        idStrategy = "PRIMARY_KEY"
27        idFields = ["name"]
28        properties = ["name", "age"]
29      }
30    ]
31  }
32}
1./bin/seatunnel.sh --config ./config/sql2graph-person.conf -m local

Check the result in Hubble or Gremlin. You should find marko and vadas with their ages:

1g.V().hasLabel('person').valueMap('name', 'age')

idFields = ["name"] uses the name to generate the primary key. Importing the same name again writes to the same vertex. properties lists the source fields to write.

3.2 Import edges

Prepare the relation table. Its two endpoint fields correspond to person.name from the vertex job:

1CREATE TABLE knows (
2  source_name VARCHAR(64) NOT NULL,
3  target_name VARCHAR(64) NOT NULL,
4  since INT NOT NULL
5);
6INSERT INTO knows VALUES ('marko', 'vadas', 2010);
Expand the configuration and save it as config/sql2graph-knows.conf
 1env {
 2  job.mode = "BATCH"
 3}
 4
 5source {
 6  Jdbc {
 7    url = "jdbc:mysql://mysql:3306/demo?useSSL=false&serverTimezone=UTC"
 8    driver = "com.mysql.cj.jdbc.Driver"
 9    username = "seatunnel"
10    password = "change_me"
11    query = "SELECT source_name, target_name, since FROM knows ORDER BY source_name, target_name"
12  }
13}
14
15sink {
16  HugeGraph {
17    host = "hugegraph"
18    port = 8080
19    graph_name = "hugegraph"
20    graph_space = "DEFAULT"
21    batch_failure_fallback = false
22    check_vertex = true
23    mappings = [
24      {
25        type = "EDGE"
26        label = "knows"
27        sourceConfig = {
28          label = "person"
29          idFields = ["source_name"]
30        }
31        targetConfig = {
32          label = "person"
33          idFields = ["target_name"]
34        }
35        fieldMapping = {
36          source_name = "name"
37          target_name = "name"
38        }
39        properties = ["since"]
40      }
41    ]
42  }
43}

After the vertex job succeeds, run the edge job:

1./bin/seatunnel.sh --config ./config/sql2graph-knows.conf -m local

The following query should return a knows edge from marko to vadas with since set to 2010:

1g.V().has('person', 'name', 'marko').outE('knows').where(inV().has('name', 'vadas')).valueMap()

sourceConfig and targetConfig identify the endpoint fields. fieldMapping maps them to the vertex primary key name, and properties = ["since"] writes only the edge property. The example enables check_vertex = true and disables per-record fallback after a batch failure (batch_failure_fallback = false), so a missing endpoint or write failure causes the job to fail.

If the relation table has only numeric foreign keys while the graph uses names as primary keys, join the names in SQL before passing the records to the Sink. See MySQL CDC Source[13] for MySQL CDC integration.

4 Import from Kafka (kafka2graph)

Kafka is useful for a continuous stream of events. Create the user-events topic and publish the following JSON message. Each message becomes one person vertex:

1{"name":"marko","age":29}

Save the following as config/kafka2graph.conf:

 1env {
 2  job.mode = "STREAMING"
 3  checkpoint.interval = 10000
 4  sink.flush.interval = 5000
 5}
 6
 7source {
 8  Kafka {
 9    bootstrap.servers = "kafka:9092"
10    topic = "user-events"
11    consumer.group = "hugegraph-import"
12    start_mode = "earliest"
13    format = "json"
14    schema = {
15      fields = {
16        name = "string"
17        age = "int"
18      }
19    }
20  }
21}
22
23sink {
24  HugeGraph {
25    host = "hugegraph"
26    port = 8080
27    graph_name = "hugegraph"
28    graph_space = "DEFAULT"
29    batch_failure_fallback = false
30    mappings = [
31      {
32        type = "VERTEX"
33        label = "person"
34        idStrategy = "PRIMARY_KEY"
35        idFields = ["name"]
36        properties = ["name", "age"]
37      }
38    ]
39  }
40}
1./bin/seatunnel.sh --config ./config/kafka2graph.conf -m local

Use the Gremlin query from section 3.1 to check the data. The streaming job keeps running. checkpoint.interval saves job state every 10 seconds, while sink.flush.interval asks Zeta to flush every 5 seconds so a small number of messages does not wait for a full batch.

HugeGraph Sink writes with at-least-once semantics, so recovery can replay records. PRIMARY_KEY sends the same name to the same vertex, but it does not make every update exactly-once. Scheduled flushing is provided by Zeta and does not apply to Spark or Flink engines.

5 Common configuration and troubleshooting

The following table applies to the SeaTunnel 3.0+ version used by this guide:

ConfigurationPurpose
host, portSet the HugeGraph host and port
graph_name, graph_spaceSelect an existing graph and graph space
mappingsDefine how input fields become vertices or edges
propertiesList the source fields written by each mapping
schema_save_modemappings creates missing schema by default; existing schema must still be compatible
batch_sizeNumber of records per batch; default 500
env.sink.flush.intervalZeta scheduled flush interval in milliseconds
check_vertexCheck edge endpoints; the edge job in this guide sets it to true
batch_failure_fallback[2]Defaults to true, so a failed batch falls back to record-by-record retries, capped by max_insert_errors; the examples explicitly set false so a batch failure stops the job
max_insert_errorsNumber of failed records that record-by-record fallback may skip; default 500, -1 for unlimited, and only applies when batch_failure_fallback is enabled

Use these checks when a job fails:

  • mappings is unknown or HugeGraph Source is missing: Check that the engine and HugeGraph connector come from the same SeaTunnel 3.0+ build.
  • Connection failure: Check the host, port, graph space, authentication details, and whether the SeaTunnel runtime can reach the service.
  • Schema incompatibility: Check the ID strategy, property types, and edge endpoints. Automatic creation does not change an existing PRIMARY_KEY label into CUSTOMIZE_STRING.
  • Small Kafka batches do not appear promptly: Confirm that the job uses Zeta and set sink.flush.interval in env. In this version, batch_interval_ms is retained only for compatibility and cannot replace it.

6 Choosing a tool

Choose a tool based on the work to complete. Use Tools for graph management, Gremlin, backup, or cloning. Use Loader for a direct graph import. Choose SeaTunnel when you need to reuse a Source, Transform, and Sink pipeline. For SeaTunnel graph reads and migrations, prepare the environment using the SeaTunnel 3.0+ version used by this guide.

Choosing a tool: Tools for graph management, Loader for direct imports, and SeaTunnel for reusable data pipelines

7 References

HugeGraph connectors

[1] HugeGraph Source
[2] HugeGraph Sink

Configuration and deployment

[3] HOCON job configuration
[4] SeaTunnel 3.0+ release download
[5] Connector plugin installation
[6] SeaTunnel local deployment

Execution engines

[7] SeaTunnel Engine Overview
[8] SeaTunnel Spark Engine
[9] SeaTunnel Flink Engine
[10] Connector V2 multi-engine support

Data source connectors

[11] JDBC Source
[12] Kafka Source
[13] MySQL CDC Source

2.4 - HugeGraph-Loader Quick Start

Bulk import graph data into HugeGraph with Loader from files, HDFS, relational databases, Kafka, and other HugeGraph graphs.

This guide aligns with the Toolchain master branch (currently based on 1.8, unreleased). Released packages and Docker latest may differ; check your version before using unreleased features.

Quick start: local file import

Walk through a minimal import workflow: import two people and one relationship, then verify the data in Hubble. First install Loader and start HugeGraph Server. The commands below run from the Loader directory, importing into the default graph hugegraph. For an authenticated Server, append --username and --password (or --token).

  1. Create the data files.

    These CSV files have no header row. The mapping in the next step supplies the column names.

    1mkdir -p demo-loader
    2cat > demo-loader/people.csv <<'EOF'
    3docs-smoke-alice,29
    4docs-smoke-bob,31
    5EOF
    6cat > demo-loader/knows.csv <<'EOF'
    7docs-smoke-alice,docs-smoke-bob
    8EOF
  2. Create the schema and mapping.

    Dedicated docs_* names keep this example separate from the built-in person data. File paths are relative to the directory where you run Loader.

     1cat > demo-loader/schema.groovy <<'EOF'
     2schema.propertyKey("docs_name").asText().ifNotExist().create()
     3schema.propertyKey("docs_age").asInt().ifNotExist().create()
     4schema.vertexLabel("docs_smoke_person").properties("docs_name", "docs_age")
     5      .primaryKeys("docs_name").ifNotExist().create()
     6schema.edgeLabel("docs_smoke_knows").sourceLabel("docs_smoke_person")
     7      .targetLabel("docs_smoke_person").ifNotExist().create()
     8EOF
     9cat > demo-loader/struct.json <<'EOF'
    10{
    11  "version": "2.0",
    12  "structs": [
    13    {
    14      "id": "people",
    15      "input": {
    16        "type": "FILE",
    17        "path": "demo-loader/people.csv",
    18        "format": "CSV",
    19        "header": ["docs_name", "docs_age"]
    20      },
    21      "vertices": [{"label": "docs_smoke_person"}]
    22    },
    23    {
    24      "id": "knows",
    25      "input": {
    26        "type": "FILE",
    27        "path": "demo-loader/knows.csv",
    28        "format": "CSV",
    29        "header": ["source_name", "target_name"]
    30      },
    31      "edges": [{
    32        "label": "docs_smoke_knows",
    33        "source": ["source_name"],
    34        "target": ["target_name"],
    35        "field_mapping": {"source_name": "docs_name", "target_name": "docs_name"}
    36      }]
    37    }
    38  ]
    39}
    40EOF
  3. Run the import.

    Keep the same working directory. Change the host and port if Server runs elsewhere.

    1bin/hugegraph-loader.sh \
    2  --graphspace DEFAULT --graph hugegraph \
    3  --file demo-loader/struct.json --schema demo-loader/schema.groovy \
    4  --host 127.0.0.1 --port 8080

    A first successful run imports two vertices and one edge. Check the success and failure counts in the summary.

  4. Query the result in Hubble.

    Connect Hubble to the same Server and graph, then run these queries separately in the Gremlin workspace.

    1g.V().hasLabel('docs_smoke_person').valueMap('docs_name', 'docs_age')
    2g.V().has('docs_smoke_person', 'docs_name', 'docs-smoke-alice')
    3 .out('docs_smoke_knows').values('docs_name')

    The first query returns Alice with age 29 and Bob with age 31. The second returns docs-smoke-bob.

For other inputs, continue with the mapping reference. The bundled full example also includes software vertices and edge properties.

1 HugeGraph-Loader Overview

HugeGraph-Loader converts source data into graph vertices and edges and imports them into HugeGraph in batches.

Currently supported data sources include:

  • Local disk file or directory, supports TEXT, CSV and JSON format files, supports compressed files
  • HDFS file or directory supports compressed files
  • Mainstream relational databases, such as MySQL, PostgreSQL, Oracle, SQL Server
  • Kafka topic
  • An existing HugeGraph graph, used to copy data from one graph into another

Local disk files and HDFS files support resumable uploads.

Note: HugeGraph-Loader requires HugeGraph Server service, please refer to HugeGraph-Server Quick Start to download and start Server

Concurrent imports: Loader on the master branch supports LOADING owner protection on the same Server instance when the Server’s GET .../mode/owners returns protocol: 1. Servers without this endpoint retain legacy import behavior without this protection; see the compatibility notes below.

Testing Guide: For running HugeGraph-Loader tests locally, please refer to HugeGraph Toolchain Local Testing Guide

2 Get HugeGraph-Loader

HugeGraph-Loader is available in the following three ways:

2.1 Use Docker image (Convenient for Test/Dev)

We can deploy the loader service using docker run -itd --name loader hugegraph/loader:latest. For the data that needs to be loaded, it can be copied into the loader container either by mounting -v /path/to/data/file:/loader/file or by using docker cp.

Alternatively, to start the loader using docker-compose, the command is docker-compose up -d. An example of the docker-compose.yml is as follows:

This combination is for local anonymous testing; the Server API is published only on the local host.

 1version: '3'
 2
 3services:
 4  server:
 5    image: hugegraph/hugegraph:latest
 6    container_name: server
 7    ports:
 8      - 127.0.0.1:8080:8080
 9
10  loader:
11    image: hugegraph/loader:latest
12    container_name: loader
13    # mount your own data here
14    # volumes:
15      # - /path/to/data/file:/loader/file

The specific data loading process can be referenced under 4.5 User Docker to load data

Note:

  1. The docker image of hugegraph-loader is a convenience release to start hugegraph-loader quickly, but not official distribution artifacts. You can find more details from ASF Release Distribution Policy.

  2. Pin a published version tag or image digest in production. latest is mutable and does not guarantee alignment with master.

2.2 Download the compiled archive

Choose a published Toolchain archive from the download page and extract it. A release may not include all master functionality; use the source build below for the implementation described here.

2.3 Clone source code to compile and install

Building Loader from the master branch requires Java 17, Maven 3.6.3+, and Python 3.9+. Clone the repository and build:

1git clone --branch master --single-branch https://github.com/apache/hugegraph-toolchain.git
2cd hugegraph-toolchain
3mvn package -pl hugegraph-client,hugegraph-loader -am -DskipTests -ntp
4cd hugegraph-loader/apache-hugegraph-loader-*

To build against candidate Server SDKs during development, refer to the Java 17 migration guide.

For Oracle input, obtain a JDBC driver JAR compatible with your Oracle/JDK versions and place it in the extracted Loader lib/ directory. The startup script loads JARs there; installing a driver only in the local Maven repository does not put it on Loader’s runtime classpath.

3 How to use

Importing a dataset into HugeGraph follows four ordered steps:

  1. Construct the graph schema — Define property keys, vertex labels, and edge labels in Groovy.
  2. Prepare data files — Format data records (TEXT, CSV, JSON, HDFS, or DB tables).
  3. Write input source mapping file — Create the JSON mapping specification.
  4. Execute data import — Run hugegraph-loader.sh to load data into the graph.

3.1 Construct graph schema

Define the vertex labels, edge labels and properties needed to represent your data before writing the schema.

The example below uses person and software vertices, connected by knows and created edges. People have names, ages and cities; software has a name and price. Each edge records a date.

Example graph with person and software vertices connected by knows and created edges

graph model example

Save the schema definitions in schema.groovy.

 1// Create some properties
 2schema.propertyKey("name").asText().ifNotExist().create();
 3schema.propertyKey("age").asInt().ifNotExist().create();
 4schema.propertyKey("city").asText().ifNotExist().create();
 5schema.propertyKey("date").asText().ifNotExist().create();
 6schema.propertyKey("price").asDouble().ifNotExist().create();
 7
 8// Create the person vertex type, which has three attributes: name, age, city, and the primary key is name
 9schema.vertexLabel("person").properties("name", "age", "city").primaryKeys("name").ifNotExist().create();
10// Create a software vertex type, which has two properties: name, price, the primary key is name
11schema.vertexLabel("software").properties("name", "price").primaryKeys("name").ifNotExist().create();
12
13// Create the knows edge type, which goes from person to person
14schema.edgeLabel("knows").sourceLabel("person").targetLabel("person").properties("date").ifNotExist().create();
15// Create the created edge type, which points from person to software
16schema.edgeLabel("created").sourceLabel("person").targetLabel("software").ifNotExist().create();

See HugeGraph Client for the schema API.

3.2 Prepare data

The data sources currently supported by HugeGraph-Loader include:

  • local disk file or directory
  • HDFS file or directory
  • Partial relational database
  • Kafka topic
  • An existing HugeGraph graph
3.2.1 Data source structure
3.2.1.1 Local disk file or directory

Set the input path to a local file or a directory. A single input source accepts one directory, not a list of directories.

For data split across part-0, part-1 and other files, place those files in one directory and set the mapping’s path to that directory.

Supported file formats include:

  • TEXT
  • CSV
  • JSON

TEXT uses a configurable delimiter. Each data row represents a record; its columns supply IDs, labels or properties. Column names can come from a header row or from the mapping’s header setting.

An example is as follows:

1id|name|lang|price|ISBN
21|lop|java|328|ISBN978-7-107-18618-5
32|ripple|java|199|ISBN978-7-100-13678-5

CSV is a TEXT file with commas , as delimiters. When a column value itself contains a comma, the column value needs to be enclosed in double quotes, for example:

1marko,29,Beijing
2"li,nary",26,"Wu,han"

The JSON file requires that each line is a JSON string, and the format of each line needs to be consistent.

1{"source_name": "marko", "target_name": "vadas", "date": "20160110", "weight": 0.5}
2{"source_name": "marko", "target_name": "josh", "date": "20130220", "weight": 1.0}
3.2.1.2 HDFS file or directory

Users can also specify HDFS files or directories as data sources, all of the above requirements for local disk files or directories apply here. In addition, since HDFS usually stores compressed files, loader also provides support for compressed files, and local disk file or directory also supports compressed files.

Currently supported compressed file types include: GZIP, BZ2, XZ, LZMA, SNAPPY_RAW, SNAPPY_FRAMED, Z, DEFLATE, LZ4_BLOCK, LZ4_FRAMED, ORC, and PARQUET.

3.2.1.3 Mainstream relational database

The loader also supports some relational databases as data sources, and currently supports MySQL, PostgreSQL, Oracle, and SQL Server.

Loader does not follow foreign keys to query other tables while parsing a row. Each input row must provide the values needed to construct its graph elements. Prepare any joins in the source query first.

Consider three tables named person, software and created.

1// person schema
2id | name | age | city
1// software schema
2id | name | lang | price
1// created schema
2id | p_id | s_id | date

If person and software use PRIMARY_KEY IDs with name as the primary key, each edge needs its endpoints’ names. The created table contains only p_id and s_id; Loader will not query the names from those IDs. Use either of these ID designs.

  1. Keep PRIMARY_KEY and use each vertex table’s id column as its primary key property. Loader can then build endpoint IDs from p_id, s_id and the vertex labels.
  2. Use custom IDs, taking each vertex table’s id column as the vertex ID. The edge mapping can use p_id and s_id directly.

Both designs let edge rows supply endpoint IDs without another lookup.

3.2.2 Prepare vertex and edge data
3.2.2.1 Vertex Data

The vertex data file consists of data line by line. Generally, each line is used as a vertex, and each column is used as a vertex attribute. The following description uses CSV format as an example.

  • person vertex data (the data itself does not contain a header)
1Tom,48,Beijing
2Jerry,36,Shanghai
  • software vertex data (the data itself contains the header)
1name,price
2Photoshop,999
3Office,388
3.2.2.2 Edge data

The edge data file consists of data line by line. Generally, each line is used as an edge. Some columns are used as the IDs of the source and target vertices, and other columns are used as edge attributes. The following uses JSON format as an example.

  • knows edge data
1{"source_name": "Tom", "target_name": "Jerry", "date": "2008-12-12"}
  • created edge data
1{"source_name": "Tom", "target_name": "Photoshop"}
2{"source_name": "Tom", "target_name": "Office"}
3{"source_name": "Jerry", "target_name": "Office"}

3.3 Write data source mapping file

3.3.1 Mapping file overview

The mapping file uses JSON. Each item in structs contains one input and its vertices and edges mappings.

input describes the data location, format and delimiter. Vertex and edge mappings define labels, IDs, properties and field conversions.

Loader 0.11.0 introduced mapping format 2.0; earlier mappings use format 1.0. This page uses 2.0 unless stated otherwise.

Click to expand/collapse the skeleton of the map file for version 2.0
 1{
 2  "version": "2.0",
 3  "structs": [
 4    {
 5      "id": "1",
 6      "input": {
 7      },
 8      "vertices": [
 9        {},
10        {}
11      ],
12      "edges": [
13        {},
14        {}
15      ]
16    }
17  ]
18}

Two versions of the mapping file are given directly here (the above graph model and data file are described)

Click to expand/collapse the mapping file for version 2.0
  1{
  2  "version": "2.0",
  3  "structs": [
  4    {
  5      "id": "1",
  6      "skip": false,
  7      "input": {
  8        "type": "FILE",
  9        "path": "vertex_person.csv",
 10        "file_filter": {
 11          "extensions": [
 12            "*"
 13          ]
 14        },
 15        "format": "CSV",
 16        "delimiter": ",",
 17        "date_format": "yyyy-MM-dd HH:mm:ss",
 18        "time_zone": "GMT+8",
 19        "skipped_line": {
 20          "regex": "(^#|^//).*|"
 21        },
 22        "compression": "NONE",
 23        "header": [
 24          "name",
 25          "age",
 26          "city"
 27        ],
 28        "charset": "UTF-8",
 29        "list_format": {
 30          "start_symbol": "[",
 31          "elem_delimiter": "|",
 32          "end_symbol": "]"
 33        }
 34      },
 35      "vertices": [
 36        {
 37          "label": "person",
 38          "skip": false,
 39          "id": null,
 40          "unfold": false,
 41          "field_mapping": {},
 42          "value_mapping": {},
 43          "selected": [],
 44          "ignored": [],
 45          "null_values": [
 46            ""
 47          ],
 48          "update_strategies": {}
 49        }
 50      ],
 51      "edges": []
 52    },
 53    {
 54      "id": "2",
 55      "skip": false,
 56      "input": {
 57        "type": "FILE",
 58        "path": "vertex_software.csv",
 59        "file_filter": {
 60          "extensions": [
 61            "*"
 62          ]
 63        },
 64        "format": "CSV",
 65        "delimiter": ",",
 66        "date_format": "yyyy-MM-dd HH:mm:ss",
 67        "time_zone": "GMT+8",
 68        "skipped_line": {
 69          "regex": "(^#|^//).*|"
 70        },
 71        "compression": "NONE",
 72        "header": null,
 73        "charset": "UTF-8",
 74        "list_format": {
 75          "start_symbol": "",
 76          "elem_delimiter": ",",
 77          "end_symbol": ""
 78        }
 79      },
 80      "vertices": [
 81        {
 82          "label": "software",
 83          "skip": false,
 84          "id": null,
 85          "unfold": false,
 86          "field_mapping": {},
 87          "value_mapping": {},
 88          "selected": [],
 89          "ignored": [],
 90          "null_values": [
 91            ""
 92          ],
 93          "update_strategies": {}
 94        }
 95      ],
 96      "edges": []
 97    },
 98    {
 99      "id": "3",
100      "skip": false,
101      "input": {
102        "type": "FILE",
103        "path": "edge_knows.json",
104        "file_filter": {
105          "extensions": [
106            "*"
107          ]
108        },
109        "format": "JSON",
110        "delimiter": null,
111        "date_format": "yyyy-MM-dd HH:mm:ss",
112        "time_zone": "GMT+8",
113        "skipped_line": {
114          "regex": "(^#|^//).*|"
115        },
116        "compression": "NONE",
117        "header": null,
118        "charset": "UTF-8",
119        "list_format": null
120      },
121      "vertices": [],
122      "edges": [
123        {
124          "label": "knows",
125          "skip": false,
126          "source": [
127            "source_name"
128          ],
129          "unfold_source": false,
130          "target": [
131            "target_name"
132          ],
133          "unfold_target": false,
134          "field_mapping": {
135            "source_name": "name",
136            "target_name": "name"
137          },
138          "value_mapping": {},
139          "selected": [],
140          "ignored": [],
141          "null_values": [
142            ""
143          ],
144          "update_strategies": {}
145        }
146      ]
147    },
148    {
149      "id": "4",
150      "skip": false,
151      "input": {
152        "type": "FILE",
153        "path": "edge_created.json",
154        "file_filter": {
155          "extensions": [
156            "*"
157          ]
158        },
159        "format": "JSON",
160        "delimiter": null,
161        "date_format": "yyyy-MM-dd HH:mm:ss",
162        "time_zone": "GMT+8",
163        "skipped_line": {
164          "regex": "(^#|^//).*|"
165        },
166        "compression": "NONE",
167        "header": null,
168        "charset": "UTF-8",
169        "list_format": null
170      },
171      "vertices": [],
172      "edges": [
173        {
174          "label": "created",
175          "skip": false,
176          "source": [
177            "source_name"
178          ],
179          "unfold_source": false,
180          "target": [
181            "target_name"
182          ],
183          "unfold_target": false,
184          "field_mapping": {
185            "source_name": "name",
186            "target_name": "name"
187          },
188          "value_mapping": {},
189          "selected": [],
190          "ignored": [],
191          "null_values": [
192            ""
193          ],
194          "update_strategies": {}
195        }
196      ]
197    }
198  ]
199}

Click to expand/collapse the mapping file for version 1.0
 1{
 2  "vertices": [
 3    {
 4      "label": "person",
 5      "input": {
 6        "type": "file",
 7        "path": "vertex_person.csv",
 8        "format": "CSV",
 9        "header": ["name", "age", "city"],
10        "charset": "UTF-8"
11      }
12    },
13    {
14      "label": "software",
15      "input": {
16        "type": "file",
17        "path": "vertex_software.csv",
18        "format": "CSV"
19      }
20    }
21  ],
22  "edges": [
23    {
24      "label": "knows",
25      "source": ["source_name"],
26      "target": ["target_name"],
27      "input": {
28        "type": "file",
29        "path": "edge_knows.json",
30        "format": "JSON"
31      },
32      "field_mapping": {
33        "source_name": "name",
34        "target_name": "name"
35      }
36    },
37    {
38      "label": "created",
39      "source": ["source_name"],
40      "target": ["target_name"],
41      "input": {
42        "type": "file",
43        "path": "edge_created.json",
44        "format": "JSON"
45      },
46      "field_mapping": {
47        "source_name": "name",
48        "target_name": "name"
49      }
50    }
51  ]
52}

The 1.0 version of the mapping file is centered on the vertex and edge, and sets the input source; while the 2.0 version is centered on the input source, and sets the vertex and edge mapping. Some input sources (such as a file) can generate both vertices and edges. If you write in the 1.0 format, you need to write an input block in each of the vertex and edge mapping blocks. The two input blocks are exactly the same; and the 2.0 version only needs to write input once. Therefore, compared with version 1.0, version 2.0 can save some repetitive writing of input.

In the bin directory of hugegraph-loader-{version}, there is a script tool mapping-convert.sh that can directly convert the mapping file of version 1.0 to version 2.0. The usage is as follows:

1bin/mapping-convert.sh struct.json

A struct-v2.json will be generated in the same directory as struct.json.

The bin directory also ships utf8-bom-to-utf8.sh, which strips the UTF-8 BOM from a single data file, or from every file under a directory. It is useful when a CSV or TEXT file exported by a Windows tool fails to parse because its first header column carries an invisible BOM:

1bin/utf8-bom-to-utf8.sh /path/to/file-or-dir
3.3.2 Input Source

Input sources are currently divided into five categories: FILE, HDFS, JDBC, KAFKA and GRAPH, which are distinguished by the type node. We call them local file input sources, HDFS input sources, JDBC input sources, KAFKA input sources and GRAPH input source, which are described below.

3.3.2.1 Local file input source
  • id: The id of the input source. This field is used to support some internal functions. It is not required (it will be automatically generated if it is not filled in). It is strongly recommended to write it, which is very helpful for debugging;
  • skip: whether to skip the input source, because the JSON file cannot add comments, if you do not want to import an input source during a certain import, but do not want to delete the configuration of the input source, you can set it to true to skip it, the default is false, not required;
  • input: input source map block, composite structure
    • type: an input source type, file or FILE must be filled;
    • path: the path of the local file or directory, an absolute path or a path relative to the Loader process working directory (not the mapping file). An absolute path is recommended; required;
    • file_filter: filter files with compound conditions from path, compound structure, currently only supports configuration extensions, represented by child node extensions, the default is “*”, which means to keep all files;
    • format: the format of the local file, the optional values are CSV, TEXT and JSON, which must be uppercase, the default is CSV, optional;
    • header: column names; if omitted, the first data line supplies them. With an explicit header, an identical first line is still skipped by default (see has_header). JSON does not require a header; optional;
    • has_header: for CSV and TEXT, the first line of every file is dropped when it is identical to the header, so a header repeated in each part file of a directory is not imported as data. Set this to false to turn that check off when the first line of a file is real data that happens to equal the header, optional;
    • delimiter: The column delimiter of the file line. The default depends on format: a comma "," for CSV and a tab "\t" for TEXT; a CSV file accepts no other delimiter. The JSON file does not need to be specified, optional;
    • charset: the encoded character set of the file, the default is UTF-8, optional;
    • date_format: custom date format, the default value is yyyy-MM-dd HH:mm:ss, optional; if the date is presented in the form of a timestamp, this item must be written as timestamp (fixed writing);
    • extra_date_formats: a list of fallback date formats, tried when a value does not match date_format, empty by default, optional;
    • time_zone: Set which time zone the date data is in, the default value is GMT+8, optional;
    • skipped_line: The line to be skipped, compound structure, currently only the regular expression of the line to be skipped can be configured, described by the child node regex. The default regex is (^#|^//).*|, which skips lines starting with # or // and empty lines; to keep such lines, set regex to a pattern that matches nothing, optional;
    • compression: The compression format of the file, the optional values ​​are NONE, GZIP, BZ2, XZ, LZMA, SNAPPY_RAW, SNAPPY_FRAMED, Z, DEFLATE, LZ4_BLOCK, LZ4_FRAMED, ORC and PARQUET, the default is NONE, which means a non-compressed file, optional; for ORC and PARQUET the header is matched case-insensitively;
    • list_format: When a column of the file (non-JSON) is a collection structure (the Cardinality of the PropertyKey in the corresponding figure is Set or List), you can use this item to set the start character, separator, and end character of the column, compound structure :
      • start_symbol: The start character of the collection structure column (the default value is the empty string "", JSON format currently does not support specification)
      • elem_delimiter: the delimiter of the collection structure column (the default value is |, and it must differ from delimiter; JSON format currently only supports native , delimiter)
      • end_symbol: the end character of the collection structure column (the default value is the empty string "", the JSON format does not currently support specification)
      • ignored_elems: the elements dropped after the column is split, the default value is [""], so empty elements are ignored
3.3.2.2 HDFS input source

The nodes and meanings of the above local file input source are basically applicable here. Only the different and unique nodes of the HDFS input source are listed below.

  • type: input source type, must fill in hdfs or HDFS, required;
  • path: the path of the HDFS file or directory, it must be the absolute path of HDFS, required;
  • core_site_path: the path of the core-site.xml file of the HDFS cluster, the key point is to specify the address of the NameNode (fs.default.name) and the implementation of the file system (fs.hdfs.impl), required;
  • hdfs_site_path: the path of the hdfs-site.xml file of the HDFS cluster, optional;
  • dir_filter: when path is a directory, decides which sub-directories are walked into, compound structure, optional:
    • include_regex: only directories whose name matches this regular expression are read, empty by default, which places no restriction;
    • exclude_regex: directories whose name matches this regular expression are skipped, empty by default;
  • kerberos_config: how to authenticate against a Kerberos-secured HDFS cluster, compound structure, optional:
    • enable: whether to authenticate with Kerberos, the default is false;
    • krb5_conf: the path of the krb5.conf file, required when enable is true;
    • principal: the Kerberos principal, required when enable is true;
    • keytab: the path of the keytab file, required when enable is true;
3.3.2.3 JDBC input source

As mentioned above, it supports multiple relational databases, but because their mapping structures are very similar, they are collectively referred to as JDBC input sources, and then use the vendor node to distinguish different databases.

  • type: input source type, must fill in jdbc or JDBC, required;
  • vendor: database type, optional options are [MySQL, PostgreSQL, Oracle, SQLServer], case-insensitive, required;
  • driver: the JDBC driver class, optional; when it is left out, the default driver of the vendor listed in the tables below is used;
  • url: the url of the database that jdbc wants to connect to, required;
  • database: the name of the database to be connected, required;
  • schema: The name of the schema to be connected, different databases have different requirements, and the details are explained below;
  • table: the name of the table to be connected, at least one of table or custom_sql is required;
  • custom_sql: custom SQL statement, at least one of table or custom_sql is required;
  • username: username to connect to the database, required;
  • password: password for connecting to the database, required;
  • where: an extra condition appended to the generated select statement, written without the where keyword, optional;
  • batch_size: The size of one page when obtaining table data by page, the default is 500, optional;

MYSQL

NodeFixed value or common value
vendorMYSQL
drivercom.mysql.cj.jdbc.Driver
urljdbc:mysql://127.0.0.1:3306

schema: nullable, if filled in, it must be the same as the value of database

POSTGRESQL

NodeFixed value or common value
vendorPOSTGRESQL
driverorg.postgresql.Driver
urljdbc:postgresql://127.0.0.1:5432

schema: nullable, default is “public”

ORACLE

NodeFixed value or common value
vendorORACLE
driveroracle.jdbc.driver.OracleDriver
urljdbc:oracle:thin:@127.0.0.1:1521

schema: nullable, the default value is the username in upper case

SQLSERVER

NodeFixed value or common value
vendorSQLSERVER
drivercom.microsoft.sqlserver.jdbc.SQLServerDriver
urljdbc:sqlserver://127.0.0.1:1433

schema: required

3.3.2.4 Kafka input source
  • type: input source type, kafka or KAFKA, required;
  • bootstrap_server: the list of kafka bootstrap servers, required;
  • topic: the topic to subscribe to, required;
  • group: group of Kafka consumers, required;
  • from_beginning: sets auto.offset.reset to earliest when true, or latest when false (default). It applies only when no valid committed offset exists; otherwise consumption resumes from the committed offset. Optional;
  • format: format of each message, options are CSV, TEXT and JSON, must be uppercase, required;
  • header: column name of each column of a message; no header line is read from the topic, so it has to be given for CSV and TEXT, while JSON messages do not need it;
  • delimiter: delimiter of the message columns, used by TEXT only, since CSV always splits on ,, optional;
  • charset: encoding charset of the messages, default is UTF-8, optional;
  • date_format: customized date format, default value is yyyy-MM-dd HH:mm:ss, optional; if the date is presented in the form of timestamp, this item must be written as timestamp (fixed);
  • extra_date_formats: a customized list of another date formats, empty by default, optional; each item in the list is an alternate date format to the date_format specified date format;
  • time_zone: set which time zone the date data is in, default is GMT+8, optional;
  • skipped_line: current master KafkaReader does not implement this filter. Kafka messages go directly to the parser, so configuring this field does not skip blank or comment messages; preprocess them upstream or in the consumer to avoid parse failures;
  • batch_size: the maximum number of records fetched in one poll (max.poll.records), default is 500, optional;
  • early_stop: the record pulled from Kafka broker at a certain time is empty, stop the task, default is false, only for debugging, optional;
3.3.2.5 GRAPH input Source

The GRAPH input source reads vertices and edges out of another HugeGraph graph, reached through HugeGraph-PD, and writes them into the target graph. When a mapping file contains a GRAPH input source, every input source in it that is not skipped has to be a GRAPH input source as well, and the loader puts the target graph into RESTORING mode for the duration of the import.

  • type: Data source type; must be filled in as graph or GRAPH (required);
  • graphspace: Source graphSpace name (required);
  • graph: Source graph name (required);
  • username: HugeGraph username; the --username command-line option is used when this is empty;
  • password: HugeGraph password; the --password command-line option is used when this is empty;
  • selected_vertices: the vertex labels to copy, each item written as {"label": "...", "properties": [...], "query": {...}}, where properties narrows the copied properties and query is an optional filter passed to the source graph;
  • ignored_vertices: the vertex labels to skip, each item written as {"label": "...", "properties": [...]};
  • selected_edges: the edge labels to copy, items have the same shape as in selected_vertices;
  • ignored_edges: the edge labels to skip, items have the same shape as in ignored_vertices;
  • pd-peers: HugeGraph-PD node addresses of the source cluster; the --pd-peers option is used when this is empty;
  • meta-endpoints: Meta service endpoints of the source cluster; the --meta-endpoints option is used when this is empty;
  • cluster: Source cluster name; the --cluster option is used when this is empty;
  • batch_size: Batch size for reading data from the source graph; default is 500;
3.3.3 Vertex and Edge Mapping

The nodes of vertex and edge mapping (a key in the JSON file) have a lot of the same parts. The same parts are introduced first, and then the unique nodes of vertex map and edge map are introduced respectively.

Nodes of the same section

  • label: label to which the vertex/edge data to be imported belongs, required;
  • skip: whether to skip this vertex/edge mapping while the input source and the other mappings stay active, the default is false, optional;
  • field_mapping: Map the column name of the input source column to the attribute name of the vertex/edge, optional;
  • value_mapping: map the data value of the input source to the attribute value of the vertex/edge, optional;
  • selected: select some columns to insert, other unselected ones are not inserted, cannot exist at the same time as ignored, optional;
  • ignored: ignore some columns so that they do not participate in insertion, cannot exist at the same time as selected, optional;
  • null_values: You can specify some strings to represent null values, such as “NULL”. If the vertex/edge attribute corresponding to this column is also a nullable attribute, the value of this attribute will not be set when constructing the vertex/edge, optional ;
  • update_strategies: If the data needs to be updated in batches in a specific way, you can specify a specific update strategy for each attribute (see below for details), optional;
  • unfold: Whether to unfold the column, each unfolded column will form a row with other columns, which is equivalent to unfolding into multiple rows; for example, the value of a certain column (id column) of the file is [1,2,3], The values ​​of other columns are 18,Beijing. When unfold is set, this row will become 3 rows, namely: 1,18,Beijing, 2,18,Beijing and 3,18, Beijing. Note that this will only expand the column selected as id. Default false, optional;

Update strategy supports 8 types: (requires all uppercase)

  1. Value accumulation: SUM
  2. Take the greater of the two numbers/dates: BIGGER
  3. Take the smaller of two numbers/dates: SMALLER
  4. Set property takes union: UNION
  5. Set attribute intersection: INTERSECTION
  6. List attribute append element: APPEND
  7. List/Set attribute delete element: ELIMINATE
  8. Override an existing property: OVERRIDE

Note: If the newly imported attribute value is empty, the existing old data will be used instead of the empty value. For the effect, please refer to the following example

 1// The update strategy is specified in the JSON file as follows
 2{
 3  "vertices": [
 4    {
 5      "label": "person",
 6      "update_strategies": {
 7        "age": "SMALLER",
 8        "set": "UNION"
 9      },
10      "input": {
11        "type": "file",
12        "path": "vertex_person.txt",
13        "format": "TEXT",
14        "header": ["name", "age", "set"]
15      }
16    }
17  ]
18}
19
20// 1. Write a line of data with the OVERRIDE update strategy (null means empty here)
21'a b null null'
22
23// 2. Write another line
24'null null c d'
25
26// 3. Finally we can get
27'a b c d'   
28
29// If there is no update strategy, you will get
30'null null c d'

Note : After adopting the batch update strategy, the number of disk read requests will increase significantly, and the import speed will be several times slower than that of pure write coverage (at this time HDD disk [IOPS](https://en.wikipedia .org/wiki/IOPS) will be the bottleneck, SSD is recommended for speed)

Unique Nodes for Vertex Maps

  • id: Specify a column as the id column of the vertex. When the vertex id policy is CUSTOMIZE, it is required; when the id policy is PRIMARY_KEY, it must be empty;

Unique Nodes for Edge Maps

  • source: Select certain columns of the input source as the id column of source vertex. When the id policy of the source vertex is CUSTOMIZE, a certain column must be specified as the id column of the vertex; when the id policy of the source vertex is When PRIMARY_KEY, one or more columns must be specified for splicing the id of the generated vertex, that is, no matter which id strategy is used, this item is required;
  • target: Specify certain columns as the id columns of target vertex, similar to source, so I won’t repeat them;
  • unfold_source: Whether to unfold the source column of the file, the effect is similar to that in the vertex map, and will not be repeated;
  • unfold_target: Whether to unfold the target column of the file, the effect is similar to that in the vertex mapping, and will not be repeated;

3.4 Execute command import

With the schema, data and mapping ready, run Loader with the options below.

3.4.1 Parameter description
ParameterDefault valueRequired or notDescription
-f or --fileYPath to configure script
-g or --graphhugegraphGraph name
--graphspaceDEFAULTGraph space name
-s or --schemaSchema file path; optional when the Schema already exists
-h or --host or -ilocalhostAddress of HugeGraphServer
-p or --port8080Port number of HugeGraphServer
--usernamenullWhen HugeGraphServer enables permission authentication, the username of the current graph
--passwordnullWhen HugeGraphServer enables permission authentication, the password of the current graph
--create-graphfalseWhether to automatically create the graph if it does not exist
--tokennullWhen HugeGraphServer has enabled authorization authentication, the token of the current graph
--protocolhttpProtocol for sending requests to the server, optional http or https
--pd-peersPD service node addresses
--pd-tokenToken for accessing PD service
--meta-endpointsMeta information storage service addresses
--directfalseStore direct mode is disabled in current master; imports still use Server API. Keep the default
--route-typeNODE_PORTRoute selection method (optional values: NODE_PORT / DDS / BOTH)
--clusterhgCluster name
--trust-store-fileWhen the request protocol is https, the client’s certificate file path
--trust-store-passwordWhen the request protocol is https, the client certificate password
--clear-all-datafalseWhether to clear the original data on the server before importing data
--clear-timeout240Timeout for clearing the original data on the server before importing data
--incremental-modefalseWhether to use the breakpoint resume mode; only input sources FILE and HDFS support this mode. Enabling this mode allows starting the import from where the last import stopped
--failure-modefalseWhen failure mode is true, previously failed data will be imported. Generally, the failed data file needs to be manually corrected and edited before re-importing
--batch-insert-threadsCPUsBatch insert thread pool size (CPUs is the number of logical cores available to the current OS)
--single-insert-threads8Size of single insert thread pool
--max-conn4 * CPUsThe maximum number of HTTP connections between HugeClient and HugeGraphServer; while it is left at its default, it is raised automatically to 4 * --batch-insert-threads
--max-conn-per-route2 * CPUsThe maximum number of HTTP connections for each route between HugeClient and HugeGraphServer; while it is left at its default, it is raised automatically to 2 * --batch-insert-threads
--batch-size500The number of data items in each batch when importing data
--max-parse-errors1The maximum number of data parsing errors allowed (per line); the program exits when this value is reached
--max-insert-errors500The maximum number of data insertion errors allowed (per row); the program exits when this value is reached
--timeout60Timeout (seconds) for insert result return
--shutdown-timeout10Waiting time for multithreading to stop (seconds)
--retry-times3Maximum number of retries after a timeout
--retry-interval10Interval before retry (seconds)
--check-vertexfalseWhether to check if the vertices connected by the edge exist when inserting the edge
--print-progresstrueWhether to print the number of imported items in real time on the console
--dry-runfalseEnable this mode to only parse data without importing; usually used for testing
--help or -helpfalsePrint help information
--parser-threads or --parallel-countmax(2,CPUs/2)Number of parallel read pipelines; --parallel-count is deprecated
--start-file0Start file index for partial loading
--end-file-1End file index for partial loading
--scatter-sourcesfalseScatter multiple sources for I/O optimization
--cdc-flush-interval30000The flush interval for Flink CDC
--cdc-sink-parallelism1The sink parallelism for Flink CDC
--max-read-errors1The maximum number of read error lines before exiting
--max-read-lines-1LThe maximum number of read lines, task stops when reached
--test-modefalseWhether the loader works in test mode
--use-prefilterfalseWhether to filter vertex in advance
--short-idMap a customized vertex ID to a shorter generated ID, written as label:field:type, where type is one of boolean, byte, int, long, float, double, text, blob, date and uuid; repeat the option to cover several labels
--vertex-edge-limit-1LThe maximum number of vertex’s edges
--sink-typetruespark-loader only: true writes through the HugeGraph server API, false generates HFiles and bulk-loads them into HBase
--vertex-partitions64The number of partitions of the HBase vertex table, used with --sink-type false
--edge-partitions64The number of partitions of the HBase edge table, used with --sink-type false
--vertex-table-nameHBase vertex table name, used with --sink-type false
--edge-table-nameHBase edge table name, used with --sink-type false
--hbase-zk-quorumHBase ZooKeeper quorum, used with --sink-type false
--hbase-zk-portHBase ZooKeeper port, used with --sink-type false
--hbase-zk-parentHBase ZooKeeper parent, used with --sink-type false
--restorefalseSet graph mode to RESTORING
--backendhstoreThe backend store type when creating graph if not exists
--serializerbinaryThe serializer type when creating graph if not exists
--scheduler-typedistributedThe task scheduler type when creating graph if not exists
--batch-failure-fallbacktrueWhether to fallback to single insert when batch insert fails

The mapping file option --file (-f) is required; other options use their defaults. A valid minimal invocation is:

1bin/hugegraph-loader.sh --file struct.json

The launcher also accepts JCommander’s native @file syntax. Put each option and value on separate lines in loader.args:

1--file
2struct.json
1bin/hugegraph-loader.sh @loader.args

This is equivalent to the minimal invocation above. Relative path values resolve from the Loader process working directory. If the argument file contains credentials, restrict it to mode 0600 with chmod 0600 loader.args. The @file form keeps credential values out of process arguments. Loader must be able to create and write the mapping-prefix directory used for progress output. Copy a mapping mounted from a read-only Kubernetes ConfigMap to a writable volume before running Loader.

3.4.2 Breakpoint Continuation Mode

For a long import, use a progress file to resume after an interruption.

Set --incremental-mode to true to resume from a progress file. Loader records progress when it closes normally after a successful import or a normal stop; an unexpected failure may prevent this write. The progress file is in ${struct} and named load-progress_${timestamp}, where ${struct} is the mapping-file prefix and ${timestamp} is the import start time, formatted as yyyyMMdd-HHmmss. For example, an import started at 2019-10-10 12:30:30 with mapping file struct-example.json writes struct-example/load-progress_20191010-123030. When the directory holds several progress files, the resumed import reads the last one in name order, which is the most recent one.

Note: A progress file is written after a successful import or a normal stop (including an error limit), regardless of --incremental-mode or --print-progress. An import aborted by an unexpected failure may not write one.

After stopping an import with Ctrl+C or kill, set --incremental-mode true to continue from the recorded position. A forced stop with kill -9 cannot save progress.

Invalid data or connection failures can exhaust --max-read-errors, --max-parse-errors or --max-insert-errors. Correct the rows in the failure files, then use --failure-mode true to retry them. This leaves the original data files unchanged. Rows that still fail are recorded again and deduplicated when the failure file closes. Failure mode lifts these three limits and scans the entire file.

Each input source, that is each item of structs in the mapping file, gets its own failure file. The file is named after the id of that input source with the suffix .error and is stored in the ${struct}/failure-data directory. Every failed line is written as a pair of lines: a tip line starting with #### READ ERROR:, #### PARSE ERROR: or #### INSERT ERROR:, followed by the original data line. When the input source has a header, that header is written as JSON to a sibling ${id}.header file, so the failure file can be read back with the right columns. For example, if the mapping file has an input source with id 1 holding a vertex mapping person and an input source with id 3 holding an edge mapping knows, each of which has some error lines, you will see the following files in the ${struct}/failure-data directory when the Loader exits:

  • 1.error: the failed lines of input source 1, each preceded by its tip line
  • 1.header: the header of input source 1, written only when that input source has a header
  • 3.error: the failed lines of input source 3
  • 3.header: the header of input source 3

A .error file that turns out to be empty is deleted when the Loader exits, so only the input sources that really had failed lines leave a file behind. In incremental mode new failures are appended to the existing file, otherwise the file is rewritten from scratch.

3.4.3 logs directory file description

The log and error data during program execution will be written into the hugegraph-loader.log file.

3.4.4 Execute command

Run bin/hugegraph-loader.sh and pass in parameters

1bin/hugegraph-loader.sh -g {GRAPH_NAME} -f ${INPUT_DESC_FILE} -s ${SCHEMA_FILE} -h {HOST} -p {PORT}

The script runs the JVM under JAVA_HOME when that variable is set, and java from the PATH otherwise. It passes the contents of the JVM_OPTS environment variable, then -Xmx10g and the class path built from lib/, to that JVM, so JVM_OPTS is the place to add JVM flags. Logging is configured by conf/log4j2.xml.

3.4.5 Minimal Local-File Import

Follow the local-file import steps at the start of this page.

3.4.6 Concurrent imports and LOADING ownership

In the master branch, Loader implements a LOADING ownership protocol coordinated with Server’s /mode/owners endpoint. When the deployed Server provides this endpoint and returns protocol: 1, Loader enables ownership protection automatically; released Server 1.7.0 and older versions continue using the legacy mode switching flow.

graph TD
    A["Graph Mode: NONE"] -->|First Loader acquires owner| B["LOADING Mode (Acquires Token)"]
    B -->|Concurrent Loaders join| B
    B -->|Loader finishes & releases| B
    B -->|Last Loader releases| C["Restores to NONE"]
    B -.->|Abnormal exit / kill -9| D["Manual Release / Recovery / Restart"]
    D -.->|"Last owner released, or recover/restart"| A
    D -.->|"Other owners still active"| B
  1. Probe and Acquire: Loader probes /mode/owners. The first Loader obtains an ownership handle (instance/epoch/token), transitioning graph mode from NONE to LOADING; concurrent Loaders can join.
  2. Data Import: Loaders holding valid handles execute concurrent imports while preserving LOADING mode.
  3. Graceful Release: When an import completes, Loader drains its workers and releases its handle. When the last owner releases, graph mode automatically returns to NONE.
  4. Failure Recovery: If abnormal exit (e.g. kill -9) leaves an orphaned owner, stop and drain all related imports and graph writes, then perform administrator recovery via the RESTful API recovery endpoint or restart the Server. Manual release applies only when the caller still holds its token (e.g. a direct REST client).
Note

Compatibility and Operational Notes

  • Legacy Compatibility: During ordinary imports, only an unsupported-route 404 status on /mode/owners falls back to the legacy enum LOADING/NONE flow (maintains compatibility with Server 1.7, without multi-loader protection). Network or permission errors, or a 404 for a missing graph, do not trigger fallback.
  • Restore and Graph Source Imports: Retain the RESTORING flow without ownership probes or automatic resets. After a restore import completes, manually reset graph mode to NONE before starting ordinary imports. Do not run restore, graph-source, Spark connector, or older Loader imports while owned imports are active.
  • Scope & Lifecycle: Ownership is bound to the target Server instance without cross-node synchronization or automatic lease expiry. For embedded programmatic integrations, callers must invoke HugeGraphLoader.shutdown() to attempt a single graceful release. If release fails, the handle is not retried; the owner may remain active on the Server, and the administrator recovery step above may be required.

4 Complete example

The Loader distribution includes this dataset in example/file/. See the source files.

4.1 Prepare data

Vertex file: example/file/vertex_person.csv

1marko,29,Beijing
2vadas,27,Hongkong
3josh,32,Beijing
4peter,35,Shanghai
5"li,nary",26,"Wu,han"
6tom,null,NULL

Vertex file: example/file/vertex_software.txt

1id|name|lang|price|ISBN
21|lop|java|328|ISBN978-7-107-18618-5
32|ripple|java|199|ISBN978-7-100-13678-5

Edge file: example/file/edge_knows.json

1{"source_name": "marko", "target_name": "vadas", "date": "20160110", "weight": 0.5}
2{"source_name": "marko", "target_name": "josh", "date": "20130220", "weight": 1.0}

Edge file: example/file/edge_created.json

1{"source_name": "marko", "target_id": 1, "date": "2017-12-10", "weight": 0.4}
2{"source_name": "josh", "target_id": 1, "date": "2009-11-11", "weight": 0.4}
3{"source_name": "josh", "target_id": 2, "date": "2017-12-10", "weight": 1.0}
4{"source_name": "peter", "target_id": 1, "date": "2017-03-24", "weight": 0.2}

4.2 Write schema

Click to expand/collapse the schema file: example/file/schema.groovy
 1schema.propertyKey("name").asText().ifNotExist().create();
 2schema.propertyKey("age").asInt().ifNotExist().create();
 3schema.propertyKey("city").asText().ifNotExist().create();
 4schema.propertyKey("weight").asDouble().ifNotExist().create();
 5schema.propertyKey("lang").asText().ifNotExist().create();
 6schema.propertyKey("date").asText().ifNotExist().create();
 7schema.propertyKey("price").asDouble().ifNotExist().create();
 8
 9schema.vertexLabel("person").properties("name", "age", "city").primaryKeys("name").nullableKeys("age", "city").ifNotExist().create();
10schema.vertexLabel("software").useCustomizeNumberId().properties("name", "lang", "price").ifNotExist().create();
11
12schema.indexLabel("personByAge").onV("person").by("age").range().ifNotExist().create();
13schema.indexLabel("personByCity").onV("person").by("city").secondary().ifNotExist().create();
14schema.indexLabel("personByAgeAndCity").onV("person").by("age", "city").secondary().ifNotExist().create();
15schema.indexLabel("softwareByPrice").onV("software").by("price").range().ifNotExist().create();
16
17schema.edgeLabel("knows").sourceLabel("person").targetLabel("person").properties("date", "weight").ifNotExist().create();
18schema.edgeLabel("created").sourceLabel("person").targetLabel("software").properties("date", "weight").ifNotExist().create();
19
20schema.indexLabel("createdByDate").onE("created").by("date").secondary().ifNotExist().create();
21schema.indexLabel("createdByWeight").onE("created").by("weight").range().ifNotExist().create();
22schema.indexLabel("knowsByWeight").onE("knows").by("weight").range().ifNotExist().create();

person uses name as its primary key. Tom’s age and city are removed by null_values, so both properties must be nullable. software uses custom numeric IDs: mapping id supplies the vertex ID and created.target_id supplies the edge’s target ID. Run this example in an empty graph without these schema labels; ifNotExist() does not replace incompatible existing definitions.

4.3 Write the input source mapping file example/file/struct.json

Click to expand/collapse the input source mapping file example/file/struct.json
 1{
 2  "vertices": [
 3    {
 4      "label": "person",
 5      "input": {
 6        "type": "file",
 7        "path": "example/file/vertex_person.csv",
 8        "format": "CSV",
 9        "header": ["name", "age", "city"],
10        "charset": "UTF-8",
11        "skipped_line": {
12          "regex": "(^#|^//).*"
13        }
14      },
15      "null_values": ["NULL", "null", ""]
16    },
17    {
18      "label": "software",
19      "input": {
20        "type": "file",
21        "path": "example/file/vertex_software.txt",
22        "format": "TEXT",
23        "delimiter": "|",
24        "charset": "GBK"
25      },
26      "id": "id",
27      "ignored": ["ISBN"]
28    }
29  ],
30  "edges": [
31    {
32      "label": "knows",
33      "source": ["source_name"],
34      "target": ["target_name"],
35      "input": {
36        "type": "file",
37        "path": "example/file/edge_knows.json",
38        "format": "JSON",
39        "date_format": "yyyyMMdd"
40      },
41      "field_mapping": {
42        "source_name": "name",
43        "target_name": "name"
44      }
45    },
46    {
47      "label": "created",
48      "source": ["source_name"],
49      "target": ["target_id"],
50      "input": {
51        "type": "file",
52        "path": "example/file/edge_created.json",
53        "format": "JSON",
54        "date_format": "yyyy-MM-dd"
55      },
56      "field_mapping": {
57        "source_name": "name"
58      }
59    }
60  ]
61}

4.4 Command to import

1sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy

After the import is complete, statistics similar to the following will appear:

 1vertices/edges has been loaded this time : 8/6
 2--------------------------------------------------
 3count metrics
 4     input read success            : 14
 5     input read failure            : 0
 6     vertex parse success          : 8
 7     vertex parse failure          : 0
 8     vertex insert success         : 8
 9     vertex insert failure         : 0
10     edge parse success            : 6
11     edge parse failure            : 0
12     edge insert success           : 6
13     edge insert failure           : 0

4.5 Use Docker to load data

4.5.1 Use docker exec to load data directly
4.5.1.1 Prepare data

For a first import, use the built-in example dataset.

For custom data, first copy the files into the Loader container.

Prepare the data, schema and mapping as described in sections 4.1 to 4.3, then copy them with docker cp.

The following example stores the files in hugegraph-dataset.

1tree -f hugegraph-dataset/
2
3hugegraph-dataset
4├── hugegraph-dataset/edge_created.json
5├── hugegraph-dataset/edge_knows.json
6├── hugegraph-dataset/schema.groovy
7├── hugegraph-dataset/struct.json
8├── hugegraph-dataset/vertex_person.csv
9└── hugegraph-dataset/vertex_software.txt

Copy the files into the container.

1docker cp hugegraph-dataset loader:/loader/dataset
2docker exec -it loader ls /loader/dataset
3
4edge_created.json  edge_knows.json  schema.groovy  struct.json  vertex_person.csv  vertex_software.txt
4.5.1.2 Data loading

Run this command to import the built-in dataset.

For custom data, set -f to your mapping file and -s to your schema file.

You can refer to 3.4.1-Parameter description for the rest of the parameters.

1docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h server -p 8080

If loading a custom dataset, following the previous example, you would use:

1docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f /loader/dataset/struct.json -s /loader/dataset/schema.groovy -h server -p 8080

Also update every input.path in the custom struct.json to its actual container path, such as /loader/dataset/vertex_person.csv. Changing only -f and -s does not change where Loader looks for data files.

If loader and server are in the same Docker network, you can specify -h {server_container_name}; otherwise, you need to specify the IP of the server host (in our example, server_container_name is server).

Then we can see the result:

 1HugeGraphLoader worked in NORMAL MODE
 2vertices/edges loaded this time : 8/6
 3--------------------------------------------------
 4count metrics
 5    input read success            : 14                  
 6    input read failure            : 0                   
 7    vertex parse success          : 8                   
 8    vertex parse failure          : 0                   
 9    vertex insert success         : 8                   
10    vertex insert failure         : 0                   
11    edge parse success            : 6                   
12    edge parse failure            : 0                   
13    edge insert success           : 6                   
14    edge insert failure           : 0                   
15--------------------------------------------------
16meter metrics
17    total time                    : 0.199s              
18    read time                     : 0.046s              
19    load time                     : 0.153s              
20    vertex load time              : 0.077s              
21    vertex load rate(vertices/s)  : 103                 
22    edge load time                : 0.112s              
23    edge load rate(edges/s)       : 53   

Query the imported data with Hubble or curl.

These requests use the GraphSpace-aware Server API. For an older Server, omit /graphspaces/DEFAULT from the path.

1curl --compressed "http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/graph/vertices"
2{"vertices":[{"id":1,"label":"software","type":"vertex","properties":{"name":"lop","lang":"java","price":328.0}},{"id":2,"label":"software","type":"vertex","properties":{"name":"ripple","lang":"java","price":199.0}},{"id":"1:tom","label":"person","type":"vertex","properties":{"name":"tom"}},{"id":"1:josh","label":"person","type":"vertex","properties":{"name":"josh","age":32,"city":"Beijing"}},{"id":"1:marko","label":"person","type":"vertex","properties":{"name":"marko","age":29,"city":"Beijing"}},{"id":"1:peter","label":"person","type":"vertex","properties":{"name":"peter","age":35,"city":"Shanghai"}},{"id":"1:vadas","label":"person","type":"vertex","properties":{"name":"vadas","age":27,"city":"Hongkong"}},{"id":"1:li,nary","label":"person","type":"vertex","properties":{"name":"li,nary","age":26,"city":"Wu,han"}}]}

If you want to check the import result of edges, you can use curl --compressed "http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/graph/edges".

4.5.2 Enter the docker container to load data

You can also open a shell in the container and run the same import commands.

Open the shell with docker exec -it loader bash, then run the import.

1sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h server -p 8080

The results of the execution will be similar to those shown in 4.5.1.

4.6 Import data by spark-loader

The current master API sink uses Spark 3.5.8, Scala 2.12 and Java 17 on the driver and executors. The POM’s Spark 3.2.2 dependency supplies compile-time APIs; it is not the runtime engine. Set SPARK_HOME and JAVA_HOME, then use the distribution’s shaded Loader JAR with its launcher.

Create the schema first and make data files readable by the executors. CSV and TEXT inputs need headerless UTF-8 data and an explicit mapping header. Other charsets and has_header: true are rejected. Load vertex data before edge data when the edges depend on those vertices. Replace the mapping path below with your prepared vertex mapping.

1bin/hugegraph-spark-loader.sh \
2  --master 'local[2]' --driver-memory 1g \
3  --file /path/to/vertex-mapping.json \
4  --host 127.0.0.1 --port 8080 --graph hugegraph --sink-type true

Pass Spark options such as --master, --conf and --jars to the launcher alongside Loader options. Additional JDBC drivers need explicit --jars; copying them into lib/ does not distribute them to executors. Query the target graph to verify counts and properties; the API sink’s insertion counters do not report distributed writes.

This path rejects --incremental-mode true and --failure-mode true. Standalone Spark supports client deployment only. HBase bulkload (--sink-type false) is unsupported on Spark 3.5.8 / Java 17; keep an existing bulkload deployment on its separately validated released Loader/JVM/Spark/HBase environment. For details, see the Spark migration guide.

The current master integration targets Flink 2.2.1, MySQL CDC 3.6.0-2.2 and Java 17. Older examples using Flink 1.13.5 and CDC 2.2.1 belong to the previous implementation. Set FLINK_HOME; the submission JVM, JobManager and TaskManager must all use Java 17. Keep Flink’s module-access options and exactly one shaded Loader JAR in the distribution’s lib/.

Each mapping needs a MySQL JDBC input with url, database, table, username and password. Set MySQL binlog_row_image=FULL. UPDATE events require explicit, nonempty update_strategies and unchanged graph IDs; changes to vertex IDs, edge endpoints or sort keys are rejected. CDC also rejects --dry-run true and --use-prefilter true.

Without a separator, all arguments go to Loader. To set Flink options, put them before -- and Loader options after it. For a prepared mapping at mysql-cdc.json, submit with one sink writer.

1bin/hugegraph-flinkcdc-loader.sh -p 1 -- \
2  --file ./mysql-cdc.json --host 127.0.0.1 --port 8080 \
3  --graph hugegraph --cdc-sink-parallelism 1

Each mapping uses a single source reader and sink writer. --cdc-sink-parallelism defaults to 1; other values are rejected. After --, Loader’s -p means the Server port. Checkpoint flushing does not provide transactional graph writes or an exactly-once guarantee. Validate edge CDC and broader mappings in your test environment before deploying to production. For more details, see the Flink CDC documentation.

2.5 - Graph export and migration

Choose an export or migration tool when you need to back up, export, or move data between graphs. Tools is suited to standalone operations and backups; SeaTunnel Source connects graph reads to an expandable data pipeline.

2.5.1 - Export and Migrate Graph Data with SeaTunnel Source

If you need to copy data from one HugeGraph graph to another, use graph2graph: HugeGraph Source reads vertices and edges from the source graph (graph A), and HugeGraph Sink writes them to the target graph (graph B), with an optional Transform in between. The data path is graph A → HugeGraph Source → (optional Transform) → HugeGraph Sink → graph B. If you need to export graph data to a file, JDBC, Kafka, or another system, use graph2any, where a downstream Sink receives the records read by HugeGraph Source. This page covers both job types.

Version requirement: This guide targets SeaTunnel 3.0+

SeaTunnel 3.0+ provides HugeGraph Source, with schema auto-discovery and multi-label reads. See the official Source documentation for parallel-scan backend requirements and configuration limits.

Before starting, complete the shared environment and configuration steps on the import page. They cover JDK, HOCON, plugin installation, and the sample graph model.

With HugeGraph Helm, follow the SeaTunnel deployment guide to run a persistent standalone engine or cluster with pre-mounted Source/Sink examples. Addresses and credentials for a remote source graph still require separate configuration.

1 Migrate a HugeGraph graph (graph2graph)

The following example migrates person vertices and knows edges from a source graph. Use a separate target graph. This section uses CUSTOMIZE_STRING to preserve vertex IDs. Do not reuse the person label created earlier with PRIMARY_KEY.

These two jobs migrate only the selected labels and properties. They do not copy every source schema setting, such as indexes and TTLs. Pause writes to the source graph during the migration so both jobs read a consistent point in time. Afterward, compare vertex and edge counts and sample properties.

To retain schema IDs and index definitions for ordinary edge labels, see the Chart’s schema copy and per-label examples. The helper rejects inherited PARENT/SUB edge labels before accessing the target; it does not migrate inheritance automatically.

Graph Migration Sample Layout

  • helm/hugegraph/examples/seatunnel/clone/
    • copy-schema.pyCopies schema from source to target graph (preserves IDs & indexes)
    • person.confVertex data migration job (CUSTOMIZE_STRING preserves IDs)
    • knows.confEdge data migration job (depends on vertex ID consistency)
Warning

Check the source Server’s timezone for DATE properties. REST date strings read by SeaTunnel 3.0.0 have no timezone. Set timeZone inside every Sink mappings entry to the source Server’s timezone, for example timeZone = "UTC". Do not rely on the worker default: Source time_zone does not correct these strings. The Chart’s copy examples assume a UTC source Server. Adjust non-UTC sources and compare the original timestamps.

Regenerating a primary key can change 1:marko to 2:marko; CUSTOMIZE_STRING preserves the original ID so edge endpoints still resolve

1.1 Migrate vertices first

Source adds a ~id column for the original ID, and Sink stores it as a string. Do not declare ~id in schema.fields; manually declaring this reserved column is rejected.

Expand the configuration and save it as config/graph2graph-person.conf
 1env {
 2  job.mode = "BATCH"
 3}
 4
 5source {
 6  HugeGraph {
 7    host = "source-hugegraph"
 8    port = 8080
 9    graph_name = "hugegraph"
10    graph_space = "DEFAULT"
11    label = "person"
12    label_type = "VERTEX"
13    schema = {
14      fields = {
15        name = "string"
16        age = "int"
17      }
18    }
19  }
20}
21
22sink {
23  HugeGraph {
24    host = "target-hugegraph"
25    port = 8080
26    graph_name = "hugegraph"
27    graph_space = "DEFAULT"
28    batch_failure_fallback = false
29    mappings = [
30      {
31        type = "VERTEX"
32        label = "person"
33        idStrategy = "CUSTOMIZE_STRING"
34        idFields = ["~id"]
35        properties = ["name", "age"]
36      }
37    ]
38  }
39}
1./bin/seatunnel.sh --config ./config/graph2graph-person.conf -m local

1.2 Migrate edges second

After the vertex job succeeds, use the ~source_id and ~target_id columns added by Source to locate endpoints. Because the previous job preserved the original IDs, these columns can refer directly to vertices in the target graph.

Expand the configuration and save it as config/graph2graph-knows.conf
 1env {
 2  job.mode = "BATCH"
 3}
 4
 5source {
 6  HugeGraph {
 7    host = "source-hugegraph"
 8    port = 8080
 9    graph_name = "hugegraph"
10    graph_space = "DEFAULT"
11    label = "knows"
12    label_type = "EDGE"
13    schema = {
14      fields = {
15        since = "int"
16      }
17    }
18  }
19}
20
21sink {
22  HugeGraph {
23    host = "target-hugegraph"
24    port = 8080
25    graph_name = "hugegraph"
26    graph_space = "DEFAULT"
27    check_vertex = true
28    batch_failure_fallback = false
29    mappings = [
30      {
31        type = "EDGE"
32        label = "knows"
33        sourceConfig = {
34          label = "person"
35          idFields = ["~source_id"]
36        }
37        targetConfig = {
38          label = "person"
39          idFields = ["~target_id"]
40        }
41        properties = ["since"]
42      }
43    ]
44  }
45}
1./bin/seatunnel.sh --config ./config/graph2graph-knows.conf -m local

This example checks endpoints and makes write errors fail the job. The default check_vertex = false does not guarantee a consistent result: a missing endpoint can create a dangling edge, so a successful job is not a substitute for checking the migrated graph.

Why preserve IDs? A HugeGraph PRIMARY_KEY ID contains the internal ID of the vertex label, and that internal ID can differ between graphs. For example, a source vertex can be 1:marko, while regenerating the primary key in the target graph can produce 2:marko. Reusing the source edge endpoints after regenerating vertex IDs can connect edges to the wrong vertices. This example stores the original ID as a string, which changes the target graph’s ID strategy

When Source reads every label, omit label to read all labels of label_type (default VERTEX). It produces one output table per label. Bind each Sink mapping to its table with sourceTable, for example sourceTable = "default.person"; use the full table name shown in the Writer log for the exact value. Do not reuse the single-label configuration from this section. See the HugeGraph Source documentation for other limitations.

2 Export to another system (graph2any)

graph2any uses HugeGraph Source[1] to read vertices or edges and sends them to a downstream Sink. The example below exports person vertices to local JSON files; to export to JDBC, Kafka, or another system, replace LocalFile[2] and its options.

 1env {
 2  job.mode = "BATCH"
 3}
 4
 5source {
 6  HugeGraph {
 7    host = "hugegraph"
 8    port = 8080
 9    graph_name = "hugegraph"
10    graph_space = "DEFAULT"
11    label = "person"
12    label_type = "VERTEX"
13    schema = {
14      fields = {
15        name = "string"
16        age = "int"
17      }
18    }
19  }
20}
21
22sink {
23  LocalFile {
24    path = "/tmp/hugegraph-export/${table_name}"
25    file_format_type = "json"
26  }
27}

Save this as config/graph2file-person.conf and run it from the SeaTunnel installation directory:

1./bin/seatunnel.sh --config ./config/graph2file-person.conf -m local

To export edges, change the Source label to an edge label, set label_type = "EDGE", and declare the edge properties in schema.fields. Source also outputs the reserved columns ~source_id, ~source_label, ~target_id, and ~target_label; write them to the file or pass them to downstream transforms as needed.

This page covers row reads and writes. It does not copy source indexes, TTLs, or other schema settings. For the complete Source options and shared environment guidance, return to the SeaTunnel graph import guide[3].

3 References

Connectors

[1] HugeGraph Source
[2] LocalFile Sink

Related guide

[3] SeaTunnel graph import guide

2.6 - Tools Quick Start

1 HugeGraph-Tools Overview

HugeGraph-Tools manages graphs and tasks and provides backup and restore commands. It also retains deployment commands for legacy Server/Studio versions.

Testing Guide: For running HugeGraph-Tools tests locally, please refer to HugeGraph Toolchain Local Testing Guide

2 Get HugeGraph-Tools

HugeGraph-Tools is included in the Toolchain distribution. You can obtain it via:

2.1 Download the compiled archive

The following example uses the released HugeGraph-Toolchain 1.7.0 package. Check the download page for releases.

1export VERSION=1.7.0
2export ARCHIVE="apache-hugegraph-toolchain-incubating-${VERSION}"
3wget "https://downloads.apache.org/hugegraph/${VERSION}/${ARCHIVE}.tar.gz"
4tar zxf "${ARCHIVE}.tar.gz"
5# hugegraph-tools ships inside the toolchain package, in a directory
6# whose version suffix is the same as the archive's
7cd "${ARCHIVE}/apache-hugegraph-tools-incubating-${VERSION}"

2.2 Clone source code to compile and install

Choose one source below. Both build the released 1.7.0 Tools; install Maven and JDK 11 first. The archive route also requires wget. Each example ends in the generated Tools directory, where bin/hugegraph help lists the available commands.

Git tag
1git clone --branch 1.7.0 --single-branch https://github.com/apache/hugegraph-toolchain.git
2cd hugegraph-toolchain
3mvn package -pl hugegraph-tools -am -DskipTests -ntp
4cd hugegraph-tools/apache-hugegraph-tools-incubating-1.7.0
5bin/hugegraph help
Source archive
1export VERSION=1.7.0
2export ARCHIVE="apache-hugegraph-toolchain-incubating-${VERSION}"
3wget "https://downloads.apache.org/hugegraph/${VERSION}/${ARCHIVE}-src.tar.gz"
4tar zxf "${ARCHIVE}-src.tar.gz"
5cd "${ARCHIVE}-src"
6mvn package -pl hugegraph-tools -am -DskipTests -ntp
7cd "hugegraph-tools/apache-hugegraph-tools-incubating-${VERSION}"
8bin/hugegraph help

The build also writes hugegraph-tools/target/apache-hugegraph-tools-incubating-1.7.0.tar.gz relative to the source root.

2.3 Build current source

To use unreleased features from the master branch, build Tools from source (master is currently based on 1.8; requires JDK 17, Maven 3.6.3+, and Python 3.9+):

  1. Clone Toolchain source

    1git clone --branch master --single-branch https://github.com/apache/hugegraph-toolchain.git
    2cd hugegraph-toolchain
  2. Build matching Server SDK

    Run the script below to download Server source and install the SDK into an isolated Maven repository:

    1java -version
    2mvn -version
    3candidate_dir=$(mktemp -d)
    4mkdir -p "$candidate_dir/m2"
    5bash hugegraph-client/assembly/travis/install-candidate-sdk.sh \
    6  master "$candidate_dir/server" "$candidate_dir/m2"

    See the Java 17 migration guide for details.

  3. Package Tools

    1mvn -Dmaven.repo.local="$candidate_dir/m2" \
    2  -Dsdk.validation.mode=candidate \
    3  package -pl hugegraph-tools -am -DskipTests -Dmaven.javadoc.skip=true -ntp
    4cd hugegraph-tools/apache-hugegraph-tools-*
    5bin/hugegraph help

Keep the SDK repository while using this source build. The master build directory does not contain incubating.

3 How to use

3.1 Function overview

From the extracted or generated Tools directory, run bin/hugegraph help for the command list, or bin/hugegraph help <sub-command> for one command. The commands cover these tasks.

  • Graph management type, graph-mode-set, graph-mode-get, graph-list, graph-get, graph-clear, graph-create, graph-clone and graph-drop
  • Asynchronous task management type, task-list, task-get, task-delete, task-cancel and task-clear
  • Gremlin type, gremlin-execute and gremlin-schedule
  • Backup/Restore type, backup, restore, migrate, schedule-backup and dump
  • Authentication data backup/restore type, auth-backup and auth-restore
  • Legacy Server/Studio deployment, deploy, clear, start-all and stop-all
1Usage: hugegraph [options] [command] [command options]
3.2 [options]-Global Variable

Set these global connection options on the command line or through the environment variables below.

  • --graph selects the graph, default hugegraph.
  • --url sets the Server address, default http://127.0.0.1:8080.
  • --user supplies the username when Server authentication is enabled.
  • --password supplies that user’s password.
  • --timeout sets the connection timeout, default 30 seconds.
  • --trust-store-file selects the HTTPS truststore. An empty value uses the built-in conf/hugegraph.truststore.
  • --trust-store-password supplies its password. An empty value uses the built-in truststore password.
  • --throw-mode defaults to false, which prints errors and exits. Set it to true to throw exceptions, mainly for tests.

The scheme in --url selects the protocol. Use https://... for HTTPS and set truststore options only with that scheme. Supply both --user and --password, or omit both.

Use export to set an option for the current shell and the commands launched from it.

Global VariableEnvironment VariableExample
–urlHUGEGRAPH_URLexport HUGEGRAPH_URL=http://127.0.0.1:8080
–graphHUGEGRAPH_GRAPHexport HUGEGRAPH_GRAPH=hugegraph
–userHUGEGRAPH_USERNAMEexport HUGEGRAPH_USERNAME=admin
–passwordHUGEGRAPH_PASSWORDexport HUGEGRAPH_PASSWORD=test
–timeoutHUGEGRAPH_TIMEOUTexport HUGEGRAPH_TIMEOUT=30
–trust-store-fileHUGEGRAPH_TRUST_STORE_FILEexport HUGEGRAPH_TRUST_STORE_FILE=/tmp/trust-store
–trust-store-passwordHUGEGRAPH_TRUST_STORE_PASSWORDexport HUGEGRAPH_TRUST_STORE_PASSWORD=xxxx

Both forms below count vertices in the same graph; choose either form. The example uses a local Server without authentication. For an authenticated Server, also provide the username and password described above.

CLI flags
1./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph gremlin-execute --script 'g.V().count()'
Environment variables
1export HUGEGRAPH_URL=http://127.0.0.1:8080
2export HUGEGRAPH_GRAPH=hugegraph
3./bin/hugegraph gremlin-execute --script 'g.V().count()'

You can also set these variables in bin/hugegraph.

 1#!/bin/bash
 2
 3# Set environment here if needed
 4#export HUGEGRAPH_URL=
 5#export HUGEGRAPH_GRAPH=
 6#export HUGEGRAPH_USERNAME=
 7#export HUGEGRAPH_PASSWORD=
 8#export HUGEGRAPH_TIMEOUT=
 9#export HUGEGRAPH_TRUST_STORE_FILE=
10#export HUGEGRAPH_TRUST_STORE_PASSWORD=

bin/hugegraph also reads JAVA_HOME (a warning is printed when it is not set, and it is needed for https) and JAVA_OPTIONS (JVM options; when it is empty the script uses -Xms512m plus an -Xmx computed from the free memory of the machine).

3.3 Graph Management Type, graph-mode-set, graph-mode-get, graph-list, graph-get, graph-clear, graph-create, graph-clone and graph-drop
  • graph-mode-set, set graph restore mode
    • –graph-mode or -m, required, specifies the mode to be set, legal values include [NONE, RESTORING, MERGING, LOADING]
  • graph-mode-get, get graph restore mode
  • graph-list, list all graphs in a HugeGraph-Server
  • graph-get, get a graph and its storage backend type
  • graph-clear, clear all schema and data of a graph
    • –confirm-message or -c, required, delete confirmation information, manual input is required, double confirmation to prevent accidental deletion, “I’m sure to delete all data”, including double quotes
  • graph-create, create a new graph with configuration file
    • –name or -n, optional, the name of the new graph, default is g
    • –file or -f, the path to the graph configuration file, the content of the file is sent to HugeGraph-Server as the config of the new graph
  • graph-clone, clone an existing graph
    • –name or -n, optional, the name of the cloned graph, default is g
    • –clone-graph-name, optional, the name of the source graph to clone from, default is hugegraph
  • graph-drop, drop a graph (different from graph-clear, this completely removes the graph)
    • –confirm-message or -c, required, confirmation message “I’m sure to drop the graph”, including double quotes

graph-create, graph-clone, graph-clear and graph-drop raise –timeout to at least 300 seconds.

Before restoring into a new graph, set its mode to RESTORING. To merge into an existing graph, use MERGING.

3.4 Asynchronous task management Type,task-list、task-get、task-delete、task-cancel and task-clear
  • task-list,List the asynchronous tasks in a graph, which can be filtered according to the status of the tasks
    • –status,Optional, specify the status of the task to view, i.e. filter tasks by status, legal values include [UNKNOWN, NEW, QUEUED, RESTORING, RUNNING, SUCCESS, CANCELLED, FAILED] (case insensitive)
    • –limit,Optional, specify the number of tasks to be obtained, the default is -1, which means to obtain all eligible tasks, a value passed explicitly must be positive
  • task-get,Get detailed information about an asynchronous task
    • –task-id,Required, specifies the ID of the asynchronous task
  • task-delete,Delete information about an asynchronous task
    • –task-id,Required, specifies the ID of the asynchronous task
  • task-cancel,Cancel the execution of an asynchronous task
    • –task-id,Required, the ID of the asynchronous task to cancel
  • task-clear,Clean up completed asynchronous tasks
    • –force,Optional. When set, it means to clean up all asynchronous tasks. Unfinished ones are canceled first, and then all asynchronous tasks are cleared. By default, only completed asynchronous tasks are cleaned up
3.5 Gremlin Type,gremlin-execute and gremlin-schedule

⚠️ SEC Reminder: The execution of Gremlin depends on the actual logic of the statements, which may involve scenarios such as large-scale data modification and high-risk system calls with potential implicit hazards. Please use this tool only in secure and trusted network environments. It is imperative to configure and secure HugeGraph-Server with the Authentication System (Auth) and an IP Whitelist to restrict execution requests on the server side. Never hand over the tool or expose the execution entry to unauthorized personnel.

  • gremlin-execute, send Gremlin statements to HugeGraph-Server to execute query or modification operations, execute synchronously, and return results after completion
    • –file or -f, specify the script file to execute, UTF-8 encoding, mutually exclusive with –script
    • –script or -s, specifies the script string to execute, mutually exclusive with –file
    • –aliases or -a, Gremlin alias settings, the format is: key1=value1,key2=value2,…
    • –bindings or -b, Gremlin binding settings, the format is: key1=value1,key2=value2,…
    • –language or -l, the language of the Gremlin script, the default is gremlin-groovy

    –file and –script are mutually exclusive, one of them must be set

  • gremlin-schedule, send Gremlin statements to HugeGraph-Server to perform query or modification operations, asynchronous execution, and return the asynchronous task id immediately after the task is submitted
    • –file or -f, specify the script file to execute, UTF-8 encoding, mutually exclusive with –script
    • –script or -s, specifies the script string to execute, mutually exclusive with –file
    • –bindings or -b, Gremlin binding settings, the format is: key1=value1,key2=value2,…
    • –language or -l, the language of the Gremlin script, the default is gremlin-groovy

    –file and –script are mutually exclusive, one of them must be set

3.6 Backup/Restore Type
  • backup, back up the schema or data in a certain graph out of the HugeGraph system, and store it on the local disk or HDFS in the form of JSON
    • –format, the backup format, optional values include [json, text], the default is json
    • –all-properties, whether to back up all properties of vertices/edges, only valid when –format is text, default false
    • –label, the vertex label or edge label to be backed up, only applied when –format is text; when it is set, –huge-types must name exactly one type and that type must be vertex or edge, otherwise the command fails
    • –properties, properties of vertices/edges to be backed up, separated by commas, only valid when –format is text, valid only when backing up vertices or edges
    • –compress, whether to compress data during backup, the default is true
    • –directory or -d, the directory to store schema or data, the default is ‘./{graphName}’ for local directory, and ‘{fs.default.name}/{graphName}’ for HDFS
    • –huge-types or -t, the data types to be backed up, separated by commas, the optional value is ‘all’ or a combination of one or more [vertex, edge, vertex_label, edge_label, property_key, index_label], ‘all’ Represents all 6 types, namely vertices, edges and all schemas, ‘schema’ represents the 4 schema types [vertex_label, edge_label, property_key, index_label]
    • –log or -l, specify the log directory, the default is ./logs
    • –retry, specify the number of failed retries, the default is 3
    • –thread-num or -T, the number of threads to use, default is Math.min(10, Math.max(4, CPUs / 2))
    • –split-size or -s, specifies the size of splitting vertices or edges when backing up, the default is 1048576, and it must be at least 1048576 (1M)
    • -D, use the mode of -Dkey=value to specify dynamic parameters, and specify HDFS configuration items when backing up data to HDFS, for example: -Dfs.default.name=hdfs://localhost:9000

    If –timeout is less than 120 seconds, backup (and the backup step of migrate) uses 120 seconds

  • restore, restore schema or data stored in JSON format to a new graph (RESTORING mode) or merge into an existing graph (MERGING mode)
    • –directory or -d, the directory to store schema or data, the default is ‘./{graphName}’ for local directory, and ‘{fs.default.name}/{graphName}’ for HDFS
    • –clean, whether to delete the directory specified by –directory after the recovery map is completed, the default is false
    • –huge-types or -t, data types to restore, separated by commas, optional value is ‘all’ or a combination of one or more [vertex, edge, vertex_label, edge_label, property_key, index_label], ‘all’ Represents all 6 types, namely vertices, edges and all schemas, ‘schema’ represents the 4 schema types [vertex_label, edge_label, property_key, index_label]
    • –log or -l, specify the log directory, the default is ./logs
    • –retry, specify the number of failed retries, the default is 3
    • –thread-num or -T, the number of threads to use, default is Math.min(10, Math.max(4, CPUs / 2))
    • -D, use the mode of -Dkey=value to specify dynamic parameters, which are used to specify HDFS configuration items when restoring graphs from HDFS, for example: -Dfs.default.name=hdfs://localhost:9000

    restore accepts backups made with --format json. restore requires the graph to be in RESTORING or MERGING mode (set it with graph-mode-set first), otherwise the command fails

  • migrate, migrate the currently connected graph to another HugeGraphServer
    • –target-graph, the name of the target graph, the default is hugegraph
    • –target-url, the HugeGraphServer where the target graph is located, the default is http://127.0.0.1:8081
    • –target-user, the username used to access the target graph
    • –target-password, the password to access the target map
    • –target-timeout, the timeout for accessing the target map
    • –target-trust-store-file, access the truststore file used by the target graph
    • –target-trust-store-password, the password to access the truststore used by the target map
    • –directory or -d, during the migration process, the directory where the schema or data of the source graph is stored. For a local directory, the default is ‘./{graphName}’; for HDFS, the default is ‘{fs.default.name}/ {graphName}’
    • –huge-types or -t, the data types to be migrated, separated by commas, the optional value is ‘all’ or a combination of one or more [vertex, edge, vertex_label, edge_label, property_key, index_label], ‘all’ Represents all 6 types, namely vertices, edges and all schemas, ‘schema’ represents the 4 schema types [vertex_label, edge_label, property_key, index_label]
    • –log or -l, specify the log directory, the default is ./logs
    • –retry, specify the number of failed retries, the default is 3
    • –thread-num or -T, the number of threads to use, default is Math.min(10, Math.max(4, CPUs / 2))
    • –split-size or -s, specify the size of the vertex or edge block when backing up the source graph during the migration process, the default is 1048576, and it must be at least 1048576 (1M)
    • -D, use the mode of -Dkey=value to specify dynamic parameters, which are used to specify HDFS configuration items when the data needs to be backed up to HDFS during the migration process, for example: -Dfs.default.name=hdfs://localhost: 9000
    • –graph-mode or -m, the mode to set the target graph when restoring the source graph to the target graph, legal values include [RESTORING, MERGING], the default is RESTORING. The target graph is switched to this mode during the migration and switched back to its original mode afterwards
    • –keep-local-data, whether to keep the backup of the source map generated in the process of migrating the map, the default is false, that is, the backup of the source map is not kept after the default migration map ends
    Warning

    Backup & Migration Failures: If any vertex or edge data page fails after configured retries, the backup halts immediately with an error, and migrate does not proceed to restore the target graph. Partial backup files and shard diagnostics (files ending in _shards, such as vertex_failed_shards or edge_timeout_shards) are preserved in ./logs for root-cause diagnosis. The backup directory (--directory) and log directory (--log, default ./logs) are independent. Note that task initialization automatically clears files ending in _shards from the selected log directory. Before retrying, copy diagnostic files elsewhere or select a new --log directory as well as a clean --directory.

  • schedule-backup, periodically back up the graph and keep a certain number of the latest backups (currently only supports local file systems)
    • –directory or -d, required, specifies the directory of the backup data
    • –backup-num, optional, specifies the number of latest backups to save, defaults to 3
    • –interval, an optional item, specifies the backup cycle, the format is the same as the Linux crontab format, the default is “0 0 * * *” (every day at 00:00)

    schedule-backup adds a crontab entry that runs backup -t all into {directory}/{graph}/hugegraph-backup-{yyMMddHHmm}/ and keeps only the latest –backup-num backups. A relative –directory is resolved against the hugegraph-tools home directory, and {directory}/{graph} must not exist yet

  • dump, export all vertices and edges in the graph, using the vertex vertex-edge1 vertex-edge2... JSON format by default. To customize the format, implement a Formatter subclass such as CustomFormatter under hugegraph-tools/src/main/java/org/apache/hugegraph/formatter, then select it when running the command: bin/hugegraph dump -f CustomFormatter
    • –formatter or -f, specify the formatter to use, the default is JsonFormatter
    • –directory or -d, the directory where schema or data is stored, the default is ‘./{graphName}’ for local directory, and ‘{fs.default.name}/{graphName}’ for HDFS
    • –log or -l, specify the log directory, the default is ./logs
    • –retry, specify the number of failed retries, the default is 3
    • –thread-num or -T, the number of threads to use, default is Math.min(10, Math.max(4, CPUs / 2))
    • –split-size or -s, specifies the size of splitting vertices or edges when backing up, the default is 1048576, and it must be at least 1048576 (1M)
    • -D, use the mode of -Dkey=value to specify dynamic parameters, and specify HDFS configuration items when backing up data to HDFS, for example: -Dfs.default.name=hdfs://localhost:9000
3.7 Authentication data backup/restore type
  • auth-backup, backup authentication data to a specified directory
    • –types or -t, types of authentication data to back up, separated by commas, optional value is ‘all’ or a combination of one or more [user, group, target, belong, access], ‘all’ represents all 5 types; ‘belong’ requires ‘user’ and ‘group’ to be included, ‘access’ requires ‘group’ and ’target’ to be included
    • –directory, directory to store backup data, the default is ‘./auth-backup-restore’ for local directory, and ‘{fs.default.name}/auth-backup-restore’ for HDFS (this option has no -d short form)
    • –retry, specify the number of failed retries, the default is 3
    • -D, use the mode of -Dkey=value to specify dynamic parameters, and specify HDFS configuration items when backing up data to HDFS, for example: -Dfs.default.name=hdfs://localhost:9000
  • auth-restore, restore authentication data from a specified directory
    • –types or -t, types of authentication data to restore, separated by commas, optional value is ‘all’ or a combination of one or more [user, group, target, belong, access], ‘all’ represents all 5 types; ‘belong’ requires ‘user’ and ‘group’ to be included, ‘access’ requires ‘group’ and ’target’ to be included
    • –directory, directory where backup data is stored, the default is ‘./auth-backup-restore’ for local directory, and ‘{fs.default.name}/auth-backup-restore’ for HDFS (this option has no -d short form)
    • –retry, specify the number of failed retries, the default is 3
    • –strategy, conflict handling strategy, optional values are [stop, ignore], default is stop. stop means stop restoring when encountering conflicts, ignore means ignore conflicts and continue restoring
    • –init-password, initial password to set when restoring users, required when –types includes user
    • -D, use the mode of -Dkey=value to specify dynamic parameters, which are used to specify HDFS configuration items when restoring data from HDFS, for example: -Dfs.default.name=hdfs://localhost:9000
3.8 Install the deployment type

Note: These deployment commands only apply to legacy versions (Server 0.6.10.10.4 and Studio 0.6.10.10.0), and do not support current HugeGraph Server or Hubble. For modern deployments, follow Server Quick Start and Hubble Quick Start. The options below are retained only for maintaining legacy environments.

  • deploy, one-click download, install and start HugeGraph-Server and HugeGraph-Studio
    • -v, required, specifies the HugeGraph-Server and HugeGraph-Studio version to install, must be one of the versions listed in bin/version-map.yaml (0.6, 0.7, 0.8, 0.9, 0.10), which maps it to the matching server and studio release versions
    • -p, required, specifies the installed HugeGraph-Server and HugeGraph-Studio directories
    • -u, optional, specifies the link to download the HugeGraph-Server and HugeGraph-Studio compressed packages
  • clear, clean up HugeGraph-Server and HugeGraph-Studio directories and tarballs (refuses to run while a matching server or studio process is still alive, and prompts before each removal)
    • -p, required, specifies the directory of HugeGraph-Server and HugeGraph-Studio to be cleaned
  • start-all, start HugeGraph-Server and HugeGraph-Studio with one click
    • -v, required, specifies the installed HugeGraph-Server and HugeGraph-Studio version to start, same values as deploy
    • -p, required, specifies the directory where HugeGraph-Server and HugeGraph-Studio are installed
  • stop-all, close HugeGraph-Server and HugeGraph-Studio with one click

deploy, start-all, clear and stop-all are handed by bin/hugegraph straight to the shell scripts bin/deploy.sh, bin/start-all.sh, bin/clear.sh and bin/stop-all.sh, so the global options and environment variables in 3.2 do not apply to them.

deploy -u overrides the download URL and saves it in ~/hugegraph-download-url-prefix. Later runs without -u reuse the saved URL. Without either setting, the script uses https://github.com/hugegraph.

3.9 Specific command parameters

The help output below lists each subcommand and its options.

  1Usage: hugegraph [options] [command] [command options]
  2  Options:
  3    --graph
  4      Name of graph
  5      Default: hugegraph
  6    --password
  7      Password of user
  8    --throw-mode
  9      Whether the hugegraph-tools work to throw an exception
 10      Default: false
 11    --timeout
 12      Connection timeout
 13      Default: 30
 14    --trust-store-file
 15      The path of client truststore file used when https protocol is enabled
 16    --trust-store-password
 17      The password of the client truststore file used when the https protocol 
 18      is enabled
 19    --url
 20      The URL of HugeGraph-Server
 21      Default: http://127.0.0.1:8080
 22    --user
 23      Name of user
 24  Commands:
 25    graph-create      Create graph with config
 26      Usage: graph-create [options]
 27        Options:
 28          --file, -f
 29            Creating graph config file
 30          --name, -n
 31            The name of new created graph, default is g
 32            Default: g
 33
 34    graph-clone      Clone graph
 35      Usage: graph-clone [options]
 36        Options:
 37          --clone-graph-name
 38            The name of cloned graph, default is hugegraph
 39            Default: hugegraph
 40          --name, -n
 41            The name of new created graph, default is g
 42            Default: g
 43
 44    graph-list      List all graphs
 45      Usage: graph-list
 46
 47    graph-get      Get graph info
 48      Usage: graph-get
 49
 50    graph-clear      Clear graph schema and data
 51      Usage: graph-clear [options]
 52        Options:
 53        * --confirm-message, -c
 54            Confirm message of graph clear is "I'm sure to delete all data". 
 55            (Note: include "")
 56
 57    graph-drop      Drop graph
 58      Usage: graph-drop [options]
 59        Options:
 60        * --confirm-message, -c
 61            Confirm message of graph clear is "I'm sure to drop the graph". 
 62            (Note: include "")
 63
 64    graph-mode-set      Set graph mode
 65      Usage: graph-mode-set [options]
 66        Options:
 67        * --graph-mode, -m
 68            Graph mode, include: [NONE, RESTORING, MERGING]
 69            Possible Values: [NONE, RESTORING, MERGING, LOADING]
 70
 71    graph-mode-get      Get graph mode
 72      Usage: graph-mode-get
 73
 74    task-list      List tasks
 75      Usage: task-list [options]
 76        Options:
 77          --limit
 78            Limit number, no limit if not provided
 79            Default: -1
 80          --status
 81            Status of task
 82
 83    task-get      Get task info
 84      Usage: task-get [options]
 85        Options:
 86        * --task-id
 87            Task id
 88            Default: 0
 89
 90    task-delete      Delete task
 91      Usage: task-delete [options]
 92        Options:
 93        * --task-id
 94            Task id
 95            Default: 0
 96
 97    task-cancel      Cancel task
 98      Usage: task-cancel [options]
 99        Options:
100        * --task-id
101            Task id
102            Default: 0
103
104    task-clear      Clear completed tasks
105      Usage: task-clear [options]
106        Options:
107          --force
108            Force to clear all tasks, cancel all uncompleted tasks firstly, 
109            and delete all completed tasks
110            Default: false
111
112    gremlin-execute      Execute Gremlin statements
113      Usage: gremlin-execute [options]
114        Options:
115          --aliases, -a
116            Gremlin aliases, valid format is: 'key1=value1,key2=value2...'
117            Default: {}
118          --bindings, -b
119            Gremlin bindings, valid format is: 'key1=value1,key2=value2...'
120            Default: {}
121          --file, -f
122            Gremlin Script file to be executed, UTF-8 encoded, exclusive to 
123            --script 
124          --language, -l
125            Gremlin script language
126            Default: gremlin-groovy
127          --script, -s
128            Gremlin script to be executed, exclusive to --file
129
130    gremlin-schedule      Execute Gremlin statements as asynchronous job
131      Usage: gremlin-schedule [options]
132        Options:
133          --bindings, -b
134            Gremlin bindings, valid format is: 'key1=value1,key2=value2...'
135            Default: {}
136          --file, -f
137            Gremlin Script file to be executed, UTF-8 encoded, exclusive to 
138            --script 
139          --language, -l
140            Gremlin script language
141            Default: gremlin-groovy
142          --script, -s
143            Gremlin script to be executed, exclusive to --file
144
145    backup      Backup graph schema/data. If directory is on HDFS, use -D to 
146            set HDFS params. For example: 
147            -Dfs.default.name=hdfs://localhost:9000 
148      Usage: backup [options]
149        Options:
150          --all-properties
151            All properties to be backup flag
152            Default: false
153          --compress
154            compress flag
155            Default: true
156          --directory, -d
157            Directory of graph schema/data, default is './{graphname}' in 
158            local file system or '{fs.default.name}/{graphname}' in HDFS
159          --format
160            File format, valid is [json, text]
161            Default: json
162          --huge-types, -t
163            Type of schema/data. Concat with ',' if more than one. Other types 
164            include 'all' and 'schema'. 'all' means all vertices, edges and 
165            schema. In other words, 'all' equals with 'vertex, edge, 
166            vertex_label, edge_label, property_key, index_label'. 'schema' 
167            equals with 'vertex_label, edge_label, property_key, index_label'.
168            Default: [PROPERTY_KEY, VERTEX_LABEL, EDGE_LABEL, INDEX_LABEL, VERTEX, EDGE]
169          --label
170            Vertex label or edge label, only valid when type is vertex or edge
171          --log, -l
172            Directory of log
173            Default: ./logs
174          --properties
175            Vertex or edge properties to backup, only valid when type is 
176            vertex or edge
177            Default: []
178          --retry
179            Retry times, default is 3
180            Default: 3
181          --split-size, -s
182            Split size of shard
183            Default: 1048576
184          --thread-num, -T
185            Threads number to use, default is Math.min(10, Math.max(4, CPUs / 
186            2)) 
187            Default: 0
188          -D
189            HDFS config parameters
190            Syntax: -Dkey=value
191            Default: {}
192
193    schedule-backup      Schedule backup task
194      Usage: schedule-backup [options]
195        Options:
196          --backup-num
197            The number of latest backups to keep
198            Default: 3
199        * --directory, -d
200            The directory of backups stored
201          --interval
202            The interval of backup, format is: "a b c d e". 'a' means minute 
203            (0 - 59), 'b' means hour (0 - 23), 'c' means day of month (1 - 
204            31), 'd' means month (1 - 12), 'e' means day of week (0 - 6) 
205            (Sunday=0), "*" means all
206            Default: "0 0 * * *"
207
208    dump      Dump graph to files
209      Usage: dump [options]
210        Options:
211          --directory, -d
212            Directory of graph schema/data, default is './{graphname}' in 
213            local file system or '{fs.default.name}/{graphname}' in HDFS
214          --formatter, -f
215            Formatter to customize format of vertex/edge
216            Default: JsonFormatter
217          --log, -l
218            Directory of log
219            Default: ./logs
220          --retry
221            Retry times, default is 3
222            Default: 3
223          --split-size, -s
224            Split size of shard
225            Default: 1048576
226          --thread-num, -T
227            Threads number to use, default is Math.min(10, Math.max(4, CPUs / 
228            2)) 
229            Default: 0
230          -D
231            HDFS config parameters
232            Syntax: -Dkey=value
233            Default: {}
234
235    restore      Restore graph schema/data. If directory is on HDFS, use -D to 
236            set HDFS params if needed. For 
237            example:-Dfs.default.name=hdfs://localhost:9000 
238      Usage: restore [options]
239        Options:
240          --clean
241            Whether to remove the directory of graph data after restored
242            Default: false
243          --directory, -d
244            Directory of graph schema/data, default is './{graphname}' in 
245            local file system or '{fs.default.name}/{graphname}' in HDFS
246          --huge-types, -t
247            Type of schema/data. Concat with ',' if more than one. Other types 
248            include 'all' and 'schema'. 'all' means all vertices, edges and 
249            schema. In other words, 'all' equals with 'vertex, edge, 
250            vertex_label, edge_label, property_key, index_label'. 'schema' 
251            equals with 'vertex_label, edge_label, property_key, index_label'.
252            Default: [PROPERTY_KEY, VERTEX_LABEL, EDGE_LABEL, INDEX_LABEL, VERTEX, EDGE]
253          --log, -l
254            Directory of log
255            Default: ./logs
256          --retry
257            Retry times, default is 3
258            Default: 3
259          --thread-num, -T
260            Threads number to use, default is Math.min(10, Math.max(4, CPUs / 
261            2)) 
262            Default: 0
263          -D
264            HDFS config parameters
265            Syntax: -Dkey=value
266            Default: {}
267
268    migrate      Migrate graph
269      Usage: migrate [options]
270        Options:
271          --directory, -d
272            Directory of graph schema/data, default is './{graphname}' in 
273            local file system or '{fs.default.name}/{graphname}' in HDFS
274          --graph-mode, -m
275            Mode used when migrating to target graph, include: [RESTORING, 
276            MERGING] 
277            Default: RESTORING
278            Possible Values: [NONE, RESTORING, MERGING, LOADING]
279          --huge-types, -t
280            Type of schema/data. Concat with ',' if more than one. Other types 
281            include 'all' and 'schema'. 'all' means all vertices, edges and 
282            schema. In other words, 'all' equals with 'vertex, edge, 
283            vertex_label, edge_label, property_key, index_label'. 'schema' 
284            equals with 'vertex_label, edge_label, property_key, index_label'.
285            Default: [PROPERTY_KEY, VERTEX_LABEL, EDGE_LABEL, INDEX_LABEL, VERTEX, EDGE]
286          --keep-local-data
287            Whether to keep the local directory of graph data after restored
288            Default: false
289          --log, -l
290            Directory of log
291            Default: ./logs
292          --retry
293            Retry times, default is 3
294            Default: 3
295          --split-size, -s
296            Split size of shard
297            Default: 1048576
298          --target-graph
299            The name of target graph to migrate
300            Default: hugegraph
301          --target-password
302            The password of target graph to migrate
303          --target-timeout
304            The timeout to connect target graph to migrate
305            Default: 0
306          --target-trust-store-file
307            The trust store file of target graph to migrate
308          --target-trust-store-password
309            The trust store password of target graph to migrate
310          --target-url
311            The url of target graph to migrate
312            Default: http://127.0.0.1:8081
313          --target-user
314            The username of target graph to migrate
315          --thread-num, -T
316            Threads number to use, default is Math.min(10, Math.max(4, CPUs / 
317            2)) 
318            Default: 0
319          -D
320            HDFS config parameters
321            Syntax: -Dkey=value
322            Default: {}
323
324    deploy      Install HugeGraph-Server and HugeGraph-Studio
325      Usage: deploy [options]
326        Options:
327        * -p
328            Install path of HugeGraph-Server and HugeGraph-Studio
329          -u
330            Download url prefix path of HugeGraph-Server and HugeGraph-Studio
331        * -v
332            Version of HugeGraph-Server and HugeGraph-Studio
333
334    start-all      Start HugeGraph-Server and HugeGraph-Studio
335      Usage: start-all [options]
336        Options:
337        * -p
338            Install path of HugeGraph-Server and HugeGraph-Studio
339        * -v
340            Version of HugeGraph-Server and HugeGraph-Studio
341
342    clear      Clear HugeGraph-Server and HugeGraph-Studio
343      Usage: clear [options]
344        Options:
345        * -p
346            Install path of HugeGraph-Server and HugeGraph-Studio
347
348    stop-all      Stop HugeGraph-Server and HugeGraph-Studio
349      Usage: stop-all
350
351    auth-backup      null
352      Usage: auth-backup [options]
353        Options:
354          --directory
355            Directory of auth information, default is 
356            './{auth-backup-restore}' in local file system or 
357            '{fs.default.name}/{auth-backup-restore}' in HDFS
358          --retry
359            Retry times, default is 3
360            Default: 3
361          --types, -t
362            Type of auth data to restore and backup, concat with ',' if more 
363            than one. 'all' means all auth information. In other words, 'all' 
364            equals with 'user, group, target, belong, access'. In addition, 
365            'belong' or 'access' can not backup or restore alone, if type 
366            contains 'belong' then should contains 'user' and 'group'. If type 
367            contains 'access' then should contains 'group' and 'target'.
368            Default: [TARGET, GROUP, USER, ACCESS, BELONG]
369          -D
370            HDFS config parameters
371            Syntax: -Dkey=value
372            Default: {}
373
374    auth-restore      null
375      Usage: auth-restore [options]
376        Options:
377          --directory
378            Directory of auth information, default is 
379            './{auth-backup-restore}' in local file system or 
380            '{fs.default.name}/{auth-backup-restore}' in HDFS
381          --init-password
382            Init user password, if restore type include 'user', please specify 
383            the init-password of users.
384            Default: <empty string>
385          --retry
386            Retry times, default is 3
387            Default: 3
388          --strategy
389            The strategy needs to be chosen in the event of a conflict when 
390            restoring. Valid strategies include 'stop' and 'ignore', default 
391            is 'stop'. 'stop' means if there a conflict, stop restore. 
392            'ignore' means if there a conflict, ignore and continue to 
393            restore. 
394            Default: STOP
395            Possible Values: [STOP, IGNORE]
396          --types, -t
397            Type of auth data to restore and backup, concat with ',' if more 
398            than one. 'all' means all auth information. In other words, 'all' 
399            equals with 'user, group, target, belong, access'. In addition, 
400            'belong' or 'access' can not backup or restore alone, if type 
401            contains 'belong' then should contains 'user' and 'group'. If type 
402            contains 'access' then should contains 'group' and 'target'.
403            Default: [TARGET, GROUP, USER, ACCESS, BELONG]
404          -D
405            HDFS config parameters
406            Syntax: -Dkey=value
407            Default: {}
408
409    help      Print usage
410      Usage: help
3.10 Specific command example
1. gremlin statement
1# Execute gremlin synchronously
2./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph gremlin-execute --script 'g.V().count()'
3
4# Execute gremlin asynchronously
5./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph gremlin-schedule --script 'g.V().count()'
2. Show task status
1./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph task-list
2
3./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph task-list --limit 5
4
5./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph task-list --status success
3. Set and show graph mode
1./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-mode-set -m RESTORING
2
3./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-mode-get
4
5./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-list
4. Cleanup Graph
1./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-clear -c "I'm sure to delete all data"
5. Backup Graph
1./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph backup -t all --directory ./backup-test
6. Periodic Backup Graph
1./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph schedule-backup -d ./backup --interval "*/2 * * * *"
7. Recovery Graph
1# set graph mode
2./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-mode-set -m RESTORING
3
4# recovery graph
5./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph restore -t all --directory ./backup-test
6
7# restore graph mode
8./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-mode-set -m NONE
8. Graph Migration
1./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph migrate --target-url http://127.0.0.1:8090 --target-graph hugegraph
Note

If a data page fails during the backup phase of migration, migrate terminates immediately and leaves the target graph untouched. Inspect shard diagnostic files in ./logs/ (*_shards) and ensure a clean directory is provided before retrying.

2.7 - HugeGraph-Spark-Connector Quick Start

1 HugeGraph-Spark-Connector Overview

HugeGraph-Spark-Connector uses the Spark DataFrame API to write bulk data to HugeGraph. The current implementation provides vertex and edge writers.

Reading from HugeGraph is not implemented yet: the table only implements SupportsWrite, so spark.read.format(...) is not supported. The connector supports the CUSTOMIZE and PRIMARY_KEY vertex id strategies; the AUTOMATIC strategy is rejected.

2 Environment Requirements

  • Java 8+
  • Maven 3.6+
  • Spark 3.2.x (the module is built against Spark 3.2.2 with provided scope, so your Spark runtime must supply the Spark jars)
  • Scala 2.12 (built with Scala 2.12.11)

3 Building

3.1 Build without executing tests

1git clone https://github.com/apache/hugegraph-toolchain.git
2cd hugegraph-toolchain
3mvn clean package -pl hugegraph-spark-connector -am -DskipTests -ntp

3.2 Build with default tests

1mvn clean package -pl hugegraph-spark-connector -am -ntp

Both commands produce a fat jar at hugegraph-spark-connector/target/hugegraph-spark-connector-${revision}-jar-with-dependencies.jar (Spark itself is not bundled). Pass it to spark-submit --jars when you do not manage the dependency through Maven.

4 Usage

Add the dependency to pom.xml, replacing ${revision} with the release version you use:

1<dependency>
2    <groupId>org.apache.hugegraph</groupId>
3    <artifactId>hugegraph-spark-connector</artifactId>
4    <version>${revision}</version>
5</dependency>

The format string must be the full class name org.apache.hugegraph.spark.connector.DataSource; the connector does not register a short name with Spark’s DataSourceRegister service loader. When HugeGraphServer has authentication enabled, add .option("username", ...) and .option("token", ...) to the examples below.

4.1 Schema Definition Example

If we have a graph, the schema is defined as follows:

 1schema.propertyKey("name").asText().ifNotExist().create()
 2schema.propertyKey("age").asInt().ifNotExist().create()
 3schema.propertyKey("city").asText().ifNotExist().create()
 4schema.propertyKey("weight").asDouble().ifNotExist().create()
 5schema.propertyKey("lang").asText().ifNotExist().create()
 6schema.propertyKey("date").asText().ifNotExist().create()
 7schema.propertyKey("price").asDouble().ifNotExist().create()
 8
 9schema.vertexLabel("person")
10        .properties("name", "age", "city")
11        .useCustomizeStringId()
12        .nullableKeys("age", "city")
13        .ifNotExist()
14        .create()
15
16schema.vertexLabel("software")
17        .properties("name", "lang", "price")
18        .primaryKeys("name")
19        .ifNotExist()
20        .create()
21
22schema.edgeLabel("knows")
23        .sourceLabel("person")
24        .targetLabel("person")
25        .properties("date", "weight")
26        .ifNotExist()
27        .create()
28
29schema.edgeLabel("created")
30        .sourceLabel("person")
31        .targetLabel("software")
32        .properties("date", "weight")
33        .ifNotExist()
34        .create()

4.2 Vertex Sink (Scala)

 1val df = sparkSession.createDataFrame(Seq(
 2  Tuple3("marko", 29, "Beijing"),
 3  Tuple3("vadas", 27, "HongKong"),
 4  Tuple3("Josh", 32, "Beijing"),
 5  Tuple3("peter", 35, "ShangHai"),
 6  Tuple3("li,nary", 26, "Wu,han"),
 7  Tuple3("Bob", 18, "HangZhou"),
 8)) toDF("name", "age", "city")
 9
10df.show()
11
12df.write
13  .format("org.apache.hugegraph.spark.connector.DataSource")
14  .option("host", "127.0.0.1")
15  .option("port", "8080")
16  .option("graph", "hugegraph")
17  .option("data-type", "vertex")
18  .option("label", "person")
19  .option("id", "name")
20  .option("batch-size", 2)
21  .mode(SaveMode.Overwrite)
22  .save()

4.3 Edge Sink (Scala)

 1val df = sparkSession.createDataFrame(Seq(
 2  Tuple4("marko", "vadas", "20160110", 0.5),
 3  Tuple4("peter", "Josh", "20230801", 1.0),
 4  Tuple4("peter", "li,nary", "20130220", 2.0)
 5)).toDF("source", "target", "date", "weight")
 6
 7df.show()
 8
 9df.write
10  .format("org.apache.hugegraph.spark.connector.DataSource")
11  .option("host", "127.0.0.1")
12  .option("port", "8080")
13  .option("graph", "hugegraph")
14  .option("data-type", "edge")
15  .option("label", "knows")
16  .option("source-name", "source")
17  .option("target-name", "target")
18  .option("batch-size", 2)
19  .mode(SaveMode.Overwrite)
20  .save()

4.4 Vertex Sink with PRIMARY_KEY id strategy (Scala)

For a vertex label that uses primaryKeys(...), do not set the id option: the id is spliced from the primary key columns. Columns that are not part of the schema can be dropped with ignored-fields.

 1val df = sparkSession.createDataFrame(Seq(
 2  Tuple4("lop", "java", 328L, "ISBN978-7-107-18618-5"),
 3  Tuple4("ripple", "python", 199L, "ISBN978-7-100-13678-5"),
 4)).toDF("name", "lang", "price", "ISBN")
 5
 6df.write
 7  .format("org.apache.hugegraph.spark.connector.DataSource")
 8  .option("host", "127.0.0.1")
 9  .option("port", "8080")
10  .option("graph", "hugegraph")
11  .option("data-type", "vertex")
12  .option("label", "software")
13  .option("ignored-fields", "ISBN")
14  .option("batch-size", 2)
15  .mode(SaveMode.Overwrite)
16  .save()

4.5 Edge Sink with mixed id strategies (Scala)

source-name and target-name follow the id strategy of their own vertex label. Below, person uses a customized string id (one column) while software uses a primary key (its name column):

 1val df = sparkSession.createDataFrame(Seq(
 2  Tuple4("marko", "lop", "20171210", 0.5),
 3  Tuple4("Josh", "lop", "20091111", 0.4),
 4  Tuple4("peter", "ripple", "20171210", 1.0),
 5  Tuple4("vadas", "lop", "20171210", 0.2)
 6)).toDF("source", "name", "date", "weight")
 7
 8df.write
 9  .format("org.apache.hugegraph.spark.connector.DataSource")
10  .option("host", "127.0.0.1")
11  .option("port", "8080")
12  .option("graph", "hugegraph")
13  .option("data-type", "edge")
14  .option("label", "created")
15  .option("source-name", "source") // customize id
16  .option("target-name", "name")   // primary key
17  .option("batch-size", 2)
18  .mode(SaveMode.Overwrite)
19  .save()

Note on save modes: SaveMode.Overwrite and SaveMode.Append both insert the rows. The overwrite path does not delete existing data from the graph first.

5 Configuration Parameters

Option keys are matched case-insensitively and trimmed. data-type and label are always required; source-name and target-name are required when data-type is edge; all other options have defaults.

5.1 Client Configs

Client Configs are used to configure hugegraph-client.

ParameterDefault ValueDescription
hostlocalhostAddress of HugeGraphServer. A bare host name or IP, or a full http:// / https:// prefix
port8080Port of HugeGraphServer
graphhugegraphGraph name
protocolhttpProtocol for sending requests to the server, optional http or https
usernamenullUsername of the current graph when HugeGraphServer enables permission authentication. When unset, the graph name is used as the username
tokennullToken of the current graph when HugeGraphServer has enabled authorization authentication
timeout60Timeout (seconds) for inserting results to return
max-connCPUS * 4The maximum number of HTTP connections between HugeClient and HugeGraphServer
max-conn-per-routeCPUS * 2The maximum number of HTTP connections for each route between HugeClient and HugeGraphServer
trust-store-filenullThe client’s certificate file path when the request protocol is https. When unset under https, the connector reads conf/hugegraph.truststore under the directory given by the JVM system property connector.home.path, which must then be set
trust-store-tokennullThe client’s certificate password when the request protocol is https. When unset under https, hugegraph is used

5.2 Graph Data Configs

Graph Data Configs describe how DataFrame columns map to vertices or edges.

ParameterDefault ValueDescription
data-typeRequired. Graph data type, must be vertex or edge
labelRequired. Label to which the vertex/edge data to be imported belongs
idSpecify a column as the id column of the vertex. When the vertex id policy is CUSTOMIZE, it is required; when the id policy is PRIMARY_KEY, it must be empty. The AUTOMATIC id policy is not supported
source-nameRequired when data-type is edge. Select certain columns of the input source as the id column of source vertex. When the id policy of the source vertex is CUSTOMIZE, a certain column must be specified as the id column of the vertex; when the id policy of the source vertex is PRIMARY_KEY, one or more columns must be specified for splicing the id of the generated vertex, that is, no matter which id strategy is used, this item is required. Multiple columns are separated by , (the delimiter option does not apply here)
target-nameRequired when data-type is edge. Specify certain columns as the id columns of target vertex, similar to source-name
selected-fieldsSelect some columns to insert, other unselected ones are not inserted, cannot exist at the same time as ignored-fields
ignored-fieldsIgnore some columns so that they do not participate in insertion, cannot exist at the same time as selected-fields
batch-size500The number of data items in each batch when importing data. Applied per Spark task: each partition writer flushes its buffer to the server once it holds this many vertices/edges, and again at commit for the remainder

5.3 Common Configs

Common Configs contains some common configurations.

ParameterDefault ValueDescription
delimiter,Separator of selected-fields and ignored-fields. source-name and target-name are always split on ,

6 Notes and Limitations

  • Each Spark write task opens its own HugeClient, switches the graph to LOADING mode before writing and sets it back to NONE at commit or abort.
  • Vertex ids are limited to 128 bytes (UTF-8). This applies to customized string ids and to ids spliced from primary keys.
  • The AUTOMATIC vertex id strategy is not supported; the write fails with an IllegalArgumentException when the writer is created.
  • Properties with SET or LIST cardinality are not supported yet; only SINGLE cardinality values are converted.
  • Date properties: string values must use the format yyyy-MM-dd HH:mm:ss and are parsed in the GMT+8 time zone; numeric values are treated as epoch milliseconds.
  • Boolean properties given as strings accept true, 1, yes, y and false, 0, no, n (case-insensitive).
  • Rows whose customized string id, or any primary key value, is an empty string are skipped. A null id or primary key value raises an error instead.

7 License

The same as HugeGraph, hugegraph-spark-connector is also licensed under Apache 2.0 License.

3 - HugeGraph-AI

hugegraph-ai provides Python clients for HugeGraph, graph machine learning tools, and LLM tools for knowledge graph construction and GraphRAG applications.

Apache License 2.0 · Ask DeepWiki

Modules

  • hugegraph-llm-sdk: standalone Python SDK for schema design, validation, and reviewed property-graph extraction.
  • hugegraph-mcp: graph queries, schema tools, and controlled writes for AI clients through MCP.
  • hugegraph-llm: knowledge graph construction, GraphRAG, and natural-language graph queries.
  • hugegraph-ml: reads graph data from HugeGraph and runs graph learning models.
  • hugegraph-python-client: a Python SDK for managing schemas and graph data and running Gremlin queries.
  • vermeer-python-client: a Python SDK for the Vermeer graph computing service.

The repository uses a uv workspace whose members are hugegraph-llm-sdk, hugegraph-llm, and hugegraph-python-client. hugegraph-ml and vermeer-python-client are editable path dependencies rather than workspace members. The current repository version is 1.8.0. The client source directory is named hugegraph-python-client, but its distribution name is hugegraph-python.

Requirements

  • HugeGraph-AI root workspace: Python >= 3.12 (Python 3.12–3.14 supported)
  • HugeGraph-LLM & HugeGraph-LLM-SDK: Python >= 3.12 (Python 3.12–3.14 supported)
  • HugeGraph-ML: Python 3.10 or later
  • PyPI hugegraph-python 1.5.0: Python 3.9 or later; current repository source uses Python >= 3.12 with the workspace
  • Vermeer Python client: current source requires Python 3.10 or later, although package metadata still says >=3.9; see the client guide
  • uv 0.7 or later
  • HugeGraph Server 1.5.0 or later; the current workspace client rejects detectable older versions

Optional Dependency Groups

The root project declares one extra per module plus a few combined ones:

ExtraInstalls
llmhugegraph-llm
mlhugegraph-ml
python-clienthugegraph-python (source directory hugegraph-python-client)
vermeervermeer-python-client
devpytest, pytest-cov, coverage, pylint, ruff, mypy, ty, pre-commit
nk-llmhugegraph-llm, hugegraph-python-client, and Nuitka for the compiled image
allall four module packages

hugegraph-llm itself declares a vectordb extra that adds pymilvus and qdrant-client.

Deploy with Docker Compose

The repository includes a Compose file that starts both HugeGraph Server and the RAG service:

1git clone https://github.com/apache/hugegraph-ai.git
2cd hugegraph-ai
3cp docker/env.template docker/.env
4# Edit docker/.env and set PROJECT_PATH to the absolute path of this repository
5touch hugegraph-llm/.env
6# Set GRAPH_URL=server:8080 in hugegraph-llm/.env and supply matching Server credentials
7cd docker
8docker compose -f docker-compose-network.yml up -d

Default addresses:

  • HugeGraph Server: http://localhost:8080
  • RAG service and Web UI: http://localhost:8001

Start the RAG Service from Source

1git clone https://github.com/apache/hugegraph-ai.git
2cd hugegraph-ai
3uv sync --extra llm
4source .venv/bin/activate
5cd hugegraph-llm
6python -m hugegraph_llm.demo.rag_demo.app

uv sync creates .venv at the repository root. Installing from the root resolves workspace members and path dependencies together. The repository does not track uv.lock; uv sync resolves the declarations and version constraints in pyproject.toml.

Install ML Dependencies

1cd hugegraph-ai
2uv sync --extra ml
3source .venv/bin/activate
4cd hugegraph-ml/src

Example scripts are under hugegraph-ml/src/hugegraph_ml/examples/.

Next Steps

3.1 - HugeGraph-LLM

HugeGraph-LLM supports knowledge graph construction, GraphRAG, and natural-language graph queries. Its demo service hosts Gradio and FastAPI in one process. Source launches listen locally at 127.0.0.1:8001, accessible at http://localhost:8001; explicitly use --host 127.0.0.1 for local-only access. Dockerfile.llm overrides this to 0.0.0.0:8001; do not use loopback inside the container, or published ports will not reach the service.

Warning

Production requires HugeGraph-LLM login (ENABLE_LOGIN=True, replacing USER_TOKEN and ADMIN_TOKEN) and a source IP allowlist at the firewall or network entry point. Separately enable Server authentication and authorization, retain Server audit logs (normally audit-*.log), and grant GRAPH_USER only the permissions this service needs. AI service tokens authenticate the LLM UI and API, not HugeGraph Server.

Requirements

AI-generated project documentation: Ask DeepWiki

  • Python >= 3.12 (Python 3.12–3.14 supported)
  • uv 0.7 or later
  • HugeGraph Server 1.5.0 or later; the current workspace client rejects detectable older versions

Choose your deployment method below:

Deploy with Docker Compose

Prepare the environment files from the HugeGraph-AI repository root:

1git clone https://github.com/apache/hugegraph-ai.git
2cd hugegraph-ai
3cp docker/env.template docker/.env
4# Edit docker/.env and set PROJECT_PATH to the absolute path of this repository
5touch hugegraph-llm/.env
6# Set GRAPH_URL=server:8080 and matching GRAPH_USER / GRAPH_PWD in hugegraph-llm/.env
7cd docker
8docker compose -f docker-compose-network.yml up -d
9docker compose -f docker-compose-network.yml ps

After startup, HugeGraph Server is available at http://localhost:8080, and the RAG service and Web UI are available at http://localhost:8001.

The Compose file mounts ${PROJECT_PATH}/hugegraph-llm/.env into the container at /home/work/hugegraph-llm/.env, so the file has to exist before the container starts. The resource directory hugegraph-llm/src/hugegraph_llm/resources can be mounted the same way; the mount is commented out by default.

The application reads GRAPH_URL; Compose-provided HUGEGRAPH_HOST and HUGEGRAPH_PORT do not override it. Set GRAPH_URL=server:8080 and matching Server credentials in the container’s .env.

Container Images

Build recipeDescription
docker/Dockerfile.llmSource runtime image recipe; starts python -m hugegraph_llm.demo.rag_demo.app --host 0.0.0.0 --port 8001
docker/Dockerfile.nkNuitka binary image recipe using the nk-llm extra; starts ./app.dist/app.bin

Compose references untagged hugegraph/rag, which resolves to latest. scripts/build_llm_image.sh builds docker/Dockerfile.llm locally as hugegraph/graphrag:1.7.0. These are different images: building locally does not replace Compose’s image. To run the build, change Compose’s image to hugegraph/graphrag:1.7.0. The script builds but does not push. Both Dockerfiles expose 8001, run as non-root user work, declare a resource-directory volume, and use curl -f http://localhost:8001/ as their health check.

Deploy on Kubernetes

docker/charts/hg-llm is a Helm chart for the RAG service. It deploys the hugegraph/graphrag image and, by default, publishes a NodePort service that maps node port 8039 and service port 8080 onto container port 8001. The release name is fixed to hg-llm-service. Ingress and horizontal pod autoscaling are present but disabled by default.

The chart deploys only the RAG service, not HugeGraph Server. Set GRAPH_URL in the mounted .env to a Server address reachable from the Pod, with matching credentials.

image.tag still defaults to v0.0.1. The build script produces local hugegraph/graphrag:1.7.0; before deployment, push the image to a registry reachable by the cluster or load it onto cluster nodes, then set matching tag and pull policy values.

The .env and prompt YAML mounts are commented out in values.yaml. Prompt YAML can use a ConfigMap, but .env may contain secrets and passwords: use a Secret and change the env-config volume from configMap to secret. Create them separately:

1kubectl create secret generic hugegraph-llm-env --from-file=.env=/path/to/.env
2kubectl create configmap hugegraph-llm-prompt-config --from-file=/path/to/config_prompt.yaml

Uncomment the volumeMounts entries and enable the prompt ConfigMap volume/mount if needed. Helm renders these volume definitions directly from values.yaml.

Start from Source

  1. Clone the repository:

    1git clone https://github.com/apache/hugegraph-ai.git
    2cd hugegraph-ai
  2. Install dependencies and activate environment:

    1uv sync --extra llm
    2source .venv/bin/activate
  3. Launch the Web application:

    1cd hugegraph-llm
    2python -m hugegraph_llm.demo.rag_demo.app

To use a custom address and port:

1python -m hugegraph_llm.demo.rag_demo.app \
2  --host 127.0.0.1 \
3  --port 18001

Set HG_DEV_RELOAD=1 to start uvicorn with auto-reload during development.

The service stores model, HugeGraph, and login settings in hugegraph-llm/.env. Prompts are stored separately in hugegraph-llm/src/hugegraph_llm/resources/demo/config_prompt.yaml. The configuration code creates missing files with default values.

The .env location is resolved in this order: HUGEGRAPH_LLM_ENV_PATH if it is set, then hugegraph-llm/.env when the package runs from a source checkout, then .env in the current working directory.

Main Capabilities

Build RAG Indexes

The first Web UI tab splits text into a chunk vector index, extracts vertices and edges according to a schema, writes the graph to HugeGraph, and updates the vertex vector index. Input can be typed into the text tab or uploaded through the file tab, which accepts .txt, .docx, and .pdf files and allows selecting several at once. Encrypted PDFs and scanned PDFs without an extractable text layer are rejected.

The schema can be inline JSON or the name of an existing graph. Through the REST API, a graph name requires a matching client_config.graph; inline JSON neither connects to HugeGraph nor accepts client_config.

The tab also carries two generators. Graph Schema Generator derives a schema from query examples plus a few-shot example. Graph Extraction Prompt Generator writes an extraction prompt from a described scenario and a selected reference example. A Graph Extraction Split Type dropdown chooses document, paragraph, or sentence granularity before extraction.

GraphRAG

The query pipeline can combine direct LLM answers, chunk-vector retrieval, and graph retrieval. Graph retrieval first extracts keywords and matches vertices, then attempts Text2Gremlin. If generation or execution fails, it can fall back to predefined graph traversals. Request parameters control result limits, vector distance thresholds, template counts, and reranking.

The same tab has a batch back-testing panel that reads questions from an .xlsx or .csv file, answers each one, and returns a downloadable file. A template file is offered for download next to the upload control.

Knowledge graph builder

Text2Gremlin

POST /text2gremlin generates Gremlin from natural language, the graph schema, and optional examples. A custom prompt must retain {query}, {schema}, {example}, and {vertices}.

The matching UI tab can first build the example vector index from a .json or .csv file of question and Gremlin pairs. The bundled resources/demo/text2gremlin.csv is used when no file is supplied.

Graph and Admin Tools

The Graph Tools tab runs a Gremlin query directly, triggers a manual graph backup, and can initialize demo data in HugeGraph. The Admin Tools tab shows the last lines of logs/llm-server.log behind an ADMIN_TOKEN prompt, and can refresh or clear that file.

Two background tasks run for the lifetime of the process: a cron job that backs up the graph every day at 01:00, and a task that keeps vertex-id embeddings up to date.

Models and Vector Backends

Chat, information extraction, and Text2Gremlin can independently use an OpenAI-compatible endpoint, Ollama, or LiteLLM. The embedding model is configured separately and supports the same three providers. Reranking supports Cohere and SiliconFlow.

FAISS is the default vector index. CUR_VECTOR_INDEX selects Faiss, Milvus, or Qdrant, and the same choice is available in the 5. Set up the vector engine. panel of the Web UI. Milvus and Qdrant require the optional dependencies:

1cd hugegraph-ai
2uv sync --package hugegraph-llm --extra vectordb

See the workflow guide, the configuration reference, and the REST API for details.

Programmatic Use

The former RAGPipeline and KgBuilder classes were replaced by a pipeline scheduler. Call a flow by name through SchedulerSingleton:

 1from hugegraph_llm.flows.scheduler import SchedulerSingleton
 2
 3scheduler = SchedulerSingleton.get_instance()
 4res = scheduler.schedule_flow(
 5    "rag_graph_only",
 6    query="Tell me about Al Pacino.",
 7    graph_only_answer=True,
 8    vector_only_answer=False,
 9    raw_answer=False,
10    gremlin_tmpl_num=-1,
11    gremlin_prompt=None,
12)
13print(res.get("graph_only_answer"))

The registered flow names are rag_raw, rag_vector_only, rag_graph_only, rag_graph_vector, text2gremlin, build_examples_index, build_vector_index, graph_extract, import_graph_data, update_vid_embeddings, get_graph_index_info, build_schema, and prompt_generate. schedule_stream_flow is the async streaming variant.

Development Checks

Install the module and the development tools from the repository root, then run the checks that mirror CI:

 1cd hugegraph-ai
 2uv sync --extra llm --extra dev
 3uv run ruff format --check .
 4uv run ruff check .
 5
 6cd hugegraph-llm
 7SKIP_EXTERNAL_SERVICES=true uv run pytest src/tests/config/ src/tests/document/ src/tests/middleware/ \
 8  src/tests/operators/ src/tests/models/ src/tests/indices/ src/tests/test_utils.py -v --tb=short
 9SKIP_EXTERNAL_SERVICES=true uv run pytest src/tests/integration/test_graph_rag_pipeline.py \
10  src/tests/integration/test_kg_construction.py src/tests/integration/test_rag_pipeline.py -v --tb=short

Git hooks are available through pre-commit:

1cd hugegraph-ai
2pre-commit install
3pre-commit run --all-files

3.2 - HugeGraph-ML

HugeGraph-ML reads graph data from HugeGraph and converts it to DGL graphs for tasks such as node embedding, node classification, graph classification, link prediction and fraud detection. Model implementations are under hugegraph-ml/src/hugegraph_ml/models/.

Requirements

  • Python 3.10 or later
  • HugeGraph Server 1.5.0 or later; the current workspace client rejects detectable older versions
  • uv 0.7 or later

All server access goes through hugegraph-python-client (the pyhugegraph package) from the same repository. HugeGraph2DGL pulls vertices and edges over the Gremlin endpoint with g.V().hasLabel(...) and g.E().hasLabel(...), and the dataset importers write through the schema and batch vertex/edge APIs in batches of 500.

The repository root declares ML version constraints under [tool.uv] constraint-dependencies:

PackageConstraint
torch==2.2.0
dgl~=2.1.0
ogb~=1.3.6
torchdata~=0.7.0
catboost~=1.2.3
category-encoders~=2.6.3
numpy~=1.24.4
pandas~=2.2.3

These constraints limit versions but do not select CPU or CUDA builds. Tasks that support gpu default to -1 for CPU. GPU use requires mutually compatible CUDA builds of torch and dgl.

Installation

1git clone https://github.com/apache/hugegraph-ai.git
2cd hugegraph-ai
3uv sync --extra ml
4source .venv/bin/activate
5cd hugegraph-ml/src

HugeGraph-ML is a path dependency of the root project but is not a uv workspace member. Select the ml extra at the repository root so uv installs it and its local client dependency according to workspace configuration.

Implemented Models

Every module below lives in hugegraph-ml/src/hugegraph_ml/models/. models/__init__.py re-exports nothing, so import from the module file directly.

ModelModuleEntry classUsed forPaper
AGNNagnn.pyAGNNNode classification1803.03735
APPNPappnp.pyAPPNPNode classification1810.05997
ARMAarma.pyARMA4NCNode classification1901.01343
BGNNbgnn.pyBGNNPredictorGradient boosting over node features combined with a GNN; the bundled example runs regression2101.08543
BGRLbgrl.pyBGRLSelf-supervised node embedding2102.06514
CARE-GNNcare_gnn.pyCAREGNNFraud detection2008.08692
Cluster-GCNcluster_gcn.pySAGENode classification with subgraph sampling1905.07953
C&Scorrect_and_smooth.pyMLP, CorrectAndSmooth, LabelPropagationCorrecting and smoothing base predictions2010.13993
DAGNNdagnn.pyDAGNNNode classification2007.09296
DeeperGCNdeepergcn.pyDeeperGCNNode classification with edge features2006.07739
DGIdgi.pyDGISelf-supervised node embedding1809.10341
DiffPooldiffpool.pyDiffPoolGraph classification1806.08804
GATNEgatne.pyDGLGATNEHeterogeneous network embedding1905.01669
GINgin_global_pool.pyGINGraph classification
GRACEgrace.pyGRACESelf-supervised node embedding2006.04131
GRANDgrand.pyGRANDNode classification2005.11079
JKNetjknet.pyJKNetNode classification1806.03536
MLPmlp.pyMLPClassifierDownstream classifier over learned embeddings
P-GNNpgnn.pyPGNNLink predictionyou19b
SEALseal.pyDGCNN, SEALDataLink prediction1802.09691

GIN accepts pooling values sum (default), mean, max, global_attention and set2set.

Reading Graph Data

HugeGraph2DGL in hugegraph-ml/src/hugegraph_ml/data/hugegraph2dgl.py opens a PyHugeClient and converts query results into DGL objects:

1from hugegraph_ml.data.hugegraph2dgl import HugeGraph2DGL
2
3hg2d = HugeGraph2DGL(
4    url="http://127.0.0.1:8080",
5    graph="hugegraph",
6    user="",
7    pwd="",
8    graphspace=None,
9)
MethodReturnsNotes
convert_graph(vertex_label, edge_label, feat_key="feat", label_key="label", mask_keys=None)dgl.DGLGraphmask_keys falls back to ["train_mask", "val_mask", "test_mask"]
convert_hetero_graph(vertex_labels, edge_labels, feat_key="feat", label_key="label", mask_keys=None)DGL heterographTakes lists of labels
convert_graph_dataset(graph_vertex_label, vertex_label, edge_label, feat_key="feat", label_key="label")HugeGraphDatasetFills info with n_graphs, max_n_nodes, n_feat_dim, n_classes
convert_graph_nx(vertex_label, edge_label)networkx.GraphUsed by P-GNN
convert_graph_with_edge_feat(vertex_label, edge_label, node_feat_key="feat", edge_feat_key="edge_feat", label_key="label", mask_keys=None)dgl.DGLGraphAlso fills edata["feat"]
convert_graph_ogb(vertex_label, edge_label, split_label)(dgl.DGLGraph, split_edge)Used by SEAL
convert_hetero_graph_bgnn(vertex_labels, edge_labels, feat_key="feat", label_key="class", cat_key="cat_features", mask_keys=None)DGL heterographUsed by BGNN

Node features land in ndata["feat"], labels in ndata["label"] and each mask in ndata[<mask key>]. NodeEmbed requires feat only; NodeClassify, NodeClassifyWithEdge and NodeClassifyWithSample require feat, label, train_mask, val_mask and test_mask and raise ValueError when one is missing.

Importing Sample Datasets

hugegraph_ml.utils.dgl2hugegraph_utils writes DGL, OGB and NetworkX datasets into HugeGraph so the conversion layer has something to read. Every function takes the same url, graph, user, pwd and graphspace arguments as HugeGraph2DGL, and most upper-case the dataset name before matching it.

FunctionAccepted datasetsLabels created
import_graph_from_dglCORA, CITESEER, PUBMED<NAME>_vertex, <NAME>_edge
import_graphs_from_dglMUTAG, COLLAB, NCI1, PROTEINS, PTC, ENZYMES, DD<NAME>_graph_vertex, <NAME>_vertex, <NAME>_edge
import_hetero_graph_from_dglACM<NAME>_<ntype>_v, <NAME>_<etype>_e
import_hetero_graph_from_dgl_no_featAMAZONGATNE<NAME>_<ntype>_v, <NAME>_<etype>_e
import_hetero_graph_from_dgl_bgnnAVAZU<NAME>_<ntype>_v, <NAME>_<etype>_e
import_graph_from_nxCAVEMAN<NAME>_vertex, <NAME>_edge
import_graph_from_dgl_with_edge_featCORA, CITESEER, PUBMED<NAME>_edge_feat_vertex, <NAME>_edge_feat_edge
import_graph_from_ogbogbl-collab, matched without upper-casing<NAME>_vertex, <NAME>_edge
import_split_edge_from_ogbogbl-collab, matched without upper-casing<NAME>_split_edge

Any other name raises ValueError("dataset not supported"). import_split_edge_from_ogb additionally requires the idx_to_vertex_id mapping and a max_nodes cap returned by the vertex import.

clear_all_data() drops every vertex and edge in the target graph. The test fixture calls it, loads CORA, MUTAG and ACM, and calls it again on teardown.

AMAZONGATNE and AVAZU are not fetched automatically. Their archive URLs are recorded in comments above import_hetero_graph_from_dgl_no_feat and import_hetero_graph_from_dgl_bgnn.

Tasks

Task classes live in hugegraph-ml/src/hugegraph_ml/tasks/. Each one takes the converted graph and a model instance.

ClassModuleEntry points
NodeEmbednode_embed.pytrain_and_embed(add_self_loop=True, lr=1e-3, weight_decay=0, n_epochs=200, patience=inf, gpu=-1) returns the graph with ndata["feat"] replaced by the embedding
NodeClassifynode_classify.pytrain(lr, weight_decay, n_epochs, patience, early_stopping_monitor, gpu) then evaluate(), which returns {"accuracy": ..., "loss": ...}
NodeClassifyWithEdgenode_classify_with_edge.pySame shape, for models that also read edata["feat"]
NodeClassifyWithSamplenode_classify_with_sample.pyCluster-GCN style training on ClusterGCNSampler partitions; runs on CPU and takes no gpu argument
GraphClassifygraph_classify.pytrain(batch_size=20, lr, weight_decay, n_epochs, patience, early_stopping_monitor, clip=2.0, gpu) over a HugeGraphDataset, split 70/20/10
DetectorCaregnnfraud_detector_caregnn.pyCARE-GNN training; evaluate() reports recall and ROC AUC and reads ndata["feature"] rather than ndata["feat"]
HeteroSampleEmbedGATNEhetero_sample_embed_gatne.pytrain_and_embed(lr=1e-3, n_epochs=200, gpu=-1)
LinkPredictionPGNNlink_prediction_pgnn.pytrain(lr, weight_decay, n_epochs, gpu)
LinkPredictionSeallink_prediction_seal.pyThe constructor calls data_prepare() itself, then train(lr=1e-3, n_epochs=200, gpu=-1)

patience defaults to float("inf"). EarlyStopping in utils/early_stopping.py monitors either loss or accuracy, keeps a copy of the best weights and restores them when training stops.

Runnable Examples

Scripts sit in hugegraph-ml/src/hugegraph_ml/examples/. From hugegraph-ml/src, run one with:

1python ./hugegraph_ml/examples/dgi_example.py

Each script also exposes a function of the same name, so it can be imported and called with a smaller epoch count.

ScriptModelTaskReads
agnn_example.pyAGNNNodeClassifyCORA_vertex, CORA_edge
appnp_example.pyAPPNPNodeClassifyCORA_vertex, CORA_edge
arma_example.pyARMA4NCNodeClassifyCORA_vertex, CORA_edge
bgnn_example.pyBGNNPredictorIts own fit()AVAZU__N_v, AVAZU__E_e
bgrl_example.pyBGRLNodeEmbed, NodeClassifyCORA_vertex, CORA_edge
care_gnn_example.pyCAREGNNDetectorCaregnnAMAZON_user_v plus AMAZON_net_upu_e, AMAZON_net_usu_e, AMAZON_net_uvu_e
cluster_gcn_example.pySAGENodeClassifyWithSampleCORA_vertex, CORA_edge
correct_and_smooth_example.pyMLP from correct_and_smoothNodeClassifyCORA_vertex, CORA_edge
dagnn_example.pyDAGNNNodeClassifyCORA_vertex, CORA_edge
deepergcn_example.pyDeeperGCNNodeClassifyWithEdgeCORA_vertex, CORA_edge through convert_graph_with_edge_feat
dgi_example.pyDGINodeEmbed, NodeClassifyCORA_vertex, CORA_edge
diffpool_example.pyDiffPoolGraphClassifyMUTAG_graph_vertex, MUTAG_vertex, MUTAG_edge
gatne_example.pyDGLGATNEHeteroSampleEmbedGATNEAMAZONGATNE__N_v, AMAZONGATNE_1_e, AMAZONGATNE_2_e
gin_example.pyGINGraphClassifyMUTAG_graph_vertex, MUTAG_vertex, MUTAG_edge
grace_example.pyGRACENodeEmbed, NodeClassifyCORA_vertex, CORA_edge
grand_example.pyGRANDNodeClassifyCORA_vertex, CORA_edge
jknet_example.pyJKNetNodeClassifyCORA_vertex, CORA_edge
pgnn_example.pyPGNNLinkPredictionPGNNCAVEMAN_vertex, CAVEMAN_edge
seal_example.pyDGCNNLinkPredictionSealogbl-collab_vertex, ogbl-collab_edge, ogbl-collab_split_edge

DGI Node Embedding Example

First import DGL’s Cora dataset into HugeGraph. The name is upper-cased before use, so cora and CORA both produce the CORA_vertex and CORA_edge labels:

1from hugegraph_ml.utils.dgl2hugegraph_utils import import_graph_from_dgl
2
3import_graph_from_dgl("cora")

Read the graph and train DGI:

 1from hugegraph_ml.data.hugegraph2dgl import HugeGraph2DGL
 2from hugegraph_ml.models.dgi import DGI
 3from hugegraph_ml.models.mlp import MLPClassifier
 4from hugegraph_ml.tasks.node_classify import NodeClassify
 5from hugegraph_ml.tasks.node_embed import NodeEmbed
 6
 7hg2d = HugeGraph2DGL()
 8graph = hg2d.convert_graph(vertex_label="CORA_vertex", edge_label="CORA_edge")
 9
10embed_model = DGI(n_in_feats=graph.ndata["feat"].shape[1])
11embed_task = NodeEmbed(graph=graph, model=embed_model)
12embedded_graph = embed_task.train_and_embed(
13    add_self_loop=True, n_epochs=300, patience=30
14)
15
16classifier = MLPClassifier(
17    n_in_feat=embedded_graph.ndata["feat"].shape[1],
18    n_out_feat=embedded_graph.ndata["label"].unique().shape[0],
19)
20classify_task = NodeClassify(graph=embedded_graph, model=classifier)
21classify_task.train(lr=1e-3, n_epochs=400, patience=40)
22print(classify_task.evaluate())

evaluate() returns a dictionary such as {'accuracy': 0.82, 'loss': 0.5714246034622192}. The complete script is hugegraph-ml/src/hugegraph_ml/examples/dgi_example.py.

GRAND Node Classification Example

 1from hugegraph_ml.data.hugegraph2dgl import HugeGraph2DGL
 2from hugegraph_ml.models.grand import GRAND
 3from hugegraph_ml.tasks.node_classify import NodeClassify
 4
 5hg2d = HugeGraph2DGL()
 6graph = hg2d.convert_graph(vertex_label="CORA_vertex", edge_label="CORA_edge")
 7model = GRAND(
 8    n_in_feats=graph.ndata["feat"].shape[1],
 9    n_out_feats=graph.ndata["label"].unique().shape[0],
10)
11task = NodeClassify(graph, model)
12task.train(lr=1e-2, weight_decay=5e-4, n_epochs=2000, patience=100)
13print(task.evaluate())

GRAND returns a list of logits per augmentation sample, and NodeClassify masks each element of that list before computing the loss. The complete script is hugegraph-ml/src/hugegraph_ml/examples/grand_example.py.

Troubleshooting

  • Connection failures: check the HugeGraph Server address, port, and credentials.
  • Schema mismatches: the examples use CORA_vertex and CORA_edge; pass the actual labels for your own data.
  • ValueError: Graph is missing required node attribute ...: the node classification tasks need feat, label, train_mask, val_mask and test_mask in ndata. Import a dataset that carries masks, or pass your own mask_keys to convert_graph.
  • ValueError: dataset not supported: the importer only accepts the names in the table above, and import_graph_from_ogb matches ogbl-collab without upper-casing.
  • DGL or PyTorch import failures: rerun uv sync --extra ml from the repository root and confirm that Python comes from the root .venv.
  • bgrl_example.py currently fails on import: it asks for MLP_Predictor from hugegraph_ml.models.bgrl, but that module defines the class as MLPPredictor.
  • care_gnn_example.py reads AMAZON_user_v and the three AMAZON_net_*_e edge labels. No bundled importer creates them, so load that dataset yourself before running the script.

3.3 - HugeGraph-LLM Workflow

Follow the Web UI to build indexes, query documents, and generate Gremlin. See HugeGraph-LLM for startup instructions.

Prepare a reachable HugeGraph Server, a generation model, and an embedding model. Viewing settings or running handwritten Gremlin in Graph Tools needs no model. Document vectors require embeddings; graph extraction and answers also require an LLM. The reranker is optional. Start with the local bleu method.

TaskUI tabPrerequisites
Build indexes from documents1. Build RAG IndexDocuments and embeddings; graph extraction also needs a schema and LLM
Answer from documents or a graph2. (Graph)RAG & User FunctionsThe corresponding indexes and an LLM
Convert a question to Gremlin3. Text2gremlinGraph schema and LLM; examples also need embeddings and an example index
Check graph data directly4. Graph ToolsA HugeGraph connection and handwritten Gremlin

The screenshots show the HugeGraph-LLM Web UI. The built-in prompt language can be toggled via LANGUAGE in .env, while UI controls remain in English.

Missing batch-question template when starting from source

If startup raises FileNotFoundError for questions_template.xlsx, create the input template from the repository root after installing dependencies, then restart. This preserves an existing file. The template has all 6 columns, with a test question and expected answer; the 4 model-answer columns stay empty.

 1uv run --extra llm python - <<'PYTHON'
 2from pathlib import Path
 3from openpyxl import Workbook
 4
 5path = Path("hugegraph-llm/src/hugegraph_llm/resources/demo/questions_template.xlsx")
 6if not path.exists():
 7    workbook = Workbook()
 8    sheet = workbook.active
 9    sheet.append([
10        "Question", "Expected Answer", "Basic LLM Answer",
11        "Vector-only Answer", "Graph-only Answer", "Graph-Vector Answer",
12    ])
13    sheet.append(["Who is Sarah?", "Sarah is a 30-year-old attorney.", None, None, None, None])
14    workbook.save(path)
15PYTHON

0. Configuration Panel

Above the tabs sits a collapsible configuration panel with five sections: 1. Set up the HugeGraph server., 2. Set up the LLM., 3. Set up the Embedding., 4. Set up the Reranker., and 5. Set up the vector engine.. Each section has its own apply button, and applying a change writes the supported fields back to .env. The header also shows the current prompt language.

Set the answer, extraction, and query-generation models in the chat, mini_tasks, and text2gql sub-tabs of the LLM settings. The screenshots use local Ollama models qwen2.5:3b and mxbai-embed-large, with Faiss as the vector engine. Rebuild indexes after changing embedding models, since vector dimensions or meaning can differ.

1. Build RAG Indexes

The first tab splits documents into a chunk vector index. It also extracts vertices and edges according to a schema, writes them to HugeGraph, and maintains a vertex vector index.

flowchart TD
    A[Input document] --> B[Split text]
    B --> C[Generate chunk vectors]
    C --> D[Write vector index]
    B --> E[LLM extracts vertices and edges from schema]
    E --> F[Write to HugeGraph]
    F --> G[Update vertex vector index]

Input comes from either the text sub-tab or the file sub-tab. Uploads accept .txt, .docx, and .pdf, and several files can be selected at once.

For a first run, paste this short document into Input Doc(s). Select a dedicated test graph before importing it.

1Sarah is a 30-year-old attorney. James is a journalist. Sarah has been James's roommate since 2010.

Paste this JSON into Graph Schema and keep the default extraction prompt. It allows people and roommate relationships, with optional age and occupation.

 1{
 2  "vertexlabels": [
 3    {"id": 1, "name": "person", "id_strategy": "PRIMARY_KEY",
 4     "primary_keys": ["name"], "properties": ["name", "age", "occupation"],
 5     "nullable_keys": ["age", "occupation"]}
 6  ],
 7  "edgelabels": [
 8    {"name": "roommate", "source_label": "person", "target_label": "person", "properties": ["date"]}
 9  ],
10  "propertykeys": [
11    {"name": "name", "data_type": "TEXT", "cardinality": "SINGLE"},
12    {"name": "age", "data_type": "INT", "cardinality": "SINGLE"},
13    {"name": "occupation", "data_type": "TEXT", "cardinality": "SINGLE"},
14    {"name": "date", "data_type": "TEXT", "cardinality": "SINGLE"}
15  ]
16}

Follow these steps to build the indexes. For vector-only retrieval, stop after step 1.

  1. Click Import into Vector to split the document and build the chunk vector index.
  2. Enter schema JSON or an existing graph name in Graph Schema, then click Extract Graph Data (1). Check the vertices, edges, and properties in Output Info.
  3. Click Load into GraphDB (2) after checking the extraction. This writes to HugeGraph and refreshes vertex embeddings.
  4. Open Graph Tools. Run g.V().count() and g.E().count(), then query the properties to confirm that the data was written.

This example should produce 2 vertices and 1 edge, with Sarah’s age set to 30 and occupation to attorney. If the graph already contains data, use Update Vid Embedding to refresh its vertex index separately. The Graph Extraction Split Type dropdown selects document, paragraph, or sentence. document uses the whole input; the other options split it before extraction.

The page can also inspect or clear chunk indexes, vertex indexes, and graph data. Clearing removes existing data, so first confirm that the current graph and indexes are not still used by other queries.

Two collapsed helpers sit below the main controls:

  • Graph Schema Generator takes query examples and a few-shot example and produces a schema for the Graph Schema field.
  • Graph Extraction Prompt Generator takes an expected scenario, such as social relationships or a financial knowledge graph, and a selected reference example, and produces a Graph Extract Prompt Header.

2. GraphRAG Queries

The second tab offers four answer modes. Choose the indexes needed for your question.

ControlAnswer source
Basic LLM AnswerThe LLM alone, without document or graph retrieval
Vector-only AnswerRetrieved document chunks
Graph-only AnswerRetrieved graph data
Graph-Vector AnswerRetrieved graph data and document chunks

Using Graph-Vector Answer is recommended, as it combines graph traversal and vector search for more comprehensive fact retrieval. Other scopes are summarized in the table above.

The screenshots use these short prompts. Paste them into Query Prompt and Keywords Extraction Prompt, respectively. The service fills the placeholders. Keep the KEYWORDS: prefix and ASCII colons in keyword output, and preserve name capitalization.

1Answer the question in English using only the retrieved context. Preserve names and state when the context is insufficient.
2Context: {context_str}
3Question: {query_str}
1Extract person names from the question and preserve their case. Return at most {max_keywords} names.
2Return one line in this exact ASCII format: KEYWORDS:name:1.0
3Question: {question}
  1. Enter Who is Sarah?, set Graph-only Answer to False, and set Graph-Vector Answer to True.
  2. Keep Template Num=-1 and Rerank method=bleu for the first check, then click Answer Question.
  3. Read Graph-Vector Answer and compare facts such as age and occupation with the source document and graph.
flowchart TD
    Q[Question] --> V[Query chunk vector index]
    Q --> K[Extract keywords]
    K --> M[Match graph vertices]
    M --> D{Text2Gremlin enabled}
    D -->|Yes| T[Generate and execute Gremlin]
    D -->|No| B[BFS graph traversal]
    T -->|Failure| B
    T --> R[Prepare graph results]
    B --> R
    V --> S[Merge and rerank]
    R --> S
    S --> A[Generate answer]

Graph retrieval first matches HugeGraph vertices exactly by keyword and then uses vector similarity if no exact match exists. The matched vertices are passed to Text2Gremlin. If generation or execution fails, the pipeline can fall back to a predefined traversal.

Template Num controls how Text2Gremlin participates in graph retrieval:

  • A negative value skips Text2Gremlin entirely, so graph retrieval goes straight to the predefined traversal.
  • 0 uses baseline generation without retrieving examples.
  • A positive value retrieves that many similar examples from the example index and uses the template-guided result. The example count is clamped to the range 0 to 10.

Other controls on this tab are Rerank method (bleu or reranker), Graph Ratio, Near neighbor first, and Query related information, plus editable Query Prompt and Keywords Extraction Prompt fields.

Below the single-question panel is a batch back-testing panel. Upload an .xlsx or .csv file of questions, set Max Lines To Show, and click Generate Answer (Batch). The answers appear in a preview table and can be downloaded as a file. A template file is offered next to the upload control.

3. Text2Gremlin

The third tab converts natural language into Gremlin and executes the generated queries. The upper panel builds an example vector index from question and Gremlin pairs in .json or .csv files. When no file is uploaded, it uses resources/demo/text2gremlin.csv. Match example labels and properties to the current graph.

Tip

If building the example index does not automatically retrieve similar templates, you can provide few-shot examples directly in the Prompt template below.

Use these inputs to reproduce the Sarah query.

  1. Save the CSV below as sarah-gremlin.csv, upload it, and click Build Example Vector Index. Set Number of refer examples to 1.
  2. Paste the simplified schema below into Schema. It contains only the labels and properties needed for this query.
  3. Paste the prompt below into Prompt, enter Who is Sarah?, then click Text2Gremlin.
  4. Check both generated queries and their Query ... Output fields. This example returns Sarah with age=30 and occupation=attorney.
1query,gremlin
2Who is Sarah?,"g.V().has('person', 'name', 'Sarah').valueMap()"
1{"vertexlabels":[{"name":"person","properties":["name","age","occupation"]}],"edgelabels":[]}
 1Translate the question into a read-only HugeGraph Gremlin query.
 2For person-name lookups, use g.V().has('person', 'name', '<name from question>').valueMap().
 3Preserve name capitalization. Query by name; do not use vertex IDs.
 4Example question: Who is Peter?
 5Example Gremlin: g.V().has('person', 'name', 'Peter').valueMap()
 6Graph schema: {schema}
 7Additional examples: {example}
 8Known vertices: {vertices}
 9Question: {query}
10Return only one ```gremlin``` code block.

A small model can generate incorrect IDs, change name capitalization, or use filters that do not meet the graph’s index requirements. Check the returned data before treating a query as successful. The Web UI executes generated queries directly, so use a dedicated test graph. To generate a query without executing it, select the corresponding output types in the REST API.

A custom prompt must retain {query}, {schema}, {example}, and {vertices}. The REST API rejects it if any placeholder is missing.

4. Graph and Administration Tools

Graph Tools runs a Gremlin query directly against the configured graph, triggers a manual graph backup, and can initialize demo data in HugeGraph through a beta action. A background job also backs up the graph every day at 01:00, and a second background task keeps vertex-id embeddings up to date while the process runs. Initializing demo data first clears the current graph. Use this action only on a dedicated test graph.

Admin Tools is password protected. Entering the configured ADMIN_TOKEN reveals the tail of logs/llm-server.log, which refreshes every 60 seconds, along with buttons to refresh or clear it. Access is refused while ADMIN_TOKEN is empty or still set to the placeholder xxxx.

When ENABLE_LOGIN=True, the Web UI asks for basic credentials with the fixed user name rag and USER_TOKEN as the password, and the REST API requires USER_TOKEN as a Bearer token. The log endpoint additionally requires a separately configured, secure ADMIN_TOKEN.

Warning

In production, enable HugeGraph-LLM login, replace USER_TOKEN and ADMIN_TOKEN, and enforce a source IP allowlist at the firewall or network entry point. Separately enable Server authentication and authorization, retain Server audit logs (normally audit-*.log), and grant GRAPH_USER minimum required permissions. AI service tokens do not replace Server authentication.

5. Prompt Language

Set LANGUAGE=EN or LANGUAGE=CN in hugegraph-llm/.env, then restart the service. This selects the language of built-in prompts; it does not translate input documents and is not a field in the /rag request body.

6. REST Calls

The Web UI and REST API use the same pipeline. For application integration, use /rag, /rag/graph, /graph/extract, and /text2gremlin; see the REST API for request formats.

3.4 - Configuration Reference

HugeGraph-LLM reads runtime configuration from .env and prompts from config_prompt.yaml. These files have different path-resolution rules; the prompt file is not part of .env.

The .env path is resolved in this order:

  1. HUGEGRAPH_LLM_ENV_PATH, if that environment variable is set. A leading ~ is expanded.
  2. hugegraph-llm/.env, when the package runs from a source checkout.
  3. .env in the current working directory, for an installed package.

The path is selected when the configuration module is imported. Set HUGEGRAPH_LLM_ENV_PATH before starting Python, not inside the .env file that will be loaded. Relative overrides are resolved against the process working directory.

The prompt YAML path is resolved in this order:

  1. HUGEGRAPH_LLM_PROMPT_CONFIG_PATH from the process environment, expanding a leading ~.
  2. hugegraph-llm/src/hugegraph_llm/resources/demo/config_prompt.yaml when running from source.
  3. ${XDG_CONFIG_HOME:-~/.config}/hugegraph-llm/config_prompt.yaml for an installed package.

Set path overrides before starting the process. Relative paths use its working directory. The source Docker image points PYTHONPATH at the source tree, so its default prompt path remains under hugegraph-llm/src/hugegraph_llm/resources/demo/.

Create or update files from configuration-class defaults with:

1cd hugegraph-ai/hugegraph-llm
2python -m hugegraph_llm.config.generate --update

--update is enabled by default, so running without arguments has the same effect. On first configuration-module import, missing .env and prompt YAML files are created from defaults. The generator asks interactively before overwriting existing files. It handles HugeGraph, administrator, LLM, index, and prompt settings without overwriting existing files silently.

.env contains keys and passwords. Do not commit it to version control.

Basic Options

SettingDefaultDescription
LANGUAGEENPrompt language: EN or CN
CHAT_LLM_TYPEopenaiAnswer model: openai, litellm, or ollama/local
EXTRACT_LLM_TYPEopenaiInformation extraction model; same choices as above
TEXT2GQL_LLM_TYPEopenaiText2Gremlin model; same choices as above
EMBEDDING_TYPEopenaiEmbedding model; same choices as above, or empty
RERANKER_TYPEemptycohere or siliconflow
KEYWORD_EXTRACT_TYPEllmllm, textrank, or hybrid
WINDOW_SIZE3TextRank window size, from 1 to 10
HYBRID_LLM_WEIGHTS0.5Weight of LLM results in hybrid mode, from 0 to 1

OpenAI-Compatible APIs

Chat, extraction, and Text2Gremlin can use different endpoints, keys, and models.

PurposeAPI baseKeyModelDefault maximum tokens
AnswerOPENAI_CHAT_API_BASEOPENAI_CHAT_API_KEYOPENAI_CHAT_LANGUAGE_MODELOPENAI_CHAT_TOKENS=8192
ExtractionOPENAI_EXTRACT_API_BASEOPENAI_EXTRACT_API_KEYOPENAI_EXTRACT_LANGUAGE_MODELOPENAI_EXTRACT_TOKENS=256
Text2GremlinOPENAI_TEXT2GQL_API_BASEOPENAI_TEXT2GQL_API_KEYOPENAI_TEXT2GQL_LANGUAGE_MODELOPENAI_TEXT2GQL_TOKENS=4096
EmbeddingOPENAI_EMBEDDING_API_BASEOPENAI_EMBEDDING_API_KEYOPENAI_EMBEDDING_MODELNot applicable

The default API base is https://api.openai.com/v1. The default language model for all three tasks is gpt-4.1-mini, and the default embedding model is text-embedding-3-small.

OPENAI_BASE_URL and OPENAI_API_KEY provide general fallback values. Embeddings also support OPENAI_EMBEDDING_BASE_URL and OPENAI_EMBEDDING_API_KEY as fallback values.

LiteLLM

PurposeAPI baseKeyModelDefault maximum tokens
AnswerLITELLM_CHAT_API_BASELITELLM_CHAT_API_KEYLITELLM_CHAT_LANGUAGE_MODELLITELLM_CHAT_TOKENS=8192
ExtractionLITELLM_EXTRACT_API_BASELITELLM_EXTRACT_API_KEYLITELLM_EXTRACT_LANGUAGE_MODELLITELLM_EXTRACT_TOKENS=256
Text2GremlinLITELLM_TEXT2GQL_API_BASELITELLM_TEXT2GQL_API_KEYLITELLM_TEXT2GQL_LANGUAGE_MODELLITELLM_TEXT2GQL_TOKENS=4096
EmbeddingLITELLM_EMBEDDING_API_BASELITELLM_EMBEDDING_API_KEYLITELLM_EMBEDDING_MODELNot applicable

The default language model is openai/gpt-4.1-mini, and the default embedding model is openai/text-embedding-3-small. Model names generally use the provider/model form; supported values depend on the LiteLLM service.

Ollama

PurposeHostPortModel
AnswerOLLAMA_CHAT_HOSTOLLAMA_CHAT_PORTOLLAMA_CHAT_LANGUAGE_MODEL
ExtractionOLLAMA_EXTRACT_HOSTOLLAMA_EXTRACT_PORTOLLAMA_EXTRACT_LANGUAGE_MODEL
Text2GremlinOLLAMA_TEXT2GQL_HOSTOLLAMA_TEXT2GQL_PORTOLLAMA_TEXT2GQL_LANGUAGE_MODEL
EmbeddingOLLAMA_EMBEDDING_HOSTOLLAMA_EMBEDDING_PORTOLLAMA_EMBEDDING_MODEL

The default host is 127.0.0.1 and the default port is 11434. Model names have no defaults; pull the required models in Ollama before use.

Reranking

SettingDefaultDescription
COHERE_BASE_URLhttps://api.cohere.com/v1/rerankCohere rerank endpoint; CO_API_URL is a fallback
RERANKER_API_KEYemptyCohere or SiliconFlow key
RERANKER_MODELemptyModel name supported by the service

HugeGraph Connection and Retrieval Limits

SettingDefaultDescription
GRAPH_URL127.0.0.1:8080HugeGraph address; it is not split into IP and port
GRAPH_NAMEhugegraphGraph name
GRAPH_USERadminUser name
GRAPH_PWDxxxPassword
GRAPH_SPACEemptyGraphSpace name
LIMIT_PROPERTYFalseWhether to limit returned properties; read as a string by the configuration class
MAX_GRAPH_PATH10Maximum graph path length
MAX_GRAPH_ITEMS30Maximum number of graph retrieval items
EDGE_LIMIT_PRE_LABEL8Result limit for each edge label
VECTOR_DIS_THRESHOLD0.9Results beyond this vector-distance threshold are ignored
TOPK_PER_KEYWORD1Candidates per keyword
TOPK_RETURN_RESULTS20Results returned after reranking

Vector Index Backend

SettingDefaultDescription
CUR_VECTOR_INDEXFaissActive vector store: Faiss, Milvus, or Qdrant
QDRANT_HOSTempty
QDRANT_PORT6333
QDRANT_API_KEYempty
MILVUS_HOSTempty
MILVUS_PORT19530
MILVUS_USERempty
MILVUS_PASSWORDempty

FAISS is local and needs no extra dependency. Selecting Milvus or Qdrant without the optional dependencies raises an error that names the missing package, so install them first:

1cd hugegraph-ai
2uv sync --package hugegraph-llm --extra vectordb

The same choice is available in the 5. Set up the vector engine. panel of the Web UI, which also persists the connection settings for the selected engine.

Login and Log API

SettingDefaultDescription
ENABLE_LOGINFalseWhether to require a Bearer token; read as a string by the configuration class
USER_TOKEN4321Token for the Web UI and regular APIs
ADMIN_TOKENxxxxAdministrator token used by /logs

/logs returns 403 when ADMIN_TOKEN is empty or still set to xxxx.

Warning

In production, set ENABLE_LOGIN=True, replace USER_TOKEN and ADMIN_TOKEN, and enforce a source IP allowlist at the firewall or network entry point. This protects only HugeGraph-LLM. Separately enable Server authentication and authorization, retain Server audit logs (normally audit-*.log), and grant GRAPH_USER minimum required permissions. The two services use different credentials.

Minimal OpenAI Configuration

 1LANGUAGE=EN
 2CHAT_LLM_TYPE=openai
 3EXTRACT_LLM_TYPE=openai
 4TEXT2GQL_LLM_TYPE=openai
 5EMBEDDING_TYPE=openai
 6
 7OPENAI_API_KEY=your-api-key
 8OPENAI_BASE_URL=https://api.openai.com/v1
 9OPENAI_CHAT_LANGUAGE_MODEL=gpt-4.1-mini
10OPENAI_EXTRACT_LANGUAGE_MODEL=gpt-4.1-mini
11OPENAI_TEXT2GQL_LANGUAGE_MODEL=gpt-4.1-mini
12OPENAI_EMBEDDING_MODEL=text-embedding-3-small
13
14GRAPH_URL=127.0.0.1:8080
15GRAPH_NAME=hugegraph
16GRAPH_USER=admin
17GRAPH_PWD=your-password

Configuration Loading

Configuration classes supply code defaults. During initialization, values from the selected .env are written into the process environment before the configuration objects are created, overriding same-named shell variables. Missing keys use defaults; empty values and unknown keys are ignored. The Web UI and configuration APIs can update settings and write supported fields back to .env. Restart after manual .env edits; prompt YAML is read at service startup or page load.

Keys are matched case-insensitively.

Configuration definitions are in:

  • hugegraph-llm/src/hugegraph_llm/config/llm_config.py
  • hugegraph-llm/src/hugegraph_llm/config/hugegraph_config.py
  • hugegraph-llm/src/hugegraph_llm/config/index_config.py
  • hugegraph-llm/src/hugegraph_llm/config/admin_config.py
  • hugegraph-llm/src/hugegraph_llm/config/prompt_config.py
  • hugegraph-llm/src/hugegraph_llm/config/models/base_config.py for the loading and file-sync behaviour

3.5 - HugeGraph-LLM REST API

The HugeGraph-LLM demo process serves both the Web UI and REST API. The default address is http://localhost:8001:

1cd hugegraph-ai/hugegraph-llm
2python -m hugegraph_llm.demo.rag_demo.app \
3  --host 127.0.0.1 \
4  --port 8001

All endpoints are POST:

PathSuccess statusPurpose
/rag200Answer a question with the selected retrieval modes
/rag/graph200Graph retrieval only, without a final answer
/graph/extract200Extract vertices and edges from text
/text2gremlin200Generate Gremlin from natural language
/config/graph201Update the HugeGraph connection
/config/llm201Update the language model
/config/embedding201Update the embedding model
/config/rerank201Update the reranker
/logs200Stream the server log

Authentication

Enable login in .env:

1ENABLE_LOGIN=True
2USER_TOKEN=replace-with-a-secret

Requests then require a Bearer token:

1Authorization: Bearer replace-with-a-secret

The same setting puts the Gradio UI behind basic authentication, with the fixed user name rag and USER_TOKEN as the password. A wrong token returns 401 with a WWW-Authenticate: Bearer header. When ENABLE_LOGIN is left at False, every endpoint is open.

Warning

Production requires HugeGraph-LLM login and a source IP allowlist at the firewall or network entry point. This protects only the LLM UI and REST API. Separately enable Server authentication and authorization, retain Server audit logs (normally audit-*.log), and grant GRAPH_USER minimum permissions. USER_TOKEN does not replace Server authentication.

RAG

POST /rag

Returns one or more answer types according to the switches. When none is explicitly selected, only graph_only is enabled.

 1curl -X POST http://localhost:8001/rag \
 2  -H 'Content-Type: application/json' \
 3  -d '{
 4    "query": "Which movies feature Al Pacino?",
 5    "raw_answer": false,
 6    "vector_only": false,
 7    "graph_only": true,
 8    "graph_vector_answer": false,
 9    "max_graph_items": 30,
10    "topk_return_results": 20,
11    "vector_dis_threshold": 0.9,
12    "topk_per_keyword": 1,
13    "gremlin_tmpl_num": 1,
14    "client_config": {
15      "url": "127.0.0.1:8080",
16      "graph": "hugegraph",
17      "user": "admin",
18      "pwd": "admin",
19      "gs": "DEFAULT"
20    }
21  }'

The response contains only enabled answer fields:

1{
2  "query": "Which movies feature Al Pacino?",
3  "graph_only": "..."
4}

Other optional parameters include graph_ratio (default 0.5), rerank_method (bleu or reranker, default bleu), near_neighbor_first (default false), custom_priority_info, and the three custom prompt fields answer_prompt, keywords_extract_prompt, and gremlin_prompt. Omitting a prompt field uses the value from config_prompt.yaml.

gremlin_tmpl_num selects how Text2Gremlin runs during graph retrieval. A negative value skips Text2Gremlin and goes straight to the predefined traversal, 0 generates Gremlin without examples, and a positive value retrieves that many examples from the example index.

An empty or whitespace-only query returns 400.

POST /rag/graph

Runs graph retrieval without generating a final natural-language answer:

1curl -X POST http://localhost:8001/rag/graph \
2  -H 'Content-Type: application/json' \
3  -d '{
4    "query": "Which movies feature Al Pacino?",
5    "get_vertex_only": false,
6    "gremlin_tmpl_num": 1,
7    "rerank_method": "bleu"
8  }'

graph_recall in the response can contain query, keywords, match_vids, graph_result_flag, gremlin, graph_result, and vertex_degree_list. Set get_vertex_only=true to return immediately after vertex matching; the endpoint then replaces match_vids with the full vertex details.

An empty query returns 400, a type error in the request returns 400, and any other failure returns 500.

Graph Extraction

POST /graph/extract

An inline schema does not connect to HugeGraph:

 1curl -X POST http://localhost:8001/graph/extract \
 2  -H 'Content-Type: application/json' \
 3  -d '{
 4    "texts": ["Alice works at Acme."],
 5    "schema": {
 6      "vertexlabels": [
 7        {"name": "person", "properties": ["name"]},
 8        {"name": "company", "properties": ["name"]}
 9      ],
10      "edgelabels": [
11        {
12          "name": "works_at",
13          "source_label": "person",
14          "target_label": "company",
15          "properties": []
16        }
17      ]
18    },
19    "language": "en",
20    "split_type": "sentence",
21    "include_meta": true
22  }'

Request fields:

FieldDefaultNotes
textsrequiredA string or an array of strings; empty or blank entries are dropped and an empty result is rejected
schemarequiredInline JSON object or string, or the name of an existing graph
example_promptprompt YAML valueExtraction prompt header
extract_typeproperty_graphOnly value currently accepted
languagezhzh or en, used for chunk splitting
split_typedocumentdocument, paragraph, or sentence
include_metafalseAdds vertex_count, edge_count, and text_count to meta
review_optionsnone (disabled)Optional JSON object enabling model review and bounded repair: max_reviews, max_fixes, max_workers
client_confignoneOnly allowed with a graph-name schema

An inline schema must be an object with vertexlabels and edgelabels lists. Every vertex label needs a non-empty name and a non-empty properties list; every edge label needs a non-empty name, source_label, and target_label. propertykeys is optional and must be a list when present.

When schema is an existing graph name, also pass client_config, and make client_config.graph match that name. client_config here accepts only graph, user, pwd, and gs; unknown fields are rejected, and there is no url field:

 1{
 2  "texts": "Alice works at Acme.",
 3  "schema": "hugegraph",
 4  "client_config": {
 5    "graph": "hugegraph",
 6    "user": "admin",
 7    "pwd": "admin",
 8    "gs": "DEFAULT"
 9  }
10}

Model Review and Repair (review_options)

Providing review_options enables iterative model review and bounded schema repair for each text chunk:

1{
2  "texts": ["Alice works at Acme Corp.", "Bob works at Beta LLC."],
3  "schema": {...},
4  "review_options": {
5    "max_reviews": 2,
6    "max_fixes": 1,
7    "max_workers": 4
8  }
9}
  • max_reviews (default 2, range 0–5): Maximum review iterations per chunk.
  • max_fixes (default 1, range 0–3): Maximum bounded repair attempts per chunk.
  • max_workers (default 4, range 1–8): Number of chunks executed concurrently.
  • Passing {} uses the defaults. Omitting review_options or setting it to null uses legacy extraction without review.

Response Format

When review_options is omitted, status is always succeeded on HTTP 200. When review_options is enabled:

  • The overall status reflects cumulative chunk results: succeeded (all chunks final), partial (some chunks final), or failed (no chunks final).
  • Only final chunks are merged into top-level result.vertices and result.edges.
  • The response includes chunk_results detailing each chunk’s execution:
    • index: 0-based chunk position.
    • status: final (passed validation & review), candidate (passed validation; review budget exhausted), blocked (schema constraints unresolved), or failed (exception or truncation).
    • graph: The extracted sub-graph object (vertices and edges), or null on failure.
    • issues: List of issues found during validation or review, including path, message, and optional evidence.
    • reason and stage: Failure explanation and execution stage (prepare, extract, validate, review, or repair).
    • reviews_used, fixes_used, and llm_calls: Consumed model call counters.

Text2Gremlin

POST /text2gremlin

1curl -X POST http://localhost:8001/text2gremlin \
2  -H 'Content-Type: application/json' \
3  -d '{
4    "query": "Find all person vertices",
5    "example_num": 1,
6    "output_types": ["template_gremlin", "template_execution_result"]
7  }'

output_types can contain:

  • match_result
  • template_gremlin
  • raw_gremlin
  • template_execution_result
  • raw_execution_result

If omitted, only template_gremlin is returned by default. An empty array lets the implementation return all outputs. A custom gremlin_prompt must contain {query}, {schema}, {example}, and {vertices}; a missing placeholder fails request validation and names the placeholders that are absent.

example_num defaults to 0, which means no templates, and is clamped to the range 0 to 10. client_config overrides the HugeGraph connection for the request; the schema used for generation is the active graph name. An empty query returns 400, and a generation failure returns 500.

Runtime Configuration

POST /config/graph

1{
2  "url": "127.0.0.1:8080",
3  "graph": "hugegraph",
4  "user": "admin",
5  "pwd": "admin",
6  "gs": "DEFAULT"
7}

user and pwd default to empty strings, and gs is optional.

POST /config/llm and POST /config/embedding

Both endpoints use the same request model. /config/llm sets chat_llm_type, extract_llm_type, and text2gql_llm_type to the same value; per-task types can only be set separately through .env or the Web UI. OpenAI or LiteLLM example:

1{
2  "llm_type": "openai",
3  "api_key": "your-key",
4  "api_base": "https://api.openai.com/v1",
5  "language_model": "gpt-4.1-mini",
6  "max_tokens": "4096"
7}

Ollama requests still require the common fields; api_key and api_base can be empty strings:

1{
2  "llm_type": "ollama/local",
3  "api_key": "",
4  "api_base": "",
5  "language_model": "qwen2.5:7b",
6  "host": "127.0.0.1",
7  "port": "11434"
8}

POST /config/rerank

1{
2  "reranker_type": "siliconflow",
3  "reranker_model": "BAAI/bge-reranker-v2-m3",
4  "api_key": "your-key"
5}

reranker_type accepts cohere or siliconflow. Cohere also accepts cohere_base_url.

All four configuration endpoints return 201 on success. They change the process’s active configuration and may write values back to .env. /config/llm, /config/embedding, and /config/rerank restore the previous values if applying a change raises; /config/graph does not.

client_config in /rag, /rag/graph, and /text2gremlin overrides the HugeGraph connection for one request, and only the fields actually present in the request are applied. The current implementation still changes process-global settings temporarily, so do not issue long-running requests with different connections concurrently.

Logs

POST /logs

This endpoint requires ADMIN_TOKEN in .env to be changed to a secure value. Example request body:

1{
2  "admin_token": "replace-with-an-admin-secret",
3  "log_file": "llm-server.log"
4}

log_file defaults to llm-server.log, must be a file name under logs/, and cannot be absolute, contain path separators, or resolve to . or ... Invalid names return 400.

An unset or placeholder ADMIN_TOKEN returns 403 before the token is even compared, and a wrong token returns a 403 body with the message Invalid admin_token.

The successful response is a text/plain stream that first replays the last 125 lines of the file and then follows it, in the manner of tail -f.

3.6 - Vermeer Python Client

vermeer-python-client is the Python SDK for Vermeer, the memory-first graph computing engine written in Go. The SDK wraps the REST API of the Vermeer master so you can list graphs, submit load and compute tasks, and read task state from Python. The import package is pyvermeer.

The module does not pin a Vermeer server version. It talks to the Vermeer master over HTTP using the endpoints listed in API Surface.

Requirements

  • Current source requires Python 3.10 or later because its type annotations use Python 3.10 features, although packaging metadata still declares >=3.9.
  • A running Vermeer master reachable over HTTP on its default port 6688. Docker deployments must publish 6688:6688; see the Vermeer quick start.
  • uv (recommended) or pip

Runtime dependencies: requests, urllib3, python-dateutil, decorator, rich, and setuptools.

Installation

The distribution name in the packaging metadata is vermeer-python-client and the version is managed independently of the repository version. The package is not published on PyPI yet, so install it from source.

From the root of the HugeGraph-AI repository, the vermeer extra installs it into the shared virtual environment:

1git clone https://github.com/apache/hugegraph-ai.git
2cd hugegraph-ai
3uv sync --extra vermeer
4source .venv/bin/activate

vermeer-python-client is wired in as an editable path dependency rather than a uv workspace member, so a plain uv sync at the repository root does not install it. You have to ask for the extra (or for --all-extras).

To install the module standalone:

1git clone https://github.com/apache/hugegraph-ai.git
2cd hugegraph-ai/vermeer-python-client
3uv sync
4source .venv/bin/activate

Connect to a Vermeer Master

1from pyvermeer.client.client import PyVermeerClient
2
3client = PyVermeerClient(
4    ip="127.0.0.1",
5    port=6688,
6    token="",
7    timeout=(0.5, 15.0),
8    log_level="INFO",
9)

Constructor parameters:

ParameterTypeDefaultDescription
ipstrrequiredHost name or IP address of the Vermeer master
portintrequiredREST port of the Vermeer master
tokenstrrequiredSent verbatim as the Authorization request header
timeout(float, float) or NoneNoneConnect and read timeouts in seconds
log_levelstr"INFO"Level applied to the shared VermeerClient logger

For a client running on the host, use a master address reachable from that host. For a container client, use its container-network address. The client connects only to the master HTTP API; PD and Store addresses must be reachable from the Vermeer nodes that perform loading.

Behavior worth knowing before you connect:

  • token may be an empty string when the master does not check authorization, but it cannot be None. The session raises ValueError("Vermeer Token must be provided.") in that case.
  • timeout is a (connect, read) pair. VermeerConfig has its own default of (0.5, 15.0), but the client always forwards its own argument, so omitting timeout stores None and the request waits without a deadline. Pass the pair explicitly if you want one.
  • The base URL is always built as http://{ip}:{port}/, so the client speaks plain HTTP.
  • Every request sets Content-Type: application/json and serializes params into the request body, including for GET requests.
  • The session configures up to 3 retries on HTTP 500, 502, and 504 with backoff factor 0.1. Status retries follow urllib3’s default allowed methods, which exclude POST; do not rely on automatic retries for task submission.
  • log_level sets the level of the shared logger named VermeerClient. Its console handler is fixed at INFO, so DEBUG records are not printed to the console today.

End-to-End Example

The module ships vermeer-python-client/src/pyvermeer/demo/task_demo.py. This extended example polls with a deadline, waits for loading, reads the graph, and submits PageRank. It reads PD peers and the HugeGraph password from environment variables. The worker group must match the task-space allocation; a single-worker quick start can use worker_group=$. Bind a named group to the task space first, as shown in the Vermeer quick start.

 1import os
 2import time
 3
 4from pyvermeer.client.client import PyVermeerClient
 5from pyvermeer.structure.task_data import TaskCreateRequest
 6
 7client = PyVermeerClient(
 8    ip="127.0.0.1",
 9    port=6688,
10    token=os.getenv("VERMEER_TOKEN", ""),
11    timeout=(0.5, 15.0),
12    log_level="INFO",
13)
14
15# List tasks on the master
16tasks = client.tasks.get_tasks()
17print(tasks.to_dict())
18
19# Load graph data from HugeGraph into Vermeer
20create_response = client.tasks.create_task(
21    create_task=TaskCreateRequest(
22        task_type="load",
23        graph_name="DEFAULT-example",
24        params={
25            "load.hg_pd_peers": os.environ["VERMEER_PD_PEERS"],
26            "load.hugegraph_name": "DEFAULT/example/g",
27            "load.hugegraph_username": "admin",
28            "load.hugegraph_password": os.environ["HUGEGRAPH_PASSWORD"],
29            "load.parallel": "10",
30            "load.type": "hugegraph",
31        },
32    )
33)
34print(create_response.errcode, create_response.message)
35if create_response.errcode != 0:
36    raise RuntimeError(f"Could not create load task: {create_response.message}")
37
38def wait_task(task_id, success_state, poll_timeout=300.0):
39    deadline = time.monotonic() + poll_timeout
40    while True:
41        response = client.tasks.get_task(task_id)
42        if response.errcode != 0:
43            raise RuntimeError(f"Could not read task {task_id}: {response.message}")
44        task = response.task
45        print(task_id, task.state)
46        if task.state == success_state:
47            return task
48        if task.state in ("error", "canceled"):
49            raise RuntimeError(f"Task {task_id} ended with state {task.state}")
50        remaining = deadline - time.monotonic()
51        if remaining <= 0:
52            raise TimeoutError(f"Task {task_id} did not finish within {poll_timeout}s")
53        time.sleep(min(1.0, remaining))
54
55
56# Wait for loading before reading the graph
57wait_task(create_response.task.id, success_state="loaded")
58
59# Inspect the loaded graph
60print(client.graph.get_graph("DEFAULT-example").to_dict())
61
62# Submit PageRank on the loaded graph
63compute_response = client.tasks.create_task(
64    create_task=TaskCreateRequest(
65        task_type="compute",
66        graph_name="DEFAULT-example",
67        params={
68            "compute.algorithm": "pagerank",
69            "compute.parallel": "10",
70            "compute.max_step": "10",
71            "output.type": "local",
72            "output.parallel": "1",
73            "output.file_path": "result/pagerank",
74        },
75    )
76)
77if compute_response.errcode != 0:
78    raise RuntimeError(f"Could not create compute task: {compute_response.message}")
79wait_task(compute_response.task.id, success_state="complete")

Loading succeeds with loaded; computation succeeds with complete. error or canceled stops the example. Set VERMEER_PD_PEERS to a JSON array string, for example export VERMEER_PD_PEERS='["hugegraph-pd:8686"]'. The master must reach these PD addresses, and workers must reach the Store addresses returned by PD. Set VERMEER_TOKEN only if the master uses token authentication; an empty string works without it.

output.type=local writes results to the executing worker’s local filesystem with file-name prefix result/pagerank. Mount its result directory to read container output from the host. poll_timeout is the client polling deadline; requests between checks remain subject to HTTP timeouts and SDK retries, so completion can exceed that deadline. A timeout stops client waiting, not the Server task.

Never hardcode a real HugeGraph password into a script or configuration file. Use an environment variable or credential store.

Save and Run the Documentation Example

Save the complete extended example above as vermeer_client_example.py at the hugegraph-ai/ repository root. Run it with the workspace’s vermeer extra:

1uv sync --extra vermeer
2export VERMEER_PD_PEERS='["hugegraph-pd:8686"]'
3read -s -r HUGEGRAPH_PASSWORD
4export HUGEGRAPH_PASSWORD
5uv run --extra vermeer python vermeer_client_example.py

Enter the password and press Enter. Replace hugegraph-pd:8686 with a PD address reachable from the Vermeer master. If token authentication is enabled, supply VERMEER_TOKEN through your local credential-management method. These commands run the extended documentation example, not the bundled demo.

Original Bundled Demo

vermeer-python-client/src/pyvermeer/demo/task_demo.py is a separate minimal example. It hardcodes client port 8688, PD address 127.0.0.1:8686, and placeholder credentials xxx. It does not read the extended example’s environment variables, poll task state, or compute PageRank. Adjust its addresses and credentials separately if running it; it does not replace the documentation commands above.

API Surface

PyVermeerClient exposes its API groups as attributes. Two groups are registered today, graph and tasks.

client.graph

MethodVermeer endpointReturns
get_graphs()GET /graphsGraphsResponse
get_graph(graph_name)GET /graphs/{graph_name}GraphResponse

client.tasks

MethodVermeer endpointReturns
get_tasks()GET /tasksTasksResponse
get_task(task_id)GET /task/{task_id}TaskResponse
create_task(create_task)POST /tasks/createTaskCreateResponse

pyvermeer/api/master.py and pyvermeer/api/worker.py contain only the license header, and neither group is registered on the client. Master and worker information is therefore not reachable from the client yet, even though MasterResponse and WorkersResponse already exist under pyvermeer/structure/.

client.send_request(method, endpoint, params) is the shared entry point behind both groups. You can call it directly to reach a Vermeer endpoint that has no wrapper yet; it returns the decoded JSON body as a plain dict.

Requests and Responses

TaskCreateRequest(task_type, graph_name, params) is serialized as {"task_type": ..., "graph": ..., "params": ...}. Note that graph_name becomes graph on the wire, which matches the payload documented for the Vermeer REST API.

Every response type extends BaseResponse and exposes errcode and message, plus a to_dict() helper. errcode is 0 on success and 1 on error; -1 means the field was missing from the response body.

  • GraphsResponse.graphs and GraphResponse.graph yield VermeerGraph objects with name, space_name, status, create_time, update_time, vertex_count, edge_count, workers, worker_group, use_out_edges, use_property, use_out_degree, use_undirected, on_disk, and backend_option.
  • TasksResponse.tasks, TaskResponse.task, and TaskCreateResponse.task yield TaskInfo objects with id, state, create_user, create_type, create_time, start_time, update_time, graph_name, space_name, params, workers, and error_message.
  • Timestamps are parsed with python-dateutil into datetime objects. An empty timestamp string becomes None.

Server returns the task type as task_type, but SDK TaskInfo.type reads type, so that property is currently empty. The SDK also drops Server statistics_result. Use TaskInfo.state for polling. This is a field-contract mismatch in current source.

Task Parameters

The client does not validate params. Keys and values are passed straight through to Vermeer, so the accepted names come from the engine, not from the SDK. For the load parameters and the parameters of the supported algorithms, see the Vermeer quick start.

The usual sequence is the same as with the REST API directly: create a load task to read the graph into Vermeer, wait for it to finish, then create computation tasks against the loaded graph.

Errors

pyvermeer.utils.exception defines four exceptions, all raised from the underlying requests or JSON failure:

ExceptionRaised when
ConnectErrorrequests.ConnectionError, the master is unreachable
TimeOutErrorrequests.Timeout, the connect or read deadline expired
JsonDecodeErrorThe response body is not valid JSON
UnknownErrorAny other failure during the request
1from pyvermeer.utils.exception import ConnectError, TimeOutError
2
3try:
4    graphs = client.graph.get_graphs()
5except (ConnectError, TimeOutError) as error:
6    print(error)

The client does not check the HTTP status code of the response, so inspect errcode and message on the returned object to tell success from a Vermeer-side error.

Development Checks

Run the formatting and static checks from the root of the HugeGraph-AI repository:

1./style/code_format_and_analysis.sh

The source lives under vermeer-python-client/src/pyvermeer/. A few structure tests are available under src/tests/structure/test_task_data.py; they do not cover HTTP API integration.

References

3.7 - HugeGraph-MCP

hugegraph-mcp exposes HugeGraph tools to AI clients through the Model Context Protocol (MCP). It supports schema inspection, structured graph queries, controlled writes, and optional natural-language query generation and graph extraction. The MCP client starts a local process and communicates over STDIO. This process is not HugeGraph Server and does not expose an HTTP MCP endpoint.

Ordinary graph queries connect directly to HugeGraph Server. Query generation, extraction, and embedding refresh also require HugeGraph-LLM, where model credentials are configured.

Requirements

  • Python >= 3.10 for the pinned 1.7.1 release (Python >= 3.12 for master / 1.8.0+). A separate HugeGraph-LLM service must also meet that module’s Python requirements.
  • uv and an MCP client that supports STDIO.
  • HugeGraph Server 1.7.0 or later, reachable from the MCP process. The default graph path is DEFAULT/hugegraph and uses the graphspace API.
  • An existing graph and an account with the required permissions. See authentication and authorization.

Installation and Connection

These examples pin the published hugegraph-mcp==1.7.1 package. Cloning the repository is not required. Replace the address, graph path, and credentials with your deployment values. Verify the connection in read-only mode before enabling supported writes.

For clients that accept a mcpServers JSON configuration, add:

 1{
 2  "mcpServers": {
 3    "hugegraph": {
 4      "command": "uvx",
 5      "args": ["hugegraph-mcp@1.7.1"],
 6      "env": {
 7        "HUGEGRAPH_URL": "http://127.0.0.1:8080",
 8        "HUGEGRAPH_GRAPH_PATH": "DEFAULT/hugegraph",
 9        "HUGEGRAPH_USER": "your-user",
10        "HUGEGRAPH_PASSWORD": "your-password",
11        "HUGEGRAPH_MCP_READONLY": "true"
12      }
13    }
14  }
15}

Configuration locations and formats vary between clients. Use your client’s instructions to set the same command, arguments, and environment variables. If the client cannot find uvx, use its absolute path for command. Restart the MCP process after changing the configuration.

You can also start it in a terminal with the environment variables above already configured:

1uvx hugegraph-mcp@1.7.1

The process waits for protocol messages from the MCP client on stdin; it does not open a web page. stdout carries MCP JSON-RPC. Do not write extra shell output to that channel.

Verify the Connection

List tools through the client, then call inspect_graph_tool. The following JSON is the parameter body for MCP tools/call, not a request to send directly to the HugeGraph REST API.

1{
2  "name": "inspect_graph_tool",
3  "arguments": {"include_raw_schema": true}
4}

Check for ok=true and data.hugegraph_server_status="available". Choose an existing vertex label from the schema and replace person below with that label:

1{
2  "name": "query_graph_data_tool",
3  "arguments": {
4    "target": "vertex",
5    "operation": "page",
6    "label": "person",
7    "limit": 5
8  }
9}

This tool belongs to the default set of 16 tools. To paginate, pass the returned next_page as page on the next call, keeping the same query conditions. An empty result does not necessarily mean the connection failed; check the graph path, label, and existing data.

Toolsets and Versions

A toolset selects a group of tools within the same package. It is not a package version, HugeGraph Server version, or MCP protocol version. Version 1.7.1 accepts the following configuration names:

HUGEGRAPH_MCP_TOOLSETToolsPurpose
Unset, or v2_core16Default toolset, including structured queries and persistent write-plan tools
v110Compatibility toolset retaining earlier tool entry points

Any other value, including v2, core, legacy or an empty string, falls back to the 10-tool v1 set in 1.7.1 and logs an error. Both toolsets report mcp_tool_contract_version="2.0". This field describes the tool contract, not the package version.

Toolset Naming Evolution

In upcoming releases, the default toolset is renamed to core and the compatibility toolset to legacy. For the current 1.7.1 release package, continue using the configurations documented above.

  • v2_core maps to core, and v1 maps to legacy. Legacy identifiers are retained as backward-compatible aliases with migration warnings.
  • Inspection endpoints report core or legacy. Clients checking data.toolset or meta.toolset should update their checks accordingly.
  • An unset variable defaults to core. Empty or unrecognized values halt startup rather than silently falling back.
  • Toolset selection is resolved at launch; modifying configuration requires restarting the client. Compatibility mode does not bypass security or read-only controls.

Tool Capabilities

A registered tool does not imply that every operation is enabled. The current capability boundaries are summarized below. Use the tool schema returned to your client for exact parameters.

FeatureToolsScope
Inspection and designinspect_graph_tool, inspect_schema_tool, design_schema_toolConnection status, schema, and design guidance
Structured readsquery_graph_data_toolRead vertices and edges by ID, ID list, page, or conditions
Schema creationapply_schema_toolValidate and preview; the default toolset can confirm one property key, vertex label, or edge label creation
Write plansconfirm_write_tool, get_write_status_tool, reconcile_write_toolConfirm, inspect, and reconcile persistent plans
Graph importimport_graph_data_toolAI-backed extract; preview-only ingest; table is currently disabled
Graph deletiondelete_graph_data_toolExact edge deletion can be confirmed; vertex deletion is preview-only
Property changesmutate_graph_properties_toolPreview-only; confirmation is currently disabled
AI generation and extractiongenerate_gremlin_tool, extract_graph_data_toolGenerate queries or graph data without executing generated queries
Embedding refreshrefresh_vid_embeddings_toolRequires AI, admin mode, writes enabled, and confirmation; see service gates below
Raw Gremlinexecute_gremlin_read_tool, execute_gremlin_write_toolBoth return FEATURE_DISABLED; admin mode cannot override this

The compatibility toolset excludes inspect_schema_tool, query_graph_data_tool, mutate_graph_properties_tool, and the three write-plan tools. Switching to compatibility mode does not enable raw Gremlin execution.

Controlled Writes

To use writes, set HUGEGRAPH_MCP_READONLY=false and use a HugeGraph account with the appropriate permissions. For a confirmable operation, first run a dry-run, review its exact targets, changes, and warnings, then confirm the returned plan_id:

1{
2  "name": "confirm_write_tool",
3  "arguments": {"plan_id": "<PLAN_ID>"}
4}

Use get_write_status_tool(plan_id) to inspect the persisted result. APPLIED means all operations are confirmed successful. PARTIAL or UNKNOWN means the workflow is incomplete or its outcome is uncertain. Inspect and reconcile status before retrying; do not blindly create a replacement plan. Confirming the same completed plan again returns its recorded result without repeating the write.

The built-in SQLite state store requires a single writable MCP instance and a persistent state directory. When multiple clients start separate processes, keep them read-only initially; multi-instance writes require a shared transactional store design. Operations returning confirmable=false cannot be forced through by disabling read-only mode or enabling admin mode.

Configuration

The following defaults apply to published version 1.7.1. HugeGraph credentials and the HugeGraph-LLM API token are separate credentials.

Environment variableDefaultDescription
HUGEGRAPH_URLhttp://127.0.0.1:8080HugeGraph address reachable from the MCP process
HUGEGRAPH_GRAPH_PATHDEFAULT/hugegraphGRAPH_SPACE/GRAPH_NAME
HUGEGRAPH_GRAPHSPACEUnsetOverrides the graphspace in HUGEGRAPH_GRAPH_PATH
HUGEGRAPH_GRAPHUnsetOverrides the graph name in HUGEGRAPH_GRAPH_PATH
HUGEGRAPH_USERadminHugeGraph user
HUGEGRAPH_PASSWORDEmptyHugeGraph password
HUGEGRAPH_MCP_TOOLSETv2_coreThis release accepts v2_core and v1
HUGEGRAPH_MCP_READONLYtrueDisable controlled writes
HUGEGRAPH_MCP_ALLOW_AIfalseAllow calls to HugeGraph-LLM
HUGEGRAPH_MCP_ADMIN_MODEfalseEnable eligible administrative operations
HUGEGRAPH_AI_URLhttp://127.0.0.1:8001HugeGraph-LLM service address
HUGEGRAPH_AI_TOKENUnsetHugeGraph-LLM bearer token
HUGEGRAPH_AI_GRAPH_URLUnsetDoes not change the AI target; ignored in current source
HUGEGRAPH_AI_TIMEOUT_SECONDS30AI HTTP request timeout in seconds
HUGEGRAPH_MCP_STATE_DIRXDG state directory or ~/.local/state/hugegraph-mcpPersistent directory for write plans and receipts

Optional AI Features

Configure and start the model service using the HugeGraph-LLM guide, then add these entries to the MCP client’s env:

1{
2  "HUGEGRAPH_MCP_ALLOW_AI": "true",
3  "HUGEGRAPH_AI_URL": "http://127.0.0.1:8001",
4  "HUGEGRAPH_AI_TOKEN": "your-ai-service-token"
5}

Merge these entries into the earlier env object. Generation and extraction can remain read-only. Embedding refresh additionally requires:

  • Client settings: HUGEGRAPH_MCP_READONLY=false, HUGEGRAPH_MCP_ADMIN_MODE=true, and confirm=true in the tool call.
  • Service settings (HugeGraph-LLM): ENABLE_LOGIN=true, HUGEGRAPH_LLM_ENABLE_THIN_WRITES=true, a strong non-default USER_TOKEN (avoiding the well-known default 4321), and valid authentication passed via MCP’s HUGEGRAPH_AI_TOKEN. Without these service-side gates, refresh requests are rejected with FEATURE_DISABLED (“Legacy Thin API writes are disabled”).

MCP and HugeGraph-LLM must connect to the same underlying HugeGraph instance. The capability check protocol only verifies graph_name and graphspace, while the actual embedding refresh executes entirely using the AI service’s independent GRAPH_URL. Modifying MCP’s HUGEGRAPH_URL does not redirect the AI service’s data target. If MCP runs on the host and AI runs in a container, they may need different addresses to reach HugeGraph: configure GRAPH_URL on the AI service to an address reachable from its container, such as http://server:8080, and ensure GRAPH_NAME and GRAPH_SPACE match MCP. Do not use the container’s 127.0.0.1 to reach the host graph service.

Capability check protocol: The updated protocol requires GET /vid-embeddings/refresh-capabilities to return protocol_version=1, transmitting the target graph and graphspace for server-side validation before refreshing embeddings. Services without this endpoint will reject refresh requests. This validation protects multi-tenant boundary integrity in future releases.

Troubleshooting

  • No tools after client startup: check the uvx path, Python version, and process stderr. Waiting for protocol input in a terminal is normal.
  • Structured query tool missing: check whether the compatibility toolset was selected. For 1.7.1, use v2_core or leave the variable unset.
  • FEATURE_DISABLED response: check the capability table and read-only, AI, and admin settings. Some operations are disabled because of backend capability limits and cannot be enabled through configuration alone.
  • AI calls fail: check the service address, token, container networking, and AI logs. An unavailable-service error does not necessarily mean a model was retired.
  • Embedding calls return 400 or 429: check the provider’s input-count and request-rate limits. Batch size and concurrent request count are different limits; handle them in HugeGraph-LLM’s embedding and index-building paths, not by changing MCP toolsets.

An ok=false tool response indicates failure. Inspect error.type, error.retryable, and warnings; an established MCP connection or successful HTTP status alone does not prove that a business operation succeeded.

Next Steps

Documentation on the repository’s main branch follows source changes. Check version applicability when using a pinned package; the installation examples on this page use 1.7.1.

3.8 - HugeGraph-LLM SDK Quick Start

Standalone Python SDK for schema design, entity/relation extraction, graph validation, and reviewed graph construction from text.

hugegraph-llm-sdk is a lightweight, standalone Python SDK for programmatically designing schemas, extracting property graphs from unstructured text, validating schemas against HugeGraph rules, and performing iterative model review and bounded repair.

Unlike the full hugegraph-llm application, the SDK is completely headless and does not require HugeGraph Server, Gradio Web UI, FastAPI, or a vector database. Its core runtime depends only on pydantic>=2.14,<3.

Apache License 2.0 · Ask DeepWiki

Requirements and Installation

  • Python: >=3.12 (tested and supported on Python 3.12, 3.13, and 3.14).
  • Package name: hugegraph-llm-sdk
  • Import name: hugegraph_llm_sdk

Install from source or a local wheel

The SDK is currently provided by the hugegraph-ai source tree and has not yet been published to a public package index. Install it from a checkout or a locally built wheel.

1pip install -e /path/to/hugegraph-ai/hugegraph-llm-sdk
2# or
3uv add --editable /path/to/hugegraph-ai/hugegraph-llm-sdk
4
5uv build --no-sources /path/to/hugegraph-ai/hugegraph-llm-sdk --out-dir /tmp/llm-sdk-dist
6pip install /tmp/llm-sdk-dist/hugegraph_llm_sdk-1.8.0-py3-none-any.whl

Provider Extras

Install provider extras from the same checkout:

1pip install -e "/path/to/hugegraph-ai/hugegraph-llm-sdk[openai]"
2pip install -e "/path/to/hugegraph-ai/hugegraph-llm-sdk[ollama]"
3pip install -e "/path/to/hugegraph-ai/hugegraph-llm-sdk[litellm]"
4# uv example: uv add --editable /path/to/hugegraph-ai/hugegraph-llm-sdk --extra openai

Quick Start: Minimal Example

Here is a complete example demonstrating schema design, schema validation, and reviewed graph extraction using a custom model function:

 1from hugegraph_llm_sdk import (
 2    FunctionModel,
 3    design_schema,
 4    validate_schema,
 5    extract_graph,
 6)
 7
 8# 1. Define or wrap your LLM caller
 9model = FunctionModel(generate=lambda prompt: my_agent_model.invoke(prompt).content)
10
11# 2. Design a graph schema from sample texts and target questions
12draft = design_schema(
13    texts=["Alice works for Acme Corp. Bob reports to Alice."],
14    questions=["Which company does each person work for? Who reports to whom?"],
15    model=model,
16)
17
18# Inspect any structural or conceptual issues flagged during design
19if draft.issues:
20    print(f"Schema draft issues: {draft.issues}")
21
22schema = draft.schema
23
24# 3. Validate schema structure against HugeGraph rules
25validation_issues = validate_schema(schema)
26assert not validation_issues, f"Validation failed: {validation_issues}"
27
28# 4. Extract property graph from text
29result = extract_graph(
30    texts="Alice works for Acme Corp as Chief Engineer.",
31    schema=schema,
32    model=model,
33)
34
35# 5. Process chunk outcomes
36print(f"Overall status: {result.status}")  # 'succeeded', 'partial', or 'failed'
37for chunk in result.chunks:
38    if chunk.status == "final":
39        print(f"Extracted graph: {chunk.graph}")
40    else:
41        print(f"Chunk {chunk.index} status: {chunk.status}, reason: {chunk.reason}, issues: {chunk.issues}")

Core API Capabilities

FunctionSignatureDescription
design_schema(texts, *, model, questions=(), example_schema=None, prompt=None) -> SchemaDraftGenerates a candidate schema (SchemaDraft(schema, issues)) from texts and domain queries.
adesign_schema(texts, *, model, questions=(), example_schema=None, prompt=None) -> SchemaDraftAsynchronous schema design variant.
validate_schema(schema) -> list[Issue]Validates schema structure against HugeGraph property graph constraints without mutating the input.
extract_graph(texts, *, schema, model=None, model_factory=None, review=True, options=None, prompts=None, validators=()) -> ExtractionResultExecutes chunked graph extraction, validation, and review/repair loops.
aextract_graph(texts, *, schema, model=None, model_factory=None, review=True, options=None, prompts=None, validators=()) -> ExtractionResultAsynchronous graph extraction variant.

Schema Validation Rules

  • validate_schema inspects vertex labels, edge labels, ID strategies (DEFAULT, AUTOMATIC, PRIMARY_KEY, CUSTOMIZE_STRING, CUSTOMIZE_NUMBER, CUSTOMIZE_UUID), and property declarations.
  • When propertykeys is omitted, global property definitions are not strictly checked. When a label omits properties, its property set is unconstrained.
  • An explicit empty properties: [] indicates that no properties are allowed for that label.
  • For edge labels, sort_keys cannot be members of nullable_keys. When frequency is SINGLE or DEFAULT, sort keys are disallowed; MULTIPLE requires non-empty sort keys.

Model Adapters

The SDK includes standard model adapters under hugegraph_llm_sdk.providers:

1from hugegraph_llm_sdk.providers.openai import OpenAIClient
2
3model = OpenAIClient(
4    api_key="your-api-key",
5    api_base="https://api.openai.com/v1",
6    model_name="gpt-4o-mini",
7)
  • FunctionModel: Wraps any custom function generate(prompt: str) -> str or async agenerate(prompt: str) -> str.
  • OpenAIClient: Adapter for OpenAI and OpenAI-compatible endpoints (supports close() and await aclose()).
  • OllamaClient: Adapter for local Ollama instances.
  • LiteLLMClient: Adapter using LiteLLM connection routing.
  • ThreadedModel: Wraps a synchronous model so calls execute inside worker threads during async execution without blocking the event loop.

Review and Bounded Repair

By default, extraction executes an iterative review and bounded repair loop:

1extract -> validate -> review
2              ^           |
3              |           +-- pass  -> final
4              |           +-- block -> blocked
5              +-- repair <- fix

Review options are configured via ExtractionOptions:

 1from hugegraph_llm_sdk import ExtractionOptions, extract_graph
 2
 3options = ExtractionOptions(
 4    max_reviews=2,  # Maximum review attempts per chunk (0–5, default: 2)
 5    max_fixes=1,    # Maximum repair attempts per chunk (0–3, default: 1)
 6    max_workers=1,  # Worker concurrency (1–8, default: 1)
 7)
 8
 9result = extract_graph(
10    texts="Bob manages Charlie at Beta Corp.",
11    schema=schema,
12    model=model,
13    options=options,
14)

Chunk and Result Statuses

  • Chunk status:
    • final: Graph passed structural validation and model review. Only final chunks are merged into result.graph.
    • candidate: Validation passed, but review budget was exhausted or review could not conclude.
    • blocked: Graph could not pass validation within the repair budget or was explicitly rejected by review.
    • failed: An exception or unrecoverable error occurred during execution.
  • Aggregate status:
    • succeeded: All chunks achieved final status.
    • partial: At least one chunk is final, but one or more chunks failed or remain candidate/blocked.
    • failed: No chunk reached final status.

Concurrency and Async Processing

When setting max_workers > 1, you must pass a stateless model_factory callable instead of a single model instance. This ensures thread-safety and isolated model state per worker:

 1from hugegraph_llm_sdk import ExtractionOptions, extract_graph
 2from hugegraph_llm_sdk.providers.openai import OpenAIClient
 3
 4chunks = [
 5    "Alice joined Acme Corp in 2022.",
 6    "Bob joined Beta LLC in 2023.",
 7    "Charlie works with Alice at Acme Corp.",
 8]
 9
10result = extract_graph(
11    texts=chunks,
12    schema=schema,
13    model_factory=lambda: OpenAIClient(api_key="your-api-key", model_name="gpt-4o-mini"),
14    options=ExtractionOptions(max_workers=3),
15)
16
17print(f"Total vertices: {len(result.graph.get('vertices', []))}")
18print(f"Total edges: {len(result.graph.get('edges', []))}")

The SDK automatically manages the lifecycle of models created by model_factory, disposing of them when each worker batch finishes.

Relationship with HugeGraph-LLM

hugegraph-llm-sdk and hugegraph-llm reside in the same repository under separate modules:

1hugegraph-ai/
2├── hugegraph-llm-sdk/      # Standalone headless Python SDK (pydantic-only baseline)
3├── hugegraph-llm/          # Full application layer (FastAPI, Gradio UI, GraphRAG, vector DB)
4├── hugegraph-python-client/# Low-level HugeGraph Server client (pyhugegraph)
5└── hugegraph-mcp/          # Model Context Protocol server
  • hugegraph-llm-sdk: Embeddable library for Python pipelines, LLM agent toolchains, and data transformation scripts requiring schema design and graph extraction.
  • hugegraph-llm: Production application service providing Web UI, REST endpoints (POST /graph/extract, POST /rag), vector index persistence, and automated ingestion into HugeGraph Server.

4 - HugeGraph Computing (OLAP)

The HugeGraph-Computer repository contains two graph computing systems with different deployment and runtime models. Start with Go Vermeer for general graph algorithms; use Computer when you need distributed Java BSP/Pregel computation. Both can connect to HugeGraph, but their configuration and job entry points are not interchangeable.

Default entry: Go Vermeer

flowchart TB
    Master["Master"] --> Workers["Workers"]
    Master -->|PD| PD["PD"]
    Workers -->|Scan| Store["Store"]
    Workers -.->|REST| Server["Server"]

With load.type=hugegraph, the Vermeer master queries PD for partition metadata and workers scan HStore Store partitions directly. Workers write results through the Server REST API only when output.type=hugegraph; other input sources or output settings do not require those connections.

Java Computer

flowchart TB
    Config["Job configuration"] --> Master["Master"] --> Workers["Workers"]
    Master -. BSP .-> Etcd["etcd"]
    Workers <-->|REST| Server["Server"]
    Workers <-->|HDFS| HDFS["HDFS (optional)"]

Job configuration can be submitted through the Kubernetes Operator or YARN. Workers read graph data through the HugeGraph Server REST API and can write results back according to the output configuration. With HDFS input or output, workers access HDFS directly. The master uses etcd to coordinate BSP jobs.

4.1 - HugeGraph-Vermeer Quick Start

Vermeer high-performance in-memory graph computing: start once, execute repeatedly, with 15+ OLAP algorithms, seconds-to-minutes execution, deployment, loading, PageRank, and community detection.

1. Overview of Vermeer

1.1 Architecture

Vermeer is a high-performance, memory-first graph computing framework written in Go: start once and execute repeatedly. It supports fast execution of 15+ OLAP algorithms, often in seconds to minutes; actual time depends on graph size, algorithm parameters, and resources. A single master currently schedules multiple workers.

The master handles communication, forwarding, and aggregation, with modest computation and resource usage. Workers store graph data and execute tasks, consuming most memory and CPU. gRPC handles internal communication; REST provides external APIs.

At startup, built-in defaults are loaded first, followed by the [default] section of config/<env>.ini under the working directory, then explicit command-line overrides. For example, --env=master reads config/master.ini. The Docker image copies repository configuration to /go/bin/config/ and works from /go/bin/. A host bind mount hides the bundled files, so it must contain every ini file used. Ordinary INI keys are not mapped automatically from environment variables. Dedicated deployment credential settings also support VERMEER_PD_PASSWORD_FILE, VERMEER_PD_PEERS, VERMEER_HUGEGRAPH_CREDENTIALS_FILE and the documented token-factor/password file settings.

Default ports:

RoleConfiguration keyDefault addressPurpose
masterhttp_peer0.0.0.0:6688REST API for host-side clients
mastergrpc_peer0.0.0.0:6689Workers connect to master
workerhttp_peer0.0.0.0:6788Worker HTTP service
workergrpc_peer0.0.0.0:6789Worker gRPC listener, also advertised to master for node communication

The Docker examples publish only master HTTP on host loopback 127.0.0.1:6688:6688; master and worker gRPC ports stay within the container network.

flowchart LR
  Client["curl / Python client"] -->|"HTTP :6688"| Master["master"]
  Worker["worker"] <-->|"Bidirectional gRPC: master :6689, worker :6789"| Master
  Master -->|"gRPC partition lookup"| PD["HugeGraph PD"]
  Worker -->|"gRPC partition scan"| Store["HugeGraph Store"]

1.2 Running Vermeer

Helm: Deploy Vermeer with HugeGraph and Hubble

The HugeGraph Helm Chart supports deploying the native Vermeer in-memory graph computing engine, disabled by default. From the HugeGraph repository root, enable Vermeer with:

1helm install hugegraph ./helm/hugegraph -n hugegraph --create-namespace \
2  -f helm/hugegraph/values-single.yaml \
3  --set vermeer.enabled=true --set vermeer.mode=standalone \
4  --wait --timeout 20m

Vermeer Core Files and Layout

  • helm/hugegraph/
    • examples/
      • values-vermeer-standalone.yamlStandalone preset (Master and Worker run in a single Pod)
      • values-vermeer-cluster.yamlCluster preset (dedicated Master Pod + distributed Worker Pods)
    • templates/
      • vermeer-master.yamlMaster scheduling and coordination workload
      • vermeer-worker.yamlWorker in-memory computing workload
      • vermeer-auth-secret.yamlAdmin credentials and Hubble internal service tokens
    • docs/
      • vermeer.mdAdvanced algorithm parameters, API mapping and operations

Architecture & Topologies:

  • Standalone Mode (standalone): Master and Worker run as distinct containers in the same Pod with minimal overhead, suitable for local exploration and small-scale testing.
  • Cluster Mode (cluster): Set vermeer.mode=cluster and vermeer.worker.replicas=3 to provision a dedicated Master coordinator Pod and multiple distributed Worker computing Pods.
  • Graph Service Integration: Master queries PD via gRPC for partition topology, while Workers scan graph slices directly from Store via gRPC. Hubble automatically injects the Master URL and communication token, allowing users to load graphs and execute 15+ graph algorithms directly from the UI.

Key Operational Considerations:

  1. Preserve Metadata Namespace: Retain the Chart default metadataCluster: hg-test. This key synchronizes Server HG_SERVER_CLUSTER and Store PD_CLUSTER so that Store can properly resolve schemas. Do not arbitrarily change this value on existing clusters to avoid Schema missing errors during load tasks.
  2. Access Native Web Dashboard: In easy mode, open http://<reachable-node-ip>:<assigned-http-node-port>/ for Service <release>-vermeer-master. Strict mode defaults to ClusterIP; use kubectl -n hugegraph port-forward svc/<release>-vermeer-master 6688:6688 for local access. vermeer.service.type: null inherits the mode; explicit ClusterIP stays private and explicit NodePort exposes authenticated HTTP/UI. Internal Master/Worker RPC and Worker HTTP stay private; standalone / cluster does not change this access choice. Enter the <release>-vermeer-admin Secret’s password into Admin credential (no username) for a 24-hour session. New passwords use 6 random alphanumeric characters in easy mode or 16 in strict mode; upgrades and mode changes retain existing credentials.
  3. For image pinning, persistent volume sizing, and recovery procedures, refer to the Helm Optional Engines Guide and Vermeer Deployment Guide.
Warning

In production, enable Server authentication and authorization, an IP allowlist, and minimum permissions; retain audit-*.log. Server Auth does not protect independent Vermeer, PD, or Store APIs. Restrict these HTTP/gRPC ports to trusted networks and callers, and configure access controls at the Vermeer external entry point.

Manual standalone deployments default to auth=none in master.ini, and the Docker examples below publish to loopback only. When deploying manually for remote access, always enable auth=token. The Helm deployment requires token authentication in both modes and exposes only native HTTP/UI through NodePort in easy mode by default.

Both Docker options need a host configuration directory. From the Vermeer repository root, copy the supplied master.ini and worker.ini templates. The mount hides image configuration under /go/bin/config, so never mount an empty directory or your entire home directory there:

1CONFIG_DIR="$HOME/vermeer-config"
2mkdir -p "$CONFIG_DIR"
3cp config/master.ini config/worker.ini "$CONFIG_DIR/"

Keep master HTTP/gRPC listeners at 0.0.0.0:6688 and 0.0.0.0:6689. The Compose example assigns 172.20.0.10 to master and 172.20.0.11 to worker; set these options in the copied worker.ini:

1[default]
2http_peer=0.0.0.0:6788
3grpc_peer=172.20.0.11:6789
4master_peer=172.20.0.10:6689
5run_mode=worker
6worker_group=$

This single-worker example uses literal worker_group=$, the general group used when no named group is bound. The repository template defaults to worker_group=default; if retaining that named group, bind it to the task space or graph before submitting tasks. For default space $DEFAULT, use:

1curl --fail --show-error -X POST \
2  'http://localhost:6688/admin/workers/alloc/default/%24DEFAULT'

An errcode=0 response confirms binding; with token authentication, also supply the authorization header. master_peer must reach master gRPC from the worker network. grpc_peer is both the listener and advertised worker address, so do not advertise 0.0.0.0, which other containers cannot reach. Update these IPs when changing the subnet or addresses; each additional worker needs a unique, mutually reachable grpc_peer.

  1. Option 1: Docker Compose (Recommended)

Run from the Vermeer repository root. Modify existing docker-compose.yaml or use the following example. The repository file currently mounts all of ~/ as configuration and does not publish master HTTP; replace the mount and add the port mapping before starting:

 1services:
 2  vermeer-master:
 3    image: hugegraph/vermeer
 4    container_name: vermeer-master
 5    ports:
 6      - "127.0.0.1:6688:6688"
 7    volumes:
 8      - /home/user/vermeer-config:/go/bin/config:ro
 9    command: --env=master
10    networks:
11      vermeer_network:
12        ipv4_address: 172.20.0.10 # Static master address
13
14  vermeer-worker:
15    image: hugegraph/vermeer
16    container_name: vermeer-worker
17    volumes:
18      - /home/user/vermeer-config:/go/bin/config:ro
19    command: --env=worker
20    networks:
21      vermeer_network:
22        ipv4_address: 172.20.0.11 # Static worker address
23
24networks:
25  vermeer_network:
26    driver: bridge
27    ipam:
28      config:
29        - subnet: 172.20.0.0/24 # Change the subnet as needed

Replace /home/user/vermeer-config with your actual absolute configuration directory. Changing the subnet or static IPs of vermeer_network also requires updating grpc_peer and master_peer in worker.ini. Do not use the default grpc_peer=0.0.0.0:6789 as an advertised container address.

Build and start from the project directory:

1# Build the image from the Vermeer root directory
2docker build -t hugegraph/vermeer .
3
4# Start from the Vermeer root directory
5docker-compose up -d
6# Or use the newer CLI:
7# docker compose up -d

View logs / stop / remove:

1docker-compose logs -f
2docker-compose down
  1. Option 2: Separate docker run Commands (Manual Network and Static IPs)

Set CONFIG_DIR to the prepared configuration directory, with grpc_peer and master_peer matching the static container addresses. Replace the example CONFIG_DIR with your actual absolute path.

Build the image:

1docker build -t hugegraph/vermeer .

Create a custom bridge network (once):

1docker network create --driver bridge \
2  --subnet 172.20.0.0/24 \
3  vermeer_network

Run master (use your absolute CONFIG_DIR and adjust IPs as needed):

1CONFIG_DIR=/home/user/vermeer-config
2
3docker run -d \
4  --name vermeer-master \
5  --network vermeer_network --ip 172.20.0.10 \
6  -p 127.0.0.1:6688:6688 \
7  -v ${CONFIG_DIR}:/go/bin/config:ro \
8  hugegraph/vermeer \
9  --env=master

Run worker:

1docker run -d \
2  --name vermeer-worker \
3  --network vermeer_network --ip 172.20.0.11 \
4  -v ${CONFIG_DIR}:/go/bin/config:ro \
5  hugegraph/vermeer \
6  --env=worker

View logs / stop / remove:

1docker logs -f vermeer-master
2docker logs -f vermeer-worker
3
4docker stop vermeer-master vermeer-worker
5docker rm vermeer-master vermeer-worker
6
7# Delete the custom network if needed
8docker network rm vermeer_network
  1. Option 3: Build from Source

Build following the Vermeer README.

1go build

Start from the Vermeer root directory with ./vermeer --env=master and ./vermeer --env=worker01. In worker01.ini, set grpc_peer to an address bindable locally and reachable by master and other workers; point master_peer to master gRPC.

After starting master, check its HTTP port from the host:

1curl --fail --show-error http://localhost:6688/graphs

Expect HTTP 200 and JSON errcode=0.

1.3 Authentication, Token Tooling, and Deployment Credentials

When auth=token is enabled in master.ini (or -auth token via CLI), protect the Master signing factor in a dedicated file (0600) instead of command-line arguments. File-backed authentication uses v2 HMAC-SHA256 Base58-encoded tokens with constant-time MAC verification, preventing legacy downgrade attacks.

1. Generating File-Backed Signing Factor

Create a protected signing factor file before starting Master:

1umask 077
2openssl rand -hex 32 > config/token-factor

Set the environment variable VERMEER_AUTH_TOKEN_FACTOR_FILE=/path/to/token-factor, or set auth_token_factor_file in Master’s master.ini. The environment variable takes precedence over the INI file, and either file setting takes precedence over inline auth_token_factor. The file must contain one valid UTF-8 line with at least 16 characters and cannot consist solely of whitespace. An invalid or unreadable file stops Master before opening network listeners (fails closed), and it never falls back to inline or default factors.

2. Generating Auth Tokens with /go/bin/vermeer-token

Vermeer container images bundle the native /go/bin/vermeer-token CLI utility (built from cmd/vermeer-token). It generates v2 Base58 tokens without requiring a running Master or Kubernetes API access:

Docker CLI
1docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/config:/tokens" \
2  --entrypoint /go/bin/vermeer-token \
3  hugegraph/vermeer:latest -user admin -space '$DEFAULT' \
4  -factor-file /tokens/token-factor -output-file /tokens/admin-token -i -quiet
Go Source
1go run ./cmd/vermeer-token -user admin -space '$DEFAULT' \
2  -factor-file config/token-factor -output-file config/admin-token -i -quiet

CLI Parameters:

  • -user: Username (default: foo).
  • -space: Graph space name (default: bar; use $DEFAULT for system default graph space).
  • -factor-file: Path to the signing factor file (required; inline factors are rejected).
  • -output-file: Atomically writes the Base58 token with file mode 0600 without echoing secrets to stdout.
  • -quiet: Suppresses informational logs and outputs only the token when -output-file is omitted.
  • -client: Client identifier (default: hg).
  • -i: Generates an immortal token that never expires (standard tokens expire after 24 hours).

To authenticate HTTP requests, pass the generated token in the header:

1curl -H "Authorization: Bearer <token>" http://localhost:6688/graphs

3. Kubernetes Secret Password Files and Credential Isolation

To integrate with Kubernetes Secrets without wrapping raw credentials in JSON, configure Worker with raw password files:

  • Set VERMEER_HUGEGRAPH_USERNAME and VERMEER_HUGEGRAPH_PASSWORD_FILE (or hugegraph_username and hugegraph_password_file in worker.ini). Both values must be configured together; if either is set, a missing or unreadable file fails graph loading closed.
  • Raw password file bytes are preserved exactly, including quotes, backslashes, spaces, and terminal newlines.
  • When an explicit JSON credentials file is set (VERMEER_HUGEGRAPH_CREDENTIALS_FILE or hugegraph_credentials_file), it takes precedence over the raw pair.
  • For PD authentication, mount VERMEER_PD_PASSWORD_FILE (or pd_password_file in INI). When deployment credentials are used, VERMEER_PD_PEERS (pd_peers) is required to discover Schema exclusively through trusted PD endpoints.
  • Secret credentials remain confined to deployment configuration and worker processes; they are never persisted in task metadata or returned in query APIs.
  • Dial timeouts can be bounded with grpc_dial_timeout or VERMEER_GRPC_DIAL_TIMEOUT (default 5s).

2. Task Creation REST API

2.1 Introduction

Submit a load task, wait for loading to finish, then submit a compute task. A loaded graph can support repeated computations and is not deleted after completion. Asynchronous APIs return creation results and task information, including ID, before completion; synchronous APIs wait for success or failure, so client and proxy HTTP timeouts must be sufficiently long. Query task states: loaded for successful loading, complete for successful computation, error for failure, and canceled for cancellation; other states remain waiting or running. Graphs loading or in an error state cannot be computed. Deletion requires a deletable graph state and no current usage.

Available URLs:

  • Asynchronous: POST http://master_ip:port/tasks/create; read the ID from response task.id.
  • Synchronous: POST http://master_ip:port/tasks/create/sync; returns after the task ends.
  • Query task: GET http://master_ip:port/task/{task_id}. errcode=0 indicates a successful query; inspect task.state for task status. Set a client polling deadline for asynchronous jobs; reaching it stops client waiting without canceling the server task.
  • Read retained results: GET http://master_ip:port/tasks/value/{task_id} (or /api/v1/tasks/value/{id}).
  • Cluster health snapshot: GET http://master_ip:port/api/v1/admin/cluster (admin token required; see Section 4).
  • Worker availability probe: GET http://master_ip:port/api/v1/workers/availability (space-scoped availability check).

Native clients submit POST /api/v1/tasks/create with graph, task_type (load, reload or compute), and a flat string-valued params map. For computation, set params["compute.algorithm"], not a top-level algorithm. The response contains a native task.id; query it at GET /api/v1/task/{id} and read retained values at GET /api/v1/tasks/value/{id}. These paths and positive IDs differ from Hubble’s wrapper. POST /api/v1/tasks does not submit work: after the task-method fix it returns HTTP 405 and Allow: GET; older builds can return a misleading {"code":200} without creating a task. See the native API examples.

Vermeer Master HTTP follows Helm mode defaults: easy is NodePort, strict is ClusterIP; RPC and Worker ports remain internal. Native UI admin uses the password in <release>-vermeer-admin. Automation APIs use service tokens in Authorization: Bearer <token>; UI admin password cannot be used as this token. Chart automatically configures service token for Hubble.

2.2 Loading Graph Data

The examples list common load parameters. Each loader reads its own keys; unused keys do not change behavior.

Vermeer provides three loading methods:

  1. Local files

load.vertex_files and load.edge_files map worker address host portions to file paths read by those workers. With containers, mount data into the worker and use container paths. In the Compose example, mount a host data directory at worker /data (such as - /host/data:/data:ro) and use 172.20.0.11 from grpc_peer as the mapping key. Other deployments use the host portion of their advertised worker addresses.

Obtain a dataset such as Twitter-2010; the first twitter-2010.txt.gz file is sufficient.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "load",
 4 "graph": "testdb",
 5 "params": {
 6  "load.parallel": "50",
 7  "load.type": "local",
 8  "load.vertex_files": "{\"172.20.0.11\":\"/data/twitter-2010.v_[0,99]\"}",
 9  "load.edge_files": "{\"172.20.0.11\":\"/data/twitter-2010.e_[0,99]\"}",
10  "load.use_out_degree": "1",
11  "load.use_outedge": "1"
12 }
13}
  1. HugeGraph

Request example:

Replace request addresses, graph name, and credentials with your actual connection settings.

 1POST http://localhost:6688/tasks/create
 2{
 3  "task_type": "load",
 4  "graph": "testdb",
 5  "params": {
 6    "load.parallel": "50",
 7    "load.type": "hugegraph",
 8    "load.hg_pd_peers": "[\"<pd-address-reachable-from-vermeer>:8686\"]",
 9    "load.hugegraph_name": "DEFAULT/hugegraph2/g",
10    "load.hugegraph_username": "admin",
11    "load.hugegraph_password": "<your-password-here>",
12    "load.use_out_degree": "1",
13    "load.use_outedge": "1"
14  }
15}

Vermeer master connects to PD through load.hg_pd_peers to look up partitions; workers read data from the returned Store addresses. These services must be reachable from the corresponding Vermeer hosts or containers. Inside Docker, 127.0.0.1 identifies the container itself, not the host or another container.

  1. HDFS

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3  "task_type": "load",
 4  "graph": "testdb",
 5  "params": {
 6    "load.parallel": "50",
 7    "load.type": "hdfs",
 8    "load.hdfs_namenode": "name_node1:9000",
 9    "load.hdfs_conf_path": "/path/to/conf",
10    "load.krb_realm": "EXAMPLE.COM",
11    "load.krb_name": "user@EXAMPLE.COM",
12    "load.krb_keytab_path": "/path/to/keytab",
13    "load.krb_conf_path": "/path/to/krb5.conf",
14    "load.hdfs_use_krb": "1",
15    "load.vertex_files": "/data/graph/vertices",
16    "load.edge_files": "/data/graph/edges",
17    "load.use_out_degree": "1",
18    "load.use_outedge": "1"
19  }
20}

2.3 Outputting Computation Results

Current result writers support local, hdfs, and hugegraph through output.type; none disables output. Source retains an afs constant, but current master registers no AFS loader or writer. Set output.need_statistics=1 to put statistics into task information; supported statistics operators depend on each algorithm implementation.

The examples list common computation and output parameters; supported algorithm parameters depend on current Vermeer implementations.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "pagerank",
 7 "compute.parallel": "10",
 8 "compute.max_step": "10",
 9 "output.type": "local",
10 "output.parallel": "1",
11 "output.file_path": "result/pagerank"
12  }
13}

output.type=local writes to the executing worker’s local filesystem. For containers, mount the output directory if the host needs to read results.

2.4 Submit and Integrate via Hubble

With Chart-managed native integration, POST /api/v1.3/vermeer/task is used to load or reload a graph, not for computation. It accepts graphspace, graph, task_type:"load", and optional loading tuning parameters. Once the graph is loaded, submit the algorithm via the graph-level computation endpoint:

1curl -b hubble.cookies -X POST \
2  'http://localhost:8088/api/v1.3/graphspaces/DEFAULT/graphs/hugegraph/algorithms/vermeer' \
3  -H 'Content-Type: application/json' \
4  -d '{"params":{"compute.algorithm":"pagerank","compute.parallel":1,"compute.max_step":10,"pagerank.damping":0.85,"pagerank.diff_threshold":0.00001}}'

Parameters and example values come from the current Hubble PageRank form:

ParameterMeaningExample
compute.algorithmSupported algorithm namepagerank
compute.parallelComputation parallelism1
compute.max_stepMaximum iteration steps10
pagerank.dampingPageRank damping factor0.85
pagerank.diff_thresholdConvergence threshold0.00001

Current Hubble accepts pagerank, wcc, lpa, degree, closeness_centrality, betweenness_centrality, triangle_count, kcore, and sssp. Use the selected algorithm’s form for its own parameters; PageRank keys do not apply to another algorithm. The direct adapter accepts compute.*, the selected algorithm’s <algorithm>.* keys and output.need_statistics. Vermeer still validates the actual values. Hubble controls output storage and does not accept arbitrary output paths or Server credentials.

The response contains data.task_id inside the Hubble response envelope. Preserve the returned sign: Hubble maps native Vermeer task IDs to negative IDs. Poll the same graph’s GET /api/v1.3/graphspaces/DEFAULT/graphs/hugegraph/async-tasks/{id}; read result pages through GET .../async-tasks/{id}/values?cursor=0&limit=100 after successful completion. Do not send that Hubble ID to Server’s native /tasks/{id} endpoint. A graph state of loaded confirms loading only. An HTTP 400 on computation requires inspecting the exact rejected request and response.

3. Supported Algorithms

3.1 PageRank

The PageRank algorithm, also known as the web ranking algorithm, is a technique used by search engines to calculate the relevance and importance of web pages (nodes) based on their mutual hyperlinks.

  • If a web page is linked to by many other web pages, it indicates that the web page is relatively important, and its PageRank value will be relatively high.
  • If a web page with a high PageRank value links to other web pages, the PageRank value of the linked web pages will also increase accordingly.

The PageRank algorithm is suitable for scenarios such as web page ranking and identifying key figures in social networks.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "pagerank",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"1",
10 "output.file_path":"result/pagerank",
11 "compute.max_step":"10"
12 }
13}

3.2 WCC (Weakly Connected Components)

The weakly connected components algorithm calculates all connected subgraphs in an undirected graph and outputs the weakly connected subgraph ID to which each vertex belongs, indicating the connectivity between points and distinguishing different connected communities.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "wcc",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"1",
10 "output.file_path":"result/wcc",
11 "compute.max_step":"10"
12 }
13}

3.3 LPA (Label Propagation Algorithm)

The label propagation algorithm is a graph clustering algorithm commonly used in social networks to discover potential communities.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "lpa",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"1",
10 "output.file_path":"result/lpa",
11 "compute.max_step":"10"
12 }
13}

3.4 Degree Centrality

The degree centrality algorithm calculates the degree centrality value of each node in the graph, supporting both undirected and directed graphs. Degree centrality is an important indicator of node importance; the more edges a node has with other nodes, the higher its degree centrality value, and the more important the node is in the graph. In an undirected graph, degree centrality is calculated based on edge information to count the number of times a node appears, resulting in the degree centrality value of the node. In a directed graph, it is based on the direction of the edges, filtering based on input or output-edge information to count the number of times a node appears, resulting in the in-degree or out-degree value of the node. It indicates the importance of each point, with more important points having higher degrees.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "degree",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"1",
10 "output.file_path":"result/degree",
11 "degree.direction":"both"
12 }
13}

3.5 Closeness Centrality

Closeness centrality is used to calculate the inverse of the shortest distance from a node to all other reachable nodes, accumulating and normalizing the value. Closeness centrality can be used to measure the time it takes for information to be transmitted from the node to other nodes. The larger the closeness centrality of a node, the closer its position in the graph is to the center, suitable for scenarios such as identifying key nodes in social networks.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "closeness_centrality",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"1",
10 "output.file_path":"result/closeness_centrality",
11 "closeness_centrality.sample_rate":"0.01"
12 }
13}

3.6 Betweenness Centrality

The betweenness centrality algorithm determines the value of a node as a “bridge” node; the larger the value, the more likely it is to be a necessary path between two points in the graph. Typical examples include mutual followers in social networks. It is suitable for measuring the degree of aggregation around a node in a community.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "betweenness_centrality",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"1",
10 "output.file_path":"result/betweenness_centrality",
11 "betweenness_centrality.sample_rate":"0.01"
12 }
13}

3.7 Triangle Count

The triangle count algorithm calculates the number of triangles passing through each vertex, suitable for calculating the relationships between users and whether the associations form triangles. The more triangles, the higher the degree of association between nodes in the graph, and the tighter the organizational relationship. In social networks, triangles indicate cohesive communities, and identifying triangles helps understand clustering and interconnections among individuals or groups in the network. In financial or transaction networks, the presence of triangles may indicate suspicious or fraudulent activities, and triangle counting can help identify transaction patterns that may require further investigation.

The output result is the Triangle Count corresponding to each vertex, i.e., the number of triangles the vertex is part of.

Note: This algorithm is for undirected graphs and ignores edge directions.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "triangle_count",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"1",
10 "output.file_path":"result/triangle_count"
11 }
12}

3.8 K-Core

The K-Core algorithm marks all vertices with a degree of K, suitable for graph pruning and finding the core part of the graph.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "kcore",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"1",
10 "output.file_path":"result/kcore",
11 "kcore.degree_k":"5"
12 }
13}

3.9 SSSP (Single Source Shortest Path)

The single source the shortest path algorithm calculates the shortest distance from one point to all other points.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "sssp",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"1",
10 "output.file_path":"result/degree",
11 "sssp.source":"tom"
12 }
13}

3.10 KOUT

Starting from a point, get the k-layer nodes of this point.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "kout",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"1",
10 "output.file_path":"result/kout",
11 "kout.source":"tom",
12 "compute.max_step":"6"
13 }
14}

3.11 Louvain

The Louvain algorithm is a community detection algorithm based on modularity. The basic idea is that nodes in the network try to traverse all neighbor community labels and choose the community label that maximizes the modularity increment. After maximizing modularity, each community is regarded as a new node, and the process is repeated until the modularity no longer increases.

The distributed Louvain algorithm implemented on Vermeer is affected by factors such as node order and parallel computation. Due to the random traversal order of the Louvain algorithm, community compression also has a certain randomness, leading to different results in multiple executions. However, the overall trend will not change significantly.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "louvain",
 7 "compute.parallel":"10",
 8 "compute.max_step":"1000",
 9 "louvain.threshold":"0.0000001",
10 "louvain.resolution":"1.0",
11 "louvain.step":"10",
12 "output.type":"local",
13 "output.parallel":"1",
14 "output.file_path":"result/louvain"
15  }
16 }

3.12 Jaccard Similarity Coefficient

The Jaccard index, also known as the Jaccard similarity coefficient, is used to compare the similarity and diversity between finite sample sets. The larger the Jaccard coefficient value, the higher the similarity of the samples. It is used to calculate the Jaccard similarity coefficient between a given source point and all other points in the graph.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "jaccard",
 7 "compute.parallel":"10",
 8 "compute.max_step":"2",
 9 "jaccard.source":"123",
10 "output.type":"local",
11 "output.parallel":"1",
12 "output.file_path":"result/jaccard"
13 }
14}

3.13 Personalized PageRank

The goal of personalized PageRank is to calculate the relevance of all nodes relative to user u. Starting from the node corresponding to user u, at each node, there is a probability of 1-d to stop walking and start again from u, or a probability of d to continue walking, randomly selecting a node from the nodes pointed to by the current node to walk down. It is used to calculate the personalized PageRank score starting from a given starting point, suitable for scenarios such as social recommendations.

Since the calculation requires using out-degree, load.use_out_degree needs to be set to 1 when reading the graph.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "ppr",
 7 "compute.parallel":"100",
 8 "compute.max_step":"10",
 9 "ppr.source":"123",
10 "ppr.damping":"0.85",
11 "ppr.diff_threshold":"0.00001",
12 "output.type":"local",
13 "output.parallel":"1",
14 "output.file_path":"result/ppr"
15 }
16}

3.14 Global Kout

Calculate the k-degree neighbors of all nodes in the graph (excluding themselves and 1~k-1 degree neighbors). Due to the severe memory expansion of the global kout algorithm, k is currently limited to 1 and 2. Additionally, the global kout algorithm supports filtering functions (parameters such as “compute.filter”:“risk_level==1”), and the filtering condition is judged when calculating the k-degree. The final result set includes those that meet the filtering condition. The algorithm’s final output is the number of neighbors that meet the condition.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "kout_all",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"10",
10 "output.file_path":"result/kout",
11 "compute.max_step":"2",
12 "compute.filter":"risk_level==1"
13 }
14}

3.15 Clustering Coefficient

The clustering coefficient represents the coefficient of the clustering degree of nodes in a graph. In real networks, especially in specific networks, nodes tend to establish a tightly organized relationship due to relatively high-density connection points. The clustering coefficient algorithm (Cluster Coefficient) is used to calculate the clustering degree of nodes in the graph. This algorithm is for local clustering coefficients. The local clustering coefficient can measure the clustering degree around each node in the graph.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "clustering_coefficient",
 7 "compute.parallel":"100",
 8 "compute.max_step":"10",
 9 "output.type":"local",
10 "output.parallel":"1",
11 "output.file_path":"result/cc"
12 }
13}

3.16 SCC (Strongly Connected Components)

In the mathematical theory of directed graphs, if every vertex of a graph can be reached from any other point in the graph, the graph is said to be strongly connected. The parts of any directed graph that can achieve strong connectivity are called strongly connected components. It indicates the connectivity between points and distinguishes different connected communities.

Request example:

 1POST http://localhost:6688/tasks/create
 2{
 3 "task_type": "compute",
 4 "graph": "testdb",
 5 "params": {
 6 "compute.algorithm": "scc",
 7 "compute.parallel":"10",
 8 "output.type":"local",
 9 "output.parallel":"1",
10 "output.file_path":"result/scc",
11 "compute.max_step":"200"
12 }
13}

4. Cluster Monitoring and Telemetry

Vermeer provides dedicated REST endpoints for operations dashboards (such as Apache HugeGraph Hubble) and cloud-native container orchestrators (such as Kubernetes / Helm availability probes).

4.1 Cluster Health Snapshot (GET /api/v1/admin/cluster)

GET /api/v1/admin/cluster returns a read-only, versioned cluster health snapshot (schema_version: 1). In deployments where auth=token is enabled, this endpoint requires an administrator token (Authorization: Bearer <admin-token>) and enforces the standard /admin authentication filters.

Health and Readiness Semantics

  • Registration alone is not readiness. A Worker must be registered, connected to Master, and have load, compute, and superstep streams ready.
  • Cluster status is DEGRADED for an unready Master, no Workers, or any unready Worker; aggregate reasons are master_initializing / service_offline, no_workers, or workers_not_ready. Otherwise status is UP.
  • metric_status is an object with availability (AVAILABLE, UNAVAILABLE, or UNSUPPORTED), observed_at, and optional reason.
  • Master checks are http_ready, initialized, and service_online; Worker checks are registered, connection_ready, and the three *_stream_ready fields.
  • Worker telemetry uses one 800 ms aggregate deadline outside the Worker manager lock and does not affect readiness.

Example Request and Response

1curl -H "Authorization: Bearer <admin-token>" \
2  http://localhost:6688/api/v1/admin/cluster
 1{
 2  "schema_version": 1,
 3  "observed_at": 1730419200000,
 4  "status": "UP",
 5  "registered_workers": 1,
 6  "ready_workers": 1,
 7  "master": {
 8    "id": "master", "name": "Vermeer Master", "role": "MASTER",
 9    "grpc_peer": "127.0.0.1:6689", "version": "1.8.0",
10    "launch_time": "2026-11-01T08:00:00Z", "status": "UP", "ready": true,
11    "checks": {"http_ready": true, "initialized": true, "service_online": true},
12    "metrics": {"cpu_cores": 0.15, "rss_bytes": 134217728, "sample_window_seconds": 10.0},
13    "metric_status": {"availability": "AVAILABLE", "observed_at": 1730419200000}
14  },
15  "workers": [{
16    "id": "worker-1", "name": "worker-1", "role": "WORKER", "group": "default",
17    "grpc_peer": "127.0.0.1:6789", "version": "1.8.0",
18    "launch_time": "2026-11-01T08:00:05Z", "status": "UP", "ready": true,
19    "checks": {"registered": true, "connection_ready": true, "load_stream_ready": true,
20      "compute_stream_ready": true, "superstep_stream_ready": true},
21    "metrics": {"cpu_cores": 1.42, "rss_bytes": 2147483648, "sample_window_seconds": 10.0},
22    "metric_status": {"availability": "AVAILABLE", "observed_at": 1730419200000}
23  }]
24}

4.2 Space Worker Availability Probe (GET /api/v1/workers/availability)

GET /api/v1/workers/availability provides a space-scoped readiness check for Kubernetes and Helm availability hooks.

  • Request: Sent with caller authentication credentials representing a specific graph space.
  • Response: Returns {"available": true} when the target space has available computing capacity, or {"available": false} otherwise.
  • Availability criteria: An accessible common, space, or graph Worker group must contain at least one Worker, and every registered member in that group must pass all readiness checks. Workers registered exclusively in other graph spaces do not fulfill availability for the caller’s space.

🚧, further updates and improvements will be made at any time. Suggestions and feedback are welcome.

4.2 - HugeGraph-Computer Quick Start

1. Component Overview

HugeGraph-Computer is a Java distributed graph computing framework based on BSP (Bulk Synchronous Parallel), with algorithms running in iterative supersteps. Kubernetes Operator or YARN can schedule jobs; master and worker processes can also run on one machine for small trial jobs.

Computer and Vermeer, in the same repository, are separate implementations: Computer uses a Java/BSP runtime for distributed computing, while Vermeer is a Go in-memory graph computing platform with a master-worker architecture. They share HugeGraph data sources, but their deployment and job configurations are not interchangeable.

Computer reads graph data from HugeGraph or HDFS and writes results to either system. The runtime can spill some data to disk; whether a job completes still depends on input size, resources, and configuration. Disk spilling does not remove resource limits.

2. Prerequisites and Connections

Building and running Computer requires JDK 11 for the pinned 1.7.0 release (or JDK 17+ for 1.8.0+ and master, across build hosts and cluster workers); source builds also require Maven 3.6.3 or later (Maven 3.5+ for 1.7.0). When embedding Computer 1.8.0+ in custom Java applications, you must specify the required JVM options: --add-opens=java.base/java.io=ALL-UNNAMED (for buffered file duplication) and --add-opens=java.base/java.lang=ALL-UNNAMED (for SOFA RPC proxy generation). The distribution launcher script bin/start-computer.sh and Kubernetes Operator automatically configure these JVM options. The PageRank example needs a running HugeGraph-Server with graph data, plus etcd reachable by the master and workers.

ServiceExample address or portPurpose
HugeGraph-Serverhttp://127.0.0.1:8080Read graph data and write algorithm results; configured by hugegraph.url.
etcdhttp://127.0.0.1:2379BSP job coordination; configured by bsp.etcd_endpoints.
Computer master RPCTCP 8190Workers connect to master; port in the distribution configuration.
Computer worker data transferOS-assigned locally; K8s Operator defaults to 8099Transfer vertices and messages between workers. Advertised addresses and ports must be reachable across hosts.
MinIO (K8s manifests)HTTP 9000Input partition snapshots. Computer 1.7.0 manifests incorrectly map Service port 9000 to MinIO Console port 9090; change targetPort to 9000 before enabling snapshots. Defaults are snapshot.write=false and snapshot.load=false, so MinIO access is unnecessary when snapshots are disabled.
cert-manager (K8s)In-cluster serviceOperator manifests use cert-manager.io/v1 Certificate, Issuer, and CA injection. Install a compatible version before deploying the Operator.
HDFSCluster-specific (Hadoop 3.5.0 in CI)Required only when input or output uses HDFS. CI tests use Java 17-compatible Hadoop 3.5.0.

For Kubernetes jobs, hugegraph.url must be reachable from all computing Pods; do not use a localhost address available only on your computer. If HugeGraph authentication is enabled, configure the username and password and supply matching credentials for REST queries. For containerized deployments, ensure the container runtime supports Java 17 container base images (such as hugegraph/hugegraph-computer:latest).

Warning

In production, enable Server authentication and authorization, retain Server audit logs (normally audit-*.log), give Computer a dedicated account with only the read/write permissions required by its jobs, and configure a source IP allowlist for Server. hugegraph.username and hugegraph.password are credentials for connecting to Server, not a replacement for Server authentication.

See the Computer configuration reference for more options.

3. Obtain Source and Build a Distribution

The Apache HugeGraph download directory provides versioned Computer source archives, without separate precompiled Computer binaries. This example uses VERSION=1.7.0, whose source archive and build output names include -incubating-. For other versions, use the actual filenames in the download directory.

 1VERSION=1.7.0 # Replace with the release to download
 2ARCHIVE="apache-hugegraph-computer-incubating-${VERSION}-src.tar.gz"
 3DOWNLOAD_BASE="https://downloads.apache.org/hugegraph/${VERSION}"
 4curl -fLO "${DOWNLOAD_BASE}/${ARCHIVE}"
 5curl -fLO "${DOWNLOAD_BASE}/${ARCHIVE}.sha512"
 6curl -fLO "${DOWNLOAD_BASE}/${ARCHIVE}.asc"
 7curl -fLO https://downloads.apache.org/hugegraph/KEYS
 8shasum -a 512 -c "${ARCHIVE}.sha512"
 9gpg --import KEYS
10gpg --verify "${ARCHIVE}.asc" "${ARCHIVE}"
11tar -xzf "${ARCHIVE}"
12cd "apache-hugegraph-computer-incubating-${VERSION}-src/computer"
13mvn clean package -DskipTests
14tar -xzf "target/apache-hugegraph-computer-incubating-${VERSION}.tar.gz"
15cd "apache-hugegraph-computer-incubating-${VERSION}"

To build current master, clone and package the source:

1git clone https://github.com/apache/hugegraph-computer.git
2cd hugegraph-computer/computer
3mvn clean package -DskipTests

The computer/computer-dist module assembles the distribution, including bin/start-computer.sh, runtime dependencies in lib/, built-in algorithms in algorithm/builtin-algorithm.jar, and default conf/computer.properties and conf/log4j2.xml. The source configuration is computer/computer-dist/src/assembly/static/conf/computer.properties. Release 1.7.0 build output includes -incubating-; current master omits it and takes its version from the POM. Use archive names matching the source you built.

4. Run PageRank Locally

Edit conf/computer.properties in the distribution directory with your HugeGraph URL, graph name, and credentials; point bsp.etcd_endpoints to reachable etcd. The default configuration selects built-in PageRankParams. Master and workers must use the same configuration and job.id; concurrent jobs need distinct job IDs.

Start processes from the distribution directory in two terminals. The script reads conf/computer.properties by default; use -c to select another file:

Terminal 1: Master
1# Start Computer Master process
2bin/start-computer.sh -d local -r master
Terminal 2: Worker
1# Start Computer Worker process
2bin/start-computer.sh -d local -r worker

Master waits for the configured workers to register before running the job. Check both terminal outputs; the default logging configuration also writes master and worker logs under the distribution directory’s logs/. Process startup alone does not confirm completion: verify that input, superstep computation, and output all finish normally in the master log.

PageRank writes results to HugeGraph under page_rank. First ensure every target vertex label allows this property. Computer creates a DOUBLE, OLAP_COMMON property key but does not add it to vertex labels. If absent, create it:

1curl --fail --request POST \
2  --header 'Content-Type: application/json' \
3  --data '{"name":"page_rank","data_type":"DOUBLE","cardinality":"SINGLE","write_type":"OLAP_COMMON"}' \
4  'http://127.0.0.1:8080/graphspaces/DEFAULT/graphs/hugegraph/schema/propertykeys'

If the property key already exists, verify its type is DOUBLE and write_type is OLAP_COMMON. List vertex labels, then add nullable page_rank to every label being computed; replace person with the actual label:

1curl --fail --compressed \
2  'http://127.0.0.1:8080/graphspaces/DEFAULT/graphs/hugegraph/schema/vertexlabels'
3
4curl --fail --request PUT \
5  --header 'Content-Type: application/json' \
6  --data '{"name":"person","properties":["page_rank"],"nullable_keys":["page_rank"]}' \
7  'http://127.0.0.1:8080/graphspaces/DEFAULT/graphs/hugegraph/schema/vertexlabels/person?action=append'

If the current read mode hides OLAP writes, an administrator can set it to ALL:

1curl --fail --request PUT \
2  --header 'Content-Type: application/json' \
3  --data '"ALL"' \
4  'http://127.0.0.1:8080/graphspaces/DEFAULT/graphs/hugegraph/graph_read_mode'

Query vertices to verify the result property:

1curl --fail --compressed \
2  'http://127.0.0.1:8080/graphspaces/DEFAULT/graphs/hugegraph/graph/vertices?limit=3'

When authentication is required, add --user "$HG_USER:$HG_PASSWORD" to curl. See the graph read-mode REST API for the endpoint and permissions.

5. Run PageRank on Kubernetes

Ensure computing Pods can reach HugeGraph-Server, then install a cluster-compatible cert-manager using its official installation guide. The Computer Operator manifests deploy the Operator, etcd, and MinIO. Use the same Computer release for the CRD and Operator manifests; this example uses 1.7.0:

1VERSION=1.7.0 # Use the same release for CRD and Operator manifests
2kubectl apply -f "https://raw.githubusercontent.com/apache/hugegraph-computer/${VERSION}/computer/computer-k8s-operator/manifest/hugegraph-computer-crd.v1.yaml"
3kubectl apply -f "https://raw.githubusercontent.com/apache/hugegraph-computer/${VERSION}/computer/computer-k8s-operator/manifest/hugegraph-computer-operator.yaml"
4kubectl rollout status deployment/hugegraph-computer-operator-controller-manager \
5  -n hugegraph-computer-operator-system --timeout=120s
Optional: enable MinIO snapshots (1.7.0)

The 1.7.0 MinIO Service maps S3 API port 9000 to Console port 9090. Fix this mapping before enabling snapshots and set snapshot.minio_endpoint to http://hugegraph-computer-operator-minio.hugegraph-computer-operator-system.svc:9000. Skip this when snapshots remain disabled.

1kubectl patch service hugegraph-computer-operator-minio \
2  -n hugegraph-computer-operator-system \
3  --type=json \
4  -p='[{"op":"replace","path":"/spec/ports/0/targetPort","value":9000}]'

Replace the HugeGraph URL with a service address reachable by computing Pods, then submit a HugeGraphComputerJob. This example uses the official Docker Hub image hugegraph/hugegraph-computer:latest (built on Java 17) and its built-in PageRank JAR. When testing HDFS inputs or outputs in Kubernetes or Docker environments, pair with an Apache Hadoop 3.5.0 compatible storage cluster or image. Custom algorithm JARs can be bundled in the image and selected with jarFile, or downloaded from HTTP(S) with remoteJarUri. The partition count must be at least the worker count.

Save the following YAML as pagerank-job.yaml, then apply it:

 1apiVersion: operator.hugegraph.apache.org/v1
 2kind: HugeGraphComputerJob
 3metadata:
 4  namespace: hugegraph-computer-operator-system
 5  name: pagerank-sample
 6spec:
 7  jobId: pagerank-sample
 8  algorithmName: page_rank
 9  image: hugegraph/hugegraph-computer:latest # Official Docker Hub image
10  jarFile: /hugegraph/hugegraph-computer/algorithm/builtin-algorithm.jar
11  pullPolicy: IfNotPresent
12  workerInstances: 1
13  masterCpu: 500m
14  workerCpu: 500m
15  masterMemory: 1Gi
16  workerMemory: 2Gi
17  computerConf:
18    job.partitions_count: "1"
19    algorithm.params_class: org.apache.hugegraph.computer.algorithm.centrality.pagerank.PageRankParams
20    hugegraph.url: http://hugegraph-server:8080
21    hugegraph.name: hugegraph
1kubectl apply -f pagerank-job.yaml
2kubectl get hcjob pagerank-sample -n hugegraph-computer-operator-system --watch

SUCCEEDED means the job completed. By default, the Operator deletes the CR and computing resources after completion. Expand the following instructions when you need to retain state or investigate a failed job.

Optional: retain job state, inspect logs, or clean up resources

The Operator defaults to AUTO_DESTROY_POD=true, so short jobs may lose their CR and Pods before inspection. Disable this before submitting the job, wait for the replacement Controller Pod to become ready, then submit. Change this setting in the Java Operator’s controller container:

1kubectl set env deployment/hugegraph-computer-operator-controller-manager \
2  -n hugegraph-computer-operator-system \
3  --containers=controller AUTO_DESTROY_POD=false
4kubectl rollout status deployment/hugegraph-computer-operator-controller-manager \
5  -n hugegraph-computer-operator-system --timeout=120s

Inspect runtime information or failure logs:

1kubectl logs --follow <master-pod-name> -n hugegraph-computer-operator-system
2kubectl logs --follow <worker-pod-name> -n hugegraph-computer-operator-system

After checking results, delete the CR to clean up associated resources and restore the default policy:

1kubectl delete hcjob pagerank-sample -n hugegraph-computer-operator-system
2kubectl set env deployment/hugegraph-computer-operator-controller-manager \
3  -n hugegraph-computer-operator-system \
4  --containers=controller AUTO_DESTROY_POD=true

After PageRank writes to HugeGraph, set the read mode and query as in the previous section. For HDFS output, results are under output.hdfs_path_prefix/<job.id>/; filenames and partition layout depend on job configuration.

See the CRD configuration reference for all fields.

6. Built-in Algorithms and Development

Current built-in algorithms include:

  • Centrality: PageRank, Betweenness Centrality, Closeness Centrality, Degree Centrality.
  • Communities and structure: Clustering Coefficient, K-core, LPA, Triangle Count, WCC.
  • Paths and sampling: ring detection, filtered ring detection, single-source shortest path, Random Walk.

Implementations are in computer/computer-algorithm. Custom algorithms must follow the Computer API and be packaged as loadable JARs; see the Computer README for modules and development entry points.

4.3 - HugeGraph-Computer Configuration Reference

Computer Config Options

The defaults in the tables come from ComputerOptions.java in computer-api; explicit conf/computer.properties overrides are shown as “code default (distribution: actual value)”. Common options such as rpc.* come from HugeGraph Commons; distribution values are listed separately below.

Configuration Sources

  • Source template: computer/computer-dist/src/assembly/static/conf/computer.properties, with log4j2.xml in the same directory. Maven package copies these into distribution conf/, runtime dependencies into lib/, and built-in algorithm JARs into algorithm/. Templates are maintained in source; there is no separate configuration generator.
  • Standalone and YARN: startup scripts read distribution conf/computer.properties by default. Override it with bin/start-computer.sh -c <configuration-path>; master and workers need the same job parameters.
  • Kubernetes Operator: users supply spec.computerConf in the CRD; the Operator writes a ConfigMap mounted as computer.properties. It supplies job ID, worker count, Pod addresses, and etcd when unspecified. Unspecified or zero transfer/RPC ports become 8099/8190; transport.server_host and rpc.server_host become Pod IPs. Job startup scripts substitute only ${POD_IP}, ${HOSTNAME}, ${POD_NAME}, and ${POD_NAMESPACE}.
  • Kubernetes job images must contain the Computer runtime. Bundle algorithm JARs and select them with jarFile, or use an HTTP(S) remoteJarUri for startup download. See the Computer Quick Start and CRD table below.

Apache downloads provide versioned Computer source archives without separate precompiled binaries. Release 1.7.0 source and distribution names include -incubating-; run mvn clean package -DskipTests in the tagged computer/ project to build them. Current master omits that marker and takes its version from the POM. Source archives cannot be started as built distributions.

Warning

Empty HugeGraph credentials and example MinIO keys below are for local demonstrations. In production, enable Server authentication and authorization, retain Server audit-*.log, use a Computer account with minimum required Server permissions, and configure a Server source IP allowlist. Do not use example MinIO keys in production.


1. Basic Configuration

Core job settings for HugeGraph-Computer.

config optiondefault valuedescription
hugegraph.urlhttp://127.0.0.1:8080The HugeGraph server URL to load data and write results back.
hugegraph.namehugegraphThe graph name to load data and write results back.
hugegraph.username"" (empty)The username for HugeGraph authentication (leave empty if authentication is disabled).
hugegraph.password"" (empty)The password for HugeGraph authentication (leave empty if authentication is disabled).
job.idlocal_0001 (packaged: local_001)The job identifier on YARN cluster or K8s cluster.
job.namespace"" (empty)Optional etcd job-key prefix for namespace isolation; distinct from Kubernetes metadata.namespace and not populated automatically by the Operator.
job.workers_count1The number of workers for one graph algorithm job. In K8s, this option is set by the Operator.
job.partitions_count1The number of partitions for computing one graph algorithm job.
job.partitions_thread_nums4The number of threads for partition parallel compute.

2. Algorithm Configuration

Algorithm-specific configuration for computation logic.

config optiondefault valuedescription
algorithm.params_classComputerOptions.Null placeholder classRequired. The class used to pass algorithm parameters before the algorithm runs.
algorithm.result_classComputerOptions.Null placeholder classThe vertex value class used to store computation results.
algorithm.message_classComputerOptions.Null placeholder classThe message class passed while computing a vertex.

3. Input Configuration

Configuration for loading input data from HugeGraph or other sources.

3.1 Input Source

config optiondefault valuedescription
input.source_typehugegraph-serverThe source type to load input data, allowed values: [‘hugegraph-server’, ‘hugegraph-loader’]. The ‘hugegraph-loader’ means use hugegraph-loader to load data from HDFS or file. If using ‘hugegraph-loader’, please configure ‘input.loader_struct_path’ and ‘input.loader_schema_path’.
input.loader_struct_path"" (empty)The structure path for Loader input. It takes effect only when input.source_type=hugegraph-loader.
input.loader_schema_path"" (empty)The schema path for Loader input. It takes effect only when input.source_type=hugegraph-loader.

3.2 Input Splits

config optiondefault valuedescription
input.split_size1048576 (1 MB)The input split size in bytes.
input.split_max_splits10000000The maximum number of input splits.
input.split_page_size500The page size for streamed load input split data.
input.split_fetch_timeout300The timeout in seconds to fetch input splits.

3.3 Input Processing

config optiondefault valuedescription
input.filter_classorg.apache.hugegraph.computer.core.input.filter.DefaultInputFilterThe class to create input-filter object. Input-filter is used to filter vertex edges according to user needs.
input.edge_directionOUTThe direction of edges to load, allowed values: [OUT, IN, BOTH]. When the value is BOTH, edges in both OUT and IN directions will be loaded.
input.edge_freqMULTIPLEThe frequency of edges that can exist between a pair of vertices, allowed values: [SINGLE, SINGLE_PER_LABEL, MULTIPLE]. SINGLE means only one edge can exist between a pair of vertices (identified by sourceId + targetId); SINGLE_PER_LABEL means each edge label can have one edge between a pair of vertices (identified by sourceId + edgeLabel + targetId); MULTIPLE means many edges can exist between a pair of vertices (identified by sourceId + edgeLabel + sortValues + targetId).
input.max_edges_in_one_vertex200The maximum number of adjacent edges allowed to be attached to a vertex. The adjacent edges will be stored and transferred together as a batch unit.

3.4 Input Performance

config optiondefault valuedescription
input.send_thread_nums4The number of threads for parallel sending of vertices or edges.

4. Snapshot & Storage Configuration

HugeGraph-Computer supports snapshot functionality to save vertex/edge partitions to local storage or MinIO object storage, enabling checkpoint recovery or accelerating repeated computations.

4.1 Basic Snapshot Configuration

config optiondefault valuedescription
snapshot.writefalseWhether to write snapshots of input vertex/edge partitions.
snapshot.loadfalseWhether to load from snapshots of vertex/edge partitions.
snapshot.name"" (empty)User-defined snapshot name to distinguish different snapshots.

4.2 MinIO Integration (Optional)

MinIO can be used as a distributed object storage backend for snapshots in K8s deployments.

config optiondefault valuedescription
snapshot.minio_endpoint"" (empty)MinIO service endpoint (e.g., http://minio:9000). Required when using MinIO.
snapshot.minio_access_keyminioadminMinIO access key for authentication.
snapshot.minio_secret_keyminioadminMinIO secret key for authentication.
snapshot.minio_bucket_name"" (empty)MinIO bucket name for storing snapshot data.

Usage Scenarios:

  • Checkpoint Recovery: Resume from snapshots after job failures, avoiding data reloading
  • Repeated Computations: Load data from snapshots when running the same algorithm multiple times
  • A/B Testing: Save multiple snapshot versions of the same dataset to test different algorithm parameters

Example: Local Snapshot (in computer.properties):

1snapshot.write=true
2snapshot.name=pagerank-snapshot-20260201

Example: MinIO Snapshot (in K8s CRD computerConf):

1computerConf:
2  snapshot.write: "true"
3  snapshot.name: "pagerank-snapshot-v1"
4  snapshot.minio_endpoint: "http://minio:9000"
5  snapshot.minio_access_key: "my-access-key"
6  snapshot.minio_secret_key: "my-secret-key"
7  snapshot.minio_bucket_name: "hugegraph-snapshots"

5. Worker & Master Configuration

Configuration for worker and master computation logic.

5.1 Master Configuration

config optiondefault valuedescription
master.computation_classorg.apache.hugegraph.computer.core.master.DefaultMasterComputationMaster-computation is computation that can determine whether to continue to the next superstep. It runs at the end of each superstep on the master.

5.2 Worker Computation

config optiondefault valuedescription
worker.computation_classorg.apache.hugegraph.computer.core.config.NullThe class to create worker-computation object. Worker-computation is used to compute each vertex in each superstep.
worker.combiner_classorg.apache.hugegraph.computer.core.config.NullCombiner can combine messages into one value for a vertex. For example, PageRank algorithm can combine messages of a vertex to a sum value.
worker.partitionerorg.apache.hugegraph.computer.core.graph.partition.HashPartitionerThe partitioner that decides which partition a vertex should be in, and which worker a partition should be in.

5.3 Worker Combiners

config optiondefault valuedescription
worker.vertex_properties_combiner_classorg.apache.hugegraph.computer.core.combiner.OverwritePropertiesCombinerThe combiner can combine several properties of the same vertex into one properties at input step.
worker.edge_properties_combiner_classorg.apache.hugegraph.computer.core.combiner.OverwritePropertiesCombinerThe combiner can combine several properties of the same edge into one properties at input step.

5.4 Worker Buffers

config optiondefault valuedescription
worker.received_buffers_bytes_limit104857600 (100 MB)The limit bytes of buffers of received data. The total size of all buffers can’t exceed this limit. If received buffers reach this limit, they will be merged into a file (spill to disk).
worker.write_buffer_capacity52428800 (50 MB)The initial size of write buffer that used to store vertex or message.
worker.write_buffer_threshold52428800 (50 MB)The threshold of write buffer. Exceeding it will trigger sorting. The write buffer is used to store vertex or message.

5.5 Worker Data & Timeouts

config optiondefault valuedescription
worker.data_dirs[jobs]The directories separated by ‘,’ that received vertices and messages can persist into.
worker.wait_sort_timeout600000 (10 minutes)The max timeout (in ms) for message-handler to wait for sort-thread to sort one batch of buffers.
worker.wait_finish_messages_timeout86400000 (24 hours)The max timeout (in ms) for message-handler to wait for finish-message of all workers.

6. I/O & Output Configuration

Configuration for output computation results.

6.1 Output Class & Result

config optiondefault valuedescription
output.output_classorg.apache.hugegraph.computer.core.output.LogOutputThe class to output the computation result of each vertex. Called after iteration computation.
output.result_namevalueThe value is assigned dynamically by #name() of instance created by WORKER_COMPUTATION_CLASS.
output.result_write_typeOLAP_COMMONThe result write-type to output to HugeGraph, allowed values: [OLAP_COMMON, OLAP_SECONDARY, OLAP_RANGE].

6.2 Output Behavior

config optiondefault valuedescription
output.with_adjacent_edgesfalseWhether to output the adjacent edges of the vertex.
output.with_vertex_propertiesfalseWhether to output the properties of the vertex.
output.with_edge_propertiesfalseWhether to output the properties of the edge.

6.3 Batch Output

config optiondefault valuedescription
output.batch_size500The batch size of output.
output.batch_threads1The number of threads used for batch output.
output.single_threads1The number of threads used for single output.

6.4 HDFS Output

config optiondefault valuedescription
output.hdfs_urlhdfs://127.0.0.1:9000The HDFS URL for output.
output.hdfs_userhadoopThe HDFS user for output.
output.hdfs_path_prefix/hugegraph-computer/resultsThe directory of HDFS output results.
output.hdfs_delimiter, (comma)The delimiter of HDFS output.
output.hdfs_merge_partitionstrueWhether to merge output files of multiple partitions.
output.hdfs_replication3The replication number of HDFS.
output.hdfs_core_site_path"" (empty)The HDFS core site path.
output.hdfs_site_path"" (empty)The HDFS site path.
output.hdfs_kerberos_enablefalseWhether Kerberos authentication is enabled for HDFS.
output.hdfs_kerberos_principal"" (empty)The HDFS principal for Kerberos authentication.
output.hdfs_kerberos_keytab"" (empty)The HDFS keytab file for Kerberos authentication.
output.hdfs_krb5_conf/etc/krb5.confKerberos configuration file path.

6.5 Retry & Timeout

config optiondefault valuedescription
output.retry_times3The retry times when output fails.
output.retry_interval10The retry interval (in seconds) when output fails.
output.thread_pool_shutdown_timeout60The timeout (in seconds) of output thread pool shutdown.

7. Network & Transport Configuration

Configuration for network communication between workers and master.

7.1 Server Configuration

config optiondefault valuedescription
transport.server_host127.0.0.1Worker data listener and advertised address. It must be reachable by other workers across hosts; the Operator sets it to the Pod IP.
transport.server_port0 (distribution: 0; K8s Operator: 8099)Worker data port. Locally, 0 lets the OS assign a port; the Operator uses fixed port 8099 by default for Pod communication.
transport.server_threads4The number of transport threads for server.

7.2 Client Configuration

config optiondefault valuedescription
transport.client_threads4The number of transport threads for client.
transport.client_connect_timeout3000The timeout (in ms) of client connect to server.

7.3 Protocol Configuration

config optiondefault valuedescription
transport.provider_classorg.apache.hugegraph.computer.core.network.netty.NettyTransportProviderThe transport provider, currently only supports Netty.
transport.io_modeAUTOThe network IO mode, allowed values: [NIO, EPOLL, AUTO]. AUTO means selecting the appropriate mode automatically.
transport.tcp_keep_alivetrueWhether to enable TCP keep-alive.
transport.transport_epoll_ltfalseWhether to enable EPOLL level-trigger (only effective when io_mode=EPOLL).

7.4 Buffer Configuration

config optiondefault valuedescription
transport.send_buffer_size0The size of socket send-buffer in bytes. 0 means using system default value.
transport.receive_buffer_size0The size of socket receive-buffer in bytes. 0 means using system default value.
transport.write_buffer_high_mark67108864 (64 MB)The high water mark for write buffer in bytes. It will trigger sending unavailable if the number of queued bytes > write_buffer_high_mark.
transport.write_buffer_low_mark33554432 (32 MB)The low water mark for write buffer in bytes. It will trigger sending available if the number of queued bytes < write_buffer_low_mark.

7.5 Flow Control

config optiondefault valuedescription
transport.max_pending_requests8The max number of client unreceived ACKs. It will trigger sending unavailable if the number of unreceived ACKs >= max_pending_requests.
transport.min_pending_requests6The minimum number of client unreceived ACKs. It will trigger sending available if the number of unreceived ACKs < min_pending_requests.
transport.min_ack_interval200The minimum interval (in ms) of server reply ACK.

7.6 Timeouts

config optiondefault valuedescription
transport.close_timeout10000The timeout (in ms) of close server or close client.
transport.sync_request_timeout10000The timeout (in ms) to wait for response after sending sync-request.
transport.finish_session_timeout0The timeout (in ms) to finish session. 0 means using (transport.sync_request_timeout × transport.max_pending_requests).
transport.write_socket_timeout3000The timeout (in ms) to write data to socket buffer.
transport.server_idle_timeout360000 (6 minutes)The max timeout (in ms) of server idle.

7.7 Heartbeat

config optiondefault valuedescription
transport.heartbeat_interval20000 (20 seconds)The minimum interval (in ms) between heartbeats on client side.
transport.max_timeout_heartbeat_count120The maximum times of timeout heartbeat on client side. If the number of timeouts waiting for heartbeat response continuously > max_timeout_heartbeat_count, the channel will be closed from client side.

7.8 Advanced Network Settings

config optiondefault valuedescription
transport.max_syn_backlog511The capacity of SYN queue on server side. 0 means using system default value.
transport.recv_file_modetrueWhether to enable receive buffer-file mode. It will receive buffer and write to file from socket using zero-copy if enabled. Note: Requires OS support for zero-copy (e.g., Linux sendfile/splice).
transport.network_retries3The number of retry attempts for network communication if network is unstable.

7.9 Master RPC Configuration

These options come from HugeGraph Commons RPC configuration; the distribution template explicitly sets host and port. Kubernetes Operator advertises the Pod IP and defaults an unspecified or zero port to 8190.

OptionDistribution valueDescription
rpc.server_host127.0.0.1 (K8s: Pod IP)Master RPC address, reachable by workers.
rpc.server_port8190 (K8s default: 8190)Master RPC listener port; allow worker connections in security groups and network policies.

8. Storage & Persistence Configuration

Configuration for HGKV (HugeGraph Key-Value) storage engine and value files.

8.1 HGKV Configuration

config optiondefault valuedescription
hgkv.max_file_size2147483648 (2 GB)The max number of bytes in each HGKV file.
hgkv.max_data_block_size65536 (64 KB)The max byte size of HGKV file data block.
hgkv.max_merge_files10The max number of files to merge at one time.
hgkv.temp_file_dir/tmp/hgkvThis folder is used to store temporary files during the file merging process.

8.2 Value File Configuration

config optiondefault valuedescription
valuefile.max_segment_size1073741824 (1 GB)The max number of bytes in each segment of value-file.

9. BSP & Coordination Configuration

Configuration for Bulk Synchronous Parallel (BSP) protocol and etcd coordination.

config optiondefault valuedescription
bsp.etcd_endpointshttp://localhost:2379 (distribution: http://127.0.0.1:2379)Comma-separated etcd client endpoints. The Operator uses INTERNAL_ETCD_URL only when computerConf does not specify this key.
bsp.max_super_step10 (packaged: 2)The max super step of the algorithm.
bsp.register_timeout300000 (packaged: 100000)The max timeout (in ms) to wait for master and workers to register.
bsp.wait_workers_timeout86400000 (24 hours)The max timeout (in ms) to wait for workers BSP event.
bsp.wait_master_timeout86400000 (24 hours)The max timeout (in ms) to wait for master BSP event.
bsp.log_interval30000 (30 seconds)The log interval (in ms) to print the log while waiting for BSP event.

10. Performance Tuning Configuration

Configuration for performance optimization.

config optiondefault valuedescription
allocator.max_vertices_per_thread10000Maximum number of vertices per thread processed in each memory allocator.
sort.thread_nums4The number of threads performing internal sorting.

11. System Administration Configuration

Kubernetes Operator fills or overrides these options. Avoid overriding them unless customizing the Operator or network. job.namespace is optional user configuration; see the basic configuration table.

OptionManaged byDescription
bsp.etcd_endpointsK8s OperatorUses INTERNAL_ETCD_URL only when absent from computerConf.
transport.server_hostK8s OperatorOverrides with Pod IP; workers must reach each other.
transport.server_portK8s OperatorDefaults unspecified or zero values to 8099, not a random port.
job.idK8s OperatorSet from CRD job ID.
job.workers_countK8s OperatorSet from CRD workerInstances.
rpc.server_hostK8s OperatorOverrides with master Pod IP.
rpc.server_portK8s OperatorDefaults unspecified or zero values to 8190.
rpc.remote_urlComputer startupRemoved when configuration is read; do not set as job configuration.

Why these values must match:

  • BSP/RPC: Must match deployed etcd/RPC services for coordination.
  • Job settings: Must match the CRD to provide the correct worker count.
  • Transport: Workers need mutually reachable Pod IPs and ports; the K8s default is 8099.

K8s Operator Config Options

NOTE: Option needs to be converted through environment variable settings, e.g. k8s.internal_etcd_url => INTERNAL_ETCD_URL

config optiondefault valuedescription
k8s.auto_destroy_podtrueDelete the job CR after completion or failure; CR deletion triggers cleanup of associated computing resources.
k8s.close_reconciler_timeout120The max timeout (in ms) to close reconciler.
k8s.internal_etcd_urlCode: http://127.0.0.1:2379; manifest: http://hugegraph-computer-operator-etcd.hugegraph-computer-operator-system:2379etcd URL used by Operator jobs; the supplied manifest uses the etcd Service address.
k8s.internal_minio_urlCode: http://127.0.0.1:9000; manifest: http://hugegraph-computer-operator-minio.hugegraph-computer-operator-system:9000MinIO address supplied to jobs by the Operator; needed only for MinIO snapshots.
k8s.max_reconcile_retry3The max retry times of reconcile.
k8s.probe_backlog50The maximum backlog for serving health probes.
k8s.probe_port9892The port that the controller binds to for serving health probes.
k8s.ready_check_internal1000The time interval (ms) of check ready.
k8s.ready_timeout30000The max timeout (in ms) of check ready.
k8s.reconciler_countCode: Runtime.getRuntime().availableProcessors(); manifest: 6Maximum reconciler thread count.
k8s.resync_period600000The minimum frequency at which watched resources are reconciled.
k8s.timezoneAsia/ShanghaiThe timezone of computer job and operator.
k8s.watch_namespacehugegraph-computer-operator-systemNamespace watched for custom resources; also used by the supplied manifest. Use * to watch all namespaces.

HugeGraph-Computer CRD

1.7.0 CRD: https://github.com/apache/hugegraph-computer/blob/1.7.0/computer/computer-k8s-operator/manifest/hugegraph-computer-crd.v1.yaml

specdefault valuedescriptionrequired
algorithmNameThe name of algorithm.true
jobIdThe job id.true
imageJob container image, which must include the Computer runtime. Bundle algorithm JARs or download them with remoteJarUri.true
computerConfThe map of computer config options.true
workerInstancesThe number of worker instances, it will override the ‘job.workers_count’ option.true
pullPolicyKubernetes selects Always for latest, otherwise IfNotPresent, when unsetExplicit values: Always, Never, or IfNotPresent. The CRD has no default; see image pull policy.false
pullSecretsThe pull-secrets of Image, detail please refer to: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-podfalse
masterCpuThe cpu limit of master, the unit can be ’m’ or without unit detail please refer to: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpufalse
workerCpuThe cpu limit of worker, the unit can be ’m’ or without unit detail please refer to: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpufalse
masterMemoryThe memory limit of master, the unit can be one of Ei、Pi、Ti、Gi、Mi、Ki detail please refer to: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memoryfalse
workerMemoryThe memory limit of worker, the unit can be one of Ei、Pi、Ti、Gi、Mi、Ki detail please refer to: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memoryfalse
log4jXmlThe content of log4j.xml for computer job.false
jarFilePath to the algorithm JAR inside the image.false
remoteJarUriHTTP(S) algorithm JAR URL, downloaded and loaded by the startup script.false
jvmOptionsThe java startup parameters of computer job.false
envVarsplease refer to: https://kubernetes.io/docs/tasks/inject-data-application/define-interdependent-environment-variables/false
envFromplease refer to: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/false
masterCommandbin/start-computer.shThe run command of master, equivalent to ‘Entrypoint’ field of Docker.false
masterArgs["-r master", “-d k8s”]The run args of master, equivalent to ‘Cmd’ field of Docker.false
workerCommandbin/start-computer.shThe run command of worker, equivalent to ‘Entrypoint’ field of Docker.false
workerArgs["-r worker", “-d k8s”]The run args of worker, equivalent to ‘Cmd’ field of Docker.false
volumesPlease refer to: https://kubernetes.io/docs/concepts/storage/volumes/false
volumeMountsPlease refer to: https://kubernetes.io/docs/concepts/storage/volumes/false
secretPathsThe map of k8s-secret name and mount path.false
configMapPathsThe map of k8s-configmap name and mount path.false
podTemplateSpecPlease refer to: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-template-v1/#PodTemplateSpecfalse
securityContextPlease refer to: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/false

KubeDriver Config Options

config optiondefault valuedescription
k8s.build_image_bash_pathThe path of command used to build image.
k8s.enable_internal_algorithmtrueWhether enable internal algorithm.
k8s.framework_image_urlhugegraph/hugegraph-computer:latestThe image url of computer framework.
k8s.image_repository_passwordThe password for login image repository.
k8s.image_repository_registryThe address for login image repository.
k8s.image_repository_urlhugegraph/hugegraph-computerThe url of image repository.
k8s.image_repository_usernameThe username for login image repository.
k8s.internal_algorithm[pageRank]The name list of all internal algorithm. Note: Algorithm names use camelCase here (e.g., pageRank), but algorithm implementations return underscore_case (e.g., page_rank).
k8s.internal_algorithm_image_urlhugegraph/hugegraph-computer:latestThe image url of internal algorithm.
k8s.jar_file_dir/cache/jars/The directory where the algorithm jar will be uploaded.
k8s.kube_config~/.kube/configThe path of k8s config file.
k8s.log4j_xml_pathThe log4j.xml path for computer job.
k8s.namespacehugegraph-computer-operator-systemNamespace of the HugeGraph-Computer system.
k8s.pull_secret_names[]The names of pull-secret for pulling image.