Development & Testing

• 1: HugeGraph Toolchain Local Testing Guide
• 2: ToplingDB development integration

1 - HugeGraph Toolchain Local Testing Guide

This guide helps developers run HugeGraph toolchain tests locally.

1. Core Concepts

1.1 Core Dependency: HugeGraph Server

Integration and functional tests of the toolchain depend on HugeGraph Server, including Client, Loader, Hubble, Spark Connector, Tools, and other components.

1.2 Test Types

• Unit Tests: Test individual functions/methods, no external dependencies required
• API Tests (ApiTestSuite): Test API interfaces, requires running HugeGraph Server
• Functional Tests (FuncTestSuite): End-to-end tests, require complete system environment

2. Environment Setup

2.1 System Requirements

• Operating System: Linux / macOS (Windows use WSL2)
• JDK: >= 11, configure JAVA_HOME
• Maven: >= 3.6
• Python: >= 3.11 (only required for Hubble tests)

2.2 Clone Code

1git clone https://github.com/apache/hugegraph-toolchain.git
2cd hugegraph-toolchain

3. Deploy Test Environment

Deployment Options

• Script Deployment: Specify a Server commit to reproduce the server version used by CI
• Docker Deployment: Suitable for quick checks; if tests fail, first verify compatibility between the image and Toolchain

For detailed installation instructions, refer to Community Documentation

3.1 Script Deployment

Parameter Description

• $COMMIT_ID: Specify Server source code Git Commit ID
• $DB_DATABASE / $DB_PASS: MySQL database name and password for Loader JDBC tests

Deployment Steps

1. Install HugeGraph Server

1# Set the Server baseline; use a full commit SHA for reproducible results
2export COMMIT_ID="master"
3
4# Execute installation (script located in /assembly/travis/ directory)
5hugegraph-client/assembly/travis/install-hugegraph-from-source.sh $COMMIT_ID

• The script starts HTTP and HTTPS instances on ports 8080 and 8443 and configures admin/pa authentication.
• Ensure both ports are available before running it.

2. Install Optional Dependencies

1# Hadoop (only required for Loader HDFS tests)
2hugegraph-loader/assembly/travis/install-hadoop.sh
3
4# MySQL (only required for Loader JDBC tests)
5hugegraph-loader/assembly/travis/install-mysql.sh $DB_DATABASE $DB_PASS

3. Health Check

1curl -u admin:pa http://localhost:8080/graphspaces/DEFAULT/graphs
2# Returns {"graphs":["hugegraph"]} indicates success

3.2 Docker Deployment

Note: Docker images may have version lag, use script deployment if encountering compatibility issues

Quick Start

1docker network create hugegraph-net
2docker run -itd --name=server -p 8080:8080 --network hugegraph-net hugegraph/hugegraph:latest

docker-compose Configuration (Optional)

Complete configuration example including Server, MySQL, Hadoop services (requires Docker Compose V2):

 1version: '3.8'
 2
 3services:
 4  hugegraph-server:
 5    image: hugegraph/hugegraph:latest  # Can be replaced with a specific version, or build your own image
 6    container_name: hugegraph-server
 7    ports:
 8      - "8080:8080"  # HugeGraph Server HTTP port
 9    environment:
10      # Configure HugeGraph Server parameters as needed, e.g., backend storage
11      - HUGEGRAPH_SERVER_OPTIONS="-Dstore.backend=rocksdb"
12    volumes:
13      # If you need to persist data or mount configuration files, add volumes here
14      # - ./hugegraph-data:/opt/hugegraph/data
15    healthcheck:
16      test: ["CMD-SHELL", "curl -f http://localhost:8080/graphspaces/DEFAULT/graphs || exit 1"]
17      interval: 10s
18      timeout: 3s
19      retries: 5
20    networks:
21      - hugegraph-net
22  
23  # If you need JDBC tests for hugegraph-loader, you can add the following service
24  #   mysql:
25  #     image: mysql:5.7
26  #     container_name: mysql-db
27  #     environment:
28  #       MYSQL_ROOT_PASSWORD: ${DB_PASS:-your_mysql_root_password} # Read from environment variable, or use default
29  #       MYSQL_DATABASE: ${DB_DATABASE:-hugegraph_test_db} # Read from environment variable, or use default
30  #     ports:
31  #       - "3306:3306"
32  #     volumes:
33  #       - ./mysql-data:/var/lib/mysql # Data persistence
34  #     healthcheck:
35  #       test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-p${DB_PASS:-your_mysql_root_password}"]
36  #       interval: 5s
37  #       timeout: 3s
38  #       retries: 5
39  #     networks:
40  #       - hugegraph-net
41
42  # If you need Hadoop/HDFS tests for hugegraph-loader, you can add the following services
43  #   namenode:
44  #     image: johannestang/hadoop-namenode:2.0.0-hadoop2.8.5-java8
45  #     container_name: namenode
46  #     ports:
47  #       - "0.0.0.0:9870:9870"
48  #       - "0.0.0.0:8020:8020"
49  #     environment:
50  #       - CLUSTER_NAME=test-cluster
51  #       - HDFS_NAMENODE_USER=root
52  #       - HADOOP_CONF_DIR=/hadoop/etc/hadoop
53  #     volumes:
54  #       - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml
55  #       - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml
56  #       - namenode_data:/hadoop/dfs/name
57  #     command: bash -c "if [ ! -d /hadoop/dfs/name/current ]; then hdfs namenode -format; fi && /entrypoint.sh"
58  #     healthcheck:
59  #       test: ["CMD", "hdfs", "dfsadmin", "-report"]
60  #       interval: 5s
61  #       timeout: 3s
62  #       retries: 5
63  #     networks:
64  #       - hugegraph-net
65
66  #   datanode:
67  #     image: johannestang/hadoop-datanode:2.0.0-hadoop2.8.5-java8
68  #     container_name: datanode
69  #     depends_on:
70  #       - namenode
71  #     environment:
72  #       - CLUSTER_NAME=test-cluster
73  #       - HDFS_DATANODE_USER=root
74  #       - HADOOP_CONF_DIR=/hadoop/etc/hadoop
75  #     volumes:
76  #       - ./config/core-site.xml:/hadoop/etc/hadoop/core-site.xml
77  #       - ./config/hdfs-site.xml:/hadoop/etc/hadoop/hdfs-site.xml
78  #       - datanode_data:/hadoop/dfs/data
79  #     healthcheck:
80  #       test: ["CMD", "hdfs", "dfsadmin", "-report"]
81  #       interval: 5s
82  #       timeout: 3s
83  #       retries: 5
84  #     networks:
85  #       - hugegraph-net
86
87networks:
88  hugegraph-net:
89    driver: bridge
90volumes:
91  namenode_data:
92  datanode_data:

Hadoop Configuration Mounts

Create a ./config folder in the same directory as docker-compose.yml to mount Hadoop configuration files. You can skip this step if HDFS testing is not required.

📁 ./config/core-site.xml content:

1<configuration>
2    <property>
3        <name>fs.defaultFS</name>
4        <value>hdfs://namenode:8020</value>
5    </property>
6</configuration>

📁 ./config/hdfs-site.xml content:

 1<configuration>
 2    <property>
 3        <name>dfs.namenode.name.dir</name>
 4        <value>/hadoop/hdfs/name</value>
 5    </property>
 6    <property>
 7        <name>dfs.datanode.data.dir</name>
 8        <value>/hadoop/hdfs/data</value>
 9    </property>
10    <property>
11        <name>dfs.permissions.superusergroup</name>
12        <value>hadoop</value>
13    </property>
14    <property>
15        <name>dfs.support.append</name>
16        <value>true</value>
17    </property>
18</configuration>

Docker Operations

 1# Start services
 2docker compose up -d
 3
 4# Check status
 5docker compose ps
 6lsof -i:8080  # Server
 7lsof -i:8020  # Hadoop
 8lsof -i:3306  # MySQL
 9
10# Stop services
11docker compose down

4. Run Tests

Test process for each tool:

4.1 hugegraph-client

Compile

1mvn -e compile -pl hugegraph-client -Dmaven.javadoc.skip=true -ntp

Dependent Services

Start HugeGraph Server (refer to Section 3)

Server Authentication Configuration

ApiTest requires authentication. No additional configuration is needed when using the script in Section 3.1. For a manually deployed Server, the authentication settings and test credentials must match the test code.

 1# 1. Modify authentication mode
 2cp conf/rest-server.properties conf/rest-server.properties.backup
 3sed -i 's|#auth.authenticator=.*|auth.authenticator=org.apache.hugegraph.auth.StandardAuthenticator|' conf/rest-server.properties
 4grep auth.authenticator conf/rest-server.properties
 5sed -i 's|gremlin.graph=org.apache.hugegraph.HugeFactory|gremlin.graph=org.apache.hugegraph.auth.HugeFactoryAuthProxy|' conf/graphs/hugegraph.properties
 6
 7# 2. Set password
 8# Note: Test code uses "pa" as default password, must match for tests to work
 9bin/stop-hugegraph.sh
10export PASSWORD="pa"  # Set to test default password
11echo -e "${PASSWORD}" | bin/init-store.sh
12bin/start-hugegraph.sh

Run Tests

1# Check environment
2curl -u admin:pa http://localhost:8080/graphspaces/DEFAULT/graphs
3
4# Run tests
5cd hugegraph-client
6mvn test -Dtest=UnitTestSuite -ntp      # Unit tests
7mvn test -Dtest=ApiTestSuite -ntp       # API tests (requires Server)
8mvn test -Dtest=FuncTestSuite -ntp      # Functional tests (requires Server)

Check Server log if tests fail: logs/hugegraph-server.log

4.2 hugegraph-loader

Compile

1mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp

Dependent Services

• Required: HugeGraph Server
• Optional: Hadoop (HDFS tests), MySQL (JDBC tests); Kafka tests use Testcontainers to start Kafka and also require an available Docker daemon.

Run Tests

1cd hugegraph-loader
2mvn test -P unit -ntp   # Unit tests
3mvn test -P file -ntp   # File tests (requires Server)
4mvn test -P hdfs -ntp   # HDFS tests (requires Server + Hadoop)
5mvn test -P jdbc -ntp   # JDBC tests (requires Server + MySQL)
6mvn test -P kafka -ntp  # Kafka tests (requires Server + Docker daemon; starts a Kafka container)

4.3 hugegraph-hubble

Compile

1mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp
2cd hugegraph-hubble
3mvn -e compile -Dmaven.javadoc.skip=true -ntp

Dependent Services

1. Start Server (refer to Section 3)

2. Python Environment

1python -m venv venv
2source venv/bin/activate  # Windows: venv\Scripts\activate
3python -m pip install -r hubble-dist/assembly/travis/requirements.txt

3. Build and Verify

1mvn package -Dmaven.test.skip=true
2# Optional: Start and verify
3cd apache-hugegraph-hubble*/bin
4./start-hubble.sh -d && sleep 10
5curl http://localhost:8088/actuator/health
6./stop-hubble.sh

Run Tests

 1# Unit tests
 2mvn test -P unit-test -pl hugegraph-hubble/hubble-be -ntp
 3
 4# Legacy Python API tests (requires Server; the script installs and starts Hubble)
 5curl -u admin:pa http://localhost:8080/graphspaces/DEFAULT/graphs  # Check Server
 6cd hugegraph-hubble
 7./hubble-dist/assembly/travis/run-api-test.sh
 8
 9# Full verification entry point used by current CI (build the Hubble tarball first)
10HUBBLE_TARBALL="$(ls target/apache-hugegraph-hubble-*.tar.gz | head -n 1)"
11hubble-dist/assembly/travis/verify-hubble-issue-694.sh \
12  "$HUBBLE_TARBALL" http://127.0.0.1:8080

4.4 hugegraph-spark-connector

Compile

1mvn install -pl hugegraph-client,hugegraph-spark-connector -am -Dmaven.javadoc.skip=true -DskipTests -ntp

Run Tests

1cd hugegraph-spark-connector
2mvn test -ntp  # Requires Server running

4.5 hugegraph-tools

Compile

1mvn install -pl hugegraph-client,hugegraph-tools -am -Dmaven.javadoc.skip=true -DskipTests -ntp

Run Tests

1cd hugegraph-tools
2mvn test -Dtest=FuncTestSuite -ntp  # Requires Server running

5. Common Issues

Service Connection Issues

If Server, MySQL, or Hadoop cannot be reached:

• Confirm services are running (Server must be on port 8080)
• Check port usage: lsof -i:8080
• Docker check: docker compose ps and docker compose logs

Configuration Issues

If files cannot be found or parameters are invalid:

• Check environment variables: echo $COMMIT_ID
• Script permissions: chmod +x hugegraph-*/assembly/travis/*.sh

HDFS Test Failures

• Confirm NameNode/DataNode running normally
• Check Hadoop logs
• Verify HDFS connection: hdfs dfsadmin -report

JDBC Test Failures

• Confirm MySQL running normally
• Verify database connection: mysql -u root -p$DB_PASS
• Check MySQL logs

6. References

• HugeGraph GitHub Repository: https://github.com/apache/hugegraph
• HugeGraph Toolchain GitHub Repository: https://github.com/apache/hugegraph-toolchain
• HugeGraph Server Documentation: https://hugegraph.apache.org/docs/quickstart/hugegraph/hugegraph-server/
• CI Configuration: .github/workflows/*-ci.yml
• Dependency Installation Scripts: hugegraph-*/assembly/travis/

2 - ToplingDB development integration

Development interface: This guide applies to development builds with the optional ToplingDB runtime enabled. Existing releases and image tags do not establish support, and runtime acceptance remains a deployment prerequisite.

Choose the database owner

HugeGraph defaults to standard RocksDB. Enable Topling separately for each process that owns local storage:

A standalone Server keeps backend=rocksdb; an HStore-backed Server uses backend=hstore and needs no local Topling runtime. All local RocksDB graphs in one Server process must use the same provider. The business setting must match the runtime selected by its launcher.

Build and prepare a standard distribution

Build the normal distributions from the source repository root using Java 17 and Maven 3.6.3 or later:

1mvn clean package -Dmaven.test.skip=true -Dmaven.javadoc.skip=true

Obtain a trusted Topling Easy Migrate JNI JAR and its SHA-256 through your artifact channel. Preparation requires a Java 17 JDK, Linux x86_64, sha256sum, unzip, od, ldd, and GNU mv. The selected JAR’s native dependencies, including its glibc and libaio requirements, must be installed.

Stop the component, unpack a fresh standard distribution, and run from that component directory:

1export TOPLING_JNI_JAR=/absolute/path/to/rocksdbjni-topling.jar
2export TOPLING_JNI_SHA256='<trusted-64-character-sha256>'
3bash bin/prepare-topling.sh

The script verifies the copied JAR’s hash, Topling Java marker, one Linux x86_64 JNI entry, ELF architecture and native dependencies. Before installation, a disposable database verifies that the selected JNI actually applies EasyMigrate configuration to its persisted write-buffer size. Unsupported builds are rejected even when their marker and hash match. The probe does not certify existing-data compatibility or every production setting. A checksum receipt binds this verification to the installed JAR and native library; launchers reject missing or changed receipts. Use a fresh distribution when upgrading from an older preparation script. Launch rejects inherited RocksDB JNI preloads in default/standard mode and competing JNI in Topling mode, while preserving unrelated preloads. It installs topling/rocksdbjni.jar, its native library and optional web resources outside lib. Preparation refuses an existing topling directory; replace a runtime by preparing another stopped, fresh distribution. Repeat preparation separately for Server, PD and Store as applicable.

Keep data outside the distribution

Changing providers does not convert an existing database. Use separate, initially empty Topling directories and retain the standard-provider data. Create persistent directories writable by the service account, outside source checkouts, build outputs and unpacked distribution directories. Note that each graph requires distinct, dedicated data and WAL directories to prevent path conflicts:

For a standalone Server, edit every selected graph configuration, for example conf/graphs/hugegraph.properties (replace hugegraph with the corresponding graph name for each graph):

1backend=rocksdb
2rocksdb.provider=topling
3rocksdb.data_path=/srv/hugegraph/topling/server/hugegraph/data
4rocksdb.wal_path=/srv/hugegraph/topling/server/hugegraph/wal

Set both Server data and WAL paths, ensuring each graph uses distinct directories. Leaving WAL at its package-local default risks losing it when a build or package replacement removes that directory. Keep the external data, WAL and Raft roots across runtime/package changes.

For PD and Store, edit the existing rocksdb mapping in the provider configuration file listed above, retaining its other options:

1rocksdb:
2  provider: topling

Set their persistent paths separately in conf/application.yml; keep the normal network addresses and cluster configuration.

The published runtime directories use 0755 and regular files use 0644, so a service account different from the preparer can read the JNI. The service account also needs traversal permission on the component’s parent directories; preparation does not change those parents.

Select, start and verify

In the startup shell of each component that owns a Topling database, explicitly select the runtime:

1export TOPLINGDB_ROCKSDB_PROVIDER=topling
2# Run this component's start command from the first table.

The launcher uses that component’s conf/toplingdb.yaml. Set TOPLINGDB_EASY_MIGRATE_CONF=/absolute/path/config.yaml for an explicit override. The shipped native profile disables its HTTP server and keeps memtable_as_log_index=false, which the Java write path requires. Tune that profile for the host; it is separate from the business provider and data-path configuration.

For a fresh standalone Server graph, initialize its store once before the first startup:

1bash bin/init-store.sh
2bash bin/start-hugegraph.sh

bin/init-store.sh and bin/dump-store.sh use the same explicit runtime selection as bin/start-hugegraph.sh. For HStore, start PD, then Store. Before starting the HStore-backed Server from the same shell, run unset TOPLINGDB_ROCKSDB_PROVIDER (or export TOPLINGDB_ROCKSDB_PROVIDER=rocksdb); do not enable local Topling on that Server. A configured provider/runtime mismatch fails before the component opens its database.

Before adoption, verify the actual RocksDB Java classes and mapped JNI library belong to the prepared runtime. Use dedicated data directories to test real writes, reads, normal stop and restart; for HStore, include a Server graph operation through PD and Store. A successful build, startup or standard-JNI unit test alone does not establish Topling runtime acceptance.

Stop or switch back

Stop incoming work, then use the normal stop scripts. For HStore, stop Server with bin/stop-hugegraph.sh, Store with bin/stop-hugegraph-store.sh, and PD with bin/stop-hugegraph-pd.sh, waiting for each process to exit before stopping its dependency. A forced kill is not proof of normal native cleanup.

To use standard RocksDB again, stop the component, unset TOPLINGDB_ROCKSDB_PROVIDER, and restore its business provider to rocksdb. Select the corresponding standard-provider data and WAL paths; do not point it at a database modified by Topling.

Dedicated Topling distributions, Docker/Compose packaging, general lifecycle changes and retryable snapshot recovery are separate follow-ups. This guide covers the normal distribution launch scripts; custom IDE and embedded launchers require their own runtime setup and validation.