HugeGraph ToolChain

• 1: Graph visualization
  • 1.1: Manage an HStore Cluster with Hubble
• 2: Graph Visualization with Hubble: Standalone Quick Start
• 3: Graph import
  • 3.1: Import Graph Data with SeaTunnel Sink
• 4: HugeGraph-Loader Quick Start
• 5: Graph export and migration
  • 5.1: Export and Migrate Graph Data with SeaTunnel Source
• 6: Tools Quick Start
• 7: HugeGraph-Spark-Connector Quick Start

1 - Graph visualization

Hubble provides a Web interface for HugeGraph data, schema, queries, and imports. Start with a standalone deployment without PD, then read the distributed differences when needed.

• Hubble basics with standalone RocksDB: startup, sample graphs, modeling, imports, and queries.
• Hubble with an HStore cluster: PD discovery, GraphSpaces, and cluster operations.

1.1 - Manage an HStore Cluster with Hubble

This guide highlights the key differences between HStore + PD distributed deployments and standalone RocksDB. For general graph modeling, data import, and query workflows, see the Hubble standalone guide. Content aligns with the Toolchain, Server, PD, and Store master branches (all currently 1.8.0).

The main repository’s docker/docker-compose-hstore.yml provides complete orchestration for PD, Store, Server, and Hubble. Follow the adjacent Docker README to prepare .env and generate the local Hubble configuration, then launch services from docker/ without maintaining separate Compose files. The guidance below focuses on connectivity and operations; ensure prerequisite credentials and service readiness from the README are met. Docker latest is a moving tag. Screenshots assist in UI navigation; verify deployed versions via image digests and build details. Metrics and permission APIs vary by version.

The screenshots show the HugeGraph Hubble interface in action; features and workflows are identical in both English and Chinese.

Connect to a distributed cluster

Hubble still manages graph data through the Server graph API. In distributed mode, it discovers Servers through PD and collects cluster information from PD and Store. Hubble does not read or write graph data directly in Store.

flowchart LR
    H[Hubble] -->|Graph API| S[Server]
    S -->|Graph reads and writes| T[Store]
    H -.->|Discovery and cluster information| P[PD]
    H -.->|Node metrics| T

1. Generate Hubble configuration. Prepare .env in the main repository’s docker/ directory following the Docker README.

   1set -a
   2. ./.env
   3set +a
   4./set-hubble-pd-password.sh hstore

   The script reads HG_PD_AUTH_SECRET_KEY from .env and writes its value as operations.pd.password into the generated file docker/conf/hubble/hstore.local.properties. Edit this generated file to adjust connection or operations settings (do not overwrite the tracked .example template), ensuring operations.pd.password matches the PD secret. Compose mounts this file read-only into the container; do not edit inside the container. Both .env and the generated file contain secrets and should not be committed to Git. Re-running the script overwrites the generated file, requiring custom settings to be reapplied.

2. Start the services. Use the same docker/ directory and confirm Server registration and Store readiness.

   1docker compose -f docker-compose-hstore.yml up -d --wait

For separately deployed services, follow the PD deployment guide and HStore deployment guide. A source or binary Hubble deployment instead uses the package’s conf/hugegraph-hubble.properties. These settings match the official minimal topology; use backend-reachable addresses for other deployments:

1pd.enabled=true
2cluster=hg
3pd.peers=pd:8686
4pd.server=pd:8620

Distinguish between the two ports: 8686 is for gRPC and 8620 is for REST. Do not interchange them, and use container service names or reachable IPs rather than 127.0.0.1 for cross-container traffic. The bundled package sets pd.enabled=false. If omitted, the fallback default is true, so declare this setting explicitly in both standalone and distributed setups. After editing configuration: restart the Hubble process for source or binary deployments; for Docker Compose, recreate the Hubble container from docker/ to reload the mounted configuration (retaining the project name and all -f arguments):

1docker compose -f docker-compose-hstore.yml up -d --force-recreate hubble

In the Hubble UI, you do not need to configure a Server host and port for each graph.

Organize graphs and permissions with GraphSpaces

A GraphSpace groups graphs, Schema templates, and access permissions. For example, create sales and research spaces so each team can work with its own graphs. Modeling, importing, and querying within a space follow the standalone guide.

Create and adjust a GraphSpace

With Server authentication enabled, creating, editing, and deleting GraphSpaces require super administrator access. Start with a globally unique GraphSpace name, an optional display alias, and the maximum graph count. For example, use research as the API identifier and “Research_graph” as the display alias. The name cannot change after creation. The alias and description can change; aliases do not participate in URLs or permission matching. Choose GraphSpace administrators from existing accounts.

“Advanced deployment and resource limits” includes CPU and memory limits for graph query/write services and asynchronous compute tasks, plus the storage capacity limit. These deployment quotas neither report current usage nor resize existing containers.

Select a space before opening a graph. Check the current graph after switching spaces, especially when spaces contain identically named graphs. The list reflects account access. If a space is missing, check membership permissions before creating more graphs.

Reuse Schema templates

User-defined Schema templates require PD mode and persist reusable Groovy Schema within a GraphSpace. When several business graphs share vertex labels, edge labels, and indexes, save the model as a template and select it when creating subsequent graphs. For example, a user template in research can be reused for new graphs in that space. After switching spaces, select a template belonging to the new space.

User templates differ from the built-in sample templates in the main guide: built-in templates help explore preset models, while user templates preserve your own models for future use. A template is not an import of data; loading sample data is a separate graph creation choice. With Server authentication enabled, creating a user template requires write access to its space. Updating or deleting it also requires ownership or the corresponding administrative permission. Anonymous mode does not enforce per-user permissions or template ownership checks. Standalone mode does not provide user template management.

Assign access to a GraphSpace

For account creation, login, and personal details, see the standalone guide. Distributed mode adds “Manage GraphSpace members” to assign permissions between existing accounts and selected spaces. Servers supporting permission presets provide these common choices:

An account may have different access in different spaces, such as read-write in research and read-only in sales. Accounts and space memberships are managed separately; removing a member does not delete the global account. After creating an account, continue directly to assigning space access. When changing an existing membership, check the selected space and preset and preserve permissions still needed in other spaces. Enabling pd.enabled grants no permissions; legacy custom roles on older Servers are not interchangeable with these presets. See Server authentication and authorization.

Locate problems through the cluster overview

The overview presents Server → PD → Store in one topology. Switch to the node list to filter by type, status, or name. PD Leader, online Store count, graphs, partitions, replicas, and data size help identify cluster scale and nodes requiring attention. Partitions and replicas describe data distribution. Data size is observed usage, distinct from a GraphSpace’s configured storage limit.

Start with cluster and source status, then inspect an affected node. UP indicates a successful collection from the source; DEGRADED indicates a cluster or partial source issue, and DOWN indicates that the corresponding probe failed. Graphs may remain queryable when topology is available but some metrics are missing. The UI distinguishes unsupported, unavailable, stale, and failed collections and includes observation times. An empty value is not zero, and an older value is not a current observation.

Read node details

• 

  Start with cluster health and affected sources

• 

  Inspect the affected Store node

The PD Leader and Store Leader partitions serve different purposes: PD coordinates cluster metadata, while a Store leads the respective data partition’s Raft group. Hubble displays upstream observations and does not replace a full Raft replica consistency check. Available metrics vary by component version.

With Server authentication enabled, cluster operations require a super administrator (ADMIN level). GraphSpace administrators and ordinary members do not inherit cluster access. Use container isolation, a trusted HTTPS entrypoint, Server authentication, and network allowlists; avoid publishing Hubble or component ports directly.

Configure operations access

Hubble’s backend uses the PD/Store operations credentials, separately from the Server account used to log in through the browser. For Compose, edit the generated host file docker/conf/hubble/hstore.local.properties; other deployments use the Hubble package configuration. The script-generated PD password must match the secret in .env. Configure the Store service account when Store authentication is enabled. Do not include passwords in documentation, screenshots, or committed configuration:

For example, when a containerized Store advertises store:8520, set:

1operations.store.allowed_targets=[http://store:8520]

For multi-node clusters, list every trusted target origin. Each entry must use http or https with an explicit port and no path, credentials, or wildcard, matching the Store metric target returned by PD. Adding an origin here only authorizes access—it does not register or discover nodes.

If topology renders but node metrics are missing, check Hubble backend connectivity and credentials to PD REST, verify the Store REST and metric addresses returned by PD, and ensure targets are in operations.store.allowed_targets. A partially available overview indicates metrics collection was blocked for specific sources, not that graph APIs are down.

2 - Graph Visualization with Hubble: Standalone Quick Start

Hubble is the HugeGraph Web management and graph visualization interface. Use a single workspace to manage schema, import data, run queries, and switch smoothly between graph, table, and JSON results. This guide uses standalone RocksDB Server + Hubble, without requiring PD or Store.

For use with an HStore distributed cluster, read the foundational operations here first, then consult the distributed supplement. This guide aligns with the Toolchain and Server master branches (currently based on 1.8, unreleased). Docker latest is a moving tag; use released versions for production deployments.

The screenshots show the HugeGraph Hubble interface in action; features and workflows are identical in both English and Chinese.

Jump directly to a task below. If this is your first visit, start with the standalone deployment:

• Start and load a sample

  Start the two services and load the People & Software Demo Graph.

• Query and visualize

  Run Gremlin and inspect graph, table, or JSON results.

• Import your own data

  Prepare CSV, map fields, and verify the imported records.

Start the standalone pair

Use the main repository’s docker/docker-compose.yml instead of writing custom Compose files. It already integrates RocksDB Server and Hubble with networking, health checks, and data volumes; see the adjacent README for deployment details.

1. Prepare the official Compose configuration.

   1git clone --branch master --single-branch --depth 1 https://github.com/apache/hugegraph.git
   2cd hugegraph/docker

   If you already have the repository locally, navigate directly to its docker/ directory. Compose automatically mounts conf/hubble/standalone.properties, which pre-configures pd.enabled=false and server.direct_url=http://server:8080. Because both services communicate over a shared Docker network, you do not need to configure Server endpoints per graph. Start Compose directly from docker/ so relative file paths resolve correctly.

   Hubble publishes on host loopback port 8088 by default (controlled by HUBBLE_PUBLISH_HOST, defaulting to 127.0.0.1), whereas Server publishes 8080 across all interfaces. For a local trial, set Server’s ports in Compose to 127.0.0.1:8080:8080 to prevent exposing unauthenticated ports beyond loopback (set HUBBLE_PUBLISH_HOST=0.0.0.0 if remote access to Hubble is required).

2. Start and check the services.

   Choose an unused project name for this trial and keep these variables in the same terminal session:

   1export HUGEGRAPH_VERSION=latest
   2export HUBBLE_IMAGE=hugegraph/hubble:latest
   3export HUBBLE_DEMO_PROJECT="hubble-demo-$(date +%Y%m%d-%H%M%S)"
   4docker compose ls
   5docker compose -p "$HUBBLE_DEMO_PROJECT" -f docker-compose.yml pull
   6docker compose -p "$HUBBLE_DEMO_PROJECT" -f docker-compose.yml up -d --wait
   7docker compose -p "$HUBBLE_DEMO_PROJECT" -f docker-compose.yml ps
   8curl -fsS http://127.0.0.1:8080/versions

3. Open Hubble.

   Once services are healthy, open http://127.0.0.1:8088. When HUGEGRAPH_ADMIN_PASSWORD is unset in .env, Server permits anonymous access and Hubble opens the homepage directly. If authentication is enabled, configure an admin password and JWT secret in .env as described in the Docker README, then sign in with a Server account (Hubble has no independent user database). Profile and account management options appear only when authentication is enabled and the account holds requisite permissions.

Use latest to explore recent features; pin a released image version or digest for production. Official release archives are on the download page.

Compose’s server-data and hubble-data retain graph data and Hubble metadata respectively. For further persistence and production settings, see the Server deployment guide.

Home: find the right starting point

Home groups the workspace into graph overview, data preparation, and graph queries. Use it to understand the workflow, then return to any section through the sidebar. The top graph selector determines the target of queries, schema operations, and async tasks; check the current graph after changing pages. Standalone mode has only the DEFAULT GraphSpace and needs no PD configuration.

Interface at a glance

Compare the graph overview, schema, and query workspaces below. Click a screenshot to view the original.

• 

  Graph overview: select a graph and inspect sample data

• 

  Schema: inspect vertex labels, primary keys, and ID strategies

• 

  Graph query: run Gremlin and explore graph results

Graph Overview and details: get to know a graph

Select the default graph hugegraph in Graph Overview. Click the graph card or details to view its schema structure and data statistics, or quickly jump to modeling, data import, or query pages.

Load the People & Software Demo Graph from the graph card’s More actions menu. The sample automatically provisions the required schema and imports sample records without clearing existing data. For your initial trial, load it into an empty graph to avoid name collisions. The remaining examples use this graph to explore connections between people and software.

To create a new graph, click the create button on the overview page and provide a graph name, optional alias, and schema template or sample data. Connection details are managed globally by the Hubble backend, so you do not re-enter Server host or credentials per graph. User-defined schema templates require PD mode; see the distributed supplement.

Schema modeling: define the shape of your data

Schema defines allowable properties, vertex labels, and edge labels, while determining vertex ID strategies and query index rules. Under Schema configuration, use the list view to manage definitions or the graph view to inspect vertex-edge topology visually.

In the person/software sample: person derives its ID from the primary key name, with nullable age and city; software uses custom numeric IDs; the created edge label links people to software. This matches the Loader example.

When modeling your own graph, create elements in order: Properties → Vertex labels → Edge labels → Indexes. Deleting labels or creating/rebuilding indexes typically submits background asynchronous tasks; verify their completion in Async Tasks.

GQL workspace: query and explore relationships

Open GQL Traversal and confirm hugegraph is selected. The workspace puts the query editor and graph display together, with immediate or async execution, query favorites, and reusable execution history. Start with this Gremlin query:

1g.V().hasLabel('person').valueMap()

Inspect person properties in table or JSON view. To visualize the people-to-software relationships, run:

1g.V().hasLabel('person').outE('created').inV().path()

Graph results support 2D / 3D. Click a vertex or edge to inspect its ID, label, and properties; double-click a vertex to expand its neighbors. Layout, styling, and filtering help highlight relevant relationships, while export helps share results. Use New to create elements, or edit existing data when authorized. Layout adjustments, styling, and display bounds only affect the local canvas; adding or editing elements via the UI or executing Gremlin write statements modifies underlying Server graph data.

Press Ctrl / Command + Enter to execute queries quickly. Immediate execution is ideal for interactive exploration; submit long or resource-intensive queries as background async tasks. Cypher is available when supported by the backend Server. The keyboard shortcut acts on whichever input currently has focus: generating a query from the natural language prompt, or running statements in the code editor.

Text2GQL natural-language queries

Describe your query requirement in the Natural language input. Text2GQL combines the current graph schema to generate read-only Gremlin or Cypher and execute it automatically. Generated statements can be edited freely (read-only safety checks still apply before execution). In Async mode, both generated and handwritten queries can be submitted as background tasks. Inputs automatically restore the graph’s most recent query prompt without running it, and the execution history retains original natural-language questions.

• 

  Unconfigured model; handwritten query

• 

  Model settings

• 

  Text2GQL Prompt

The screenshots show an unconfigured model. The graph results come from handwritten Gremlin; connect a working model before generating natural-language queries.

Choose OpenAI compatible for OpenAI, DeepSeek, GLM, and similar services, or Ollama (local) for local Ollama; unauthenticated Ollama needs no Key. The Base URL must be reachable from Hubble’s backend; container localhost refers to that container. Public endpoints require HTTPS, while plain HTTP is accepted only for loopback or private-network models; when changing the Base URL in Model settings, supply a new Key or explicitly clear the saved Key. Administrators with model-settings management permission manage shared settings. When Server authentication is disabled, anyone who can access the deployment can manage them.

Under model settings, click Text2GQL Prompt in the header to customize Gremlin and Cypher generation rules and few-shot examples. Click Apply and then save model settings to persist templates to the H2 database for both built-in and temporary models. The current graph schema is appended automatically during generation.

Input footers show model generation time and query request time separately. Async submission time excludes background execution.

Built-in algorithms: explore with parameter forms

Use Built-in Algorithms when you prefer a form to writing traversal code. Search for an algorithm and supply its parameters. Neighbor exploration answers what surrounds a vertex, path algorithms connect two vertices, and similarity/ranking algorithms compare or select vertices. Start with a known vertex ID and limit direction, edge labels, depth, and result size before attempting broader computation.

Forms provide parameter guidance and documentation links, and restore common parameters when navigating away and back. Results use graph or algorithm-specific panels. Consult the help and documentation links beside the algorithm title for definitions and parameters. While the parameter form is focused, Ctrl / Command + Enter runs the current algorithm.

OLAP batch algorithms require external compute services such as Computer or Vermeer. The two containers in this example provide online graph operations, not those compute services.

For example, select K-neighbor (GET) with source=1:marko, max_depth=1, and limit=20. After parameter validation, use the run button on the card or the form shortcut to explore one-hop neighbors.

Async Tasks: confirm background results

Async Tasks lists background work for the current graph, including async queries and some schema/index operations. Filter by task type and status, inspect IDs, creation times, and execution states, open successful query results, or expand failure information. Completed task records can be deleted where the interface permits. These tasks are separate from import execution history.

For example, submit g.V().count() asynchronously, confirm success in the list, then inspect the returned count. A successful submission means the request was accepted, not that computation or indexing has finished. Use task errors and Server logs together when diagnosing failures.

Data Source Management: prepare the input

A data source defines where data comes from and how to parse it, and can be referenced by import tasks. Hubble supports FILE, HDFS, JDBC, and Kafka, each with its own path, connection, or subscription settings. FILE is an easy starting point: upload a file, configure its format, delimiter, encoding, and header, and check column names before mapping. Hubble configuration controls upload limits and permitted extensions.

Save this UTF-8 file as people.csv for the person example:

1name,age,city
2docs_alice,28,Beijing
3docs_bob,32,Shanghai

Create a FILE data source and upload it. Select CSV (comma separation and UTF-8 by default), with column names name,age,city. Header, delimiter, and encoding belong to the data source, not mapping settings. The source fields must match the file.

Data Import: turn fields into a queryable graph

Data Import converts source rows into vertices and edges. Its four configuration sections identify the target, select fields, map them to the graph, and choose execution timing. Use the preceding data source to create a person import into DEFAULT / hugegraph:

1. Basic Information Select the target graph, new data source, and a task name.
2. Source Fields Select name, age, and city, moving them to the selected field list.
3. Mapping Fields Add a person vertex mapping; use Auto Match for same-name properties, then verify types.
4. Schedule Choose one-time execution and confirm to submit the task.

person uses PRIMARY_KEY, so do not select a separate ID column. Custom ID strategies require an ID column; AUTOMATIC lets Server generate IDs, while PRIMARY_KEY derives them from mapped primary-key properties. Edge mappings need source/target fields that follow the corresponding vertex ID rules.

The task list manages configuration and execution entry points; execution history in task details shows each instance’s state, count, and errors. Periodic schedules and real-time Kafka tasks are also available; choose an execution mode compatible with the source. After completion, verify the result in the GQL workspace:

1g.V().hasLabel('person').has('name', within('docs_alice', 'docs_bob')).valueMap()

The result should include both new people. Import counts are not necessarily counts of newly created vertices: reruns may update existing elements, and header processing can affect reader counts. If the import fails, check source fields, numeric types, nullable properties, and target schema. Use Hubble for small trials and HugeGraph Loader for production bulk ingestion.

Profile and account permissions

Hubble uses Server authentication and accounts, with no separate user database. Anonymous mode hides Profile and Account Management. After authentication is enabled, Profile shows the current account’s details and permissions and allows changing your password. Editing details such as a nickname requires Server support for the personal-profile API. Changing a password ends the current session; sign in again with the new password.

Standalone account management

In standalone mode, the administrator can create, inspect, edit, delete, and batch-create accounts. Ordinary standalone accounts created through Hubble receive read, write, delete, and execute permissions across all graphs; they are not read-only or isolated to one graph. Configure finer resource permissions through Server authentication and authorization rather than relying on GraphSpace presets.

GraphSpace permissions in PD mode

With a Server supporting default-role APIs, global accounts and space access can be managed separately. The administrator manages global accounts; a space administrator manages members only within authorized spaces. Ordinary members do not receive account-management or operations entry points. The following presets are for PD mode, and are not universally editable on older or standalone Servers:

An account may have different permissions in different spaces. Select a space before adding an existing account or changing member permissions. Before replacing custom permissions with a preset, inspect the grants that need to be retained; complex permissions may not match a single preset. After changes, refresh permission context or sign in again and verify menus and space selection. Server still validates every request. Older Servers may hide or disable unsupported operations; a visible button alone does not establish resource authorization. See the distributed supplement for space management.

Operations: inspect the standalone Server

Standalone Operations provides Node Information for Server only, with no PD/Store nodes or cluster overview. Search nodes, filter health status, and open node details to inspect available version, system, JVM, and Server-backend metrics. When metrics are unavailable or stale, use collection state and the last successful observation time; a missing value is neither zero nor proof of health.

Anonymous mode can read operations information. With authentication enabled, operations are available only to administrators with the capability. This is an observation and diagnosis interface, not a start/stop or scaling console. The distributed supplement covers cluster overview and the PD/Store node hierarchy.

Keyboard shortcuts and graph interactions

Use the topbar shortcut-help button to see key bindings. Their scope differs: typing ? in an input does not trigger global help.

Diagnose connection and result issues

Configuration can affect displayed query size. gremlin.suffix_limit defaults to 250 and supplies .limit(N) appended to applicable Gremlin queries; it is not a universal hard limit. gremlin.vertex_degree_limit (100) and gremlin.edges_total_limit (500) constrain expansion. FILE uploads allow csv,txt by default, with 1 GB per file and 10 GB total. Override upload_file.* settings when needed.

Stop the trial or build from source

When finished, run this in the main repository’s docker/ directory:

1docker compose -p "${HUBBLE_DEMO_PROJECT:?}" -f docker-compose.yml down --volumes

This removes the project’s containers, network, named volumes, and anonymous volume, losing the sample data and Hubble import tasks. To retain data, omit --volumes when taking the deployment down and reuse the same project name when starting it again.

Building current master requires JDK 17, Maven 3.9+, Python 3.9+, and stable Node.js 24 (at least 24.22.0), with pnpm 12.14.0 selected through Corepack. Maven no longer installs Node/Yarn automatically.

1. Follow the Java 17 migration instructions to install the Server SDK in an isolated Maven repository.

2. Follow the Hubble source-build procedure to install Python dependencies and build Client, Loader, and Hubble with that same repository (skipping tests).

3. Open the resulting hugegraph-hubble/apache-hugegraph-hubble-* directory, configure Server connection settings in conf/hugegraph-hubble.properties, then start:

   1bin/start-hubble.sh -d

Current master uses a new H2 metadata database. Retain the old Hubble distribution and its data, and use a separate data directory for the new version. Old metadata has no automatic migration and cannot be reused directly.

The bundled configuration binds to localhost:8088. bin/stop-hubble.sh requests graceful shutdown before forcing termination on timeout. For development and testing, see the Toolchain local test guide.

3 - Graph import

Choose an import tool when you need to write file, database, or message data into HugeGraph. Use Loader for a direct import, SeaTunnel Sink when an existing Source, Transform, and Sink pipeline should be reused, or Spark Connector from a Spark job.

• Import data with Loader
• Import graph data with SeaTunnel Sink
• Read and write graph data with Spark Connector

3.1 - Import Graph Data with SeaTunnel Sink

SeaTunnel connects data sources such as databases and Kafka to HugeGraph. The connector has two parts: Source reads data and Sink writes data^[1][2], with SeaTunnel transform components available between them. To export or migrate data from HugeGraph, see the SeaTunnel Source export and migration guide.

Version requirement: This guide targets SeaTunnel 3.0+. All examples use the mappings configuration available in SeaTunnel 3.0+.

Click a diagram to view the original size.

1 Loader, Tools, and SeaTunnel

HugeGraph-Loader is suited to direct imports from common data sources. HugeGraph-Tools focuses on standalone graph management, backup, and export. SeaTunnel organizes a job as Source → Transform → Sink, so you can reuse existing connectors, transforms, and data pipelines.

Table legend

✅ Supported natively; ⚠️ conditional support or requires an extra component/external platform; ❌ not provided

SeaTunnel covers Loader’s graph-import and Tools’ export and migration scenarios in one expandable pipeline, and it also supports SQL-CDC and dozens of input and output types. Loader and Tools normally run on one machine, while SeaTunnel supports both standalone and distributed deployments and scales with data and task volume. Tools’ schedule-backup can create a crontab entry, but it does not provide unified workflow orchestration and resource management.

Existing Spark/Flink daily jobs

Both HugeGraph Source and Sink list SeaTunnel Engine (Zeta), Spark, and Flink as supported engines in SeaTunnel 3.0+. If you express the daily job as a SeaTunnel job and submit it to that engine, records can move directly from Source to Transform to Sink without an intermediate file. If you keep the existing Spark/Flink DAG, SeaTunnel does not automatically take over its in-memory DataFrame or stream. Adapt it into a SeaTunnel job or expose the data through a Source connector

Loader and Tools are focused, direct, and quick to start. Use Loader for a direct graph import; use Tools for backup, restore, export, or daily operations. If a SeaTunnel job already exists, adding HugeGraph to that pipeline is usually simpler. For higher import throughput, Loader’s bypass-server path and other import optimizations are a better fit; measured peaks of 1-2 million records/s require a specific backend, data set, and hardware configuration and are not a general performance guarantee. For new SeaTunnel jobs, use 3.0+ and mappings. Recheck the connector configuration when using another version.

2 Prepare the environment

2.1 Get SeaTunnel 3.0+

Use JDK 11 and set JAVA_HOME; the official deployment guide lists Java 8 and 11 as prerequisites. Choose a 3.0+ binary distribution from the official release page^[4] and verify its checksum or signature using the files linked there. The commands below use 3.0.0 as an example; set SEATUNNEL_VERSION to the release you downloaded.

1export SEATUNNEL_VERSION=3.0.0
2tar -xzf "apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz"
3cd "apache-seatunnel-${SEATUNNEL_VERSION}"

The binary distribution does not include connector plugins. In config/plugin_config, select the connectors needed by the examples:

1--seatunnel-connectors--
2connector-hugegraph
3connector-jdbc
4connector-kafka
5--end--

Install the matching released plugins using the upstream plugin installation instructions^[5]:

1sh bin/install-plugin.sh "${SEATUNNEL_VERSION}"

Run the remaining commands from this installation directory and keep the engine and connector plugins at the same version. This guide uses the bundled Zeta engine in local mode^[6][7]. Check that connectors/ contains HugeGraph, JDBC, and Kafka as needed^[11][12]. The JDBC examples also require the MySQL driver JAR in lib/, with driver class com.mysql.cj.jdbc.Driver.

2.2 Prepare HugeGraph and data sources

Start HugeGraph Server and create a graph for testing. The examples use the hugegraph graph in the DEFAULT graph space. Adjust these names to match the server configuration; graph space names are case-sensitive. If authentication is enabled, provide username and password in the HugeGraph Source and Sink configurations.

The following graph model is shared by the JDBC and Kafka examples. mappings creates missing PropertyKey, VertexLabel, and EdgeLabel definitions by default; existing schema definitions must be compatible.

The mysql, kafka, and hugegraph host names in the examples are placeholders. Replace them with addresses reachable from the SeaTunnel runtime. Inside a container, 127.0.0.1 points to that container; services on the same Docker network can use their service names. Set host to a host name or IP address, and set the port separately.

3 Import from a relational database (sql2graph)

Use two jobs for this import: write the person table as vertices first, then write the knows table as edges. Both edge endpoints will already exist when the edge job runs.

3.1 Import vertices

Prepare the sample data in the MySQL demo database and grant the configured account read access:

1CREATE TABLE person (
2  name VARCHAR(64) PRIMARY KEY,
3  age INT NOT NULL
4);
5INSERT INTO person VALUES ('marko', 29), ('vadas', 27);

Save the following as config/sql2graph-person.conf and replace the database user name and password:

 1env {
 2  job.mode = "BATCH"
 3}
 4
 5source {
 6  Jdbc {
 7    url = "jdbc:mysql://mysql:3306/demo?useSSL=false&serverTimezone=UTC"
 8    driver = "com.mysql.cj.jdbc.Driver"
 9    username = "seatunnel"
10    password = "change_me"
11    query = "SELECT name, age FROM person ORDER BY name"
12  }
13}
14
15sink {
16  HugeGraph {
17    host = "hugegraph"
18    port = 8080
19    graph_name = "hugegraph"
20    graph_space = "DEFAULT"
21    batch_failure_fallback = false
22    mappings = [
23      {
24        type = "VERTEX"
25        label = "person"
26        idStrategy = "PRIMARY_KEY"
27        idFields = ["name"]
28        properties = ["name", "age"]
29      }
30    ]
31  }
32}

1./bin/seatunnel.sh --config ./config/sql2graph-person.conf -m local

Check the result in Hubble or Gremlin. You should find marko and vadas with their ages:

1g.V().hasLabel('person').valueMap('name', 'age')

idFields = ["name"] uses the name to generate the primary key. Importing the same name again writes to the same vertex. properties lists the source fields to write.

3.2 Import edges

Prepare the relation table. Its two endpoint fields correspond to person.name from the vertex job:

1CREATE TABLE knows (
2  source_name VARCHAR(64) NOT NULL,
3  target_name VARCHAR(64) NOT NULL,
4  since INT NOT NULL
5);
6INSERT INTO knows VALUES ('marko', 'vadas', 2010);

 1env {
 2  job.mode = "BATCH"
 3}
 4
 5source {
 6  Jdbc {
 7    url = "jdbc:mysql://mysql:3306/demo?useSSL=false&serverTimezone=UTC"
 8    driver = "com.mysql.cj.jdbc.Driver"
 9    username = "seatunnel"
10    password = "change_me"
11    query = "SELECT source_name, target_name, since FROM knows ORDER BY source_name, target_name"
12  }
13}
14
15sink {
16  HugeGraph {
17    host = "hugegraph"
18    port = 8080
19    graph_name = "hugegraph"
20    graph_space = "DEFAULT"
21    batch_failure_fallback = false
22    check_vertex = true
23    mappings = [
24      {
25        type = "EDGE"
26        label = "knows"
27        sourceConfig = {
28          label = "person"
29          idFields = ["source_name"]
30        }
31        targetConfig = {
32          label = "person"
33          idFields = ["target_name"]
34        }
35        fieldMapping = {
36          source_name = "name"
37          target_name = "name"
38        }
39        properties = ["since"]
40      }
41    ]
42  }
43}

After the vertex job succeeds, run the edge job:

1./bin/seatunnel.sh --config ./config/sql2graph-knows.conf -m local

The following query should return a knows edge from marko to vadas with since set to 2010:

1g.V().has('person', 'name', 'marko').outE('knows').where(inV().has('name', 'vadas')).valueMap()

sourceConfig and targetConfig identify the endpoint fields. fieldMapping maps them to the vertex primary key name, and properties = ["since"] writes only the edge property. The example enables check_vertex = true and disables per-record fallback after a batch failure (batch_failure_fallback = false), so a missing endpoint or write failure causes the job to fail.

If the relation table has only numeric foreign keys while the graph uses names as primary keys, join the names in SQL before passing the records to the Sink. See MySQL CDC Source^[13] for MySQL CDC integration.

4 Import from Kafka (kafka2graph)

Kafka is useful for a continuous stream of events. Create the user-events topic and publish the following JSON message. Each message becomes one person vertex:

1{"name":"marko","age":29}

Save the following as config/kafka2graph.conf:

 1env {
 2  job.mode = "STREAMING"
 3  checkpoint.interval = 10000
 4  sink.flush.interval = 5000
 5}
 6
 7source {
 8  Kafka {
 9    bootstrap.servers = "kafka:9092"
10    topic = "user-events"
11    consumer.group = "hugegraph-import"
12    start_mode = "earliest"
13    format = "json"
14    schema = {
15      fields = {
16        name = "string"
17        age = "int"
18      }
19    }
20  }
21}
22
23sink {
24  HugeGraph {
25    host = "hugegraph"
26    port = 8080
27    graph_name = "hugegraph"
28    graph_space = "DEFAULT"
29    batch_failure_fallback = false
30    mappings = [
31      {
32        type = "VERTEX"
33        label = "person"
34        idStrategy = "PRIMARY_KEY"
35        idFields = ["name"]
36        properties = ["name", "age"]
37      }
38    ]
39  }
40}

1./bin/seatunnel.sh --config ./config/kafka2graph.conf -m local

Use the Gremlin query from section 3.1 to check the data. The streaming job keeps running. checkpoint.interval saves job state every 10 seconds, while sink.flush.interval asks Zeta to flush every 5 seconds so a small number of messages does not wait for a full batch.

HugeGraph Sink writes with at-least-once semantics, so recovery can replay records. PRIMARY_KEY sends the same name to the same vertex, but it does not make every update exactly-once. Scheduled flushing is provided by Zeta and does not apply to Spark or Flink engines.

5 Common configuration and troubleshooting

The following table applies to the SeaTunnel 3.0+ version used by this guide:

Use these checks when a job fails:

• mappings is unknown or HugeGraph Source is missing: Check that the engine and HugeGraph connector come from the same SeaTunnel 3.0+ build.
• Connection failure: Check the host, port, graph space, authentication details, and whether the SeaTunnel runtime can reach the service.
• Schema incompatibility: Check the ID strategy, property types, and edge endpoints. Automatic creation does not change an existing PRIMARY_KEY label into CUSTOMIZE_STRING.
• Small Kafka batches do not appear promptly: Confirm that the job uses Zeta and set sink.flush.interval in env. In this version, batch_interval_ms is retained only for compatibility and cannot replace it.

6 Choosing a tool

Choose a tool based on the work to complete. Use Tools for graph management, Gremlin, backup, or cloning. Use Loader for a direct graph import. Choose SeaTunnel when you need to reuse a Source, Transform, and Sink pipeline. For SeaTunnel graph reads and migrations, prepare the environment using the SeaTunnel 3.0+ version used by this guide.

7 References

HugeGraph connectors

^[1] HugeGraph Source
^[2] HugeGraph Sink

Configuration and deployment

^[3] HOCON job configuration
^[4] SeaTunnel 3.0+ release download
^[5] Connector plugin installation
^[6] SeaTunnel local deployment

Execution engines

^[7] SeaTunnel Engine Overview
^[8] SeaTunnel Spark Engine
^[9] SeaTunnel Flink Engine
^[10] Connector V2 multi-engine support

Data source connectors

^[11] JDBC Source
^[12] Kafka Source
^[13] MySQL CDC Source

4 - HugeGraph-Loader Quick Start

This guide aligns with the Toolchain master branch (currently based on 1.8, unreleased). Released packages and Docker latest may differ; check your version before using unreleased features.

Quick start: local file import

Walk through a minimal import workflow: import two people and one relationship, then verify the data in Hubble. First install Loader and start HugeGraph Server. The commands below run from the Loader directory, importing into the default graph hugegraph. For an authenticated Server, append --username and --password (or --token).

1. Create the data files.

   These CSV files have no header row. The mapping in the next step supplies the column names.

   1mkdir -p demo-loader
   2cat > demo-loader/people.csv <<'EOF'
   3docs-smoke-alice,29
   4docs-smoke-bob,31
   5EOF
   6cat > demo-loader/knows.csv <<'EOF'
   7docs-smoke-alice,docs-smoke-bob
   8EOF

2. Create the schema and mapping.

   Dedicated docs_* names keep this example separate from the built-in person data. File paths are relative to the directory where you run Loader.

    1cat > demo-loader/schema.groovy <<'EOF'
    2schema.propertyKey("docs_name").asText().ifNotExist().create()
    3schema.propertyKey("docs_age").asInt().ifNotExist().create()
    4schema.vertexLabel("docs_smoke_person").properties("docs_name", "docs_age")
    5      .primaryKeys("docs_name").ifNotExist().create()
    6schema.edgeLabel("docs_smoke_knows").sourceLabel("docs_smoke_person")
    7      .targetLabel("docs_smoke_person").ifNotExist().create()
    8EOF
    9cat > demo-loader/struct.json <<'EOF'
   10{
   11  "version": "2.0",
   12  "structs": [
   13    {
   14      "id": "people",
   15      "input": {
   16        "type": "FILE",
   17        "path": "demo-loader/people.csv",
   18        "format": "CSV",
   19        "header": ["docs_name", "docs_age"]
   20      },
   21      "vertices": [{"label": "docs_smoke_person"}]
   22    },
   23    {
   24      "id": "knows",
   25      "input": {
   26        "type": "FILE",
   27        "path": "demo-loader/knows.csv",
   28        "format": "CSV",
   29        "header": ["source_name", "target_name"]
   30      },
   31      "edges": [{
   32        "label": "docs_smoke_knows",
   33        "source": ["source_name"],
   34        "target": ["target_name"],
   35        "field_mapping": {"source_name": "docs_name", "target_name": "docs_name"}
   36      }]
   37    }
   38  ]
   39}
   40EOF

3. Run the import.

   Keep the same working directory. Change the host and port if Server runs elsewhere.

   1bin/hugegraph-loader.sh \
   2  --graphspace DEFAULT --graph hugegraph \
   3  --file demo-loader/struct.json --schema demo-loader/schema.groovy \
   4  --host 127.0.0.1 --port 8080

   A first successful run imports two vertices and one edge. Check the success and failure counts in the summary.

4. Query the result in Hubble.

   Connect Hubble to the same Server and graph, then run these queries separately in the Gremlin workspace.

   1g.V().hasLabel('docs_smoke_person').valueMap('docs_name', 'docs_age')
   2g.V().has('docs_smoke_person', 'docs_name', 'docs-smoke-alice')
   3 .out('docs_smoke_knows').values('docs_name')

   The first query returns Alice with age 29 and Bob with age 31. The second returns docs-smoke-bob.

For other inputs, continue with the mapping reference. The bundled full example also includes software vertices and edge properties.

1 HugeGraph-Loader Overview

HugeGraph-Loader converts source data into graph vertices and edges and imports them into HugeGraph in batches.

Currently supported data sources include:

• Local disk file or directory, supports TEXT, CSV and JSON format files, supports compressed files
• HDFS file or directory supports compressed files
• Mainstream relational databases, such as MySQL, PostgreSQL, Oracle, SQL Server
• Kafka topic
• An existing HugeGraph graph, used to copy data from one graph into another

Local disk files and HDFS files support resumable uploads.

Note: HugeGraph-Loader requires HugeGraph Server service, please refer to HugeGraph-Server Quick Start to download and start Server

Concurrent imports: Loader on the master branch supports LOADING owner protection on the same Server instance when the Server’s GET .../mode/owners returns protocol: 1. Servers without this endpoint retain legacy import behavior without this protection; see the compatibility notes below.

Testing Guide: For running HugeGraph-Loader tests locally, please refer to HugeGraph Toolchain Local Testing Guide

2 Get HugeGraph-Loader

HugeGraph-Loader is available in the following three ways:

• Use docker image (Convenient for Test/Dev)
• Download the compiled tarball
• Clone source code then compile and install

2.1 Use Docker image (Convenient for Test/Dev)

We can deploy the loader service using docker run -itd --name loader hugegraph/loader:latest. For the data that needs to be loaded, it can be copied into the loader container either by mounting -v /path/to/data/file:/loader/file or by using docker cp.

Alternatively, to start the loader using docker-compose, the command is docker-compose up -d. An example of the docker-compose.yml is as follows:

This combination is for local anonymous testing; the Server API is published only on the local host.

 1version: '3'
 2
 3services:
 4  server:
 5    image: hugegraph/hugegraph:latest
 6    container_name: server
 7    ports:
 8      - 127.0.0.1:8080:8080
 9
10  loader:
11    image: hugegraph/loader:latest
12    container_name: loader
13    # mount your own data here
14    # volumes:
15      # - /path/to/data/file:/loader/file

The specific data loading process can be referenced under 4.5 User Docker to load data

Note:

1. The docker image of hugegraph-loader is a convenience release to start hugegraph-loader quickly, but not official distribution artifacts. You can find more details from ASF Release Distribution Policy.

2. Pin a published version tag or image digest in production. latest is mutable and does not guarantee alignment with master.

2.2 Download the compiled archive

Choose a published Toolchain archive from the download page and extract it. A release may not include all master functionality; use the source build below for the implementation described here.

2.3 Clone source code to compile and install

Building Loader from the master branch requires Java 17, Maven 3.6.3+, and Python 3.9+. Clone the repository and build:

1git clone --branch master --single-branch https://github.com/apache/hugegraph-toolchain.git
2cd hugegraph-toolchain
3mvn package -pl hugegraph-client,hugegraph-loader -am -DskipTests -ntp
4cd hugegraph-loader/apache-hugegraph-loader-*

To build against candidate Server SDKs during development, refer to the Java 17 migration guide.

For Oracle input, obtain a JDBC driver JAR compatible with your Oracle/JDK versions and place it in the extracted Loader lib/ directory. The startup script loads JARs there; installing a driver only in the local Maven repository does not put it on Loader’s runtime classpath.

3 How to use

To import a dataset, prepare the schema, data and mapping, then run Loader.

• Write graph schema
• Prepare data files
• Write input source map files
• Execute command import

3.1 Construct graph schema

Define the vertex labels, edge labels and properties needed to represent your data before writing the schema.

The example below uses person and software vertices, connected by knows and created edges. People have names, ages and cities; software has a name and price. Each edge records a date.

graph model example

Save the schema definitions in schema.groovy.

 1// Create some properties
 2schema.propertyKey("name").asText().ifNotExist().create();
 3schema.propertyKey("age").asInt().ifNotExist().create();
 4schema.propertyKey("city").asText().ifNotExist().create();
 5schema.propertyKey("date").asText().ifNotExist().create();
 6schema.propertyKey("price").asDouble().ifNotExist().create();
 7
 8// Create the person vertex type, which has three attributes: name, age, city, and the primary key is name
 9schema.vertexLabel("person").properties("name", "age", "city").primaryKeys("name").ifNotExist().create();
10// Create a software vertex type, which has two properties: name, price, the primary key is name
11schema.vertexLabel("software").properties("name", "price").primaryKeys("name").ifNotExist().create();
12
13// Create the knows edge type, which goes from person to person
14schema.edgeLabel("knows").sourceLabel("person").targetLabel("person").properties("date").ifNotExist().create();
15// Create the created edge type, which points from person to software
16schema.edgeLabel("created").sourceLabel("person").targetLabel("software").ifNotExist().create();

See HugeGraph Client for the schema API.

3.2 Prepare data

The data sources currently supported by HugeGraph-Loader include:

• local disk file or directory
• HDFS file or directory
• Partial relational database
• Kafka topic
• An existing HugeGraph graph

3.2.1 Data source structure

3.2.1.1 Local disk file or directory

Set the input path to a local file or a directory. A single input source accepts one directory, not a list of directories.

For data split across part-0, part-1 and other files, place those files in one directory and set the mapping’s path to that directory.

Supported file formats include:

• TEXT
• CSV
• JSON

TEXT uses a configurable delimiter. Each data row represents a record; its columns supply IDs, labels or properties. Column names can come from a header row or from the mapping’s header setting.

An example is as follows:

1id|name|lang|price|ISBN
21|lop|java|328|ISBN978-7-107-18618-5
32|ripple|java|199|ISBN978-7-100-13678-5

CSV is a TEXT file with commas , as delimiters. When a column value itself contains a comma, the column value needs to be enclosed in double quotes, for example:

1marko,29,Beijing
2"li,nary",26,"Wu,han"

The JSON file requires that each line is a JSON string, and the format of each line needs to be consistent.

1{"source_name": "marko", "target_name": "vadas", "date": "20160110", "weight": 0.5}
2{"source_name": "marko", "target_name": "josh", "date": "20130220", "weight": 1.0}

3.2.1.2 HDFS file or directory

Users can also specify HDFS files or directories as data sources, all of the above requirements for local disk files or directories apply here. In addition, since HDFS usually stores compressed files, loader also provides support for compressed files, and local disk file or directory also supports compressed files.

Currently supported compressed file types include: GZIP, BZ2, XZ, LZMA, SNAPPY_RAW, SNAPPY_FRAMED, Z, DEFLATE, LZ4_BLOCK, LZ4_FRAMED, ORC, and PARQUET.

3.2.1.3 Mainstream relational database

The loader also supports some relational databases as data sources, and currently supports MySQL, PostgreSQL, Oracle, and SQL Server.

Loader does not follow foreign keys to query other tables while parsing a row. Each input row must provide the values needed to construct its graph elements. Prepare any joins in the source query first.

Consider three tables named person, software and created.

1// person schema
2id | name | age | city

1// software schema
2id | name | lang | price

1// created schema
2id | p_id | s_id | date

If person and software use PRIMARY_KEY IDs with name as the primary key, each edge needs its endpoints’ names. The created table contains only p_id and s_id; Loader will not query the names from those IDs. Use either of these ID designs.

1. Keep PRIMARY_KEY and use each vertex table’s id column as its primary key property. Loader can then build endpoint IDs from p_id, s_id and the vertex labels.
2. Use custom IDs, taking each vertex table’s id column as the vertex ID. The edge mapping can use p_id and s_id directly.

Both designs let edge rows supply endpoint IDs without another lookup.

3.2.2 Prepare vertex and edge data

3.2.2.1 Vertex Data

The vertex data file consists of data line by line. Generally, each line is used as a vertex, and each column is used as a vertex attribute. The following description uses CSV format as an example.

• person vertex data (the data itself does not contain a header)

1Tom,48,Beijing
2Jerry,36,Shanghai

• software vertex data (the data itself contains the header)

1name,price
2Photoshop,999
3Office,388

3.2.2.2 Edge data

The edge data file consists of data line by line. Generally, each line is used as an edge. Some columns are used as the IDs of the source and target vertices, and other columns are used as edge attributes. The following uses JSON format as an example.

• knows edge data

1{"source_name": "Tom", "target_name": "Jerry", "date": "2008-12-12"}

• created edge data

1{"source_name": "Tom", "target_name": "Photoshop"}
2{"source_name": "Tom", "target_name": "Office"}
3{"source_name": "Jerry", "target_name": "Office"}

3.3 Write data source mapping file

3.3.1 Mapping file overview

The mapping file uses JSON. Each item in structs contains one input and its vertices and edges mappings.

input describes the data location, format and delimiter. Vertex and edge mappings define labels, IDs, properties and field conversions.

Loader 0.11.0 introduced mapping format 2.0; earlier mappings use format 1.0. This page uses 2.0 unless stated otherwise.

 1{
 2  "version": "2.0",
 3  "structs": [
 4    {
 5      "id": "1",
 6      "input": {
 7      },
 8      "vertices": [
 9        {},
10        {}
11      ],
12      "edges": [
13        {},
14        {}
15      ]
16    }
17  ]
18}

Two versions of the mapping file are given directly here (the above graph model and data file are described)

  1{
  2  "version": "2.0",
  3  "structs": [
  4    {
  5      "id": "1",
  6      "skip": false,
  7      "input": {
  8        "type": "FILE",
  9        "path": "vertex_person.csv",
 10        "file_filter": {
 11          "extensions": [
 12            "*"
 13          ]
 14        },
 15        "format": "CSV",
 16        "delimiter": ",",
 17        "date_format": "yyyy-MM-dd HH:mm:ss",
 18        "time_zone": "GMT+8",
 19        "skipped_line": {
 20          "regex": "(^#|^//).*|"
 21        },
 22        "compression": "NONE",
 23        "header": [
 24          "name",
 25          "age",
 26          "city"
 27        ],
 28        "charset": "UTF-8",
 29        "list_format": {
 30          "start_symbol": "[",
 31          "elem_delimiter": "|",
 32          "end_symbol": "]"
 33        }
 34      },
 35      "vertices": [
 36        {
 37          "label": "person",
 38          "skip": false,
 39          "id": null,
 40          "unfold": false,
 41          "field_mapping": {},
 42          "value_mapping": {},
 43          "selected": [],
 44          "ignored": [],
 45          "null_values": [
 46            ""
 47          ],
 48          "update_strategies": {}
 49        }
 50      ],
 51      "edges": []
 52    },
 53    {
 54      "id": "2",
 55      "skip": false,
 56      "input": {
 57        "type": "FILE",
 58        "path": "vertex_software.csv",
 59        "file_filter": {
 60          "extensions": [
 61            "*"
 62          ]
 63        },
 64        "format": "CSV",
 65        "delimiter": ",",
 66        "date_format": "yyyy-MM-dd HH:mm:ss",
 67        "time_zone": "GMT+8",
 68        "skipped_line": {
 69          "regex": "(^#|^//).*|"
 70        },
 71        "compression": "NONE",
 72        "header": null,
 73        "charset": "UTF-8",
 74        "list_format": {
 75          "start_symbol": "",
 76          "elem_delimiter": ",",
 77          "end_symbol": ""
 78        }
 79      },
 80      "vertices": [
 81        {
 82          "label": "software",
 83          "skip": false,
 84          "id": null,
 85          "unfold": false,
 86          "field_mapping": {},
 87          "value_mapping": {},
 88          "selected": [],
 89          "ignored": [],
 90          "null_values": [
 91            ""
 92          ],
 93          "update_strategies": {}
 94        }
 95      ],
 96      "edges": []
 97    },
 98    {
 99      "id": "3",
100      "skip": false,
101      "input": {
102        "type": "FILE",
103        "path": "edge_knows.json",
104        "file_filter": {
105          "extensions": [
106            "*"
107          ]
108        },
109        "format": "JSON",
110        "delimiter": null,
111        "date_format": "yyyy-MM-dd HH:mm:ss",
112        "time_zone": "GMT+8",
113        "skipped_line": {
114          "regex": "(^#|^//).*|"
115        },
116        "compression": "NONE",
117        "header": null,
118        "charset": "UTF-8",
119        "list_format": null
120      },
121      "vertices": [],
122      "edges": [
123        {
124          "label": "knows",
125          "skip": false,
126          "source": [
127            "source_name"
128          ],
129          "unfold_source": false,
130          "target": [
131            "target_name"
132          ],
133          "unfold_target": false,
134          "field_mapping": {
135            "source_name": "name",
136            "target_name": "name"
137          },
138          "value_mapping": {},
139          "selected": [],
140          "ignored": [],
141          "null_values": [
142            ""
143          ],
144          "update_strategies": {}
145        }
146      ]
147    },
148    {
149      "id": "4",
150      "skip": false,
151      "input": {
152        "type": "FILE",
153        "path": "edge_created.json",
154        "file_filter": {
155          "extensions": [
156            "*"
157          ]
158        },
159        "format": "JSON",
160        "delimiter": null,
161        "date_format": "yyyy-MM-dd HH:mm:ss",
162        "time_zone": "GMT+8",
163        "skipped_line": {
164          "regex": "(^#|^//).*|"
165        },
166        "compression": "NONE",
167        "header": null,
168        "charset": "UTF-8",
169        "list_format": null
170      },
171      "vertices": [],
172      "edges": [
173        {
174          "label": "created",
175          "skip": false,
176          "source": [
177            "source_name"
178          ],
179          "unfold_source": false,
180          "target": [
181            "target_name"
182          ],
183          "unfold_target": false,
184          "field_mapping": {
185            "source_name": "name",
186            "target_name": "name"
187          },
188          "value_mapping": {},
189          "selected": [],
190          "ignored": [],
191          "null_values": [
192            ""
193          ],
194          "update_strategies": {}
195        }
196      ]
197    }
198  ]
199}

 1{
 2  "vertices": [
 3    {
 4      "label": "person",
 5      "input": {
 6        "type": "file",
 7        "path": "vertex_person.csv",
 8        "format": "CSV",
 9        "header": ["name", "age", "city"],
10        "charset": "UTF-8"
11      }
12    },
13    {
14      "label": "software",
15      "input": {
16        "type": "file",
17        "path": "vertex_software.csv",
18        "format": "CSV"
19      }
20    }
21  ],
22  "edges": [
23    {
24      "label": "knows",
25      "source": ["source_name"],
26      "target": ["target_name"],
27      "input": {
28        "type": "file",
29        "path": "edge_knows.json",
30        "format": "JSON"
31      },
32      "field_mapping": {
33        "source_name": "name",
34        "target_name": "name"
35      }
36    },
37    {
38      "label": "created",
39      "source": ["source_name"],
40      "target": ["target_name"],
41      "input": {
42        "type": "file",
43        "path": "edge_created.json",
44        "format": "JSON"
45      },
46      "field_mapping": {
47        "source_name": "name",
48        "target_name": "name"
49      }
50    }
51  ]
52}

The 1.0 version of the mapping file is centered on the vertex and edge, and sets the input source; while the 2.0 version is centered on the input source, and sets the vertex and edge mapping. Some input sources (such as a file) can generate both vertices and edges. If you write in the 1.0 format, you need to write an input block in each of the vertex and edge mapping blocks. The two input blocks are exactly the same; and the 2.0 version only needs to write input once. Therefore, compared with version 1.0, version 2.0 can save some repetitive writing of input.

In the bin directory of hugegraph-loader-{version}, there is a script tool mapping-convert.sh that can directly convert the mapping file of version 1.0 to version 2.0. The usage is as follows:

1bin/mapping-convert.sh struct.json

A struct-v2.json will be generated in the same directory as struct.json.

The bin directory also ships utf8-bom-to-utf8.sh, which strips the UTF-8 BOM from a single data file, or from every file under a directory. It is useful when a CSV or TEXT file exported by a Windows tool fails to parse because its first header column carries an invisible BOM:

1bin/utf8-bom-to-utf8.sh /path/to/file-or-dir

3.3.2 Input Source

Input sources are currently divided into five categories: FILE, HDFS, JDBC, KAFKA and GRAPH, which are distinguished by the type node. We call them local file input sources, HDFS input sources, JDBC input sources, KAFKA input sources and GRAPH input source, which are described below.

3.3.2.1 Local file input source

• id: The id of the input source. This field is used to support some internal functions. It is not required (it will be automatically generated if it is not filled in). It is strongly recommended to write it, which is very helpful for debugging;
• skip: whether to skip the input source, because the JSON file cannot add comments, if you do not want to import an input source during a certain import, but do not want to delete the configuration of the input source, you can set it to true to skip it, the default is false, not required;
• input: input source map block, composite structure
  • type: an input source type, file or FILE must be filled;
  • path: the path of the local file or directory, an absolute path or a path relative to the Loader process working directory (not the mapping file). An absolute path is recommended; required;
  • file_filter: filter files with compound conditions from path, compound structure, currently only supports configuration extensions, represented by child node extensions, the default is “*”, which means to keep all files;
  • format: the format of the local file, the optional values are CSV, TEXT and JSON, which must be uppercase, the default is CSV, optional;
  • header: column names; if omitted, the first data line supplies them. With an explicit header, an identical first line is still skipped by default (see has_header). JSON does not require a header; optional;
  • has_header: for CSV and TEXT, the first line of every file is dropped when it is identical to the header, so a header repeated in each part file of a directory is not imported as data. Set this to false to turn that check off when the first line of a file is real data that happens to equal the header, optional;
  • delimiter: The column delimiter of the file line. The default depends on format: a comma "," for CSV and a tab "\t" for TEXT; a CSV file accepts no other delimiter. The JSON file does not need to be specified, optional;
  • charset: the encoded character set of the file, the default is UTF-8, optional;
  • date_format: custom date format, the default value is yyyy-MM-dd HH:mm:ss, optional; if the date is presented in the form of a timestamp, this item must be written as timestamp (fixed writing);
  • extra_date_formats: a list of fallback date formats, tried when a value does not match date_format, empty by default, optional;
  • time_zone: Set which time zone the date data is in, the default value is GMT+8, optional;
  • skipped_line: The line to be skipped, compound structure, currently only the regular expression of the line to be skipped can be configured, described by the child node regex. The default regex is (^#|^//).*|, which skips lines starting with # or // and empty lines; to keep such lines, set regex to a pattern that matches nothing, optional;
  • compression: The compression format of the file, the optional values ​​are NONE, GZIP, BZ2, XZ, LZMA, SNAPPY_RAW, SNAPPY_FRAMED, Z, DEFLATE, LZ4_BLOCK, LZ4_FRAMED, ORC and PARQUET, the default is NONE, which means a non-compressed file, optional; for ORC and PARQUET the header is matched case-insensitively;
  • list_format: When a column of the file (non-JSON) is a collection structure (the Cardinality of the PropertyKey in the corresponding figure is Set or List), you can use this item to set the start character, separator, and end character of the column, compound structure :
    • start_symbol: The start character of the collection structure column (the default value is the empty string "", JSON format currently does not support specification)
    • elem_delimiter: the delimiter of the collection structure column (the default value is |, and it must differ from delimiter; JSON format currently only supports native , delimiter)
    • end_symbol: the end character of the collection structure column (the default value is the empty string "", the JSON format does not currently support specification)
    • ignored_elems: the elements dropped after the column is split, the default value is [""], so empty elements are ignored

3.3.2.2 HDFS input source

The nodes and meanings of the above local file input source are basically applicable here. Only the different and unique nodes of the HDFS input source are listed below.

• type: input source type, must fill in hdfs or HDFS, required;
• path: the path of the HDFS file or directory, it must be the absolute path of HDFS, required;
• core_site_path: the path of the core-site.xml file of the HDFS cluster, the key point is to specify the address of the NameNode (fs.default.name) and the implementation of the file system (fs.hdfs.impl), required;
• hdfs_site_path: the path of the hdfs-site.xml file of the HDFS cluster, optional;
• dir_filter: when path is a directory, decides which sub-directories are walked into, compound structure, optional:
  • include_regex: only directories whose name matches this regular expression are read, empty by default, which places no restriction;
  • exclude_regex: directories whose name matches this regular expression are skipped, empty by default;
• kerberos_config: how to authenticate against a Kerberos-secured HDFS cluster, compound structure, optional:
  • enable: whether to authenticate with Kerberos, the default is false;
  • krb5_conf: the path of the krb5.conf file, required when enable is true;
  • principal: the Kerberos principal, required when enable is true;
  • keytab: the path of the keytab file, required when enable is true;

3.3.2.3 JDBC input source

As mentioned above, it supports multiple relational databases, but because their mapping structures are very similar, they are collectively referred to as JDBC input sources, and then use the vendor node to distinguish different databases.

• type: input source type, must fill in jdbc or JDBC, required;
• vendor: database type, optional options are [MySQL, PostgreSQL, Oracle, SQLServer], case-insensitive, required;
• driver: the JDBC driver class, optional; when it is left out, the default driver of the vendor listed in the tables below is used;
• url: the url of the database that jdbc wants to connect to, required;
• database: the name of the database to be connected, required;
• schema: The name of the schema to be connected, different databases have different requirements, and the details are explained below;
• table: the name of the table to be connected, at least one of table or custom_sql is required;
• custom_sql: custom SQL statement, at least one of table or custom_sql is required;
• username: username to connect to the database, required;
• password: password for connecting to the database, required;
• where: an extra condition appended to the generated select statement, written without the where keyword, optional;
• batch_size: The size of one page when obtaining table data by page, the default is 500, optional;

MYSQL

schema: nullable, if filled in, it must be the same as the value of database

POSTGRESQL

schema: nullable, default is “public”

ORACLE

schema: nullable, the default value is the username in upper case

SQLSERVER

schema: required

3.3.2.4 Kafka input source

• type: input source type, kafka or KAFKA, required;
• bootstrap_server: the list of kafka bootstrap servers, required;
• topic: the topic to subscribe to, required;
• group: group of Kafka consumers, required;
• from_beginning: sets auto.offset.reset to earliest when true, or latest when false (default). It applies only when no valid committed offset exists; otherwise consumption resumes from the committed offset. Optional;
• format: format of each message, options are CSV, TEXT and JSON, must be uppercase, required;
• header: column name of each column of a message; no header line is read from the topic, so it has to be given for CSV and TEXT, while JSON messages do not need it;
• delimiter: delimiter of the message columns, used by TEXT only, since CSV always splits on ,, optional;
• charset: encoding charset of the messages, default is UTF-8, optional;
• date_format: customized date format, default value is yyyy-MM-dd HH:mm:ss, optional; if the date is presented in the form of timestamp, this item must be written as timestamp (fixed);
• extra_date_formats: a customized list of another date formats, empty by default, optional; each item in the list is an alternate date format to the date_format specified date format;
• time_zone: set which time zone the date data is in, default is GMT+8, optional;
• skipped_line: current master KafkaReader does not implement this filter. Kafka messages go directly to the parser, so configuring this field does not skip blank or comment messages; preprocess them upstream or in the consumer to avoid parse failures;
• batch_size: the maximum number of records fetched in one poll (max.poll.records), default is 500, optional;
• early_stop: the record pulled from Kafka broker at a certain time is empty, stop the task, default is false, only for debugging, optional;

3.3.2.5 GRAPH input Source

The GRAPH input source reads vertices and edges out of another HugeGraph graph, reached through HugeGraph-PD, and writes them into the target graph. When a mapping file contains a GRAPH input source, every input source in it that is not skipped has to be a GRAPH input source as well, and the loader puts the target graph into RESTORING mode for the duration of the import.

• type: Data source type; must be filled in as graph or GRAPH (required);
• graphspace: Source graphSpace name (required);
• graph: Source graph name (required);
• username: HugeGraph username; the --username command-line option is used when this is empty;
• password: HugeGraph password; the --password command-line option is used when this is empty;
• selected_vertices: the vertex labels to copy, each item written as {"label": "...", "properties": [...], "query": {...}}, where properties narrows the copied properties and query is an optional filter passed to the source graph;
• ignored_vertices: the vertex labels to skip, each item written as {"label": "...", "properties": [...]};
• selected_edges: the edge labels to copy, items have the same shape as in selected_vertices;
• ignored_edges: the edge labels to skip, items have the same shape as in ignored_vertices;
• pd-peers: HugeGraph-PD node addresses of the source cluster; the --pd-peers option is used when this is empty;
• meta-endpoints: Meta service endpoints of the source cluster; the --meta-endpoints option is used when this is empty;
• cluster: Source cluster name; the --cluster option is used when this is empty;
• batch_size: Batch size for reading data from the source graph; default is 500;

3.3.3 Vertex and Edge Mapping

The nodes of vertex and edge mapping (a key in the JSON file) have a lot of the same parts. The same parts are introduced first, and then the unique nodes of vertex map and edge map are introduced respectively.

Nodes of the same section

• label: label to which the vertex/edge data to be imported belongs, required;
• skip: whether to skip this vertex/edge mapping while the input source and the other mappings stay active, the default is false, optional;
• field_mapping: Map the column name of the input source column to the attribute name of the vertex/edge, optional;
• value_mapping: map the data value of the input source to the attribute value of the vertex/edge, optional;
• selected: select some columns to insert, other unselected ones are not inserted, cannot exist at the same time as ignored, optional;
• ignored: ignore some columns so that they do not participate in insertion, cannot exist at the same time as selected, optional;
• null_values: You can specify some strings to represent null values, such as “NULL”. If the vertex/edge attribute corresponding to this column is also a nullable attribute, the value of this attribute will not be set when constructing the vertex/edge, optional ;
• update_strategies: If the data needs to be updated in batches in a specific way, you can specify a specific update strategy for each attribute (see below for details), optional;
• unfold: Whether to unfold the column, each unfolded column will form a row with other columns, which is equivalent to unfolding into multiple rows; for example, the value of a certain column (id column) of the file is [1,2,3], The values ​​of other columns are 18,Beijing. When unfold is set, this row will become 3 rows, namely: 1,18,Beijing, 2,18,Beijing and 3,18, Beijing. Note that this will only expand the column selected as id. Default false, optional;

Update strategy supports 8 types: (requires all uppercase)

1. Value accumulation: SUM
2. Take the greater of the two numbers/dates: BIGGER
3. Take the smaller of two numbers/dates: SMALLER
4. Set property takes union: UNION
5. Set attribute intersection: INTERSECTION
6. List attribute append element: APPEND
7. List/Set attribute delete element: ELIMINATE
8. Override an existing property: OVERRIDE

Note: If the newly imported attribute value is empty, the existing old data will be used instead of the empty value. For the effect, please refer to the following example

 1// The update strategy is specified in the JSON file as follows
 2{
 3  "vertices": [
 4    {
 5      "label": "person",
 6      "update_strategies": {
 7        "age": "SMALLER",
 8        "set": "UNION"
 9      },
10      "input": {
11        "type": "file",
12        "path": "vertex_person.txt",
13        "format": "TEXT",
14        "header": ["name", "age", "set"]
15      }
16    }
17  ]
18}
19
20// 1. Write a line of data with the OVERRIDE update strategy (null means empty here)
21'a b null null'
22
23// 2. Write another line
24'null null c d'
25
26// 3. Finally we can get
27'a b c d'   
28
29// If there is no update strategy, you will get
30'null null c d'

Note : After adopting the batch update strategy, the number of disk read requests will increase significantly, and the import speed will be several times slower than that of pure write coverage (at this time HDD disk [IOPS](https://en.wikipedia .org/wiki/IOPS) will be the bottleneck, SSD is recommended for speed)

Unique Nodes for Vertex Maps

• id: Specify a column as the id column of the vertex. When the vertex id policy is CUSTOMIZE, it is required; when the id policy is PRIMARY_KEY, it must be empty;

Unique Nodes for Edge Maps

• source: Select certain columns of the input source as the id column of source vertex. When the id policy of the source vertex is CUSTOMIZE, a certain column must be specified as the id column of the vertex; when the id policy of the source vertex is When PRIMARY_KEY, one or more columns must be specified for splicing the id of the generated vertex, that is, no matter which id strategy is used, this item is required;
• target: Specify certain columns as the id columns of target vertex, similar to source, so I won’t repeat them;
• unfold_source: Whether to unfold the source column of the file, the effect is similar to that in the vertex map, and will not be repeated;
• unfold_target: Whether to unfold the target column of the file, the effect is similar to that in the vertex mapping, and will not be repeated;

3.4 Execute command import

With the schema, data and mapping ready, run Loader with the options below.

3.4.1 Parameter description

The mapping file option --file (-f) is required; other options use their defaults. A valid minimal invocation is:

1bin/hugegraph-loader.sh --file struct.json

The launcher also accepts JCommander’s native @file syntax. Put each option and value on separate lines in loader.args:

1--file
2struct.json

1bin/hugegraph-loader.sh @loader.args

This is equivalent to the minimal invocation above. Relative path values resolve from the Loader process working directory. If the argument file contains credentials, restrict it to mode 0600 with chmod 0600 loader.args. The @file form keeps credential values out of process arguments. Loader must be able to create and write the mapping-prefix directory used for progress output. Copy a mapping mounted from a read-only Kubernetes ConfigMap to a writable volume before running Loader.

3.4.2 Breakpoint Continuation Mode

For a long import, use a progress file to resume after an interruption.

Set --incremental-mode to true to resume from a progress file. Loader records progress when it closes normally after a successful import or a normal stop; an unexpected failure may prevent this write. The progress file is in ${struct} and named load-progress_${timestamp}, where ${struct} is the mapping-file prefix and ${timestamp} is the import start time, formatted as yyyyMMdd-HHmmss. For example, an import started at 2019-10-10 12:30:30 with mapping file struct-example.json writes struct-example/load-progress_20191010-123030. When the directory holds several progress files, the resumed import reads the last one in name order, which is the most recent one.

Note: A progress file is written after a successful import or a normal stop (including an error limit), regardless of --incremental-mode or --print-progress. An import aborted by an unexpected failure may not write one.

After stopping an import with Ctrl+C or kill, set --incremental-mode true to continue from the recorded position. A forced stop with kill -9 cannot save progress.

Invalid data or connection failures can exhaust --max-read-errors, --max-parse-errors or --max-insert-errors. Correct the rows in the failure files, then use --failure-mode true to retry them. This leaves the original data files unchanged. Rows that still fail are recorded again and deduplicated when the failure file closes. Failure mode lifts these three limits and scans the entire file.

Each input source, that is each item of structs in the mapping file, gets its own failure file. The file is named after the id of that input source with the suffix .error and is stored in the ${struct}/failure-data directory. Every failed line is written as a pair of lines: a tip line starting with #### READ ERROR:, #### PARSE ERROR: or #### INSERT ERROR:, followed by the original data line. When the input source has a header, that header is written as JSON to a sibling ${id}.header file, so the failure file can be read back with the right columns. For example, if the mapping file has an input source with id 1 holding a vertex mapping person and an input source with id 3 holding an edge mapping knows, each of which has some error lines, you will see the following files in the ${struct}/failure-data directory when the Loader exits:

• 1.error: the failed lines of input source 1, each preceded by its tip line
• 1.header: the header of input source 1, written only when that input source has a header
• 3.error: the failed lines of input source 3
• 3.header: the header of input source 3

A .error file that turns out to be empty is deleted when the Loader exits, so only the input sources that really had failed lines leave a file behind. In incremental mode new failures are appended to the existing file, otherwise the file is rewritten from scratch.

3.4.3 logs directory file description

The log and error data during program execution will be written into the hugegraph-loader.log file.

3.4.4 Execute command

Run bin/hugegraph-loader.sh and pass in parameters

1bin/hugegraph-loader.sh -g {GRAPH_NAME} -f ${INPUT_DESC_FILE} -s ${SCHEMA_FILE} -h {HOST} -p {PORT}

The script runs the JVM under JAVA_HOME when that variable is set, and java from the PATH otherwise. It passes the contents of the JVM_OPTS environment variable, then -Xmx10g and the class path built from lib/, to that JVM, so JVM_OPTS is the place to add JVM flags. Logging is configured by conf/log4j2.xml.

3.4.5 Minimal Local-File Import

Follow the local-file import steps at the start of this page.

3.4.6 Concurrent imports and LOADING ownership

In the master branch, Loader implements a LOADING ownership protocol coordinated with Server’s /mode/owners endpoint. When the deployed Server provides this endpoint and returns protocol: 1, Loader enables ownership protection automatically; released Server 1.7.0 and older versions continue using the legacy mode switching flow.

graph TD
    A["Graph Mode: NONE"] -->|First Loader acquires owner| B["LOADING Mode (Acquires Token)"]
    B -->|Concurrent Loaders join| B
    B -->|Loader finishes & releases| B
    B -->|Last Loader releases| C["Restores to NONE"]
    B -.->|Abnormal exit / kill -9| D["Manual Release / Recovery / Restart"]
    D -.->|"Last owner released, or recover/restart"| A
    D -.->|"Other owners still active"| B

1. Probe and Acquire: Loader probes /mode/owners. The first Loader obtains an ownership handle (instance/epoch/token), transitioning graph mode from NONE to LOADING; concurrent Loaders can join.
2. Data Import: Loaders holding valid handles execute concurrent imports while preserving LOADING mode.
3. Graceful Release: When an import completes, Loader drains its workers and releases its handle. When the last owner releases, graph mode automatically returns to NONE.
4. Failure Recovery: If abnormal exit (e.g. kill -9) leaves an orphaned owner, stop and drain all related imports and graph writes, then perform administrator recovery via the RESTful API recovery endpoint or restart the Server. Manual release applies only when the caller still holds its token (e.g. a direct REST client).

4 Complete example

The Loader distribution includes this dataset in example/file/. See the source files.

4.1 Prepare data

Vertex file: example/file/vertex_person.csv

1marko,29,Beijing
2vadas,27,Hongkong
3josh,32,Beijing
4peter,35,Shanghai
5"li,nary",26,"Wu,han"
6tom,null,NULL

Vertex file: example/file/vertex_software.txt

1id|name|lang|price|ISBN
21|lop|java|328|ISBN978-7-107-18618-5
32|ripple|java|199|ISBN978-7-100-13678-5

Edge file: example/file/edge_knows.json

1{"source_name": "marko", "target_name": "vadas", "date": "20160110", "weight": 0.5}
2{"source_name": "marko", "target_name": "josh", "date": "20130220", "weight": 1.0}

Edge file: example/file/edge_created.json

1{"source_name": "marko", "target_id": 1, "date": "2017-12-10", "weight": 0.4}
2{"source_name": "josh", "target_id": 1, "date": "2009-11-11", "weight": 0.4}
3{"source_name": "josh", "target_id": 2, "date": "2017-12-10", "weight": 1.0}
4{"source_name": "peter", "target_id": 1, "date": "2017-03-24", "weight": 0.2}

4.2 Write schema

 1schema.propertyKey("name").asText().ifNotExist().create();
 2schema.propertyKey("age").asInt().ifNotExist().create();
 3schema.propertyKey("city").asText().ifNotExist().create();
 4schema.propertyKey("weight").asDouble().ifNotExist().create();
 5schema.propertyKey("lang").asText().ifNotExist().create();
 6schema.propertyKey("date").asText().ifNotExist().create();
 7schema.propertyKey("price").asDouble().ifNotExist().create();
 8
 9schema.vertexLabel("person").properties("name", "age", "city").primaryKeys("name").nullableKeys("age", "city").ifNotExist().create();
10schema.vertexLabel("software").useCustomizeNumberId().properties("name", "lang", "price").ifNotExist().create();
11
12schema.indexLabel("personByAge").onV("person").by("age").range().ifNotExist().create();
13schema.indexLabel("personByCity").onV("person").by("city").secondary().ifNotExist().create();
14schema.indexLabel("personByAgeAndCity").onV("person").by("age", "city").secondary().ifNotExist().create();
15schema.indexLabel("softwareByPrice").onV("software").by("price").range().ifNotExist().create();
16
17schema.edgeLabel("knows").sourceLabel("person").targetLabel("person").properties("date", "weight").ifNotExist().create();
18schema.edgeLabel("created").sourceLabel("person").targetLabel("software").properties("date", "weight").ifNotExist().create();
19
20schema.indexLabel("createdByDate").onE("created").by("date").secondary().ifNotExist().create();
21schema.indexLabel("createdByWeight").onE("created").by("weight").range().ifNotExist().create();
22schema.indexLabel("knowsByWeight").onE("knows").by("weight").range().ifNotExist().create();

person uses name as its primary key. Tom’s age and city are removed by null_values, so both properties must be nullable. software uses custom numeric IDs: mapping id supplies the vertex ID and created.target_id supplies the edge’s target ID. Run this example in an empty graph without these schema labels; ifNotExist() does not replace incompatible existing definitions.

4.3 Write the input source mapping file example/file/struct.json

 1{
 2  "vertices": [
 3    {
 4      "label": "person",
 5      "input": {
 6        "type": "file",
 7        "path": "example/file/vertex_person.csv",
 8        "format": "CSV",
 9        "header": ["name", "age", "city"],
10        "charset": "UTF-8",
11        "skipped_line": {
12          "regex": "(^#|^//).*"
13        }
14      },
15      "null_values": ["NULL", "null", ""]
16    },
17    {
18      "label": "software",
19      "input": {
20        "type": "file",
21        "path": "example/file/vertex_software.txt",
22        "format": "TEXT",
23        "delimiter": "|",
24        "charset": "GBK"
25      },
26      "id": "id",
27      "ignored": ["ISBN"]
28    }
29  ],
30  "edges": [
31    {
32      "label": "knows",
33      "source": ["source_name"],
34      "target": ["target_name"],
35      "input": {
36        "type": "file",
37        "path": "example/file/edge_knows.json",
38        "format": "JSON",
39        "date_format": "yyyyMMdd"
40      },
41      "field_mapping": {
42        "source_name": "name",
43        "target_name": "name"
44      }
45    },
46    {
47      "label": "created",
48      "source": ["source_name"],
49      "target": ["target_id"],
50      "input": {
51        "type": "file",
52        "path": "example/file/edge_created.json",
53        "format": "JSON",
54        "date_format": "yyyy-MM-dd"
55      },
56      "field_mapping": {
57        "source_name": "name"
58      }
59    }
60  ]
61}

4.4 Command to import

1sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy

After the import is complete, statistics similar to the following will appear:

 1vertices/edges has been loaded this time : 8/6
 2--------------------------------------------------
 3count metrics
 4     input read success            : 14
 5     input read failure            : 0
 6     vertex parse success          : 8
 7     vertex parse failure          : 0
 8     vertex insert success         : 8
 9     vertex insert failure         : 0
10     edge parse success            : 6
11     edge parse failure            : 0
12     edge insert success           : 6
13     edge insert failure           : 0

4.5 Use Docker to load data

4.5.1 Use docker exec to load data directly

4.5.1.1 Prepare data

For a first import, use the built-in example dataset.

For custom data, first copy the files into the Loader container.

Prepare the data, schema and mapping as described in sections 4.1 to 4.3, then copy them with docker cp.

The following example stores the files in hugegraph-dataset.

1tree -f hugegraph-dataset/
2
3hugegraph-dataset
4├── hugegraph-dataset/edge_created.json
5├── hugegraph-dataset/edge_knows.json
6├── hugegraph-dataset/schema.groovy
7├── hugegraph-dataset/struct.json
8├── hugegraph-dataset/vertex_person.csv
9└── hugegraph-dataset/vertex_software.txt

Copy the files into the container.

1docker cp hugegraph-dataset loader:/loader/dataset
2docker exec -it loader ls /loader/dataset
3
4edge_created.json  edge_knows.json  schema.groovy  struct.json  vertex_person.csv  vertex_software.txt

4.5.1.2 Data loading

Run this command to import the built-in dataset.

For custom data, set -f to your mapping file and -s to your schema file.

You can refer to 3.4.1-Parameter description for the rest of the parameters.

1docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h server -p 8080

If loading a custom dataset, following the previous example, you would use:

1docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f /loader/dataset/struct.json -s /loader/dataset/schema.groovy -h server -p 8080

Also update every input.path in the custom struct.json to its actual container path, such as /loader/dataset/vertex_person.csv. Changing only -f and -s does not change where Loader looks for data files.

If loader and server are in the same Docker network, you can specify -h {server_container_name}; otherwise, you need to specify the IP of the server host (in our example, server_container_name is server).

Then we can see the result:

 1HugeGraphLoader worked in NORMAL MODE
 2vertices/edges loaded this time : 8/6
 3--------------------------------------------------
 4count metrics
 5    input read success            : 14                  
 6    input read failure            : 0                   
 7    vertex parse success          : 8                   
 8    vertex parse failure          : 0                   
 9    vertex insert success         : 8                   
10    vertex insert failure         : 0                   
11    edge parse success            : 6                   
12    edge parse failure            : 0                   
13    edge insert success           : 6                   
14    edge insert failure           : 0                   
15--------------------------------------------------
16meter metrics
17    total time                    : 0.199s              
18    read time                     : 0.046s              
19    load time                     : 0.153s              
20    vertex load time              : 0.077s              
21    vertex load rate(vertices/s)  : 103                 
22    edge load time                : 0.112s              
23    edge load rate(edges/s)       : 53   

Query the imported data with Hubble or curl.

These requests use the GraphSpace-aware Server API. For an older Server, omit /graphspaces/DEFAULT from the path.

1curl --compressed "http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/graph/vertices"
2{"vertices":[{"id":1,"label":"software","type":"vertex","properties":{"name":"lop","lang":"java","price":328.0}},{"id":2,"label":"software","type":"vertex","properties":{"name":"ripple","lang":"java","price":199.0}},{"id":"1:tom","label":"person","type":"vertex","properties":{"name":"tom"}},{"id":"1:josh","label":"person","type":"vertex","properties":{"name":"josh","age":32,"city":"Beijing"}},{"id":"1:marko","label":"person","type":"vertex","properties":{"name":"marko","age":29,"city":"Beijing"}},{"id":"1:peter","label":"person","type":"vertex","properties":{"name":"peter","age":35,"city":"Shanghai"}},{"id":"1:vadas","label":"person","type":"vertex","properties":{"name":"vadas","age":27,"city":"Hongkong"}},{"id":"1:li,nary","label":"person","type":"vertex","properties":{"name":"li,nary","age":26,"city":"Wu,han"}}]}

If you want to check the import result of edges, you can use curl --compressed "http://localhost:8080/graphspaces/DEFAULT/graphs/hugegraph/graph/edges".

4.5.2 Enter the docker container to load data

You can also open a shell in the container and run the same import commands.

Open the shell with docker exec -it loader bash, then run the import.

1sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h server -p 8080

The results of the execution will be similar to those shown in 4.5.1.

4.6 Import data by spark-loader

The current master API sink uses Spark 3.5.8, Scala 2.12 and Java 17 on the driver and executors. The POM’s Spark 3.2.2 dependency supplies compile-time APIs; it is not the runtime engine. Set SPARK_HOME and JAVA_HOME, then use the distribution’s shaded Loader JAR with its launcher.

Create the schema first and make data files readable by the executors. CSV and TEXT inputs need headerless UTF-8 data and an explicit mapping header. Other charsets and has_header: true are rejected. Load vertex data before edge data when the edges depend on those vertices. Replace the mapping path below with your prepared vertex mapping.

1bin/hugegraph-spark-loader.sh \
2  --master 'local[2]' --driver-memory 1g \
3  --file /path/to/vertex-mapping.json \
4  --host 127.0.0.1 --port 8080 --graph hugegraph --sink-type true

Pass Spark options such as --master, --conf and --jars to the launcher alongside Loader options. Additional JDBC drivers need explicit --jars; copying them into lib/ does not distribute them to executors. Query the target graph to verify counts and properties; the API sink’s insertion counters do not report distributed writes.

This path rejects --incremental-mode true and --failure-mode true. Standalone Spark supports client deployment only. HBase bulkload (--sink-type false) is unsupported on Spark 3.5.8 / Java 17; keep an existing bulkload deployment on its separately validated released Loader/JVM/Spark/HBase environment. For details, see the Spark migration guide.

4.7 Import data by flink-cdc-loader

The current master integration targets Flink 2.2.1, MySQL CDC 3.6.0-2.2 and Java 17. Older examples using Flink 1.13.5 and CDC 2.2.1 belong to the previous implementation. Set FLINK_HOME; the submission JVM, JobManager and TaskManager must all use Java 17. Keep Flink’s module-access options and exactly one shaded Loader JAR in the distribution’s lib/.

Each mapping needs a MySQL JDBC input with url, database, table, username and password. Set MySQL binlog_row_image=FULL. UPDATE events require explicit, nonempty update_strategies and unchanged graph IDs; changes to vertex IDs, edge endpoints or sort keys are rejected. CDC also rejects --dry-run true and --use-prefilter true.

Without a separator, all arguments go to Loader. To set Flink options, put them before -- and Loader options after it. For a prepared mapping at mysql-cdc.json, submit with one sink writer.

1bin/hugegraph-flinkcdc-loader.sh -p 1 -- \
2  --file ./mysql-cdc.json --host 127.0.0.1 --port 8080 \
3  --graph hugegraph --cdc-sink-parallelism 1

Each mapping uses a single source reader and sink writer. --cdc-sink-parallelism defaults to 1; other values are rejected. After --, Loader’s -p means the Server port. Checkpoint flushing does not provide transactional graph writes or an exactly-once guarantee. Validate edge CDC and broader mappings in your test environment before deploying to production. For more details, see the Flink CDC documentation.

5 - Graph export and migration

Choose an export or migration tool when you need to back up, export, or move data between graphs. Tools is suited to standalone operations and backups; SeaTunnel Source connects graph reads to an expandable data pipeline.

• Export and manage graphs with Tools
• Export and migrate graph data with SeaTunnel Source

5.1 - Export and Migrate Graph Data with SeaTunnel Source

If you need to copy data from one HugeGraph graph to another, use graph2graph: HugeGraph Source reads vertices and edges from the source graph (graph A), and HugeGraph Sink writes them to the target graph (graph B), with an optional Transform in between. The data path is graph A → HugeGraph Source → (optional Transform) → HugeGraph Sink → graph B. If you need to export graph data to a file, JDBC, Kafka, or another system, use graph2any, where a downstream Sink receives the records read by HugeGraph Source. This page covers both job types.

Version requirement: This guide targets SeaTunnel 3.0+

SeaTunnel 3.0+ provides HugeGraph Source, with schema auto-discovery and multi-label reads. See the official Source documentation for parallel-scan backend requirements and configuration limits.

Before starting, complete the shared environment and configuration steps on the import page. They cover JDK, HOCON, plugin installation, and the sample graph model.

1 Migrate a HugeGraph graph (graph2graph)

The following example migrates person vertices and knows edges from a source graph. Use a separate target graph. This section uses CUSTOMIZE_STRING to preserve vertex IDs. Do not reuse the person label created earlier with PRIMARY_KEY.

These two jobs migrate only the selected labels and properties. They do not copy every source schema setting, such as indexes and TTLs. Pause writes to the source graph during the migration so both jobs read a consistent point in time. Afterward, compare vertex and edge counts and sample properties.

1.1 Migrate vertices first

Source adds a ~id column for the original ID, and Sink stores it as a string. Do not declare ~id in schema.fields; manually declaring this reserved column is rejected.

 1env {
 2  job.mode = "BATCH"
 3}
 4
 5source {
 6  HugeGraph {
 7    host = "source-hugegraph"
 8    port = 8080
 9    graph_name = "hugegraph"
10    graph_space = "DEFAULT"
11    label = "person"
12    label_type = "VERTEX"
13    schema = {
14      fields = {
15        name = "string"
16        age = "int"
17      }
18    }
19  }
20}
21
22sink {
23  HugeGraph {
24    host = "target-hugegraph"
25    port = 8080
26    graph_name = "hugegraph"
27    graph_space = "DEFAULT"
28    batch_failure_fallback = false
29    mappings = [
30      {
31        type = "VERTEX"
32        label = "person"
33        idStrategy = "CUSTOMIZE_STRING"
34        idFields = ["~id"]
35        properties = ["name", "age"]
36      }
37    ]
38  }
39}

1./bin/seatunnel.sh --config ./config/graph2graph-person.conf -m local

1.2 Migrate edges second

After the vertex job succeeds, use the ~source_id and ~target_id columns added by Source to locate endpoints. Because the previous job preserved the original IDs, these columns can refer directly to vertices in the target graph.

 1env {
 2  job.mode = "BATCH"
 3}
 4
 5source {
 6  HugeGraph {
 7    host = "source-hugegraph"
 8    port = 8080
 9    graph_name = "hugegraph"
10    graph_space = "DEFAULT"
11    label = "knows"
12    label_type = "EDGE"
13    schema = {
14      fields = {
15        since = "int"
16      }
17    }
18  }
19}
20
21sink {
22  HugeGraph {
23    host = "target-hugegraph"
24    port = 8080
25    graph_name = "hugegraph"
26    graph_space = "DEFAULT"
27    check_vertex = true
28    batch_failure_fallback = false
29    mappings = [
30      {
31        type = "EDGE"
32        label = "knows"
33        sourceConfig = {
34          label = "person"
35          idFields = ["~source_id"]
36        }
37        targetConfig = {
38          label = "person"
39          idFields = ["~target_id"]
40        }
41        properties = ["since"]
42      }
43    ]
44  }
45}

1./bin/seatunnel.sh --config ./config/graph2graph-knows.conf -m local

This example checks endpoints and makes write errors fail the job. The default check_vertex = false does not guarantee a consistent result: a missing endpoint can create a dangling edge, so a successful job is not a substitute for checking the migrated graph.

Why preserve IDs? A HugeGraph PRIMARY_KEY ID contains the internal ID of the vertex label, and that internal ID can differ between graphs. For example, a source vertex can be 1:marko, while regenerating the primary key in the target graph can produce 2:marko. Reusing the source edge endpoints after regenerating vertex IDs can connect edges to the wrong vertices. This example stores the original ID as a string, which changes the target graph’s ID strategy

When Source reads every label, omit label to read all labels of label_type (default VERTEX). It produces one output table per label. Bind each Sink mapping to its table with sourceTable, for example sourceTable = "default.person"; use the full table name shown in the Writer log for the exact value. Do not reuse the single-label configuration from this section. See the HugeGraph Source documentation for other limitations.

2 Export to another system (graph2any)

graph2any uses HugeGraph Source^[1] to read vertices or edges and sends them to a downstream Sink. The example below exports person vertices to local JSON files; to export to JDBC, Kafka, or another system, replace LocalFile^[2] and its options.

 1env {
 2  job.mode = "BATCH"
 3}
 4
 5source {
 6  HugeGraph {
 7    host = "hugegraph"
 8    port = 8080
 9    graph_name = "hugegraph"
10    graph_space = "DEFAULT"
11    label = "person"
12    label_type = "VERTEX"
13    schema = {
14      fields = {
15        name = "string"
16        age = "int"
17      }
18    }
19  }
20}
21
22sink {
23  LocalFile {
24    path = "/tmp/hugegraph-export/${table_name}"
25    file_format_type = "json"
26  }
27}

Save this as config/graph2file-person.conf and run it from the SeaTunnel installation directory:

1./bin/seatunnel.sh --config ./config/graph2file-person.conf -m local

To export edges, change the Source label to an edge label, set label_type = "EDGE", and declare the edge properties in schema.fields. Source also outputs the reserved columns ~source_id, ~source_label, ~target_id, and ~target_label; write them to the file or pass them to downstream transforms as needed.

This page covers row reads and writes. It does not copy source indexes, TTLs, or other schema settings. For the complete Source options and shared environment guidance, return to the SeaTunnel graph import guide^[3].

3 References

Connectors

^[1] HugeGraph Source
^[2] LocalFile Sink

Related guide

^[3] SeaTunnel graph import guide

6 - Tools Quick Start

1 HugeGraph-Tools Overview

HugeGraph-Tools manages graphs and tasks and provides backup and restore commands. It also retains deployment commands for legacy Server/Studio versions.

Testing Guide: For running HugeGraph-Tools tests locally, please refer to HugeGraph Toolchain Local Testing Guide

2 Get HugeGraph-Tools

HugeGraph-Tools is included in the Toolchain distribution. You can download the distribution or build it from source.

• Download the compiled tarball
• Clone source code then compile and install

2.1 Download the compiled archive

The following example uses the released HugeGraph-Toolchain 1.7.0 package. Check the download page for releases.

1export VERSION=1.7.0
2export ARCHIVE="apache-hugegraph-toolchain-incubating-${VERSION}"
3wget "https://downloads.apache.org/hugegraph/${VERSION}/${ARCHIVE}.tar.gz"
4tar zxf "${ARCHIVE}.tar.gz"
5# hugegraph-tools ships inside the toolchain package, in a directory
6# whose version suffix is the same as the archive's
7cd "${ARCHIVE}/apache-hugegraph-tools-incubating-${VERSION}"

2.2 Clone source code to compile and install

Choose one source below. Both build the released 1.7.0 Tools; install Maven and JDK 11 first. The archive route also requires wget. Each example ends in the generated Tools directory, where bin/hugegraph help lists the available commands.

1git clone --branch 1.7.0 --single-branch https://github.com/apache/hugegraph-toolchain.git
2cd hugegraph-toolchain
3mvn package -pl hugegraph-tools -am -DskipTests -ntp
4cd hugegraph-tools/apache-hugegraph-tools-incubating-1.7.0
5bin/hugegraph help

1export VERSION=1.7.0
2export ARCHIVE="apache-hugegraph-toolchain-incubating-${VERSION}"
3wget "https://downloads.apache.org/hugegraph/${VERSION}/${ARCHIVE}-src.tar.gz"
4tar zxf "${ARCHIVE}-src.tar.gz"
5cd "${ARCHIVE}-src"
6mvn package -pl hugegraph-tools -am -DskipTests -ntp
7cd "hugegraph-tools/apache-hugegraph-tools-incubating-${VERSION}"
8bin/hugegraph help

The build also writes hugegraph-tools/target/apache-hugegraph-tools-incubating-1.7.0.tar.gz relative to the source root.

2.3 Build current source

To use unreleased features from the master branch, build Tools from source (master is currently based on 1.8; requires JDK 17, Maven 3.6.3+, and Python 3.9+):

1. Clone Toolchain source

   1git clone --branch master --single-branch https://github.com/apache/hugegraph-toolchain.git
   2cd hugegraph-toolchain

2. Build matching Server SDK

   Run the script below to download Server source and install the SDK into an isolated Maven repository:

   1java -version
   2mvn -version
   3candidate_dir=$(mktemp -d)
   4mkdir -p "$candidate_dir/m2"
   5bash hugegraph-client/assembly/travis/install-candidate-sdk.sh \
   6  master "$candidate_dir/server" "$candidate_dir/m2"

   See the Java 17 migration guide for details.

3. Package Tools

   1mvn -Dmaven.repo.local="$candidate_dir/m2" \
   2  -Dsdk.validation.mode=candidate \
   3  package -pl hugegraph-tools -am -DskipTests -Dmaven.javadoc.skip=true -ntp
   4cd hugegraph-tools/apache-hugegraph-tools-*
   5bin/hugegraph help

Keep the SDK repository while using this source build. The master build directory does not contain incubating.

3 How to use

3.1 Function overview

From the extracted or generated Tools directory, run bin/hugegraph help for the command list, or bin/hugegraph help <sub-command> for one command. The commands cover these tasks.

• Graph management type, graph-mode-set, graph-mode-get, graph-list, graph-get, graph-clear, graph-create, graph-clone and graph-drop
• Asynchronous task management type, task-list, task-get, task-delete, task-cancel and task-clear
• Gremlin type, gremlin-execute and gremlin-schedule
• Backup/Restore type, backup, restore, migrate, schedule-backup and dump
• Authentication data backup/restore type, auth-backup and auth-restore
• Legacy Server/Studio deployment, deploy, clear, start-all and stop-all

1Usage: hugegraph [options] [command] [command options]

3.2 [options]-Global Variable

Set these global connection options on the command line or through the environment variables below.

• --graph selects the graph, default hugegraph.
• --url sets the Server address, default http://127.0.0.1:8080.
• --user supplies the username when Server authentication is enabled.
• --password supplies that user’s password.
• --timeout sets the connection timeout, default 30 seconds.
• --trust-store-file selects the HTTPS truststore. An empty value uses the built-in conf/hugegraph.truststore.
• --trust-store-password supplies its password. An empty value uses the built-in truststore password.
• --throw-mode defaults to false, which prints errors and exits. Set it to true to throw exceptions, mainly for tests.