HugeGraph ToolChain

• 1: HugeGraph-Hubble Quick Start
• 2: HugeGraph-Loader Quick Start
• 3: HugeGraph-Tools Quick Start
• 4: HugeGraph-Spark-Connector Quick Start

1 - HugeGraph-Hubble Quick Start

1 HugeGraph-Hubble Overview

⚠️ Security notice: Hubble listens on plain HTTP. Do not expose it to the public Internet or untrusted networks; terminate HTTPS in front of it and restrict access with IP/port allowlists. Hubble keeps no account database of its own: when the connected HugeGraph Server has authentication enabled, Hubble shows a sign-in page and forwards the credentials to the Server; when the Server allows anonymous access, there is no sign-in and the account pages are hidden.

Version note: This page follows hugegraph-toolchain master. Features that depend on newer Server, PD or Store versions are marked below and are unavailable on older Servers.

Testing Guide: For running HugeGraph-Hubble tests locally, please refer to HugeGraph Toolchain Local Testing Guide

HugeGraph-Hubble is HugeGraph’s web management interface. It connects to one HugeGraph Server, either directly or through PD in a distributed cluster, manages GraphSpaces, graphs and schemas, imports data, runs Gremlin and Cypher queries and built-in graph algorithms, and visualizes the results.

The platform mainly includes the following modules:

Graph Overview

Graph Overview lists GraphSpaces (in PD mode) and graphs. It creates, clones and clears graphs, loads demo graphs, opens the graph detail page with statistics and schema, and jumps to the query workbench.

Metadata Modeling

Metadata Modeling manages PropertyKeys, VertexLabels, EdgeLabels and IndexLabels of one graph, in list and graph views. Schema Templates keep reusable Groovy schemas per GraphSpace that can be applied when a graph is created.

Data Import

The data import page is intended for small-scale trials. For bulk or production imports, use HugeGraph Loader.

Data Sources register FILE, HDFS, JDBC and KAFKA sources. Import tasks are configured in four steps and can run once, on a cron schedule, or continuously for Kafka.

Graph Query

Graph Query runs Gremlin and Cypher statements in immediate or asynchronous mode and displays results as a graph (2D or 3D), a table, or JSON. It keeps execution records and favorite statements.

Built-in Graph Algorithms

Built-in Graph Algorithms provides forms for the Server’s OLTP traverser APIs (interactive exploration) and for OLAP jobs (cluster batch computation through HugeGraph Computer or Vermeer).

Async Tasks

Async Tasks lists background tasks such as Gremlin and Cypher tasks, algorithm tasks, metadata removal, index creation and rebuild, and Vermeer load or compute tasks, with detail, cancel and delete actions.

System and Operations

System and Operations covers the personal profile, account management with GraphSpace permission presets, and, in PD mode, the cluster overview and node details.

1.1 Compatibility

Hubble detects the authentication mode and the capabilities of the connected Server; there is no separate authentication switch in Hubble. The supported combinations are:

Use matching Server, PD and Store minor versions in a distributed cluster.

2 Deploy

There are three ways to deploy hugegraph-hubble

• Use Docker (Convenient for Test/Dev)
• Download the Toolchain binary package
• Source code compilation

Hubble runs on Java 11: the backend is compiled with java.version=11 and the Docker image is based on eclipse-temurin:11-jre. bin/start-hubble.sh only checks that a java binary is on the PATH, so make sure the right JDK is selected.

2.1 Use docker (Convenient for Test/Dev)

Special Note: Hubble no longer asks for the Server host and port on the web page. The Server address comes from conf/hugegraph-hubble.properties: server.direct_url when pd.enabled=false, or PD discovery through pd.peers when pd.enabled=true. Inside the container 127.0.0.1 refers to the hubble container itself, so the packaged default server.direct_url=http://127.0.0.1:8080 does not reach a Server running in another container.

If hubble and server are in the same docker network, we recommend using the container_name (in our example, it is server) as the hostname, and 8080 as the port. Or you can use the host IP as the hostname, and the port is configured by the host for the server.

The image copies the packaged distribution to /hubble, rewrites server.host=0.0.0.0 and clears dashboard.address in /hubble/conf/hugegraph-hubble.properties, exposes port 8088 and runs ./bin/start-hubble.sh -f in the foreground.

Prepare a hugegraph-hubble.properties that points at your Server and keeps the container listening on all interfaces:

server.host=0.0.0.0
server.port=8088
pd.enabled=false
server.direct_url=http://server:8080

Then start hubble with that file mounted over the packaged configuration:

docker run -itd --name=hubble -p 8088:8088 \
  -v "$PWD/hugegraph-hubble.properties:/hubble/conf/hugegraph-hubble.properties" \
  hugegraph/hubble:1.7.0

Alternatively, you can use Docker Compose to start hubble. Additionally, if hubble and the graph is in the same Docker network, you can access the graph using the container name of the graph, eliminating the need for the host machine’s IP address.

Use docker-compose up -d, docker-compose.yml is following:

version: '3'
services:
  server:
    image: hugegraph/hugegraph:1.7.0
    container_name: server
    environment:
      - PASSWORD=xxx
    ports:
      - 8080:8080

  hubble:
    image: hugegraph/hubble:1.7.0
    container_name: hubble
    ports:
      - 8088:8088
    volumes:
      - ./hugegraph-hubble.properties:/hubble/conf/hugegraph-hubble.properties

Note:

1. The docker image of hugegraph-hubble is a convenience release to start hugegraph-hubble quickly, but not official distribution artifacts. You can find more details from ASF Release Distribution Policy.

2. Recommend to use release tag (like 1.7.0) for the stable version. Use latest tag to experience the newest functions in development.

2.2 Download the Toolchain binary package

hubble is in the toolchain project. First, download the binary tar tarball

export VERSION=1.7.0
export ARCHIVE="apache-hugegraph-toolchain-incubating-${VERSION}"
wget "https://downloads.apache.org/hugegraph/${VERSION}/${ARCHIVE}.tar.gz"
tar -xvf "${ARCHIVE}.tar.gz"
cd "${ARCHIVE}/apache-hugegraph-hubble-incubating-${VERSION}"

Edit conf/hugegraph-hubble.properties so that the Server address is correct, then run hubble

bin/start-hubble.sh

start-hubble.sh accepts the following options:

The script starts the JVM with -Xms512m -Dfile.encoding=UTF-8 -Dhubble.home.path=<install dir>, writes the PID to bin/pid, logs to logs/hugegraph-hubble.log and waits up to 30 seconds for http://<server.host>:<server.port>/about to answer before it returns.

The packaged default is server.host=localhost, so the service only accepts loopback connections until you change it. After startup, open http://<host>:8088.

Run bin/stop-hubble.sh to stop the service. It sends SIGTERM first so that the shutdown hooks pause running load tasks and close the embedded H2 database cleanly, and only escalates to SIGKILL if the process is still alive after STOP_TIMEOUT seconds (environment variable, default 30).

2.3 Source code compilation

Hubble’s build uses frontend-maven-plugin in hugegraph-hubble/hubble-dist/pom.xml to install Node.js v18.20.8 and Yarn v1.22.21, so neither tool needs to be installed beforehand. JDK 11 and Maven are required.

Download the toolchain source code.

git clone https://github.com/apache/hugegraph-toolchain.git

Compile hubble. It depends on the loader and client, so you need to build these dependencies in advance during the compilation process (you can skip this step later).

cd hugegraph-toolchain
python -m pip install -r hugegraph-hubble/hubble-dist/assembly/travis/requirements.txt
mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp
cd hugegraph-hubble
mvn -e compile package -Dmaven.javadoc.skip=true -Dmaven.test.skip=true -ntp
cd apache-hugegraph-hubble-*

Run hubble

bin/start-hubble.sh -d

For frontend work, run yarn dev inside hubble-fe. The backend POM does not configure spring-boot:run, so run org.apache.hugegraph.HugeGraphHubble from hubble-be/target/classes with -Dhubble.home.path pointing at a writable directory instead.

3 Platform Workflows

The home page groups the modules into three journeys: Graph Overview, Graph Import and Graph Query. It also shows whether Hubble runs in PD / cluster mode or in non-PD standalone mode. The module usage process of the platform is as follows:

4 Platform Instructions

4.1 Graph Management

In PD mode, [Graph Space Management] lists all GraphSpaces of the cluster and can create or edit one, including its alias, optional Kubernetes namespace and compute task, and resource limits. In non-PD standalone mode there is exactly one GraphSpace named DEFAULT and the GraphSpace list is skipped.

4.1.1 Graph creation

Under the graph management module, click [New Graph] and fill in the graph name, an optional alias, an optional schema template and optional sample data. The graph name is unique inside its GraphSpace and cannot be changed after creation.

Create graph by filling in the content as follows:

Special Note: The Server connection is not configured on this page. It comes from conf/hugegraph-hubble.properties, through server.direct_url or PD discovery; see section 2.1 for the Docker hostname rules. Graph creation is only offered when the connected Server exposes it (REST API 0.67 or later); on older Servers the graph list is read-only.

4.1.2 Graph Access

Realize the information access to the graph space. After entering, you can perform operations such as multidimensional query analysis, metadata management, data import, and algorithm analysis of the graph. [Open Graph Studio] opens the query workbench, [Metadata Config] opens the schema pages, and the graph detail page shows vertex and edge statistics together with the schema.

4.1.3 Graph management

1. The graph list has a card view and a list view. Search matches the graph name.
2. Per-graph actions are View Schema (with [Export Groovy Schema]), Metadata Config, Clone Graph (schema only, or schema and data), Clear Schema and Data, Delete, and, in PD mode, Set Default.
3. [Sample data and resources] builds a demo graph inside the current graph: the Red Chamber demo graph, the People & Software demo graph, or the Tiny Movie Rank demo. These demos add missing schema and elements only and never clear existing data.

4.2 Metadata Modeling (list + graph mode)

4.2.1 Module entry

Open [Metadata Config] from the graph list, or the metadata page of a graph at /graphspace/<graphspace>/graph/<graph>/meta. The page has five tabs, Property, Vertex Type, Edge Type, Vertex Index and Edge Index, and a switch between list view and graph view.

4.2.2 Property type

4.2.2.1 Create type

1. Fill in or select the property name, data type, and cardinality to complete the creation of the property.
2. Created properties can be used as properties of vertex type and edge type.

List mode:

Graph mode:

4.2.2.2 Management

1. You can delete a single item or delete it in batches in the property list. A property that is still used by a vertex or edge type cannot be deleted.
2. Deleting metadata runs as an asynchronous task; check Async Tasks for its progress.

4.2.3 Vertex type

4.2.3.1 Create type

1. Fill in or select the vertex type name, ID strategy, associated properties, primary key properties, vertex style, content displayed below the vertex in the query result, and index information: including whether to create a type index, and the specific content of the property index, complete the vertex type creation.

List mode:

Graph mode:

4.2.3.2 Administration

1. Editing operations are available. The vertex style, associated properties, vertex display content, and property index can be edited, and the rest cannot be edited. In graph mode, double-click a vertex type to edit it.

2. You can delete a single item or delete it in batches.

4.2.4 Edge Types

4.2.4.1 Create

1. Fill in or select the edge type name, the type (Normal, Parent or Sub, for edge type hierarchies), start point type, end point type, associated properties, whether to allow multiple connections, edge style, content displayed below the edge in the query result, and index information: including whether to create a type index, and the specific content of the property index, complete the creation of the edge type.

List mode:

Graph mode:

4.2.4.2 Administration

1. Editing operations are available. Edge styles, associated properties, edge display content, and property indexes can be edited, and the rest cannot be edited, the same as the vertex type.
2. You can delete a single item or delete it in batches.

4.2.5 Index Types

Displays vertex and edge indexes for vertex types and edge types. Secondary, range, search and unique indexes are supported.

4.2.6 Schema Templates

[Schema templates] at /graphspace/<graphspace>/schema keeps a reusable template library for the current GraphSpace. Example templates ship with Hubble and can be used, removed or restored; they are not stored on the Server until you save one. User templates hold Groovy schema on the Server and can be created, edited and deleted. When you create a graph, an existing template can be selected so that its schema is applied right away.

4.3 Data Import

Note: currently, we recommend to use hugegraph-loader to import data formally. The built-in import of hubble is used for testing and getting started.

The usage process of data import is as follows:

4.3.1 Module entrance

Left navigation, under Graph Import: [Data Sources] and [Data Import].

4.3.2 Data sources

1. [Data Sources] registers where an import task reads from. Four source types are supported: FILE (local upload), HDFS, Kafka and JDBC.
2. For a FILE source, upload the files that need to be composed. The accepted formats come from upload_file.format_list, which defaults to csv and txt.
3. The single file and total size limits default to 1 GB and 10 GB, and unfinished uploads are discarded after upload_file.max_uploading_time, which defaults to 12 hours.

4.3.3 Create task

1. [Data Import] > [Create Task] configures an import in four steps: Basic Information, Select Source Fields, Select Mapping Fields and Schedule.
2. Basic Information takes the task name (1 to 48 Chinese characters, letters, digits or _), the target GraphSpace and graph, the source type and the data source.
3. Multiple import tasks can be created and imported in parallel.

4.3.4 Setting up data mapping

1. Set up data mapping for the selected source, including file settings and type settings

2. File settings: check or fill in whether to include the header, separator, encoding format and other settings of the source itself, all set the default values, no need to fill in manually

3. Type setting:

   1. Vertex map and edge map:

      【Vertex Type】: Select the vertex type, and map the column data of the source for its ID;

      【Edge Type】: Select the edge type and map the column data of the source to the ID column of its start point type and end point type;

   2. Mapping settings: map the column data of the source to the properties of the selected vertex type. Here, if the property name is the same as the header name of the file, the mapping property can be automatically matched, and there is no need to manually fill in the selection.

   3. After completing the setting, the setting list will be displayed before proceeding to the next step. It supports the operations of adding, editing and deleting mappings.

Fill in the settings map:

Mapping list:

4.3.5 Import data

The last step chooses when the task runs: Run Once for a one-off import, Scheduled with a Quartz cron expression such as 0 0/5 * * * ?, or Realtime for a Kafka source.

1. Import settings

• The import setting parameter items are as shown in the figure below, all set the default value, no need to fill in manually

2. Import details

• Run a task from the task list to start the import, and pause, edit or delete it from the same list
• The execution history of a task provides the execution instance ID, the number of imported records, the average rate in records per second, the import duration and the status of each run
• If the import fails, you can view the specific reason

4.4 Graph Query

4.4.1 Module entry

Left navigation, under Graph Query: [GQL Traversal].

4.4.2 Multi-graphs switching

The top bar carries the current GraphSpace and graph, so you can flexibly switch the operation space of multiple graphs without leaving the page.

4.4.3 Graph Analysis and Processing

HugeGraph supports Gremlin, a graph traversal query language of Apache TinkerPop3. Gremlin is a general graph database query language. By entering Gremlin statements and clicking execute, you can perform query and analysis operations on graph data, and create and delete vertices/edges, modify vertex/edge properties, etc. When the connected Server supports Cypher, a Cypher tab is offered next to Gremlin. A Text2GQL tab is present as a user interface preview only: it is not connected to a model or a query service, and nothing entered there is sent or executed.

Each statement can run in one of two modes. Immediate returns the result inline and suits analyses that finish within about 30 seconds; Async submits a task instead, and its progress and result appear under Async Tasks. Ctrl/Command + Enter runs the current statement.

After the query, below is the graph result display area, which provides 3 kinds of graph result display modes: [Graph Mode], [Table Mode], [Json Mode]. The graph canvas can be rendered in 2D or 3D.

⚠️ SEC Reminder: Hubble allows the direct input and execution of native Gremlin query statements on the web interface, which grants users relatively high operational privileges. Please avoid exposing the Hubble service to public network environments. It is recommended to ensure that the graph database server has enabled the Authentication System (Auth) combined with an IP Whitelist for strict permission control when in use, preventing unauthorized access or malware execution risks.

Support zoom, center, full screen, layout and style configuration, legend, minimap, undo and redo, and export operations. The canvas can be exported as JSON, CSV or an image, and a previously exported canvas can be imported again.

【Picture Mode】

【Table mode】

【Json mode】

4.4.4 Data Details

Click the vertex/edge entity to view the data details of the vertex/edge, including vertex/edge type, vertex ID, attribute and corresponding value, expand the information display dimension of the graph, and improve the usability.

4.4.5 Multidimensional Path Query of Graph Results

In addition to the global query, an in-depth customized query and hidden operations can be performed for the vertices in the query result to realize customized mining of graph results.

Right-click a vertex, and the menu entry of the vertex appears, which can be displayed, inquired, hidden, etc.

• Expand: Click to display the vertices associated with the selected point.
• Query: By selecting the edge type and edge direction associated with the selected point, and then selecting its attributes and corresponding filtering rules under this condition, a customized path display can be realized.
• Hide: When clicked, hides the selected point and its associated edges.

Double-clicking a vertex also displays the vertex associated with the selected point.

4.4.6 Add vertex/edge

4.4.6.1 Added vertex

In the graph area, two entries can be used to dynamically add vertices, as follows:

1. Click on the graph area panel, the Add Vertex entry appears
2. Click the first icon in the action bar in the upper right corner

Complete the addition of vertices by selecting or filling in the vertex type, ID value, and attribute information.

The entry is as follows:

Add the vertex content as follows:

4.4.6.2 Add edge

Right-click a vertex in the graph result to add the outgoing or incoming edge of that point.

4.4.7 Execute the query of records and favorites

1. Record each query record at the bottom of the graph area, including: query time, execution type, content, status, time-consuming, as well as [collection] and [load] operations, to achieve a comprehensive record of graph execution, with traces to follow, and Can quickly load and reuse execution content
2. Provides the function of collecting sentences, which can be used to collect frequently used sentences, which is convenient for fast calling of high-frequency sentences.

4.5 Async Tasks

4.5.1 Module entry

Left navigation, under Graph Query: [Async Tasks].

4.5.2 Task Management

1. Provide unified management and result viewing of asynchronous tasks. The task types are:

• gremlin: Gremlin tasks
• cypher: Cypher tasks
• computer-dis: algorithm tasks
• remove_schema: remove metadata
• create_index: create an index
• rebuild_index: rebuild the index
• vermeer-task:load: Vermeer graph load tasks
• vermeer-task:compute: Vermeer graph compute tasks

2. The list displays the asynchronous task information of the current graph, including task ID, task name, task type, creation time, time-consuming, status, operation, and realizes the management of asynchronous tasks. The list refreshes every 5 seconds.
3. Support filtering by task type and status
4. Support searching for task ID and task name
5. A running task can be cancelled, and asynchronous tasks can be deleted one by one or in batches

4.5.3 Gremlin asynchronous tasks

1. Create a task

• The graph query module supports two execution modes, immediate query and asynchronous task; if the user switches to the asynchronous mode, after clicking execute, an asynchronous task will be created in the asynchronous task center. A Cypher statement creates a Cypher task in the same way;

2. Task submission

• After the task is submitted successfully, the graph area returns the submission result and task ID

3. Mission details

• Provide [View] entry, you can jump to the task details to view the specific execution of the current task After jumping to the task center, the currently executing task line will be displayed directly

Click to view the entry to jump to the task management list, as follows:

4. View the results

• The results are displayed in the form of JSON, and a compact result can be expanded inline

4.5.4 Algorithm tasks

Batch algorithms submitted from [Built-in Graph Algorithms] land here as algorithm tasks, and so do Vermeer graph load and compute tasks. Find a task by ID in the list and open it to follow its progress and result. See section 4.6 for the algorithm forms themselves.

4.5.5 Delete metadata, rebuild index

1. Create a task

• In the metadata modeling module, when deleting metadata, an asynchronous task for deleting metadata can be created

• When editing an existing vertex/edge type operation, when adding an index, an asynchronous task of creating an index can be created

2. Task details

• After confirming/saving, you can jump to the task center to view the details of the current task

4.6 Built-in Graph Algorithms

[Built-in Graph Algorithms] under Graph Query provides parameter forms for the algorithms the Server exposes, grouped by intent: explore neighborhoods, find paths and connections, compare and rank, measure importance, find communities, and analyze graph structure. Each algorithm links to its official API documentation.

Two execution modes are offered:

• Interactive exploration runs the Server’s OLTP traverser APIs and returns results directly. It covers K-out and K-neighbor, shortest path in its single source, weighted, and multi-node forms, paths and all paths, customized and template paths, rings and rays, crosspoints and customized crosspoints, same neighbors, Jaccard similarity, fusiform similarity, Adamic-Adar, resource allocation, egonet, and the rank and neighbor rank APIs.
• Cluster batch computation submits an asynchronous job over the whole graph and reports the result under Async Tasks. It covers PageRank and personal PageRank, degree, closeness and betweenness centrality, K-core, weakly connected components, label propagation, Louvain, triangle count, cluster coefficient, rings detection, subgraph matching and links, with Vermeer variants where the deployment provides Vermeer.

Batch algorithms need a HugeGraph Computer environment, including Kubernetes when the deployment requires it. When Computer cannot be reached, the page says so instead of submitting the task.

4.7 Sign-in and account management

When the connected Server has authentication enabled, Hubble opens the sign-in page at /login. Sign in with a HugeGraph Server account: Hubble forwards the credentials to the Server and keeps the returned token for the browser session, and it stores no accounts of its own. Login attempts are throttled, so after the first three failures for the same account and address, further attempts back off starting at 5 seconds and doubling up to 600 seconds. When the Server allows anonymous access, /login redirects to the home page and the profile and account pages are hidden.

[My Profile] shows the account details and changes the password. [Account Management] is available to accounts that may manage accounts or GraphSpace members. It creates accounts and assigns one of four access presets: Super Administrator, GraphSpace Read-only, GraphSpace Read-write and GraphSpace Administrator. Low-level role, target, access and belong records are not exposed in the interface.

4.8 Cluster operations

In PD mode, [System & Operations] adds [Cluster Overview] and [Node details] for accounts with the matching capabilities. Cluster Overview shows the topology, per-tier node status and cluster facts such as stores online, PD leader, capacity, data size, graphs, partitions and replicas. Node details lists every discovered node with filters for type and status, and opens a node profile with its metrics, leader role and Raft shards. Node details is also available in standalone mode; Cluster Overview needs PD.

An optional external dashboard can be linked from the navigation page through dashboard.address. It is a separate monitoring entry, and leaving it unconfigured does not affect Cluster Overview or Node details.

5 Configuration

HugeGraph-Hubble can be configured through the conf/hugegraph-hubble.properties file.

5.1 Service Configuration

5.2 Server and PD

5.3 Gremlin Query Limits

These settings control query result limits to prevent memory issues:

5.4 File Upload

These keys are not written into the packaged file; add them to override the defaults.

5.5 Cluster Operations

These keys drive the Cluster Overview and Node details pages.

The operations.store.allowed_targets default covers local testing only. A production deployment must list every trusted Store scheme, host and port explicitly, because discovery never adds an origin to this allowlist. HTTPS origins keep their configured hostname for TLS SNI and certificate verification. Supply the PD and Store passwords through a protected deployment configuration rather than the packaged file.

2 - HugeGraph-Loader Quick Start

1 HugeGraph-Loader Overview

HugeGraph-Loader is the data import component of HugeGraph, which can convert data from various data sources into graph vertices and edges and import them into the graph database in batches.

Currently supported data sources include:

• Local disk file or directory, supports TEXT, CSV and JSON format files, supports compressed files
• HDFS file or directory supports compressed files
• Mainstream relational databases, such as MySQL, PostgreSQL, Oracle, SQL Server
• Kafka topic
• An existing HugeGraph graph, used to copy data from one graph into another

Local disk files and HDFS files support resumable uploads.

It will be explained in detail below.

Note: HugeGraph-Loader requires HugeGraph Server service, please refer to HugeGraph-Server Quick Start to download and start Server

Testing Guide: For running HugeGraph-Loader tests locally, please refer to HugeGraph Toolchain Local Testing Guide

2 Get HugeGraph-Loader

HugeGraph-Loader is available in the following three ways:

• Use docker image (Convenient for Test/Dev)
• Download the compiled tarball
• Clone source code then compile and install

2.1 Use Docker image (Convenient for Test/Dev)

We can deploy the loader service using docker run -itd --name loader hugegraph/loader:1.7.0. For the data that needs to be loaded, it can be copied into the loader container either by mounting -v /path/to/data/file:/loader/file or by using docker cp.

Alternatively, to start the loader using docker-compose, the command is docker-compose up -d. An example of the docker-compose.yml is as follows:

version: '3'

services:
  server:
    image: hugegraph/hugegraph:1.7.0
    container_name: server
    ports:
      - 8080:8080

  hubble:
    image: hugegraph/hubble:1.7.0
    container_name: hubble
    ports:
      - 8088:8088

  loader:
    image: hugegraph/loader:1.7.0
    container_name: loader
    # mount your own data here
    # volumes:
      # - /path/to/data/file:/loader/file

The specific data loading process can be referenced under 4.5 User Docker to load data

Note:

1. The docker image of hugegraph-loader is a convenience release to start hugegraph-loader quickly, but not official distribution artifacts. You can find more details from ASF Release Distribution Policy.

2. Recommend to use release tag(like 1.7.0) for the stable version. Use latest tag to experience the newest functions in development.

2.2 Download the compiled archive

Download the latest version of the HugeGraph-Toolchain release package:

export VERSION=1.7.0
export ARCHIVE="apache-hugegraph-toolchain-incubating-${VERSION}"
wget "https://downloads.apache.org/hugegraph/${VERSION}/${ARCHIVE}.tar.gz"
tar zxf "${ARCHIVE}.tar.gz"

2.3 Clone source code to compile and install

Clone the latest version of HugeGraph-Loader source package:

# 1. get from github
git clone https://github.com/apache/hugegraph-toolchain.git

# 2. Download a released source package
export VERSION=1.7.0
export ARCHIVE="apache-hugegraph-toolchain-incubating-${VERSION}"
wget "https://downloads.apache.org/hugegraph/${VERSION}/${ARCHIVE}-src.tar.gz"

mvn install:install-file -Dfile=./ojdbc8.jar -DgroupId=com.oracle -DartifactId=ojdbc8 -Dversion=12.2.0.1 -Dpackaging=jar

Compile and generate tar package:

cd hugegraph-toolchain
mvn clean package -pl hugegraph-loader -am -DskipTests -ntp

3 How to use

The basic process of using HugeGraph-Loader is divided into the following steps:

• Write graph schema
• Prepare data files
• Write input source map files
• Execute command import

3.1 Construct graph schema

This step is the modeling process. Users need to have a clear idea of ​​their existing data and the graph model they want to create, and then write the schema to build the graph model.

For example, if you want to create a graph with two types of vertices and two types of edges, the vertices are “people” and “software”, the edges are “people know people” and “people create software”, and these vertices and edges have some attributes, For example, the vertex “person” has: “name”, “age” and other attributes, “Software” includes: “name”, “sale price” and other attributes; side “knowledge” includes: “date” attribute and so on.

graph model example

After designing the graph model, we can use groovy to write the definition of schema and save it to a file, here named schema.groovy.

// Create some properties
schema.propertyKey("name").asText().ifNotExist().create();
schema.propertyKey("age").asInt().ifNotExist().create();
schema.propertyKey("city").asText().ifNotExist().create();
schema.propertyKey("date").asText().ifNotExist().create();
schema.propertyKey("price").asDouble().ifNotExist().create();

// Create the person vertex type, which has three attributes: name, age, city, and the primary key is name
schema.vertexLabel("person").properties("name", "age", "city").primaryKeys("name").ifNotExist().create();
// Create a software vertex type, which has two properties: name, price, the primary key is name
schema.vertexLabel("software").properties("name", "price").primaryKeys("name").ifNotExist().create();

// Create the knows edge type, which goes from person to person
schema.edgeLabel("knows").sourceLabel("person").targetLabel("person").ifNotExist().create();
// Create the created edge type, which points from person to software
schema.edgeLabel("created").sourceLabel("person").targetLabel("software").ifNotExist().create();

Please refer to the corresponding section in hugegraph-client for the detailed description of the schema.

3.2 Prepare data

The data sources currently supported by HugeGraph-Loader include:

• local disk file or directory
• HDFS file or directory
• Partial relational database
• Kafka topic
• An existing HugeGraph graph

3.2.1 Data source structure

3.2.1.1 Local disk file or directory

The user can specify a local disk file as the data source. If the data is scattered in multiple files, a certain directory is also supported as the data source, but multiple directories are not supported as the data source for the time being.

For example, my data is scattered in multiple files, part-0, part-1 … part-n. To perform the import, it must be ensured that they are placed in one directory. Then in the loader’s mapping file, specify path as the directory.

Supported file formats include:

• TEXT
• CSV
• JSON

TEXT is a text file with custom delimiters, the first line is usually the header, and the name of each column is recorded, and no header line is allowed (specified in the mapping file). Each remaining row represents a record, which will be converted into a vertex/edge; each column of the row corresponds to a field, which will be converted into the id, label or attribute of the vertex/edge;

An example is as follows:

id|name|lang|price|ISBN
1|lop|java|328|ISBN978-7-107-18618-5
2|ripple|java|199|ISBN978-7-100-13678-5

CSV is a TEXT file with commas , as delimiters. When a column value itself contains a comma, the column value needs to be enclosed in double quotes, for example:

marko,29,Beijing
"li,nary",26,"Wu,han"

The JSON file requires that each line is a JSON string, and the format of each line needs to be consistent.

{"source_name": "marko", "target_name": "vadas", "date": "20160110", "weight": 0.5}
{"source_name": "marko", "target_name": "josh", "date": "20130220", "weight": 1.0}

3.2.1.2 HDFS file or directory

Users can also specify HDFS files or directories as data sources, all of the above requirements for local disk files or directories apply here. In addition, since HDFS usually stores compressed files, loader also provides support for compressed files, and local disk file or directory also supports compressed files.

Currently supported compressed file types include: GZIP, BZ2, XZ, LZMA, SNAPPY_RAW, SNAPPY_FRAMED, Z, DEFLATE, LZ4_BLOCK, LZ4_FRAMED, ORC, and PARQUET.

3.2.1.3 Mainstream relational database

The loader also supports some relational databases as data sources, and currently supports MySQL, PostgreSQL, Oracle, and SQL Server.

However, the requirements for the table structure are relatively strict at present. If association query needs to be done during the import process, such a table structure is not allowed. The associated query means: after reading a row of the table, it is found that the value of a certain column cannot be used directly (such as a foreign key), and you need to do another query to determine the true value of the column.

For example, Suppose there are three tables, person, software and created

// person schema
id | name | age | city

// software schema
id | name | lang | price

// created schema
id | p_id | s_id | date

If the id strategy of person or software is specified as PRIMARY_KEY when modeling (schema), choose name as the primary key (note: this is the concept of vertex-label in hugegraph), when importing edge data, the source vertex and target need to be spliced ​​out. For the id of the vertex, you must go to the person/software table with p_id/s_id to find the corresponding name. In the case of the schema that requires additional query, the loader does not support it temporarily. In this case, the following two methods can be used instead:

1. The id strategy of person and software is still specified as PRIMARY_KEY, but the id column of the person table and software table is used as the primary key attribute of the vertex, so that the id can be generated by directly splicing p_id and s_id with the label of the vertex when importing an edge;
2. Specify the id policy of person and software as CUSTOMIZE, and then directly use the id column of the person table and the software table as the vertex id, so that p_id and s_id can be used directly when importing edges;

The key point is to make the edge use p_id and s_id directly, don’t check it again.

3.2.2 Prepare vertex and edge data

3.2.2.1 Vertex Data

The vertex data file consists of data line by line. Generally, each line is used as a vertex, and each column is used as a vertex attribute. The following description uses CSV format as an example.

• person vertex data (the data itself does not contain a header)

Tom,48,Beijing
Jerry,36,Shanghai

• software vertex data (the data itself contains the header)

name,price
Photoshop,999
Office,388

3.2.2.2 Edge data

The edge data file consists of data line by line. Generally, each line is used as an edge. Some columns are used as the IDs of the source and target vertices, and other columns are used as edge attributes. The following uses JSON format as an example.

• knows edge data

{"source_name": "Tom", "target_name": "Jerry", "date": "2008-12-12"}

• created edge data

{"source_name": "Tom", "target_name": "Photoshop"}
{"source_name": "Tom", "target_name": "Office"}
{"source_name": "Jerry", "target_name": "Office"}

3.3 Write data source mapping file

3.3.1 Mapping file overview

The mapping file of the input source is used to describe how to establish the mapping relationship between the input source data and the vertex type/edge type of the graph. It is organized in JSON format and consists of multiple mapping blocks, each of which is responsible for mapping an input source. Mapped to vertices and edges.

Specifically, each mapping block contains an input source and multiple vertex mapping and edge mapping blocks, and the input source block corresponds to the local disk file or directory, HDFS file or directory and relational database are responsible for describing the basic information of the data source, such as where the data is, what format, what is the delimiter, etc. The vertex map/edge map is bound to the input source, which columns of the input source can be selected, which columns are used as ids, which columns are used as attributes, and what attributes are mapped to each column, the values ​​of the columns are mapped to what values ​​of attributes, and so on.

In the simplest terms, each mapping block describes: where is the file to be imported, which type of vertices/edges each line of the file is to be used as which columns of the file need to be imported, and the corresponding vertices/edges of these columns. what properties, etc.

Note: The format of the mapping file before version 0.11.0 and the format after 0.11.0 has changed greatly. For the convenience of expression, the mapping file (format) before 0.11.0 is called version 1.0, and the version after 0.11.0 is version 2.0. And unless otherwise specified, the “map file” refers to version 2.0.

{
  "version": "2.0",
  "structs": [
    {
      "id": "1",
      "input": {
      },
      "vertices": [
        {},
        {}
      ],
      "edges": [
        {},
        {}
      ]
    }
  ]
}

Two versions of the mapping file are given directly here (the above graph model and data file are described)

{
  "version": "2.0",
  "structs": [
    {
      "id": "1",
      "skip": false,
      "input": {
        "type": "FILE",
        "path": "vertex_person.csv",
        "file_filter": {
          "extensions": [
            "*"
          ]
        },
        "format": "CSV",
        "delimiter": ",",
        "date_format": "yyyy-MM-dd HH:mm:ss",
        "time_zone": "GMT+8",
        "skipped_line": {
          "regex": "(^#|^//).*|"
        },
        "compression": "NONE",
        "header": [
          "name",
          "age",
          "city"
        ],
        "charset": "UTF-8",
        "list_format": {
          "start_symbol": "[",
          "elem_delimiter": "|",
          "end_symbol": "]"
        }
      },
      "vertices": [
        {
          "label": "person",
          "skip": false,
          "id": null,
          "unfold": false,
          "field_mapping": {},
          "value_mapping": {},
          "selected": [],
          "ignored": [],
          "null_values": [
            ""
          ],
          "update_strategies": {}
        }
      ],
      "edges": []
    },
    {
      "id": "2",
      "skip": false,
      "input": {
        "type": "FILE",
        "path": "vertex_software.csv",
        "file_filter": {
          "extensions": [
            "*"
          ]
        },
        "format": "CSV",
        "delimiter": ",",
        "date_format": "yyyy-MM-dd HH:mm:ss",
        "time_zone": "GMT+8",
        "skipped_line": {
          "regex": "(^#|^//).*|"
        },
        "compression": "NONE",
        "header": null,
        "charset": "UTF-8",
        "list_format": {
          "start_symbol": "",
          "elem_delimiter": ",",
          "end_symbol": ""
        }
      },
      "vertices": [
        {
          "label": "software",
          "skip": false,
          "id": null,
          "unfold": false,
          "field_mapping": {},
          "value_mapping": {},
          "selected": [],
          "ignored": [],
          "null_values": [
            ""
          ],
          "update_strategies": {}
        }
      ],
      "edges": []
    },
    {
      "id": "3",
      "skip": false,
      "input": {
        "type": "FILE",
        "path": "edge_knows.json",
        "file_filter": {
          "extensions": [
            "*"
          ]
        },
        "format": "JSON",
        "delimiter": null,
        "date_format": "yyyy-MM-dd HH:mm:ss",
        "time_zone": "GMT+8",
        "skipped_line": {
          "regex": "(^#|^//).*|"
        },
        "compression": "NONE",
        "header": null,
        "charset": "UTF-8",
        "list_format": null
      },
      "vertices": [],
      "edges": [
        {
          "label": "knows",
          "skip": false,
          "source": [
            "source_name"
          ],
          "unfold_source": false,
          "target": [
            "target_name"
          ],
          "unfold_target": false,
          "field_mapping": {
            "source_name": "name",
            "target_name": "name"
          },
          "value_mapping": {},
          "selected": [],
          "ignored": [],
          "null_values": [
            ""
          ],
          "update_strategies": {}
        }
      ]
    },
    {
      "id": "4",
      "skip": false,
      "input": {
        "type": "FILE",
        "path": "edge_created.json",
        "file_filter": {
          "extensions": [
            "*"
          ]
        },
        "format": "JSON",
        "delimiter": null,
        "date_format": "yyyy-MM-dd HH:mm:ss",
        "time_zone": "GMT+8",
        "skipped_line": {
          "regex": "(^#|^//).*|"
        },
        "compression": "NONE",
        "header": null,
        "charset": "UTF-8",
        "list_format": null
      },
      "vertices": [],
      "edges": [
        {
          "label": "created",
          "skip": false,
          "source": [
            "source_name"
          ],
          "unfold_source": false,
          "target": [
            "target_name"
          ],
          "unfold_target": false,
          "field_mapping": {
            "source_name": "name",
            "target_name": "name"
          },
          "value_mapping": {},
          "selected": [],
          "ignored": [],
          "null_values": [
            ""
          ],
          "update_strategies": {}
        }
      ]
    }
  ]
}

{
  "vertices": [
    {
      "label": "person",
      "input": {
        "type": "file",
        "path": "vertex_person.csv",
        "format": "CSV",
        "header": ["name", "age", "city"],
        "charset": "UTF-8"
      }
    },
    {
      "label": "software",
      "input": {
        "type": "file",
        "path": "vertex_software.csv",
        "format": "CSV"
      }
    }
  ],
  "edges": [
    {
      "label": "knows",
      "source": ["source_name"],
      "target": ["target_name"],
      "input": {
        "type": "file",
        "path": "edge_knows.json",
        "format": "JSON"
      },
      "field_mapping": {
        "source_name": "name",
        "target_name": "name"
      }
    },
    {
      "label": "created",
      "source": ["source_name"],
      "target": ["target_name"],
      "input": {
        "type": "file",
        "path": "edge_created.json",
        "format": "JSON"
      },
      "field_mapping": {
        "source_name": "name",
        "target_name": "name"
      }
    }
  ]
}

The 1.0 version of the mapping file is centered on the vertex and edge, and sets the input source; while the 2.0 version is centered on the input source, and sets the vertex and edge mapping. Some input sources (such as a file) can generate both vertices and edges. If you write in the 1.0 format, you need to write an input block in each of the vertex and edge mapping blocks. The two input blocks are exactly the same; and the 2.0 version only needs to write input once. Therefore, compared with version 1.0, version 2.0 can save some repetitive writing of input.

In the bin directory of hugegraph-loader-{version}, there is a script tool mapping-convert.sh that can directly convert the mapping file of version 1.0 to version 2.0. The usage is as follows:

bin/mapping-convert.sh struct.json

A struct-v2.json will be generated in the same directory as struct.json.

The bin directory also ships utf8-bom-to-utf8.sh, which strips the UTF-8 BOM from a single data file, or from every file under a directory. It is useful when a CSV or TEXT file exported by a Windows tool fails to parse because its first header column carries an invisible BOM:

bin/utf8-bom-to-utf8.sh /path/to/file-or-dir

3.3.2 Input Source

Input sources are currently divided into five categories: FILE, HDFS, JDBC, KAFKA and GRAPH, which are distinguished by the type node. We call them local file input sources, HDFS input sources, JDBC input sources, KAFKA input sources and GRAPH input source, which are described below.

3.3.2.1 Local file input source

• id: The id of the input source. This field is used to support some internal functions. It is not required (it will be automatically generated if it is not filled in). It is strongly recommended to write it, which is very helpful for debugging;
• skip: whether to skip the input source, because the JSON file cannot add comments, if you do not want to import an input source during a certain import, but do not want to delete the configuration of the input source, you can set it to true to skip it, the default is false, not required;
• input: input source map block, composite structure
  • type: an input source type, file or FILE must be filled;
  • path: the path of the local file or directory, the absolute path or the relative path relative to the mapping file, it is recommended to use the absolute path, required;
  • file_filter: filter files with compound conditions from path, compound structure, currently only supports configuration extensions, represented by child node extensions, the default is “*”, which means to keep all files;
  • format: the format of the local file, the optional values are CSV, TEXT and JSON, which must be uppercase, the default is CSV, optional;
  • header: the column name of each column of the file, if not specified, the first line of the data file will be used as the header; when the file itself has a header and the header is specified, the first line of the file will be treated as a normal data line; JSON The file does not need to specify a header, optional;
  • has_header: for CSV and TEXT, the first line of every file is dropped when it is identical to the header, so a header repeated in each part file of a directory is not imported as data. Set this to false to turn that check off when the first line of a file is real data that happens to equal the header, optional;
  • delimiter: The column delimiter of the file line. The default depends on format: a comma "," for CSV and a tab "\t" for TEXT; a CSV file accepts no other delimiter. The JSON file does not need to be specified, optional;
  • charset: the encoded character set of the file, the default is UTF-8, optional;
  • date_format: custom date format, the default value is yyyy-MM-dd HH:mm:ss, optional; if the date is presented in the form of a timestamp, this item must be written as timestamp (fixed writing);
  • extra_date_formats: a list of fallback date formats, tried when a value does not match date_format, empty by default, optional;
  • time_zone: Set which time zone the date data is in, the default value is GMT+8, optional;
  • skipped_line: The line to be skipped, compound structure, currently only the regular expression of the line to be skipped can be configured, described by the child node regex. The default regex is (^#|^//).*|, which skips lines starting with # or // and empty lines; to keep such lines, set regex to a pattern that matches nothing, optional;
  • compression: The compression format of the file, the optional values ​​are NONE, GZIP, BZ2, XZ, LZMA, SNAPPY_RAW, SNAPPY_FRAMED, Z, DEFLATE, LZ4_BLOCK, LZ4_FRAMED, ORC and PARQUET, the default is NONE, which means a non-compressed file, optional; for ORC and PARQUET the header is matched case-insensitively;
  • list_format: When a column of the file (non-JSON) is a collection structure (the Cardinality of the PropertyKey in the corresponding figure is Set or List), you can use this item to set the start character, separator, and end character of the column, compound structure :
    • start_symbol: The start character of the collection structure column (the default value is the empty string "", JSON format currently does not support specification)
    • elem_delimiter: the delimiter of the collection structure column (the default value is |, and it must differ from delimiter; JSON format currently only supports native , delimiter)
    • end_symbol: the end character of the collection structure column (the default value is the empty string "", the JSON format does not currently support specification)
    • ignored_elems: the elements dropped after the column is split, the default value is [""], so empty elements are ignored

3.3.2.2 HDFS input source

The nodes and meanings of the above local file input source are basically applicable here. Only the different and unique nodes of the HDFS input source are listed below.

• type: input source type, must fill in hdfs or HDFS, required;
• path: the path of the HDFS file or directory, it must be the absolute path of HDFS, required;
• core_site_path: the path of the core-site.xml file of the HDFS cluster, the key point is to specify the address of the NameNode (fs.default.name) and the implementation of the file system (fs.hdfs.impl), required;
• hdfs_site_path: the path of the hdfs-site.xml file of the HDFS cluster, optional;
• dir_filter: when path is a directory, decides which sub-directories are walked into, compound structure, optional:
  • include_regex: only directories whose name matches this regular expression are read, empty by default, which places no restriction;
  • exclude_regex: directories whose name matches this regular expression are skipped, empty by default;
• kerberos_config: how to authenticate against a Kerberos-secured HDFS cluster, compound structure, optional:
  • enable: whether to authenticate with Kerberos, the default is false;
  • krb5_conf: the path of the krb5.conf file, required when enable is true;
  • principal: the Kerberos principal, required when enable is true;
  • keytab: the path of the keytab file, required when enable is true;

3.3.2.3 JDBC input source

As mentioned above, it supports multiple relational databases, but because their mapping structures are very similar, they are collectively referred to as JDBC input sources, and then use the vendor node to distinguish different databases.

• type: input source type, must fill in jdbc or JDBC, required;
• vendor: database type, optional options are [MySQL, PostgreSQL, Oracle, SQLServer], case-insensitive, required;
• driver: the JDBC driver class, optional; when it is left out, the default driver of the vendor listed in the tables below is used;
• url: the url of the database that jdbc wants to connect to, required;
• database: the name of the database to be connected, required;
• schema: The name of the schema to be connected, different databases have different requirements, and the details are explained below;
• table: the name of the table to be connected, at least one of table or custom_sql is required;
• custom_sql: custom SQL statement, at least one of table or custom_sql is required;
• username: username to connect to the database, required;
• password: password for connecting to the database, required;
• where: an extra condition appended to the generated select statement, written without the where keyword, optional;
• batch_size: The size of one page when obtaining table data by page, the default is 500, optional;

MYSQL

schema: nullable, if filled in, it must be the same as the value of database

POSTGRESQL

schema: nullable, default is “public”

ORACLE

schema: nullable, the default value is the username in upper case

SQLSERVER

schema: required

3.3.2.4 Kafka input source

• type: input source type, kafka or KAFKA, required;
• bootstrap_server: the list of kafka bootstrap servers, required;
• topic: the topic to subscribe to, required;
• group: group of Kafka consumers, required;
• from_beginning: whether to start from the earliest offset of the topic (auto.offset.reset=earliest) instead of the latest one, default is false, optional;
• format: format of each message, options are CSV, TEXT and JSON, must be uppercase, required;
• header: column name of each column of a message; no header line is read from the topic, so it has to be given for CSV and TEXT, while JSON messages do not need it;
• delimiter: delimiter of the message columns, used by TEXT only, since CSV always splits on ,, optional;
• charset: encoding charset of the messages, default is UTF-8, optional;
• date_format: customized date format, default value is yyyy-MM-dd HH:mm:ss, optional; if the date is presented in the form of timestamp, this item must be written as timestamp (fixed);
• extra_date_formats: a customized list of another date formats, empty by default, optional; each item in the list is an alternate date format to the date_format specified date format;
• time_zone: set which time zone the date data is in, default is GMT+8, optional;
• skipped_line: the line you want to skip, composite structure, currently can only configure the regular expression of the line to be skipped, described by the child node regex, the default is not to skip any line, optional;
• batch_size: the maximum number of records fetched in one poll (max.poll.records), default is 500, optional;
• early_stop: the record pulled from Kafka broker at a certain time is empty, stop the task, default is false, only for debugging, optional;

3.3.2.5 GRAPH input Source

The GRAPH input source reads vertices and edges out of another HugeGraph graph, reached through HugeGraph-PD, and writes them into the target graph. When a mapping file contains a GRAPH input source, every input source in it that is not skipped has to be a GRAPH input source as well, and the loader puts the target graph into RESTORING mode for the duration of the import.

• type: Data source type; must be filled in as graph or GRAPH (required);
• graphspace: Source graphSpace name (required);
• graph: Source graph name (required);
• username: HugeGraph username; the --username command-line option is used when this is empty;
• password: HugeGraph password; the --password command-line option is used when this is empty;
• selected_vertices: the vertex labels to copy, each item written as {"label": "...", "properties": [...], "query": {...}}, where properties narrows the copied properties and query is an optional filter passed to the source graph;
• ignored_vertices: the vertex labels to skip, each item written as {"label": "...", "properties": [...]};
• selected_edges: the edge labels to copy, items have the same shape as in selected_vertices;
• ignored_edges: the edge labels to skip, items have the same shape as in ignored_vertices;
• pd-peers: HugeGraph-PD node addresses of the source cluster; the --pd-peers option is used when this is empty;
• meta-endpoints: Meta service endpoints of the source cluster; the --meta-endpoints option is used when this is empty;
• cluster: Source cluster name; the --cluster option is used when this is empty;
• batch_size: Batch size for reading data from the source graph; default is 500;

3.3.3 Vertex and Edge Mapping

The nodes of vertex and edge mapping (a key in the JSON file) have a lot of the same parts. The same parts are introduced first, and then the unique nodes of vertex map and edge map are introduced respectively.

Nodes of the same section

• label: label to which the vertex/edge data to be imported belongs, required;
• skip: whether to skip this vertex/edge mapping while the input source and the other mappings stay active, the default is false, optional;
• field_mapping: Map the column name of the input source column to the attribute name of the vertex/edge, optional;
• value_mapping: map the data value of the input source to the attribute value of the vertex/edge, optional;
• selected: select some columns to insert, other unselected ones are not inserted, cannot exist at the same time as ignored, optional;
• ignored: ignore some columns so that they do not participate in insertion, cannot exist at the same time as selected, optional;
• null_values: You can specify some strings to represent null values, such as “NULL”. If the vertex/edge attribute corresponding to this column is also a nullable attribute, the value of this attribute will not be set when constructing the vertex/edge, optional ;
• update_strategies: If the data needs to be updated in batches in a specific way, you can specify a specific update strategy for each attribute (see below for details), optional;
• unfold: Whether to unfold the column, each unfolded column will form a row with other columns, which is equivalent to unfolding into multiple rows; for example, the value of a certain column (id column) of the file is [1,2,3], The values ​​of other columns are 18,Beijing. When unfold is set, this row will become 3 rows, namely: 1,18,Beijing, 2,18,Beijing and 3,18, Beijing. Note that this will only expand the column selected as id. Default false, optional;

Update strategy supports 8 types: (requires all uppercase)

1. Value accumulation: SUM
2. Take the greater of the two numbers/dates: BIGGER
3. Take the smaller of two numbers/dates: SMALLER
4. Set property takes union: UNION
5. Set attribute intersection: INTERSECTION
6. List attribute append element: APPEND
7. List/Set attribute delete element: ELIMINATE
8. Override an existing property: OVERRIDE

Note: If the newly imported attribute value is empty, the existing old data will be used instead of the empty value. For the effect, please refer to the following example

// The update strategy is specified in the JSON file as follows
{
  "vertices": [
    {
      "label": "person",
      "update_strategies": {
        "age": "SMALLER",
        "set": "UNION"
      },
      "input": {
        "type": "file",
        "path": "vertex_person.txt",
        "format": "TEXT",
        "header": ["name", "age", "set"]
      }
    }
  ]
}

// 1. Write a line of data with the OVERRIDE update strategy (null means empty here)
'a b null null'

// 2. Write another line
'null null c d'

// 3. Finally we can get
'a b c d'   

// If there is no update strategy, you will get
'null null c d'

Note : After adopting the batch update strategy, the number of disk read requests will increase significantly, and the import speed will be several times slower than that of pure write coverage (at this time HDD disk [IOPS](https://en.wikipedia .org/wiki/IOPS) will be the bottleneck, SSD is recommended for speed)

Unique Nodes for Vertex Maps

• id: Specify a column as the id column of the vertex. When the vertex id policy is CUSTOMIZE, it is required; when the id policy is PRIMARY_KEY, it must be empty;

Unique Nodes for Edge Maps

• source: Select certain columns of the input source as the id column of source vertex. When the id policy of the source vertex is CUSTOMIZE, a certain column must be specified as the id column of the vertex; when the id policy of the source vertex is When PRIMARY_KEY, one or more columns must be specified for splicing the id of the generated vertex, that is, no matter which id strategy is used, this item is required;
• target: Specify certain columns as the id columns of target vertex, similar to source, so I won’t repeat them;
• unfold_source: Whether to unfold the source column of the file, the effect is similar to that in the vertex map, and will not be repeated;
• unfold_target: Whether to unfold the target column of the file, the effect is similar to that in the vertex mapping, and will not be repeated;

3.4 Execute command import

After preparing the graph model, data file, and input source mapping relationship file, the data file can be imported into the graph database.

The import process is controlled by commands submitted by the user, and the user can control the specific process of execution through different parameters.

3.4.1 Parameter description

The loader prints its usage and exits when it is given fewer than three arguments, so -f struct.json on its own is not enough.

3.4.2 Breakpoint Continuation Mode

Usually, the Loader task takes a long time to execute. If the import interrupt process exits for some reason, and next time you want to continue the import from the interrupted point, this is the scenario of using breakpoint continuation.

The user sets the command line parameter –incremental-mode to true to open the breakpoint resume mode. The key to breakpoint continuation lies in the progress file. When the import process exits, the import progress at the time of exit will be recorded. Recorded in the progress file, the progress file is located in the ${struct} directory, the file name is like load-progress_${timestamp}, ${struct} is the prefix of the mapping file, and ${timestamp} is the start of the import moment, formatted as yyyyMMdd-HHmmss. For example, for an import task started at 2019-10-10 12:30:30, the mapping file used is struct-example.json, then the path of the progress file is the same as struct-example.json Sibling struct-example/load-progress_20191010-123030. When the directory holds several progress files, the resumed import reads the last one in name order, which is the most recent one.

Note: The generation of progress files is independent of whether –incremental-mode is turned on or not, and a progress file is generated at the end of each import.

If the data file formats are all legal and the import task is stopped by the user (CTRL + C or kill, kill -9 is not supported), that is to say, if there is no error record, the next import only needs to be set to Continue for the breakpoint.

But if the limit of –max-read-errors, –max-parse-errors or –max-insert-errors is reached because too much data is invalid or network abnormality is reached, Loader will record these original rows that failed into the failure file, after the user modifies the data lines in the failure file, set –failure-mode to true to import these “failure files” as input sources (does not affect the normal file import), Of course, if there is still a problem with the modified data line, it will be logged again to the failure file (don’t worry about duplicate lines, they are dropped when the file is closed). Failure mode lifts the three error limits above, so the whole failure file is scanned.

Each input source, that is each item of structs in the mapping file, gets its own failure file. The file is named after the id of that input source with the suffix .error and is stored in the ${struct}/failure-data directory. Every failed line is written as a pair of lines: a tip line starting with #### READ ERROR:, #### PARSE ERROR: or #### INSERT ERROR:, followed by the original data line. When the input source has a header, that header is written as JSON to a sibling ${id}.header file, so the failure file can be read back with the right columns. For example, if the mapping file has an input source with id 1 holding a vertex mapping person and an input source with id 3 holding an edge mapping knows, each of which has some error lines, you will see the following files in the ${struct}/failure-data directory when the Loader exits:

• 1.error: the failed lines of input source 1, each preceded by its tip line
• 1.header: the header of input source 1, written only when that input source has a header
• 3.error: the failed lines of input source 3
• 3.header: the header of input source 3

A .error file that turns out to be empty is deleted when the Loader exits, so only the input sources that really had failed lines leave a file behind. In incremental mode new failures are appended to the existing file, otherwise the file is rewritten from scratch.

3.4.3 logs directory file description

The log and error data during program execution will be written into the hugegraph-loader.log file.

3.4.4 Execute command

Run bin/hugegraph-loader.sh and pass in parameters

bin/hugegraph-loader.sh -g {GRAPH_NAME} -f ${INPUT_DESC_FILE} -s ${SCHEMA_FILE} -h {HOST} -p {PORT}

The script runs the JVM under JAVA_HOME when that variable is set, and java from the PATH otherwise. It passes the contents of the JVM_OPTS environment variable, then -Xmx10g and the class path built from lib/, to that JVM, so JVM_OPTS is the place to add JVM flags. Logging is configured by conf/log4j2.xml.

4 Complete example

Given below is an example in the example directory of the hugegraph-loader package. (GitHub address)

4.1 Prepare data

Vertex file: example/file/vertex_person.csv

marko,29,Beijing
vadas,27,Hongkong
josh,32,Beijing
peter,35,Shanghai
"li,nary",26,"Wu,han"
tom,null,NULL

Vertex file: example/file/vertex_software.txt

id|name|lang|price|ISBN
1|lop|java|328|ISBN978-7-107-18618-5
2|ripple|java|199|ISBN978-7-100-13678-5

Edge file: example/file/edge_knows.json

{"source_name": "marko", "target_name": "vadas", "date": "20160110", "weight": 0.5}
{"source_name": "marko", "target_name": "josh", "date": "20130220", "weight": 1.0}

Edge file: example/file/edge_created.json

{"aname": "marko", "bname": "lop", "date": "20171210", "weight": 0.4}
{"aname": "josh", "bname": "lop", "date": "20091111", "weight": 0.4}
{"aname": "josh", "bname": "ripple", "date": "20171210", "weight": 1.0}
{"aname": "peter", "bname": "lop", "date": "20170324", "weight": 0.2}

4.2 Write schema

schema.propertyKey("name").asText().ifNotExist().create();
schema.propertyKey("age").asInt().ifNotExist().create();
schema.propertyKey("city").asText().ifNotExist().create();
schema.propertyKey("weight").asDouble().ifNotExist().create();
schema.propertyKey("lang").asText().ifNotExist().create();
schema.propertyKey("date").asText().ifNotExist().create();
schema.propertyKey("price").asDouble().ifNotExist().create();

schema.vertexLabel("person").properties("name", "age", "city").primaryKeys("name").ifNotExist().create();
schema.vertexLabel("software").properties("name", "lang", "price").primaryKeys("name").ifNotExist().create();

schema.indexLabel("personByAge").onV("person").by("age").range().ifNotExist().create();
schema.indexLabel("personByCity").onV("person").by("city").secondary().ifNotExist().create();
schema.indexLabel("personByAgeAndCity").onV("person").by("age", "city").secondary().ifNotExist().create();
schema.indexLabel("softwareByPrice").onV("software").by("price").range().ifNotExist().create();

schema.edgeLabel("knows").sourceLabel("person").targetLabel("person").properties("date", "weight").ifNotExist().create();
schema.edgeLabel("created").sourceLabel("person").targetLabel("software").properties("date", "weight").ifNotExist().create();

schema.indexLabel("createdByDate").onE("created").by("date").secondary().ifNotExist().create();
schema.indexLabel("createdByWeight").onE("created").by("weight").range().ifNotExist().create();
schema.indexLabel("knowsByWeight").onE("knows").by("weight").range().ifNotExist().create();

4.3 Write the input source mapping file example/file/struct.json

{
  "vertices": [
    {
      "label": "person",
      "input": {
        "type": "file",
        "path": "example/file/vertex_person.csv",
        "format": "CSV",
        "header": ["name", "age", "city"],
        "charset": "UTF-8",
        "skipped_line": {
          "regex": "(^#|^//).*"
        }
      },
      "null_values": ["NULL", "null", ""]
    },
    {
      "label": "software",
      "input": {
        "type": "file",
        "path": "example/file/vertex_software.txt",
        "format": "TEXT",
        "delimiter": "|",
        "charset": "GBK"
      },
      "id": "id",
      "ignored": ["ISBN"]
    }
  ],
  "edges": [
    {
      "label": "knows",
      "source": ["source_name"],
      "target": ["target_name"],
      "input": {
        "type": "file",
        "path": "example/file/edge_knows.json",
        "format": "JSON",
        "date_format": "yyyyMMdd"
      },
      "field_mapping": {
        "source_name": "name",
        "target_name": "name"
      }
    },
    {
      "label": "created",
      "source": ["source_name"],
      "target": ["target_id"],
      "input": {
        "type": "file",
        "path": "example/file/edge_created.json",
        "format": "JSON",
        "date_format": "yyyy-MM-dd"
      },
      "field_mapping": {
        "source_name": "name"
      }
    }
  ]
}

4.4 Command to import

sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy

After the import is complete, statistics similar to the following will appear:

vertices/edges has been loaded this time : 8/6
--------------------------------------------------
count metrics
     input read success            : 14
     input read failure            : 0
     vertex parse success          : 8
     vertex parse failure          : 0
     vertex insert success         : 8
     vertex insert failure         : 0
     edge parse success            : 6
     edge parse failure            : 0
     edge insert success           : 6
     edge insert failure           : 0

4.5 Use Docker to load data

4.5.1 Use docker exec to load data directly

4.5.1.1 Prepare data

If you just want to try out the loader, you can import the built-in example dataset without needing to prepare additional data yourself.

If using custom data, before importing data with the loader, we need to copy the data into the container.

First, following the steps in 4.1–4.3, we can prepare the data and then use docker cp to copy the prepared data into the loader container.

Suppose we’ve prepared the corresponding dataset following the above steps, stored in the hugegraph-dataset folder with the following file structure:

tree -f hugegraph-dataset/

hugegraph-dataset
├── hugegraph-dataset/edge_created.json
├── hugegraph-dataset/edge_knows.json
├── hugegraph-dataset/schema.groovy
├── hugegraph-dataset/struct.json
├── hugegraph-dataset/vertex_person.csv
└── hugegraph-dataset/vertex_software.txt

Copy the files into the container.

docker cp hugegraph-dataset loader:/loader/dataset
docker exec -it loader ls /loader/dataset

edge_created.json  edge_knows.json  schema.groovy  struct.json  vertex_person.csv  vertex_software.txt

4.5.1.2 Data loading

Taking the built-in example dataset as an example, we can use the following command to load the data.

If you need to import your custom dataset, you need to modify the paths for -f (data script) and -s (schema) configurations.

You can refer to 3.4.1-Parameter description for the rest of the parameters.

docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h server -p 8080

If loading a custom dataset, following the previous example, you would use:

docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f /loader/dataset/struct.json -s /loader/dataset/schema.groovy -h server -p 8080

If loader and server are in the same Docker network, you can specify -h {server_container_name}; otherwise, you need to specify the IP of the server host (in our example, server_container_name is server).

Then we can see the result:

HugeGraphLoader worked in NORMAL MODE
vertices/edges loaded this time : 8/6
--------------------------------------------------
count metrics
    input read success            : 14                  
    input read failure            : 0                   
    vertex parse success          : 8                   
    vertex parse failure          : 0                   
    vertex insert success         : 8                   
    vertex insert failure         : 0                   
    edge parse success            : 6                   
    edge parse failure            : 0                   
    edge insert success           : 6                   
    edge insert failure           : 0                   
--------------------------------------------------
meter metrics
    total time                    : 0.199s              
    read time                     : 0.046s              
    load time                     : 0.153s              
    vertex load time              : 0.077s              
    vertex load rate(vertices/s)  : 103                 
    edge load time                : 0.112s              
    edge load rate(edges/s)       : 53   

You can also use curl or hubble to observe the import result. Here’s an example using curl:

> curl "http://localhost:8080/graphs/hugegraph/graph/vertices" | gunzip
{"vertices":[{"id":1,"label":"software","type":"vertex","properties":{"name":"lop","lang":"java","price":328.0}},{"id":2,"label":"software","type":"vertex","properties":{"name":"ripple","lang":"java","price":199.0}},{"id":"1:tom","label":"person","type":"vertex","properties":{"name":"tom"}},{"id":"1:josh","label":"person","type":"vertex","properties":{"name":"josh","age":32,"city":"Beijing"}},{"id":"1:marko","label":"person","type":"vertex","properties":{"name":"marko","age":29,"city":"Beijing"}},{"id":"1:peter","label":"person","type":"vertex","properties":{"name":"peter","age":35,"city":"Shanghai"}},{"id":"1:vadas","label":"person","type":"vertex","properties":{"name":"vadas","age":27,"city":"Hongkong"}},{"id":"1:li,nary","label":"person","type":"vertex","properties":{"name":"li,nary","age":26,"city":"Wu,han"}}]}

If you want to check the import result of edges, you can use curl "http://localhost:8080/graphs/hugegraph/graph/edges" | gunzip.

4.5.2 Enter the docker container to load data

Besides using docker exec directly for data import, we can also enter the container for data loading. The basic process is similar to 4.5.1.

Enter the container by docker exec -it loader bash and execute the command:

sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h server -p 8080

The results of the execution will be similar to those shown in 4.5.1.

4.6 Import data by spark-loader

The current source uses Spark 3.2.2 and Scala 2.12. Other combinations need independent verification.

The parameters of spark-loader are divided into two parts. Note: Because the abbreviations of these two-parameter names have overlapping parts, please use the full name of the parameter. And there is no need to guarantee the order between the two parameters.

• hugegraph parameters (Reference: hugegraph-loader parameter description )
• Spark task submission parameters (Reference: Submitting Applications)

Example:

sh bin/hugegraph-spark-loader.sh --master yarn \
--deploy-mode cluster --name spark-hugegraph-loader --file ./hugegraph.json \
--username admin --token admin --host xx.xx.xx.xx --port 8093 \
--graph graph-test --num-executors 6 --executor-cores 16 --executor-memory 15g

bin/hugegraph-spark-loader.sh submits org.apache.hugegraph.loader.spark.HugeGraphSparkLoader through ${SPARK_HOME}/bin/spark-submit, so SPARK_HOME has to point at a Spark installation. Every jar under lib/ is put on the class path. The Spark application name defaults to hugegraph-spark-loader and can be changed through the APP_NAME environment variable.

bin/get-params.sh splits the command line: only the options below are handed to the loader, every other argument is passed to spark-submit unchanged. The splitter matches long option names only, so short forms such as -f and -g are not recognised.

--graph --schema --host --port --username --token --protocol
--trust-store-file --trust-store-password --clear-all-data --clear-timeout
--incremental-mode --failure-mode --batch-insert-threads --single-insert-threads
--max-conn --max-conn-per-route --batch-size --max-parse-errors --max-insert-errors
--timeout --shutdown-timeout --retry-times --retry-interval --check-vertex
--print-progress --dry-run --sink-type --vertex-partitions --edge-partitions --help

--file is treated separately: with --deploy-mode cluster the mapping file is shipped to the executors through --files and the loader receives only its base name, otherwise the path is passed through as written.

In this mode the loader reads FILE, HDFS and JDBC input sources; a KAFKA or GRAPH input source is rejected.

By default (--sink-type true) each Spark partition opens a HugeClient and writes vertices and edges through the HugeGraph server API. With --sink-type false the loader generates HFiles and bulk-loads them into HBase instead, taking the table names and ZooKeeper settings from --vertex-table-name, --edge-table-name, --hbase-zk-quorum, --hbase-zk-port, --hbase-zk-parent, --vertex-partitions and --edge-partitions.

4.7 Import data by flink-cdc-loader

The current source uses Flink 1.13.5 with flink-connector-mysql-cdc 2.2.1 and Scala 2.12. Other combinations need independent verification.

bin/hugegraph-flinkcdc-loader.sh submits org.apache.hugegraph.loader.flink.HugeGraphFlinkCDCLoader through ${FLINK_HOME}/bin/flink run, so FLINK_HOME has to be set. The job captures MySQL change events with Flink CDC and applies them to the graph, which keeps the graph in step with the source tables.

The mapping file uses the same format as for the command-line loader, but every input source has to be a JDBC input source over MySQL: the loader takes url, database, table, username and password from it and parses the host and port out of url. Vertex and edge mappings work as usual. --cdc-flush-interval and --cdc-sink-parallelism in 3.4.1 apply to this mode only.

The command line is split by bin/get-params.sh in the same way as for spark-loader, with the arguments that are not loader options going to flink run.

Example:

sh bin/hugegraph-flinkcdc-loader.sh --file ./mysql-cdc.json \
--host xx.xx.xx.xx --port 8080 --graph hugegraph --username admin --token admin

3 - HugeGraph-Tools Quick Start

1 HugeGraph-Tools Overview

HugeGraph-Tools is an automated deployment, management and backup/restore component of HugeGraph.

Testing Guide: For running HugeGraph-Tools tests locally, please refer to HugeGraph Toolchain Local Testing Guide

2 Get HugeGraph-Tools

HugeGraph-Tools is included in the Toolchain distribution. You can download the distribution or build it from source.

• Download the compiled tarball
• Clone source code then compile and install

2.1 Download the compiled archive

Download the latest version of the HugeGraph-Toolchain package:

export VERSION=1.7.0
export ARCHIVE="apache-hugegraph-toolchain-incubating-${VERSION}"
wget "https://downloads.apache.org/hugegraph/${VERSION}/${ARCHIVE}.tar.gz"
tar zxf "${ARCHIVE}.tar.gz"
# hugegraph-tools ships inside the toolchain package, in a directory
# whose version suffix is the same as the archive's
cd "${ARCHIVE}/apache-hugegraph-tools-incubating-${VERSION}"

2.2 Clone source code to compile and install

Please ensure that the wget command is installed before compiling the source code

Download the latest version of the HugeGraph-Tools source package:

# 1. get from github
git clone https://github.com/apache/hugegraph-toolchain.git

# 2. Download a released source package
export VERSION=1.7.0
export ARCHIVE="apache-hugegraph-toolchain-incubating-${VERSION}"
wget "https://downloads.apache.org/hugegraph/${VERSION}/${ARCHIVE}-src.tar.gz"

Compile and generate tar package:

cd hugegraph-toolchain
mvn package -pl hugegraph-tools -am -DskipTests -ntp

The package is generated as hugegraph-tools/target/apache-hugegraph-tools-${version}.tar.gz, and the unpacked directory hugegraph-tools/apache-hugegraph-tools-${version} (containing bin/ and lib/) is created next to it.

3 How to use

3.1 Function overview

After decompression, enter the apache-hugegraph-tools-${version} directory, you can use bin/hugegraph or bin/hugegraph help to view the usage information, and bin/hugegraph help <sub-command> to view the usage of a single sub-command. mainly divided:

• Graph management type, graph-mode-set, graph-mode-get, graph-list, graph-get, graph-clear, graph-create, graph-clone and graph-drop
• Asynchronous task management type, task-list, task-get, task-delete, task-cancel and task-clear
• Gremlin type, gremlin-execute and gremlin-schedule
• Backup/Restore type, backup, restore, migrate, schedule-backup and dump
• Authentication data backup/restore type, auth-backup and auth-restore
• Install deployment type, deploy, clear, start-all and stop-all

Usage: hugegraph [options] [command] [command options]

3.2 [options]-Global Variable

options is a global variable of HugeGraph-Tools, which can be configured in hugegraph-tools/bin/hugegraph, including:

• –graph，HugeGraph-Tools The name of the graph to operate on, the default value is hugegraph
• –url，The service address of HugeGraph-Server, the default is http://127.0.0.1:8080
• –user，When HugeGraph-Server opens authentication, pass username
• –password，When HugeGraph-Server opens authentication, pass the user’s password
• –timeout，Timeout when connecting to HugeGraph-Server, the default is 30s
• –trust-store-file，The path of the certificate file, when –url uses https, the truststore file used by HugeGraph-Client, the default is empty, which means using the built-in truststore file conf/hugegraph.truststore of hugegraph-tools
• –trust-store-password，The password of the certificate file, when –url uses https, the password of the truststore used by HugeGraph-Client, the default is empty, representing the password of the built-in truststore file of hugegraph-tools
• –throw-mode, whether HugeGraph-Tools throws the exception instead of printing the error message and exiting, the default is false (mainly used by tests)

The protocol is taken from the scheme of –url: use https://... to connect over https. –trust-store-file and –trust-store-password can only be set when –url uses https, and both –user and –password must be given together or omitted together.

The above global variables can also be set through environment variables. One way is to use export on the command line to set temporary environment variables, which are valid until the command line is closed

Another way is to set the environment variable in the bin/hugegraph script:

#!/bin/bash

# Set environment here if needed
#export HUGEGRAPH_URL=
#export HUGEGRAPH_GRAPH=
#export HUGEGRAPH_USERNAME=
#export HUGEGRAPH_PASSWORD=
#export HUGEGRAPH_TIMEOUT=
#export HUGEGRAPH_TRUST_STORE_FILE=
#export HUGEGRAPH_TRUST_STORE_PASSWORD=

bin/hugegraph also reads JAVA_HOME (a warning is printed when it is not set, and it is needed for https) and JAVA_OPTIONS (JVM options; when it is empty the script uses -Xms512m plus an -Xmx computed from the free memory of the machine).

3.3 Graph Management Type, graph-mode-set, graph-mode-get, graph-list, graph-get, graph-clear, graph-create, graph-clone and graph-drop

• graph-mode-set, set graph restore mode
  • –graph-mode or -m, required, specifies the mode to be set, legal values include [NONE, RESTORING, MERGING, LOADING]
• graph-mode-get, get graph restore mode
• graph-list, list all graphs in a HugeGraph-Server
• graph-get, get a graph and its storage backend type
• graph-clear, clear all schema and data of a graph
  • –confirm-message or -c, required, delete confirmation information, manual input is required, double confirmation to prevent accidental deletion, “I’m sure to delete all data”, including double quotes
• graph-create, create a new graph with configuration file
  • –name or -n, optional, the name of the new graph, default is g
  • –file or -f, the path to the graph configuration file, the content of the file is sent to HugeGraph-Server as the config of the new graph
• graph-clone, clone an existing graph
  • –name or -n, optional, the name of the cloned graph, default is g
  • –clone-graph-name, optional, the name of the source graph to clone from, default is hugegraph
• graph-drop, drop a graph (different from graph-clear, this completely removes the graph)
  • –confirm-message or -c, required, confirmation message “I’m sure to drop the graph”, including double quotes

graph-create, graph-clone, graph-clear and graph-drop raise –timeout to at least 300 seconds.

When you need to restore the backup graph to a new graph, you need to set the graph mode to RESTORING mode; when you need to merge the backup graph into an existing graph, you need to first set the graph mode to MERGING model.

3.4 Asynchronous task management Type，task-list、task-get、task-delete、task-cancel and task-clear

• task-list，List the asynchronous tasks in a graph, which can be filtered according to the status of the tasks
  • –status，Optional, specify the status of the task to view, i.e. filter tasks by status, legal values include [UNKNOWN, NEW, QUEUED, RESTORING, RUNNING, SUCCESS, CANCELLED, FAILED] (case insensitive)
  • –limit，Optional, specify the number of tasks to be obtained, the default is -1, which means to obtain all eligible tasks, a value passed explicitly must be positive
• task-get，Get detailed information about an asynchronous task
  • –task-id，Required, specifies the ID of the asynchronous task
• task-delete，Delete information about an asynchronous task
  • –task-id，Required, specifies the ID of the asynchronous task
• task-cancel，Cancel the execution of an asynchronous task
  • –task-id，Required, the ID of the asynchronous task to cancel
• task-clear，Clean up completed asynchronous tasks
  • –force，Optional. When set, it means to clean up all asynchronous tasks. Unfinished ones are canceled first, and then all asynchronous tasks are cleared. By default, only completed asynchronous tasks are cleaned up

3.5 Gremlin Type，gremlin-execute and gremlin-schedule

⚠️ SEC Reminder: The execution of Gremlin depends on the actual logic of the statements, which may involve scenarios such as large-scale data modification and high-risk system calls with potential implicit hazards. Please use this tool only in secure and trusted network environments. It is imperative to configure and secure HugeGraph-Server with the Authentication System (Auth) and an IP Whitelist to restrict execution requests on the server side. Never hand over the tool or expose the execution entry to unauthorized personnel.

• gremlin-execute, send Gremlin statements to HugeGraph-Server to execute query or modification operations, execute synchronously, and return results after completion
  • –file or -f, specify the script file to execute, UTF-8 encoding, mutually exclusive with –script
  • –script or -s, specifies the script string to execute, mutually exclusive with –file
  • –aliases or -a, Gremlin alias settings, the format is: key1=value1,key2=value2,…
  • –bindings or -b, Gremlin binding settings, the format is: key1=value1,key2=value2,…
  • –language or -l, the language of the Gremlin script, the default is gremlin-groovy

  –file and –script are mutually exclusive, one of them must be set

• gremlin-schedule, send Gremlin statements to HugeGraph-Server to perform query or modification operations, asynchronous execution, and return the asynchronous task id immediately after the task is submitted
  • –file or -f, specify the script file to execute, UTF-8 encoding, mutually exclusive with –script
  • –script or -s, specifies the script string to execute, mutually exclusive with –file
  • –bindings or -b, Gremlin binding settings, the format is: key1=value1,key2=value2,…
  • –language or -l, the language of the Gremlin script, the default is gremlin-groovy

  –file and –script are mutually exclusive, one of them must be set

3.6 Backup/Restore Type

• backup, back up the schema or data in a certain graph out of the HugeGraph system, and store it on the local disk or HDFS in the form of JSON
  • –format, the backup format, optional values include [json, text], the default is json
  • –all-properties, whether to back up all properties of vertices/edges, only valid when –format is text, default false
  • –label, the vertex label or edge label to be backed up, only applied when –format is text; when it is set, –huge-types must name exactly one type and that type must be vertex or edge, otherwise the command fails
  • –properties, properties of vertices/edges to be backed up, separated by commas, only valid when –format is text, valid only when backing up vertices or edges
  • –compress, whether to compress data during backup, the default is true
  • –directory or -d, the directory to store schema or data, the default is ‘./{graphName}’ for local directory, and ‘{fs.default.name}/{graphName}’ for HDFS
  • –huge-types or -t, the data types to be backed up, separated by commas, the optional value is ‘all’ or a combination of one or more [vertex, edge, vertex_label, edge_label, property_key, index_label], ‘all’ Represents all 6 types, namely vertices, edges and all schemas, ‘schema’ represents the 4 schema types [vertex_label, edge_label, property_key, index_label]
  • –log or -l, specify the log directory, the default is ./logs
  • –retry, specify the number of failed retries, the default is 3
  • –thread-num or -T, the number of threads to use, default is Math.min(10, Math.max(4, CPUs / 2))
  • –split-size or -s, specifies the size of splitting vertices or edges when backing up, the default is 1048576, and it must be at least 1048576 (1M)
  • -D, use the mode of -Dkey=value to specify dynamic parameters, and specify HDFS configuration items when backing up data to HDFS, for example: -Dfs.default.name=hdfs://localhost:9000

  If –timeout is less than 120 seconds, backup (and the backup step of migrate) uses 120 seconds

• restore, restore schema or data stored in JSON format to a new graph (RESTORING mode) or merge into an existing graph (MERGING mode)
  • –directory or -d, the directory to store schema or data, the default is ‘./{graphName}’ for local directory, and ‘{fs.default.name}/{graphName}’ for HDFS
  • –clean, whether to delete the directory specified by –directory after the recovery map is completed, the default is false
  • –huge-types or -t, data types to restore, separated by commas, optional value is ‘all’ or a combination of one or more [vertex, edge, vertex_label, edge_label, property_key, index_label], ‘all’ Represents all 6 types, namely vertices, edges and all schemas, ‘schema’ represents the 4 schema types [vertex_label, edge_label, property_key, index_label]
  • –log or -l, specify the log directory, the default is ./logs
  • –retry, specify the number of failed retries, the default is 3
  • –thread-num or -T, the number of threads to use, default is Math.min(10, Math.max(4, CPUs / 2))
  • -D, use the mode of -Dkey=value to specify dynamic parameters, which are used to specify HDFS configuration items when restoring graphs from HDFS, for example: -Dfs.default.name=hdfs://localhost:9000

  restore command can be used only if –format is executed as backup for json restore requires the graph to be in RESTORING or MERGING mode (set it with graph-mode-set first), otherwise the command fails

• migrate, migrate the currently connected graph to another HugeGraphServer
  • –target-graph, the name of the target graph, the default is hugegraph
  • –target-url, the HugeGraphServer where the target graph is located, the default is http://127.0.0.1:8081
  • –target-user, the username used to access the target graph
  • –target-password, the password to access the target map
  • –target-timeout, the timeout for accessing the target map
  • –target-trust-store-file, access the truststore file used by the target graph
  • –target-trust-store-password, the password to access the truststore used by the target map
  • –directory or -d, during the migration process, the directory where the schema or data of the source graph is stored. For a local directory, the default is ‘./{graphName}’; for HDFS, the default is ‘{fs.default.name}/ {graphName}’
  • –huge-types or -t, the data types to be migrated, separated by commas, the optional value is ‘all’ or a combination of one or more [vertex, edge, vertex_label, edge_label, property_key, index_label], ‘all’ Represents all 6 types, namely vertices, edges and all schemas, ‘schema’ represents the 4 schema types [vertex_label, edge_label, property_key, index_label]
  • –log or -l, specify the log directory, the default is ./logs
  • –retry, specify the number of failed retries, the default is 3
  • –thread-num or -T, the number of threads to use, default is Math.min(10, Math.max(4, CPUs / 2))
  • –split-size or -s, specify the size of the vertex or edge block when backing up the source graph during the migration process, the default is 1048576, and it must be at least 1048576 (1M)
  • -D, use the mode of -Dkey=value to specify dynamic parameters, which are used to specify HDFS configuration items when the data needs to be backed up to HDFS during the migration process, for example: -Dfs.default.name=hdfs://localhost: 9000
  • –graph-mode or -m, the mode to set the target graph when restoring the source graph to the target graph, legal values include [RESTORING, MERGING], the default is RESTORING. The target graph is switched to this mode during the migration and switched back to its original mode afterwards
  • –keep-local-data, whether to keep the backup of the source map generated in the process of migrating the map, the default is false, that is, the backup of the source map is not kept after the default migration map ends
• schedule-backup, periodically back up the graph and keep a certain number of the latest backups (currently only supports local file systems)
  • –directory or -d, required, specifies the directory of the backup data
  • –backup-num, optional, specifies the number of latest backups to save, defaults to 3
  • –interval, an optional item, specifies the backup cycle, the format is the same as the Linux crontab format, the default is “0 0 * * *” (every day at 00:00)

  schedule-backup adds a crontab entry that runs backup -t all into {directory}/{graph}/hugegraph-backup-{yyMMddHHmm}/ and keeps only the latest –backup-num backups. A relative –directory is resolved against the hugegraph-tools home directory, and {directory}/{graph} must not exist yet

• dump, export all vertices and edges in the graph, using the vertex vertex-edge1 vertex-edge2... JSON format by default. To customize the format, implement a Formatter subclass such as CustomFormatter under hugegraph-tools/src/main/java/org/apache/hugegraph/formatter, then select it when running the command: bin/hugegraph dump -f CustomFormatter
  • –formatter or -f, specify the formatter to use, the default is JsonFormatter
  • –directory or -d, the directory where schema or data is stored, the default is ‘./{graphName}’ for local directory, and ‘{fs.default.name}/{graphName}’ for HDFS
  • –log or -l, specify the log directory, the default is ./logs
  • –retry, specify the number of failed retries, the default is 3
  • –thread-num or -T, the number of threads to use, default is Math.min(10, Math.max(4, CPUs / 2))
  • –split-size or -s, specifies the size of splitting vertices or edges when backing up, the default is 1048576, and it must be at least 1048576 (1M)
  • -D, use the mode of -Dkey=value to specify dynamic parameters, and specify HDFS configuration items when backing up data to HDFS, for example: -Dfs.default.name=hdfs://localhost:9000

3.7 Authentication data backup/restore type

• auth-backup, backup authentication data to a specified directory
  • –types or -t, types of authentication data to back up, separated by commas, optional value is ‘all’ or a combination of one or more [user, group, target, belong, access], ‘all’ represents all 5 types; ‘belong’ requires ‘user’ and ‘group’ to be included, ‘access’ requires ‘group’ and ’target’ to be included
  • –directory, directory to store backup data, the default is ‘./auth-backup-restore’ for local directory, and ‘{fs.default.name}/auth-backup-restore’ for HDFS (this option has no -d short form)
  • –retry, specify the number of failed retries, the default is 3
  • -D, use the mode of -Dkey=value to specify dynamic parameters, and specify HDFS configuration items when backing up data to HDFS, for example: -Dfs.default.name=hdfs://localhost:9000
• auth-restore, restore authentication data from a specified directory
  • –types or -t, types of authentication data to restore, separated by commas, optional value is ‘all’ or a combination of one or more [user, group, target, belong, access], ‘all’ represents all 5 types; ‘belong’ requires ‘user’ and ‘group’ to be included, ‘access’ requires ‘group’ and ’target’ to be included
  • –directory, directory where backup data is stored, the default is ‘./auth-backup-restore’ for local directory, and ‘{fs.default.name}/auth-backup-restore’ for HDFS (this option has no -d short form)
  • –retry, specify the number of failed retries, the default is 3
  • –strategy, conflict handling strategy, optional values are [stop, ignore], default is stop. stop means stop restoring when encountering conflicts, ignore means ignore conflicts and continue restoring
  • –init-password, initial password to set when restoring users, required when –types includes user
  • -D, use the mode of -Dkey=value to specify dynamic parameters, which are used to specify HDFS configuration items when restoring data from HDFS, for example: -Dfs.default.name=hdfs://localhost:9000

3.8 Install the deployment type

• deploy, one-click download, install and start HugeGraph-Server and HugeGraph-Studio
  • -v, required, specifies the HugeGraph-Server and HugeGraph-Studio version to install, must be one of the versions listed in bin/version-map.yaml (0.6, 0.7, 0.8, 0.9, 0.10), which maps it to the matching server and studio release versions
  • -p, required, specifies the installed HugeGraph-Server and HugeGraph-Studio directories
  • -u, optional, specifies the link to download the HugeGraph-Server and HugeGraph-Studio compressed packages
• clear, clean up HugeGraph-Server and HugeGraph-Studio directories and tarballs (refuses to run while a matching server or studio process is still alive, and prompts before each removal)
  • -p, required, specifies the directory of HugeGraph-Server and HugeGraph-Studio to be cleaned
• start-all, start HugeGraph-Server and HugeGraph-Studio with one click
  • -v, required, specifies the installed HugeGraph-Server and HugeGraph-Studio version to start, same values as deploy
  • -p, required, specifies the directory where HugeGraph-Server and HugeGraph-Studio are installed
• stop-all, close HugeGraph-Server and HugeGraph-Studio with one click

deploy, start-all, clear and stop-all are handed by bin/hugegraph straight to the shell scripts bin/deploy.sh, bin/start-all.sh, bin/clear.sh and bin/stop-all.sh, so the global options and environment variables in 3.2 do not apply to them.

There is an optional parameter -u in the deploy command. When provided, the specified download address will be used instead of the default download address to download the tar package, and the address will be written into the ~/hugegraph-download-url-prefix file; if no address is specified later When -u and ~/hugegraph-download-url-prefix are not specified, the tar package will be downloaded from the address specified by ~/hugegraph-download-url-prefix; if there is neither -u nor ~/hugegraph-download-url-prefix, it will be downloaded from the default download address https://github.com/hugegraph

3.9 Specific command parameters

The specific parameters of each subcommand are as follows:

Usage: hugegraph [options] [command] [command options]
  Options:
    --graph
      Name of graph
      Default: hugegraph
    --password
      Password of user
    --throw-mode
      Whether the hugegraph-tools work to throw an exception
      Default: false
    --timeout
      Connection timeout
      Default: 30
    --trust-store-file
      The path of client truststore file used when https protocol is enabled
    --trust-store-password
      The password of the client truststore file used when the https protocol 
      is enabled
    --url
      The URL of HugeGraph-Server
      Default: http://127.0.0.1:8080
    --user
      Name of user
  Commands:
    graph-create      Create graph with config
      Usage: graph-create [options]
        Options:
          --file, -f
            Creating graph config file
          --name, -n
            The name of new created graph, default is g
            Default: g

    graph-clone      Clone graph
      Usage: graph-clone [options]
        Options:
          --clone-graph-name
            The name of cloned graph, default is hugegraph
            Default: hugegraph
          --name, -n
            The name of new created graph, default is g
            Default: g

    graph-list      List all graphs
      Usage: graph-list

    graph-get      Get graph info
      Usage: graph-get

    graph-clear      Clear graph schema and data
      Usage: graph-clear [options]
        Options:
        * --confirm-message, -c
            Confirm message of graph clear is "I'm sure to delete all data". 
            (Note: include "")

    graph-drop      Drop graph
      Usage: graph-drop [options]
        Options:
        * --confirm-message, -c
            Confirm message of graph clear is "I'm sure to drop the graph". 
            (Note: include "")

    graph-mode-set      Set graph mode
      Usage: graph-mode-set [options]
        Options:
        * --graph-mode, -m
            Graph mode, include: [NONE, RESTORING, MERGING]
            Possible Values: [NONE, RESTORING, MERGING, LOADING]

    graph-mode-get      Get graph mode
      Usage: graph-mode-get

    task-list      List tasks
      Usage: task-list [options]
        Options:
          --limit
            Limit number, no limit if not provided
            Default: -1
          --status
            Status of task

    task-get      Get task info
      Usage: task-get [options]
        Options:
        * --task-id
            Task id
            Default: 0

    task-delete      Delete task
      Usage: task-delete [options]
        Options:
        * --task-id
            Task id
            Default: 0

    task-cancel      Cancel task
      Usage: task-cancel [options]
        Options:
        * --task-id
            Task id
            Default: 0

    task-clear      Clear completed tasks
      Usage: task-clear [options]
        Options:
          --force
            Force to clear all tasks, cancel all uncompleted tasks firstly, 
            and delete all completed tasks
            Default: false

    gremlin-execute      Execute Gremlin statements
      Usage: gremlin-execute [options]
        Options:
          --aliases, -a
            Gremlin aliases, valid format is: 'key1=value1,key2=value2...'
            Default: {}
          --bindings, -b
            Gremlin bindings, valid format is: 'key1=value1,key2=value2...'
            Default: {}
          --file, -f
            Gremlin Script file to be executed, UTF-8 encoded, exclusive to 
            --script 
          --language, -l
            Gremlin script language
            Default: gremlin-groovy
          --script, -s
            Gremlin script to be executed, exclusive to --file

    gremlin-schedule      Execute Gremlin statements as asynchronous job
      Usage: gremlin-schedule [options]
        Options:
          --bindings, -b
            Gremlin bindings, valid format is: 'key1=value1,key2=value2...'
            Default: {}
          --file, -f
            Gremlin Script file to be executed, UTF-8 encoded, exclusive to 
            --script 
          --language, -l
            Gremlin script language
            Default: gremlin-groovy
          --script, -s
            Gremlin script to be executed, exclusive to --file

    backup      Backup graph schema/data. If directory is on HDFS, use -D to 
            set HDFS params. For example: 
            -Dfs.default.name=hdfs://localhost:9000 
      Usage: backup [options]
        Options:
          --all-properties
            All properties to be backup flag
            Default: false
          --compress
            compress flag
            Default: true
          --directory, -d
            Directory of graph schema/data, default is './{graphname}' in 
            local file system or '{fs.default.name}/{graphname}' in HDFS
          --format
            File format, valid is [json, text]
            Default: json
          --huge-types, -t
            Type of schema/data. Concat with ',' if more than one. Other types 
            include 'all' and 'schema'. 'all' means all vertices, edges and 
            schema. In other words, 'all' equals with 'vertex, edge, 
            vertex_label, edge_label, property_key, index_label'. 'schema' 
            equals with 'vertex_label, edge_label, property_key, index_label'.
            Default: [PROPERTY_KEY, VERTEX_LABEL, EDGE_LABEL, INDEX_LABEL, VERTEX, EDGE]
          --label
            Vertex label or edge label, only valid when type is vertex or edge
          --log, -l
            Directory of log
            Default: ./logs
          --properties
            Vertex or edge properties to backup, only valid when type is 
            vertex or edge
            Default: []
          --retry
            Retry times, default is 3
            Default: 3
          --split-size, -s
            Split size of shard
            Default: 1048576
          --thread-num, -T
            Threads number to use, default is Math.min(10, Math.max(4, CPUs / 
            2)) 
            Default: 0
          -D
            HDFS config parameters
            Syntax: -Dkey=value
            Default: {}

    schedule-backup      Schedule backup task
      Usage: schedule-backup [options]
        Options:
          --backup-num
            The number of latest backups to keep
            Default: 3
        * --directory, -d
            The directory of backups stored
          --interval
            The interval of backup, format is: "a b c d e". 'a' means minute 
            (0 - 59), 'b' means hour (0 - 23), 'c' means day of month (1 - 
            31), 'd' means month (1 - 12), 'e' means day of week (0 - 6) 
            (Sunday=0), "*" means all
            Default: "0 0 * * *"

    dump      Dump graph to files
      Usage: dump [options]
        Options:
          --directory, -d
            Directory of graph schema/data, default is './{graphname}' in 
            local file system or '{fs.default.name}/{graphname}' in HDFS
          --formatter, -f
            Formatter to customize format of vertex/edge
            Default: JsonFormatter
          --log, -l
            Directory of log
            Default: ./logs
          --retry
            Retry times, default is 3
            Default: 3
          --split-size, -s
            Split size of shard
            Default: 1048576
          --thread-num, -T
            Threads number to use, default is Math.min(10, Math.max(4, CPUs / 
            2)) 
            Default: 0
          -D
            HDFS config parameters
            Syntax: -Dkey=value
            Default: {}

    restore      Restore graph schema/data. If directory is on HDFS, use -D to 
            set HDFS params if needed. For 
            example:-Dfs.default.name=hdfs://localhost:9000 
      Usage: restore [options]
        Options:
          --clean
            Whether to remove the directory of graph data after restored
            Default: false
          --directory, -d
            Directory of graph schema/data, default is './{graphname}' in 
            local file system or '{fs.default.name}/{graphname}' in HDFS
          --huge-types, -t
            Type of schema/data. Concat with ',' if more than one. Other types 
            include 'all' and 'schema'. 'all' means all vertices, edges and 
            schema. In other words, 'all' equals with 'vertex, edge, 
            vertex_label, edge_label, property_key, index_label'. 'schema' 
            equals with 'vertex_label, edge_label, property_key, index_label'.
            Default: [PROPERTY_KEY, VERTEX_LABEL, EDGE_LABEL, INDEX_LABEL, VERTEX, EDGE]
          --log, -l
            Directory of log
            Default: ./logs
          --retry
            Retry times, default is 3
            Default: 3
          --thread-num, -T
            Threads number to use, default is Math.min(10, Math.max(4, CPUs / 
            2)) 
            Default: 0
          -D
            HDFS config parameters
            Syntax: -Dkey=value
            Default: {}

    migrate      Migrate graph
      Usage: migrate [options]
        Options:
          --directory, -d
            Directory of graph schema/data, default is './{graphname}' in 
            local file system or '{fs.default.name}/{graphname}' in HDFS
          --graph-mode, -m
            Mode used when migrating to target graph, include: [RESTORING, 
            MERGING] 
            Default: RESTORING
            Possible Values: [NONE, RESTORING, MERGING, LOADING]
          --huge-types, -t
            Type of schema/data. Concat with ',' if more than one. Other types 
            include 'all' and 'schema'. 'all' means all vertices, edges and 
            schema. In other words, 'all' equals with 'vertex, edge, 
            vertex_label, edge_label, property_key, index_label'. 'schema' 
            equals with 'vertex_label, edge_label, property_key, index_label'.
            Default: [PROPERTY_KEY, VERTEX_LABEL, EDGE_LABEL, INDEX_LABEL, VERTEX, EDGE]
          --keep-local-data
            Whether to keep the local directory of graph data after restored
            Default: false
          --log, -l
            Directory of log
            Default: ./logs
          --retry
            Retry times, default is 3
            Default: 3
          --split-size, -s
            Split size of shard
            Default: 1048576
          --target-graph
            The name of target graph to migrate
            Default: hugegraph
          --target-password
            The password of target graph to migrate
          --target-timeout
            The timeout to connect target graph to migrate
            Default: 0
          --target-trust-store-file
            The trust store file of target graph to migrate
          --target-trust-store-password
            The trust store password of target graph to migrate
          --target-url
            The url of target graph to migrate
            Default: http://127.0.0.1:8081
          --target-user
            The username of target graph to migrate
          --thread-num, -T
            Threads number to use, default is Math.min(10, Math.max(4, CPUs / 
            2)) 
            Default: 0
          -D
            HDFS config parameters
            Syntax: -Dkey=value
            Default: {}

    deploy      Install HugeGraph-Server and HugeGraph-Studio
      Usage: deploy [options]
        Options:
        * -p
            Install path of HugeGraph-Server and HugeGraph-Studio
          -u
            Download url prefix path of HugeGraph-Server and HugeGraph-Studio
        * -v
            Version of HugeGraph-Server and HugeGraph-Studio

    start-all      Start HugeGraph-Server and HugeGraph-Studio
      Usage: start-all [options]
        Options:
        * -p
            Install path of HugeGraph-Server and HugeGraph-Studio
        * -v
            Version of HugeGraph-Server and HugeGraph-Studio

    clear      Clear HugeGraph-Server and HugeGraph-Studio
      Usage: clear [options]
        Options:
        * -p
            Install path of HugeGraph-Server and HugeGraph-Studio

    stop-all      Stop HugeGraph-Server and HugeGraph-Studio
      Usage: stop-all

    auth-backup      null
      Usage: auth-backup [options]
        Options:
          --directory
            Directory of auth information, default is 
            './{auth-backup-restore}' in local file system or 
            '{fs.default.name}/{auth-backup-restore}' in HDFS
          --retry
            Retry times, default is 3
            Default: 3
          --types, -t
            Type of auth data to restore and backup, concat with ',' if more 
            than one. 'all' means all auth information. In other words, 'all' 
            equals with 'user, group, target, belong, access'. In addition, 
            'belong' or 'access' can not backup or restore alone, if type 
            contains 'belong' then should contains 'user' and 'group'. If type 
            contains 'access' then should contains 'group' and 'target'.
            Default: [TARGET, GROUP, USER, ACCESS, BELONG]
          -D
            HDFS config parameters
            Syntax: -Dkey=value
            Default: {}

    auth-restore      null
      Usage: auth-restore [options]
        Options:
          --directory
            Directory of auth information, default is 
            './{auth-backup-restore}' in local file system or 
            '{fs.default.name}/{auth-backup-restore}' in HDFS
          --init-password
            Init user password, if restore type include 'user', please specify 
            the init-password of users.
            Default: <empty string>
          --retry
            Retry times, default is 3
            Default: 3
          --strategy
            The strategy needs to be chosen in the event of a conflict when 
            restoring. Valid strategies include 'stop' and 'ignore', default 
            is 'stop'. 'stop' means if there a conflict, stop restore. 
            'ignore' means if there a conflict, ignore and continue to 
            restore. 
            Default: STOP
            Possible Values: [STOP, IGNORE]
          --types, -t
            Type of auth data to restore and backup, concat with ',' if more 
            than one. 'all' means all auth information. In other words, 'all' 
            equals with 'user, group, target, belong, access'. In addition, 
            'belong' or 'access' can not backup or restore alone, if type 
            contains 'belong' then should contains 'user' and 'group'. If type 
            contains 'access' then should contains 'group' and 'target'.
            Default: [TARGET, GROUP, USER, ACCESS, BELONG]
          -D
            HDFS config parameters
            Syntax: -Dkey=value
            Default: {}

    help      Print usage
      Usage: help

3.10 Specific command example

1. gremlin statement

# Execute gremlin synchronously
./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph gremlin-execute --script 'g.V().count()'

# Execute gremlin asynchronously
./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph gremlin-schedule --script 'g.V().count()'

2. Show task status

./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph task-list

./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph task-list --limit 5

./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph task-list --status success

3. Set and show graph mode

./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-mode-set -m RESTORING

./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-mode-get

./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-list

4. Cleanup Graph

./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-clear -c "I'm sure to delete all data"

5. Backup Graph

./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph backup -t all --directory ./backup-test

6. Periodic Backup Graph

./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph schedule-backup -d ./backup --interval "*/2 * * * *"

7. Recovery Graph

# set graph mode
./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-mode-set -m RESTORING

# recovery graph
./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph restore -t all --directory ./backup-test

# restore graph mode
./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph graph-mode-set -m NONE

8. Graph Migration

./bin/hugegraph --url http://127.0.0.1:8080 --graph hugegraph migrate --target-url http://127.0.0.1:8090 --target-graph hugegraph

4 - HugeGraph-Spark-Connector Quick Start

1 HugeGraph-Spark-Connector Overview

HugeGraph-Spark-Connector uses the Spark DataFrame API to write bulk data to HugeGraph. The current implementation provides vertex and edge writers.

Reading from HugeGraph is not implemented yet: the table only implements SupportsWrite, so spark.read.format(...) is not supported. The connector supports the CUSTOMIZE and PRIMARY_KEY vertex id strategies; the AUTOMATIC strategy is rejected.

2 Environment Requirements

• Java 8+
• Maven 3.6+
• Spark 3.2.x (the module is built against Spark 3.2.2 with provided scope, so your Spark runtime must supply the Spark jars)
• Scala 2.12 (built with Scala 2.12.11)

3 Building

3.1 Build without executing tests

git clone https://github.com/apache/hugegraph-toolchain.git
cd hugegraph-toolchain
mvn clean package -pl hugegraph-spark-connector -am -DskipTests -ntp

3.2 Build with default tests

mvn clean package -pl hugegraph-spark-connector -am -ntp

Both commands produce a fat jar at hugegraph-spark-connector/target/hugegraph-spark-connector-${revision}-jar-with-dependencies.jar (Spark itself is not bundled). Pass it to spark-submit --jars when you do not manage the dependency through Maven.

4 Usage

Add the dependency to pom.xml, replacing ${revision} with the release version you use:

<dependency>
    <groupId>org.apache.hugegraph</groupId>
    <artifactId>hugegraph-spark-connector</artifactId>
    <version>${revision}</version>
</dependency>

The format string must be the full class name org.apache.hugegraph.spark.connector.DataSource; the connector does not register a short name with Spark’s DataSourceRegister service loader. When HugeGraphServer has authentication enabled, add .option("username", ...) and .option("token", ...) to the examples below.

4.1 Schema Definition Example

If we have a graph, the schema is defined as follows:

schema.propertyKey("name").asText().ifNotExist().create()
schema.propertyKey("age").asInt().ifNotExist().create()
schema.propertyKey("city").asText().ifNotExist().create()
schema.propertyKey("weight").asDouble().ifNotExist().create()
schema.propertyKey("lang").asText().ifNotExist().create()
schema.propertyKey("date").asText().ifNotExist().create()
schema.propertyKey("price").asDouble().ifNotExist().create()

schema.vertexLabel("person")
        .properties("name", "age", "city")
        .useCustomizeStringId()
        .nullableKeys("age", "city")
        .ifNotExist()
        .create()

schema.vertexLabel("software")
        .properties("name", "lang", "price")
        .primaryKeys("name")
        .ifNotExist()
        .create()

schema.edgeLabel("knows")
        .sourceLabel("person")
        .targetLabel("person")
        .properties("date", "weight")
        .ifNotExist()
        .create()

schema.edgeLabel("created")
        .sourceLabel("person")
        .targetLabel("software")
        .properties("date", "weight")
        .ifNotExist()
        .create()

4.2 Vertex Sink (Scala)

val df = sparkSession.createDataFrame(Seq(
  Tuple3("marko", 29, "Beijing"),
  Tuple3("vadas", 27, "HongKong"),
  Tuple3("Josh", 32, "Beijing"),
  Tuple3("peter", 35, "ShangHai"),
  Tuple3("li,nary", 26, "Wu,han"),
  Tuple3("Bob", 18, "HangZhou"),
)) toDF("name", "age", "city")

df.show()

df.write
  .format("org.apache.hugegraph.spark.connector.DataSource")
  .option("host", "127.0.0.1")
  .option("port", "8080")
  .option("graph", "hugegraph")
  .option("data-type", "vertex")
  .option("label", "person")
  .option("id", "name")
  .option("batch-size", 2)
  .mode(SaveMode.Overwrite)
  .save()

4.3 Edge Sink (Scala)

val df = sparkSession.createDataFrame(Seq(
  Tuple4("marko", "vadas", "20160110", 0.5),
  Tuple4("peter", "Josh", "20230801", 1.0),
  Tuple4("peter", "li,nary", "20130220", 2.0)
)).toDF("source", "target", "date", "weight")

df.show()

df.write
  .format("org.apache.hugegraph.spark.connector.DataSource")
  .option("host", "127.0.0.1")
  .option("port", "8080")
  .option("graph", "hugegraph")
  .option("data-type", "edge")
  .option("label", "knows")
  .option("source-name", "source")
  .option("target-name", "target")
  .option("batch-size", 2)
  .mode(SaveMode.Overwrite)
  .save()

4.4 Vertex Sink with PRIMARY_KEY id strategy (Scala)

For a vertex label that uses primaryKeys(...), do not set the id option: the id is spliced from the primary key columns. Columns that are not part of the schema can be dropped with ignored-fields.

val df = sparkSession.createDataFrame(Seq(
  Tuple4("lop", "java", 328L, "ISBN978-7-107-18618-5"),
  Tuple4("ripple", "python", 199L, "ISBN978-7-100-13678-5"),
)).toDF("name", "lang", "price", "ISBN")

df.write
  .format("org.apache.hugegraph.spark.connector.DataSource")
  .option("host", "127.0.0.1")
  .option("port", "8080")
  .option("graph", "hugegraph")
  .option("data-type", "vertex")
  .option("label", "software")
  .option("ignored-fields", "ISBN")
  .option("batch-size", 2)
  .mode(SaveMode.Overwrite)
  .save()

4.5 Edge Sink with mixed id strategies (Scala)

source-name and target-name follow the id strategy of their own vertex label. Below, person uses a customized string id (one column) while software uses a primary key (its name column):

val df = sparkSession.createDataFrame(Seq(
  Tuple4("marko", "lop", "20171210", 0.5),
  Tuple4("Josh", "lop", "20091111", 0.4),
  Tuple4("peter", "ripple", "20171210", 1.0),
  Tuple4("vadas", "lop", "20171210", 0.2)
)).toDF("source", "name", "date", "weight")

df.write
  .format("org.apache.hugegraph.spark.connector.DataSource")
  .option("host", "127.0.0.1")
  .option("port", "8080")
  .option("graph", "hugegraph")
  .option("data-type", "edge")
  .option("label", "created")
  .option("source-name", "source") // customize id
  .option("target-name", "name")   // primary key
  .option("batch-size", 2)
  .mode(SaveMode.Overwrite)
  .save()

Note on save modes: SaveMode.Overwrite and SaveMode.Append both insert the rows. The overwrite path does not delete existing data from the graph first.

5 Configuration Parameters

Option keys are matched case-insensitively and trimmed. data-type and label are always required; source-name and target-name are required when data-type is edge; all other options have defaults.

5.1 Client Configs

Client Configs are used to configure hugegraph-client.