Command-line Interface

The zebclient binary provides several administrative commands through its command-line interface.

$ zebclient -h
...
COMMANDS:
   config         Outputs available configuration variables
   memlimit       Shows how much memory is needed for the given config name
   mount          Mount a filesystem for the given config name
   store          Start a blockstore with given config name
   stats          Show real time performance statistics
   stat           Display inode status
   status         Show backend status
   verify-shards  Verify the integrity of local shard files
   fsck           Check consistency of a filesystem
   license        License management
   gc             Garbage collector of objects in data storage
   inlet          Access external buckets from within filesystem

Most commands require a functioning ZebClient configuration. By default, the configuration is picked up from environment variables in the process environment. The environment variables can also be loaded from a file specified with command-line option --env-file.

Agent variables use the ZM prefix and the DRPC server variables use ZB.

Multiple configurations for more than one ZebClient instance can coexist in the same environment or environment file. These configurations are distinguished by a subprefix or "configuration name". For example, an environment file containing configurations for one agent and two DRPC servers might use variable prefixes ZM_AGENT, ZB_SERVER0, and ZB_SERVER1. All commands that require a functioning configuration accept the configuration name (e.g., AGENT, or SERVER0) as the first argument on the command-line in order to find the correct configuration.

Generic Commands

The following commands are available both on agent nodes and DRPC server nodes.

zebclient config

The zebclient config command outputs a comprehensive configuration variable reference in table format. This command does not require a functioning configuration. Example:

$ zebclient config
┌──────────────────────────────────────────────────┬──────────┬────────────────────────────────────┬────────────────────┬───────────────────┬──────────────────────┬────────┐
│                       Name                       │   Type   │              Required              │      Default       │      Options      │        Range         │ Base64 │
├──────────────────────────────────────────────────┼──────────┼────────────────────────────────────┼────────────────────┼───────────────────┼──────────────────────┼────────┤
│ ZM_PROD_SERVICE_LOGLEVEL                         │ string   │ true                               │ warn               │ error             │                      │        │
│                                                  │          │                                    │                    │ warn              │                      │        │
│ Log level.                                       │          │                                    │                    │ info              │                      │        │
│                                                  │          │                                    │                    │ debug             │                      │        │
│                                                  │          │                                    │                    │ devel             │                      │        │
├──────────────────────────────────────────────────┼──────────┼────────────────────────────────────┼────────────────────┼───────────────────┼──────────────────────┼────────┤
...

Agent-only Commands

The following commands are exclusive to agent nodes.

zebclient gc

The zebclient gc command can detect and delete leaked objects i.e. shards that aren't referenced by the metadata database. This situation may occur, for instance, after restoring a metadata database backup. The system will function as normal however there will be data in the cold bucket that will never be accessed. Use the zebclient gc command to free up this space.

This command requires a functioning configuration. Example:

$ zebclient gc --env-file /etc/zebclient/agent.env --delete AGENT
...
  Listed slices count: 24                            
Deleted pending count: 0                             
Scanned objects count: 73 / 73 [==============================================================]  done      
  Valid objects count: 72                            
  Valid objects bytes: 1.00 GiB (1073742353 Bytes)   
 Leaked objects count: 1                             
 Leaked objects bytes: 29.00 b  (29 Bytes)           
Skipped objects count: 0                             
Skipped objects bytes: 0.00 b   (0 Bytes)    

Currently zebclient gc only supports direct mode and K+M=1+0.

zebclient fsck

The zebclient fsck (file system check) command detects broken inodes i.e. files that are missing ZCFS blocks. This situation can occur if multiple warm nodes are lost before clearing their upload queues. This command requires a functioning configuration.

In the example below, inode number 6 linked by path /abc is missing data:

$ zebclient fsck --env-file /etc/zebclient/agent.env AGENT
  Found blocks count: 72                            
  Found blocks bytes: 1.00 GiB (1073742353 Bytes)   
 Listed slices count: 25                            
Scanned slices count: 25 / 25 [==============================================================]  done      
Scanned slices bytes: 1.00 GiB (1073742382 Bytes)   
   Lost blocks count: 1                             
   Lost blocks bytes: 29.00 b  (29 Bytes)           
broken inode:             6  path: /abc

Note that zebclient fsck refuses to run by default if agents are mounted. Use --force to override.

Currently zebclient fsck only supports direct mode and K+M=1+0.

zebclient inlet

The zebclient inlet command manages inlets (external, read-only S3 buckets mounted within the ZCFS file system). There are three subcommands:

$ zebclient inlet -h
   add     Add an inlet, e.g. external S3 bucket
   list    List existing inlets in human-readable form
   remove  Remove an inlet

All subcommands require a functioning configuration.

zebclient inlet add

The zebclient inlet add command mounts an external S3 bucket to a path within the ZCFS file system i.e. the contents of the bucket will be visible as (read-only) directories and files under the ZCFS mount point. All external data is retrieved on-demand, both metadata and file data. Example:

$ zebclient inlet add AGENT \
    --env-file /etc/zebclient/agent.env
    --inletname=aws-mybucket \
    --inode=42 \
    --dirname=mypictures \
    --dirmode=0755 \
    --filemode=0644 \
    --uid=1000 \
    --gid=1000 \
    --protocol=s3 \
    --uri=https://s3.us-east-1.amazonaws.com \
    --region=us-east-1 \
    --bucket=mybucket \
    --accesskey=AFIOHTGJWERITYNWREMN \
    --secretkey=Xmdfgi283u4sadkmfgklasdm32498u5384sdkmfa

Explanation of the command-line options:

• --inletname is just an identifier for the inlet. It is only needed when using zebclient inlet remove.

• --inode is the inode number of the parent directory where the external bucket should be mounted. Use 1 for the root directory of the ZCFS file system. Use stat -c '%i' /zcfs/some/path to retrieve the inode number of subdirectories.

• --dirname is the name of the inlet directory i.e. the directory representing the external bucket. If missing, the directory will be created (under the --inode parent directory).

• --dirmode is the permission of the directories under the inlet directory i.e. the directories representing the folder objects in the bucket.

• --filemode is the permission of the files under the inlet directory i.e. the files representing the objects in the bucket.

• --uid is the owner of the files and directories under the inlet directory.

• --gid is the group of the files and directories under the inlet directory.

• --protocol is the bucket protocol. Currently only s3 is supported.

• --uri is the bucket endpoint.

• --region is the bucket region.

• --bucket is the name of the external bucket.

• --accesskey is the access key used to access the external bucket.

• --secretkey is the secret key used to access the external bucket.

zebclient inlet list

The zebclient inlet list command lists registered inlets. Example:

$ zebclient inlet add AGENT --env-file /etc/zebclient/agent.env
┌───────┬───────┬─────────┬──────┬──────┬─────────┬──────────┬──────────┬───────────────────────┬───────────┬──────────┐
│ Inlet │ Inode │ Dirname │ UID  │ GID  │ DirMode │ FileMode │ Protocol │          URI          │  Region   │  Bucket  │
├───────┼───────┼─────────┼──────┼──────┼─────────┼──────────┼──────────┼───────────────────────┼───────────┼──────────┤
│ minio │ 1     │ minio   │ 1000 │ 1000 │ 0755    │ 0644     │ s3       │ http://localhost:9000 │ us-east-1 │ mybucket │
└───────┴───────┴─────────┴──────┴──────┴─────────┴──────────┴──────────┴───────────────────────┴───────────┴──────────┘

zebclient inlet remove

The zebclient inlet list command removes a previously registered inlet. Example:

$ zebclient inlet remove AGENT --inletname minio
unmount request for inlet 'minio' added

zebclient license

The zebclient license command manages the ZebClient license. There are three subcommands:

$ zebclient license -h
...
   install    Install license
   status     Check license validity
   uninstall  Uninstall license

All subcommands require a functioning configuration.

zebclient license install

The zebclient license install subcommand loads a ZebClient license file into the database. Example:

$ zebclient license install --env-file /etc/zebclient/agent.env AGENT /tmp/zebclient.lic 
=== System information ===
Filesystem ID: 1bf2f84d-8a1d-4534-9778-611cb7c4aa34

=== License information ===
License source:        database
License type:          offline
Product ID:            9a9401cd-c4cc-4e5c-8acc-166a5fb11132
Key:                   f57951d3-87d8-4a5f-9142-3bddd5788588
Expires:               2024-06-18 23:37:51
Grace period ends:     2024-06-18 23:37:51
Read-only period ends: 2024-06-18 23:37:51

License installed.

Note that, as long a license file is loaded into the database, the ZM_AGENT_LICENSE_KEY environment variable is ignored.

There is no need to restart the agents after installing a license file. Any running agents will automatically detect the new license.

zebclient license status

The zebclient license install subcommand outputs the license information of the system. When using an offline license, the license properties are displayed as well. Example:

$ zebclient license status --env-file /etc/zebclient/agent.env AGENT
=== System information ===
Filesystem ID:  23002aba-212a-4bac-a687-8343fd199043
License status: valid

=== License information ===
License source:        database
License type:          offline
Product ID:            9a9401cd-c4cc-4e5c-8acc-166a5fb11132
Key:                   bfaa559a-43fb-4129-853d-8abca3694277
Expires:               2025-05-07 14:33:40
Grace period ends:     2025-05-22 14:33:40
Read-only period ends: 2025-06-21 14:33:40
Capacity:              1.0 TB
Max logical cores:     64

License OK.

When using an online license, the zebclient license status command currently just verifies that the license key itself is correctly formatted. The license status is not validated with the license server. Example:

$ zebclient license status --env-file /etc/zebclient/agent.env AGENT
=== System information ===
Filesystem ID: 23002aba-212a-4bac-a687-8343fd199043

=== License information ===
License source: environment variable
License type:   online
Product ID:     be81fecb-a6df-4a00-b3e8-5d9c7e30f25f
Key:            C6B64E-82B350-4C9687-BE0DAB-852A7A-2CD9F2

License OK.

zebclient license uninstall

The zebclient license uninstall subcommand removes any license loaded into the database. Example:

$ zebclient license uninstall --env-file /etc/zebclient/agent.env AGENT
License uninstalled.

zebclient memlimit

The zebclient memlimit command displays how much memory a certain agent configuration requires.

zebclient mount

The zebclient mount command starts the agent service which mounts the ZCFS file system. It is normally executed by systemd, see /etc/systemd/system/zebclient-agent.service. This command requires a functioning configuration. Example:

$ zebclient mount --env-file /etc/zebclient/agent.env AGENT

zebclient stat

The zebclient stat command outputs detailed slice information for inodes. This command requires a functioning configuration.

In following example we show the slice information for inode number 5:

$ zebclient stat --env-file /etc/zebclient/agent.env AGENT 5
Inode: 5
Size: 1073741824 (1.0 GiB)
Chunks: 16

Chunk 0:
Unique slices: 1 (67108864 bytes = 64.0 MiB)
Slices gaps: 0 (0 bytes = 0)
Slices references: 1
...

The inode number for a file can be retrieved with e.g. the standard stat command.

zebclient stats

The zebclient stats command outputs real-time performance statistics for an agent mount point. Example:

$ zebclient stats /zcfs
------usage------ ----------fuse--------- ----meta--- -blockcache ---object--
 cpu   mem   buf | ops   lat   read write| ops   lat | read write| get   put
 5.0%  215M 9344K|  88  26.7  3480K 1536K|  40  0.45 |2064K 1792K|   0  1285K
 6.4%  219M 8448K| 115  18.2  6420K 1664K|  55  0.54 |2688K 1792K|   0  1927K
 6.3%  221M 7884K|  93  21.2  4376K 1536K|  51  0.58 |3144K 1664K|   0  1413K
...

zebclient status

The zebclient status command reports the backend status. This command requires a functioning configuration. Example:

$ zebclient status --env-file /etc/zebclient/agent.env AGENT
┌──────────────────────┬─────────────────────────┬────────┬────────┬────────┬──────┬─────────┬───────┬─────────┬──────┐
│       Backend        │          Time           │  Size  │  Used  │ Avail  │ Use% │ ReqTime │ Flush │ Recover │ Note │
├──────────────────────┼─────────────────────────┼────────┼────────┼────────┼──────┼─────────┼───────┼─────────┼──────┤
│ drpc:server0:17303   │ Mon May 6 15:08:26 2024 │ 955.2G │ 582.6G │ 321.4G │ 60%  │ 400µs   │ idle  │ init    │      │
│ drpc:server1:17303   │ Mon May 6 15:08:26 2024 │ 955.2G │ 582.6G │ 321.4G │ 60%  │ 400µs   │ idle  │ init    │      │
│ drpc:server2:17303   │ Mon May 6 15:08:26 2024 │ 955.2G │ 582.6G │ 321.4G │ 60%  │ 400µs   │ idle  │ init    │      │
└──────────────────────┴─────────────────────────┴────────┴────────┴────────┴──────┴─────────┴───────┴─────────┴──────┘

DRPC Server-only Commands

The following commands are exclusive to DRPC server nodes.

zebclient store

The zebclient store command starts the DRPC block store service which serves shards to the agents. It is normally executed by systemd, see /etc/systemd/system/zebclient-server*.service. This command requires a functioning configuration. Example:

$ zebclient store --env-file /etc/zebclient/server0.env SERVER0

zebclient verify-shards

The zebclient verify-shards command can be executed on the DRPC server nodes in order to verify the integrity of the local shards. This command requires a functioning configuration. Example:

$ zebclient verify-shards --env-file /etc/zebclient/server0.agent SERVER0
zcfs/1e45d7a4-dec6-4635-ad0d-5297167a3850/0/chunks/0/1_0_31: OK
...

Provide command-line flags --only-failing and --only-name for an output that is suitable for piping to e.g. xargs rm.