Command syntax:

./conduit-utils.sh admin -verify [-stream (comma separated stream names)] [-mode (comma separated stream modes: {local,merge,mirror})] [-cluster (comma separated cluster names)] <-start (YYYY/MM/DD/HH/mm) | -relstart (minutes from now)> <-stop (YYYY/MM/DD/HH/mm) | -relstop (minutes from now)> [-numThreads (number of threads for parallel listing)] <-conf (conduit.xml file path)>

Overview:

This command performs validation of given stream(s) running in the given mode(s) on the given destination cluster(s) and reports any inconsistencies. The inconsistencies reported for each <stream, mode, destination cluster> are: finding missing paths between source and destination cluster, finding holes in source and destination clusters, finding duplicates in destination cluster (in case of merge mode).

Arguments:

-stream: comma separated stream names to validate; optional. If not given, all source streams will be derived from conduit.xml and considered as input.

-mode: comma separated stream modes; optional. Supported values are: local,merge,mirror. If not given, all modes will be considered for each stream.

-cluster: comma separated cluster names; optional. If not given, all clusters will be derived from conduit.xml and considered as input.

-start | - relstart: start time of source cluster to consider; mandatory. Either provide absolute start time or provide relative start time from now (and not both).

-stop | - relstop: stop time of source cluster to consider; mandatory. Either provide absolute stop time or provide relative stop time from now (and not both).

-numThreads: number of threads to be used to perform parallel recursive listing of source/destination clusters in the given time range; optional. Default value is 100.

-conf: the path containing conduit.xml; mandatory.

Command syntax:

./conduit-utils.sh admin -fix <-stream (stream name)> <-mode (stream mode: {local,merge,mirror})> <-cluster (cluster name)> <-start (YYYY/MM/DD/HH/mm)> <-stop (YYYY/MM/DD/HH/mm)> [-numThreads (number of threads for parallel listing)] <-conf (conduit.xml file path)>

Overview:

This command finds and fixes any inconsistencies of the given stream running in the given mode on the given destination cluster. The inconsistencies fixed by this command are: copying all missing paths between source and destination cluster, fixing holes in the destination cluster (in case of merge mode).

If any hole is found in source cluster (in case of mirror mode), this command will throw an exception. User should first run the command on source cluster to fix its inconsistency and then re-run on destination cluster.

NOTE: THIS COMMAND NEEDS SHUTDOWN OF THE CONDUIT WORKER RUNNING ON DESTINATION CLUSTER. The reason is that this command will actually perform data movement by copying data from source cluster to destination cluster. This data movement is not synchronized with any conduit worker runnning at the same destination cluster, and hence they both may interfere with each other.

Also in production setup, this command should be run with the same user login ('conduit') as used to start conduit worker in production cluster.

Arguments:

-stream: stream name whose inconsistency is to be fixed; mandatory.

-mode: stream mode; mandatory. Supported values are: local,merge,mirror.

-cluster: cluster name where input stream is running in input mode; mandatory.

-start: absolute start time of source cluster to consider; mandatory.

-stop: absolute stop time of source cluster to consider; mandatory.

-numThreads: number of threads to be used to perform parallel recursive listing of source/destination clusters in the given time range; optional. Default value is 100.

-conf: the path containing conduit.xml; mandatory.

Command syntax:

./conduit-utils.sh admin -checkpoint <-stream (stream name)> <-destCluster (destination cluster)> [-srcCluster (source cluster)] <-date (YYYY/MM/DD/HH/mm)> <-conf (conduit.xml file path)>

Overview:

This command creates checkpoint of the given stream on the given destination cluster corresponding to the given source cluster. The checkpoint input is taken in the above date format.

Arguments:

-stream: stream name whose checkpoint is to be created; mandatory.

-destCluster: destination cluster where checkpoint will be created and stored; mandatory.

-srcCluster: source cluster whose checkpoint is being written; mandatory.

-date: the checkpoint value of source cluster; mandatory.

-conf: the path containing conduit.xml; mandatory.