Conduit
- Conduit
- Overview
- Design
- Dev setup
- ConduitLite
- Audit
- Visualization
- Admin Script
- Monitoring
Project Documentation
%toc
Conduit Visualization project helps in visualizing data that has been delivered by conduit within a given time period. The dashboard allows one to select the time range, cluster and stream for which they wish to see the graph.
The dashboard will query the database populated by the feeder service to get relevant information. Data flows covers all tiers
Install the deb package databus-visualization.
On installing the package, conf files are copied to /usr/local/conduit-visualization/conf, the war is copied directly to /var/lib/tomcat7/webapps and the environment file setenv.sh is copied to /var/lib/tomcat7/bin
The conf files are audit-feeder.properties(same as that used for feeder service) and visualization.properties
audit-feeder.properties visualization.properties
All fields prperties in visualization.properties should be set. Mandatory fields in audit-feeder.properties are jdbc.driver.class.name, db.url, db.username, db.password, audit.table.master and rollup.tilldays
Hadoop home path should be set in /var/lib/tomcat7/bin/setenv.sh file which is copied on installing the deb, before restarting tomcat
Config Fields in visualization.properties
| Config Name | Desription | Example |
|---|---|---|
| conduit.xml.path | The path where databus.xml is stored according to which cluster and stream configuration for dashboard are loaded | /usr/local/databus/conf/databus.xml |
| percentile.string | Comma seperated list of percentiles for calculating latency. Any number of percentiles can be given but minimum is one | 80,90,95,99,99.9,99.99 |
| publisher.sla | The SLA in minutes for all the messages to reach publisher | 2 |
| agent.sla | The SLA in minutes for all the messages to reach agent | 2 |
| vip.sla | The SLA in minutes for all the messages to reach vip | 2 |
| collector.sla | The SLA in minutes for all the messages to reach collector | 2 |
| hdfs.sla | The SLA in minutes for all the messages to reach hdfs | 3 |
| local.sla | The SLA in minutes for all the messages to reach local tier | 3 |
| merge.sla | The SLA in minutes for all the messages to reach merge tier | 5 |
| mirror.sla | The SLA in minutes for all the messages to reach mirror tier | 6 |
| percentile.for.sla | Note: This value should be a part of percentile.string list | 99.99 |
| percentage.for.loss | The percentage that is to be taken as loss margin. If difference between numbers of two to-be compared entities is more than percentage.for.loss of the compared-to entity then it is considered to be in unhealthy state. | 0.2 |
| percentage.for.warn | The percentage that is to be taken as warn margin. If difference between numbers of two to-be compared entities is more than percentage.for.warn of the compared-to entity then the entity is considered to be in warning Note: percentage.for.warn should be less than percentage.for.loss | 0.02 |
| max.start.time | The maximum start time before which a start date cannot be selected in the filter. The date should be entered in the format of dd-MM-yyyy | Date before 01-08 say 31-07 cannot be selected in the date filter of dashboard. |
| max.time.range.interval.in.hours | The maximum time range in hours that user can query for in the dashboard. | 24 |
| loss.warn.threshold.diff.in.mins | Difference in minutes for loss SLA and warn SLA . The difference is the same for all tiers. | For hdfs tier, if hdfs.sla 3, then if latency >3, it is considered to be in unhealthy state. If latency <3 and latency > 3-(loss.warn.threshold.diff.in.mins) then it is in warn state, and healthy if latency <= 3-(loss.warn.threshold.diff.in.mins) |