Before start of service

Implement rollup.sql packaged with the deb in the bin directory.

psql -h <host_name> -U <user_name> -d <db name> -f rollup.sql

Configuration

Apart from the configurations needed to run feeder service, rollup service needs the below configurations to be mentioned in audit-feeder.properties file present in the conf path

Assumptions

1. Rollup service continues the creation of daily tables and the assumption is that for the first time a feeder is started in a new db, tables till current day are already present.
2. If a table with prefix hourly_ for a particular day is present, then the rollup for that date is complete

Scenarios

1. For the first run of rollup service, first time entry from the master table is picked up. So if first time entry is at some hour time say 05:10, 05:10 will be changed to first millisecond of that day i.e. 00:00 to give fromTime. toTime will be (current day - till days). toTime, as mentioned in the description for rollup.tilldays conf, is open boundary so the day table of toTime will not be rolled up and this toTime is set as the checkpoint.
2. If the service is started for the first time, service runs immediately without sleeping till rollup hour of that day and from second run onwards, the service starts a run at the rollup hour.
3. If rollup.hour = k and rollup service starts at hour m with m < k, then rollup happens twice for the same day, once during start up and second time at hour k. So, during the second run, no tables will be rolled up or created.
4. If the rollup of tables in a run is successful, then the checkpoint is returned as the first millisecond of the day from which to start next run.
5. If checkpoint is present for a run, fromTime is picked from checkpoint. fromTime is closed lower boundary so rollup of day tables will start from the day of fromTime
6. If rollup.tilldays = k(int), then k number of tables before current date(excluding current date) will have minute level data i.e k number of tables before current date will not be rolled up. Example if rollup.tilldays = 0, then all tables till current date will be rolled up.
7. If checkpoint is not found(corrupted/deleted), then the fromTime will be picked up from the master table. The way it is picked up is that, if the last time is say 23:10 of day 24, then last time will be modified to 00:00 of day 24 and it will check if the daily table corresponding to this date has been rolled up. If the date is rolled up, then date = date -1(i.e prev day), else it would return the date of the next day i.e date+1 as the fromTime
8. When rollup.tilldays conf is increased say from m to n (m < n)
   1. If checkpoint exists then for m-n runs after restart of service, no tables will be rolled up.
   2. If checkpoint doesn't exist, then also m-n runs will not roll up any tables as fromTime(t-m+1) will be after toTime(t-n)
9. When rollup.tilldays conf is decreased say from m to n (m > n)
   1. If checkpoint exists, in the first run after restarting the rollup service, more than one table will be rolled up.
   2. If checkpoint doesn't exist, then in first run, t-m+1 date will be picked up and will be rolled up till t-n, and more than one table will be rolled up.
10. If rollup service run is successful, then all the daily tables that have been rolled up will be dis-inherited but not dropped
11. If rollup fails at any point i.e at hourly_ table creation or inserting records or disinheriting master table by daily table or inheriting master table by rolled up table, then the whole transaction will be rolled back i.e either a complete rolled up table exists or no table exists. No possibility of a partially created rolled up table.
12. If rollup fails for any table, then for all tables in batch, rollup does not happen and checkpoint is not marked.
13. In the scenario when a rolled up table is dropped and we wish to create roll up table for that day then alter the daily table for that day to inherit the master table since the daily table would have been modified to no inherit
14. If creation of daily tables fails for any table, then no daily tables would be created during that run
15. If checkpoint is corrupted/deleted, then fromTime is the time from when rolled up tables do not exist. So if the tables are like till day x they are rolled up, x+1 to x+k days are not rolled up, and from x+k+1 are rolled up till x+m then x+m+1 will be the fromTime. Tables from days x+1 to x+k will not be rolled up by rollup service. This would create a hole in the tables that have been rolled up but rollup admin script can fix this by giving x+1 as start time and number of days as k-1

Rollup Daily Tables

Usage:

bin/audit-client admin -rollup -date <dd-mm-yyyy> [-n <number of days>] --conf <conf path>

Default value of number of days is 1

Example 1:

Consider the scenario when we are trying to rollup only one daily table i.e number of days is not mentioned, so default value 1 is picked up.Current date is 21-11-2013 and rollup.tilldays =1, so days after 19-11-2013 cannot be rolled up

bin/audit-client admin -rollup -date 17-11-2013 --conf conf/

-----------

SUCCESS

----------- audit_db=> \dt

  List of relations
    Schema | Name | Type | Owner 
    ----------------------------------------------------------
    public | daily_conduit_summary | table | audit
    public | daily_conduit_summary20131116 | table | audit
    public | daily_conduit_summary20131117 | table | audit
    public | daily_conduit_summary20131118 | table | audit
    public | daily_conduit_summary20131119 | table | audit
    public | daily_conduit_summary20131120 | table | audit
    public | daily_conduit_summary20131121 | table | audit
    public | hourly_daily_conduit_summary20131116 | table | audit
    public | hourly_daily_conduit_summary20131117 | table | audit

Example 2:

Consider the scenario when number of days >1. If number of days = k, then tables from date to date+k-1 will be rolled up

 bin/audit-client admin -rollup -date 18-11-2013 -n 2 --conf conf/
 ---------
 SUCCESS 
 ---------

audit_db=> \dt

   List of relations
    Schema | Name | Type | Owner 
    ---------------------------------------------------------------------------------------------
    public | daily_conduit_summary | table | audit
    public | daily_conduit_summary20131116 | table | audit
    public | daily_conduit_summary20131117 | table | audit
    public | daily_conduit_summary20131118 | table | audit
    public | daily_conduit_summary20131119 | table | audit
    public | daily_conduit_summary20131120 | table | audit
    public | daily_conduit_summary20131121 | table | audit
    public | hourly_daily_conduit_summary20131116 | table | audit
    public | hourly_daily_conduit_summary20131117 | table | audit
    public | hourly_daily_conduit_summary20131118 | table | audit
    public | hourly_daily_conduit_summary20131119 | table | audit
 

Modify/Create checkpoint

Usage:

 bin/audit-client admin -checkpoint -date <dd-mm-yyyy> --conf <conf path>

Example:

Consider the scenario when we want to mark checkpoint to date 21-11-2013 in IST TZ

 bin/audit-client admin -checkpoint -date 21-11-2013 --conf conf/
 ------------------
 SUCCESS 
 ------------------

cat ~/rollupChkPt/rollup.ck

1384972200000

Check rolledup table existence

Usage:

 bin/audit-client admin -check -rolledup -date <dd-mm-yyyy> [-n <number of days to check>] --conf <conf path>

Example 1:

Consider the same db used in above examples and number of days to check is not passed, so default value 1 is picked up

 bin/audit-client admin -check -rolledup -date 19-11-2013 --conf conf/
 ---------
 SUCCESS 
---------

 bin/audit-client admin -check -rolledup -date 21-11-2013 --conf conf/
 Table not rolled up for date: 21-11-2013
------
 FAIL 
------

Example 2:

Consider the same db used in above examples and some value k is passed as number of days to check

 bin/audit-client admin -check -rolledup -date 19-11-2013 -n 3 --conf conf/
Table not rolled up for date: 20-11-2013
Table not rolled up for date: 21-11-2013
 ------
 FAIL 
------

Check daily table existence

Usage:

 bin/audit-client admin -check -created -date <dd-mm-yyyy> [-n <number of days to check>] --conf <conf path>

Example 1:

Consider the same db used in above examples and number of days to check is not passed, so default value 1 is picked up

 bin/audit-client admin -check -created -date 24-11-2013 --conf conf/
 ------------
  SUCCESS 
 ------------

 bin/audit-client admin -check -created -date 25-11-2013 --conf conf/
 Table not created for date: 25-11-2013
 ------------
  FAIL 
 ------------

Example 2:

Consider the same db used in above examples and some value k is passed as number of days to check

 bin/audit-client admin -check -created -date 30-11-2013 -n 3 --conf conf/
    Table not created for date: 30-11-2013
    Table not created for date: 01-12-2013
    Table not created for date: 02-12-2013
  ------
   FAIL 
  ------