How will “persistency” work

There are 4 kind of "persistency" under .persist_dir\db_loader

1. .persist_dir\db_loader\main\dlq – samples that failed to insert into DB, because some not recoverable reason: wrong sample, duplicated samples, samples with time stamp older then data purging period. No limit for samples count there- only by disk space. No automatic purging of "old" files from there. Samples can be "tried" to be inserted again, but generally no reason that they will success to be inserted once they failed. Especially is data tables for sample time stamp already purged.

The rest persistency folders are

2. .persist_dir\db_loader\main\current – samples that are in loader memory now. Size is limited by memory restrictions of db loader

3. .persist_dir\db_loader\flattenfailure – hierarchy samples (trans_t) that are temporary failed to open hierarchy because DB connectivity problem. No limit – only by disk size.

4. .persist_dir\db_loader\recovery – samples that are temporary failed to be inserted – usually because DB availability issues. Limit is 509 files per sample type (8,192 samples in file) ~ 4M samples per sample type. After this limit, loader stops to work and refuses to read data from BUS


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s