Allows you to look for strange or missing values in SkySpark.

zivaDataQualityExt

Registered StackHub users may elect to receive email notifications whenever a new package version is released.

There are **0** watchers.

v2.6.0

Welcome to SkySpark's Data Quality Extension!

Find Missing Values

- This rule runs at the point level.
- Do rollups of the expected interval of the data. Then, look for empty periods.
`zivaDQTSGap()`

does the same thing, but is more advanced.- Side Effects: None.
- Input: A point, a date/date range, and an expected interval.
- Output: hisDurGrid
`zivaDQFindNulls(read(temp and discharge), pastMonth, 20min)`

Find Noise

- This rule runs at the point level.
- This rule looks for data groups that have changed by a certain percentage from a previous group.
- The percentage will be determined automatically if no delta and dur parameters are given.
- Side Effects: None.
- Input: A point, a date/date range, an optional delta, and an optional duration.
- Output: hisDurGrid
- zivaDQFindNoise(read(temp and discharge), pastMonth)
- zivaDQFindNoise(read(temp and discharge), pastMonth, 5%, 30min)

Find Oddities

- This rule runs at the point level.
- Finds nulls, NAs, or numbers less than or equal to 0.
- Side Effects: None.
- Input: A point and a date/date range.
- Output: hisDurGrid
`zivaDQFindOddities(read(temp and discharge), pastMonth)`

Find Outliers Based on Standard Deviation

- This rule runs at the point level.
- This rule has been modified to rely on the hisDQAverage and hisDQstdDev tags that come from the DQJob.
- sDs is the number of standard deviations you want this rule to allow for before a spark is detected. If a history record is below or above 3 stdDev, it will cause a spark in this form (you can change this).
- Side Effects: None.
- Input: A point, a date/date range, and optionally, how many standard deviations to go out.
- Output: hisDurGrid
- zivaDQFindOutliers(read(temp and discharge), pastMonth, 4)

Find Values too High.

- This rule runs at the point level.
- Find Outliers based on threshold and its inverse.
- Dependencies: None.
- Side Effects: None.
- Input: A point, a date/date range, and a threshold.
- Output: hisDurGrid
`zivaDQOutlierThresh(read(temp and discharge), pastMonth, 100)`

Find Outliers using ZScore

- This rule runs at the point level.
- Use tags from data quality job to look for outliers.
- Side Effects: None.
- Input: A point, a date/date range, and optional ZScore override.
- Output: hisDurGrid
`zivaDQZScoreOutofBounds(read(temp and discharge), pastMonth, 0.9)`

Data Quality Job

- Run as task
- Adds Descriptive Tags to Numeric and Boolean Points
- Run this as a job weekly
- reInitialize tells SkySpark how often to update tags
- Monthly running yearly total normalizes for many things.
- Dependencies: Calculus and Signal Analysis Pods.
- Side Effects: Adds several statistical tags to all numeric points that can be used for complex rules. Tags are also added to Boolean points.
- Input: Optional reInitialize interval and option how far back to go (default is 1mo and pastYear).
- Output: None.
`zivaDQJob()`

Data Quality Job for Booleans

- Run as task
- Adds Descriptive Tags to Boolean Points
- Run this as a task weekly
- reInitialize tells SkySpark how often to update tags
- Monthly running yearly total normalizes for many things.
- Dependencies: Calculus and Signal Analysis Pods.
- Side Effects: Adds several tags to Boolean points.
- Input: Optional reInitialize interval and option how far back to go (default is 1mo and pastYear).
- Output: None.
`zivaDQJobBool()`

Data Quality Job for Numbers

- Run as task
- Adds Descriptive Tags to Numeric Points
- Run this as a task weekly
- reInitialize tells SkySpark how often to update tags
- Monthly running yearly total normalizes for many things.
- Dependencies: Calculus and Signal Analysis Pods.
- Side Effects: Adds several statistical tags to all numeric points that can be used for complex rules.
- Input: Optional reInitialize interval and option how far back to go (default is 1mo and pastYear).
- Output: None.
`zivaDQJobNumber()`

Find Outliers

- This rule runs at the point level.
- Use tags from data quality job to look for outliers.
- Side Effects: None.
- Input: A point, a date/date range, and optional buffer.
- The buffer determines what percentage of the pastYear's absolute min or max is acceptable.
- if old: {min: 50, max: 100, buffer: 0.8} then run against: {minThresh: 62.5, maxThresh: 80} (50/0.8 = 62.5 and 100*0.8 = 80)
- Output: hisDurGrid
`zivaDQOutofRange(read(temp and discharge), pastMonth, 0.7)`

Find Rate of Change too High.

- This rule runs at the point level.
`zivaDQRateofChange()`

does the same thing, but is more advanced.- Use tags from data quality job to look for rate-of-change outliers.
- Dependencies: Calculus and Signal Analysis Pods.
- Side Effects: None.
- Input: A point, a date/date range, and an optional buffer.
- The buffer determines what percentage of the pastYear's absolute max negative or max positive rate of change is acceptable.
- if oldROC: {maxNegROC: -10, maxPosROC: 10, buffer: 0.8} then run against: {maxNegROCThresh: -8, maxPosROCThresh: 8} (-10*0.8 = -8 and 10*0.8 = 8)
- Output: hisDurGrid
`zivaDQRateofChange(read(temp and discharge), pastMonth, 0.7)`

Find Rate of Change too High.

- This rule runs at the point level.
- Use tags from data quality job to look for rate-of-change outliers.
- Dependencies: Calculus and Signal Analysis Pods.
- Side Effects: None.
- Input: A point, a date/date range, and a threshold.
- Output: hisDurGrid
`zivaDQRateofChangeThresh(read(temp and discharge), pastMonth, 10)`

Find Gaps in Data.

- This rule runs at the point level.
- Use tags from data quality job to look for missing data/intervals.
- Side Effects: None.
- Input: A point, a date/date range, and an optional buffer.
- The buffer determines what percentage of the pastYear's median ts interval is acceptable.
- if oldTSInt: {medTSInt: 15min, buffer: 1.9} then run against: {medTSIntThresh: 28.5min} (15min*1.9 = 28.5min)
- Output: hisDurGrid
`zivaDQTSGap(read(temp and discharge), pastMonth, 1.8)`

Find Where Energy Out is not in Line with other Records

- This rule runs at the point level.
- Use tags from data quality job to look for energy in/out outliers.
- This rule is currently designed to run on
`zone air temp sensor`

points and then go to the equip (usually ahu) level, find the`weather temp`

(via`siteRef`

and`weatherStationRef`

) and finally the`siteMeter energy point`

- Side Effects: None.
- Input: A
`zone air temp point`

, a date/date range, and optional buffer. - The buffer determines what percentage of the pastYear's absolute min or max is acceptable. if old: {min: 50, max: 100, buffer: 0.8} then run against: {minThresh: 62.5, maxThresh: 80} (50/0.8 = 62.5 and 100*0.8 = 80)
- Output: hisDurGrid
`zivaDQMismatchedEnergyUsage(read(zone and air and temp and sensor), pastMonth, 0.7)`

Data Quality Cleaner

- Run as task or manually
- Removes Descriptive Tags (essentially used for migration and does not need to be run regularly)
- Side Effects: Removes tags - intended effect
- Input: None
- Output: None
`zivaDQCleaner()`

Please contact adam@ziva-tech.com with any questions.

Published by Ziva Tech Solutions

Package details

Version | 2.6.0 |
---|---|

License | n/a |

Build date | 7 months ago on 22nd Nov 2023 |

Depends on | |

File name | `zivaDataQualityExt.pod` |

File size | `19.44 kB` |

MD5 | `fc95afde08d297b430e81f1cb384535b` |

SHA1 | `ea67ae186d01916afebe4730f9b75c877fd073f9` |

Published by Ziva Tech SolutionsDownload nowAlso available via SkyArc Install Manager |

Tags

Pod File