sfxDataQualityExt 2.4.3 - Allows you to look for strange or missing values in SkySpark.

sfxDataQualityExt

Registered StackHub users may elect to receive email notifications whenever a new package version is released or a comment is posted on the forum.

There are 9 watchers.

v2.4.3

Overview

Welcome to SkySpark's Data Quality Extension! As of today, there are 8 data quality functions (7 rule-ready and 1 job-ready) in this package.

Find Missing Values

Find Noise

This rule runs at the point level.
This rule looks for data groups that have changed by a certain percentage from a previous group.
The percentage will be determined automatically if no delta and dur parameters are given.
Side Effects: None.
Input: A point, a date/date range, an optional delta, and an optional duration.
Output: hisDurGrid
sfxDQFindNoise(read(temp and discharge), pastMonth)
sfxDQFindNoise(read(temp and discharge), pastMonth, 5%, 30min)

Find Oddities

Find Outliers Based on Standard Deviation

This rule runs at the point level.
This rule has been modified to rely on the hisDQAverage and hisDQstdDev tags that come from the DQJob.
sDs is the number of standard deviations you want this rule to allow for before a spark is detected. If a history record is below or above 3 stdDev, it will cause a spark in this form (you can change this).
Side Effects: None.
Input: A point, a date/date range, and optionally, how many standard deviations to go out.
Output: hisDurGrid
sfxDQFindOutliers(read(temp and discharge), pastMonth, 4)

Find Values too High.

Find Outliers using ZScore

Data Quality Job

Adds Descriptive Tags to Numeric Points
Run this as a job weekly
reInitialize tells SkySpark how often to update tags
Descriptive tags can be modified by user on the fly or with tuning function.
Monthly running yearly total normalizes for many things.
Dependencies: Calculus and Signal Analysis Pods.
Side Effects: Adds several statistical tags to all numeric points that can be used for complex rules.
Input: Optional reInitialize interval and option how far back to go (default is 1mo and pastYear).
Output: None.
sfxDQJob()

Find Outliers

This rule runs at the point level.
Use tags from data quality job to look for outliers.
Side Effects: None.
Input: A point, a date/date range, and optional buffer.
The buffer determines what percentage of the pastYear's absolute min or max is acceptable.
if old: {min: 50, max: 100, buffer: 0.8} then run against: {minThresh: 62.5, maxThresh: 80} (50/0.8 = 62.5 and 100*0.8 = 80)
Output: hisDurGrid
sfxDQOutofRange(read(temp and discharge), pastMonth, 0.7)

Find Rate of Change too High.

This rule runs at the point level.
sfxDQRateofChange() does the same thing, but is more advanced.
Use tags from data quality job to look for rate-of-change outliers.
Dependencies: Calculus and Signal Analysis Pods.
Side Effects: None.
Input: A point, a date/date range, and an optional buffer.
The buffer determines what percentage of the pastYear's absolute max negative or max positive rate of change is acceptable.
if oldROC: {maxNegROC: -10, maxPosROC: 10, buffer: 0.8} then run against: {maxNegROCThresh: -8, maxPosROCThresh: 8} (-10*0.8 = -8 and 10*0.8 = 8)
Output: hisDurGrid
sfxDQRateofChange(read(temp and discharge), pastMonth, 0.7)

Find Rate of Change too High.

Find Gaps in Data.

This rule runs at the point level.
Use tags from data quality job to look for missing data/intervals.
Side Effects: None.
Input: A point, a date/date range, and an optional buffer.
The buffer determines what percentage of the pastYear's median ts interval is acceptable.
if oldTSInt: {medTSInt: 15min, buffer: 1.9} then run against: {medTSIntThresh: 28.5min} (15min*1.9 = 28.5min)
Output: hisDurGrid
sfxDQTSGap(read(temp and discharge), pastMonth, 1.8)

Find Where Energy Out is not in Line with other Records

This rule runs at the point level.
Use tags from data quality job to look for energy in/out outliers.
This rule is currently designed to run on zone air temp sensor points and then go to the equip (usually ahu) level, find the weatherPoint temp (via siteRef and weatherRef) and finally the siteMeter energy point
Side Effects: None.
Input: A zone air temp point, a date/date range, and optional buffer.
The buffer determines what percentage of the pastYear's absolute min or max is acceptable. if old: {min: 50, max: 100, buffer: 0.8} then run against: {minThresh: 62.5, maxThresh: 80} (50/0.8 = 62.5 and 100*0.8 = 80)
Output: hisDurGrid
sfxDQMismatchedEnergyUsage(read(zone and air and temp and sensor), pastMonth, 0.7)