sfxDataQualityExt

Find Missing Values

• This rule runs at the point level.
• Do rollups of the expected interval of the data. Then, look for empty periods.
• sfxDQTSGap() does the same thing, but is more advanced.
• Side Effects: None.
• Input: A point, a date/date range, and an expected interval.
• Output: hisDurGrid
• sfxDQFindNulls(read(temp and discharge), pastMonth, 20min)

Find Oddities

• This rule runs at the point level.
• Finds nulls, NAs, or numbers less than or equal to 0.
• Side Effects: None.
• Input: A point and a date/date range.
• Output: hisDurGrid
• sfxDQFindOddities(read(temp and discharge), pastMonth)

Find Outliers Based on Standard Deviation

• This rule runs at the point level.
• This rule has been modified to rely on the hisDQAverage tag that comes from the DQJob.
• sDs is the number of standard deviations you want this rule to allow for before a spark is detected. If a history record is below or above 3 stdDev, it will cause a spark in this form (you can change this).
• Side Effects: None.
• Input: A point, a date/date range, and optionally, how many standard deviations to go out.
• Output: hisDurGrid
• sfxDQFindOutliers(read(temp and discharge), pastMonth, 4)

Data Quality Job

• Adds Descriptive Tags to Numeric Points
• Run this as a job weekly
• reInitialize tells SkySpark how often to update tags
• Descriptive tags can be modified by user on the fly or with tuning function.
• Monthly running yearly total normalizes for many things.
• Dependencies: Calculus and Signal Analysis Pods.
• Side Effects: Adds several statistical tags to all numeric points that can be used for complex rules.
• Input: Optional reInitialize interval and option how far back to go (default is 1mo and pastYear).
• Output: None.
• sfxDQJob()

Find Outliers

• This rule runs at the point level.
• Use tags from data quality job to look for outliers.
• Side Effects: None.
• Input: A point, a date/date range, and optional buffer.
• The buffer determines what percentage of the pastYear's absolute min or max is acceptable.
• if old: {min: 50, max: 100, buffer: 0.8} then run against: {minThresh: 62.5, maxThresh: 80} (50/0.8 = 62.5 and 100*0.8 = 80)
• Output: hisDurGrid
• sfxDQOutofRange(read(temp and discharge), pastMonth, 0.7)

Find Rate of Change too High

• This rule runs at the point level.
• sfxDQRateofChange() does the same thing, but is more advanced.
• Use tags from data quality job to look for rate-of-change outliers.
• Dependencies: Calculus and Signal Analysis Pods.
• Side Effects: None.
• Input: A point, a date/date range, and an optional buffer.
• The buffer determines what percentage of the pastYear's absolute max negative or max positive rate of change is acceptable.
• if oldROC: {maxNegROC: -10, maxPosROC: 10, buffer: 0.8} then run against: {maxNegROCThresh: -8, maxPosROCThresh: 8} (-10*0.8 = -8 and 10*0.8 = 8)
• Output: hisDurGrid
• sfxDQRateofChange(read(temp and discharge), pastMonth, 0.7)

Find Rate of Change too High

• This rule runs at the point level.
• Use tags from data quality job to look for rate-of-change outliers.
• Dependencies: Calculus and Signal Analysis Pods.
• Side Effects: None.
• Input: A point, a date/date range, and a threshold.
• Output: hisDurGrid
• sfxDQRateofChangeThresh(read(temp and discharge), pastMonth, 10)

Find Gaps in Data

• This rule runs at the point level.
• Use tags from data quality job to look for missing data/intervals.
• Side Effects: None.
• Input: A point, a date/date range, and an optional buffer.
• The buffer determines what percentage of the pastYear's median ts interval is acceptable.
• if oldTSInt: {medTSInt: 15min, buffer: 1.9} then run against: {medTSIntThresh: 28.5min} (15min*1.9 = 28.5min)
• Output: hisDurGrid
• sfxDQTSGap(read(temp and discharge), pastMonth, 1.8)