zivaDataQualityExt icon

zivaDataQualityExt

Allows you to look for strange or missing values in SkySpark.
zivaDataQualityExt

Registered StackHub users may elect to receive email notifications whenever a new package version is released.

There are 0 watchers.

v2.6.0

Overview

Welcome to SkySpark's Data Quality Extension!

zivaDQFindNulls(point, dateRange, duration)

Find Missing Values

  • This rule runs at the point level.
  • Do rollups of the expected interval of the data. Then, look for empty periods.
  • zivaDQTSGap() does the same thing, but is more advanced.
  • Side Effects: None.
  • Input: A point, a date/date range, and an expected interval.
  • Output: hisDurGrid
  • zivaDQFindNulls(read(temp and discharge), pastMonth, 20min)

zivaDQFindNoise(point, dateRange, delta, duration)

Find Noise

  • This rule runs at the point level.
  • This rule looks for data groups that have changed by a certain percentage from a previous group.
  • The percentage will be determined automatically if no delta and dur parameters are given.
  • Side Effects: None.
  • Input: A point, a date/date range, an optional delta, and an optional duration.
  • Output: hisDurGrid
  • zivaDQFindNoise(read(temp and discharge), pastMonth)
  • zivaDQFindNoise(read(temp and discharge), pastMonth, 5%, 30min)

zivaDQFindOddities(point, dateRange)

Find Oddities

  • This rule runs at the point level.
  • Finds nulls, NAs, or numbers less than or equal to 0.
  • Side Effects: None.
  • Input: A point and a date/date range.
  • Output: hisDurGrid
  • zivaDQFindOddities(read(temp and discharge), pastMonth)

zivaDQFindOutliers(point, dateRange, sDs)

Find Outliers Based on Standard Deviation

  • This rule runs at the point level.
  • This rule has been modified to rely on the hisDQAverage and hisDQstdDev tags that come from the DQJob.
  • sDs is the number of standard deviations you want this rule to allow for before a spark is detected. If a history record is below or above 3 stdDev, it will cause a spark in this form (you can change this).
  • Side Effects: None.
  • Input: A point, a date/date range, and optionally, how many standard deviations to go out.
  • Output: hisDurGrid
  • zivaDQFindOutliers(read(temp and discharge), pastMonth, 4)

zivaDQFindOutliers(point, dateRange, threshold)

Find Values too High.

  • This rule runs at the point level.
  • Find Outliers based on threshold and its inverse.
  • Dependencies: None.
  • Side Effects: None.
  • Input: A point, a date/date range, and a threshold.
  • Output: hisDurGrid
  • zivaDQOutlierThresh(read(temp and discharge), pastMonth, 100)

zivaDQZScoreOutofBounds(point, dateRange, buffer)

Find Outliers using ZScore

  • This rule runs at the point level.
  • Use tags from data quality job to look for outliers.
  • Side Effects: None.
  • Input: A point, a date/date range, and optional ZScore override.
  • Output: hisDurGrid
  • zivaDQZScoreOutofBounds(read(temp and discharge), pastMonth, 0.9)

zivaDQJob()

Data Quality Job

  • Run as task
  • Adds Descriptive Tags to Numeric and Boolean Points
  • Run this as a job weekly
  • reInitialize tells SkySpark how often to update tags
  • Monthly running yearly total normalizes for many things.
  • Dependencies: Calculus and Signal Analysis Pods.
  • Side Effects: Adds several statistical tags to all numeric points that can be used for complex rules. Tags are also added to Boolean points.
  • Input: Optional reInitialize interval and option how far back to go (default is 1mo and pastYear).
  • Output: None.
  • zivaDQJob()

zivaDQJobBool()

Data Quality Job for Booleans

  • Run as task
  • Adds Descriptive Tags to Boolean Points
  • Run this as a task weekly
  • reInitialize tells SkySpark how often to update tags
  • Monthly running yearly total normalizes for many things.
  • Dependencies: Calculus and Signal Analysis Pods.
  • Side Effects: Adds several tags to Boolean points.
  • Input: Optional reInitialize interval and option how far back to go (default is 1mo and pastYear).
  • Output: None.
  • zivaDQJobBool()

zivaDQJobNumber()

Data Quality Job for Numbers

  • Run as task
  • Adds Descriptive Tags to Numeric Points
  • Run this as a task weekly
  • reInitialize tells SkySpark how often to update tags
  • Monthly running yearly total normalizes for many things.
  • Dependencies: Calculus and Signal Analysis Pods.
  • Side Effects: Adds several statistical tags to all numeric points that can be used for complex rules.
  • Input: Optional reInitialize interval and option how far back to go (default is 1mo and pastYear).
  • Output: None.
  • zivaDQJobNumber()

zivaDQOutofRange(point, dateRange, buffer)

Find Outliers

  • This rule runs at the point level.
  • Use tags from data quality job to look for outliers.
  • Side Effects: None.
  • Input: A point, a date/date range, and optional buffer.
  • The buffer determines what percentage of the pastYear's absolute min or max is acceptable.
  • if old: {min: 50, max: 100, buffer: 0.8} then run against: {minThresh: 62.5, maxThresh: 80} (50/0.8 = 62.5 and 100*0.8 = 80)
  • Output: hisDurGrid
  • zivaDQOutofRange(read(temp and discharge), pastMonth, 0.7)

zivaDQRateofChange(point, dateRange, buffer)

Find Rate of Change too High.

  • This rule runs at the point level.
  • zivaDQRateofChange() does the same thing, but is more advanced.
  • Use tags from data quality job to look for rate-of-change outliers.
  • Dependencies: Calculus and Signal Analysis Pods.
  • Side Effects: None.
  • Input: A point, a date/date range, and an optional buffer.
  • The buffer determines what percentage of the pastYear's absolute max negative or max positive rate of change is acceptable.
  • if oldROC: {maxNegROC: -10, maxPosROC: 10, buffer: 0.8} then run against: {maxNegROCThresh: -8, maxPosROCThresh: 8} (-10*0.8 = -8 and 10*0.8 = 8)
  • Output: hisDurGrid
  • zivaDQRateofChange(read(temp and discharge), pastMonth, 0.7)

zivaDQRateofChangeThresh(point, dateRange, threshold)

Find Rate of Change too High.

  • This rule runs at the point level.
  • Use tags from data quality job to look for rate-of-change outliers.
  • Dependencies: Calculus and Signal Analysis Pods.
  • Side Effects: None.
  • Input: A point, a date/date range, and a threshold.
  • Output: hisDurGrid
  • zivaDQRateofChangeThresh(read(temp and discharge), pastMonth, 10)

zivaDQTSGap(point, dateRange, buffer)

Find Gaps in Data.

  • This rule runs at the point level.
  • Use tags from data quality job to look for missing data/intervals.
  • Side Effects: None.
  • Input: A point, a date/date range, and an optional buffer.
  • The buffer determines what percentage of the pastYear's median ts interval is acceptable.
  • if oldTSInt: {medTSInt: 15min, buffer: 1.9} then run against: {medTSIntThresh: 28.5min} (15min*1.9 = 28.5min)
  • Output: hisDurGrid
  • zivaDQTSGap(read(temp and discharge), pastMonth, 1.8)

zivaDQMismatchedEnergyUsage(zatsPoint, dateRange, buffer)

Find Where Energy Out is not in Line with other Records

  • This rule runs at the point level.
  • Use tags from data quality job to look for energy in/out outliers.
  • This rule is currently designed to run on zone air temp sensor points and then go to the equip (usually ahu) level, find the weather temp (via siteRef and weatherStationRef) and finally the siteMeter energy point
  • Side Effects: None.
  • Input: A zone air temp point, a date/date range, and optional buffer.
  • The buffer determines what percentage of the pastYear's absolute min or max is acceptable. if old: {min: 50, max: 100, buffer: 0.8} then run against: {minThresh: 62.5, maxThresh: 80} (50/0.8 = 62.5 and 100*0.8 = 80)
  • Output: hisDurGrid
  • zivaDQMismatchedEnergyUsage(read(zone and air and temp and sensor), pastMonth, 0.7)

zivaDQCleaner()

Data Quality Cleaner

  • Run as task or manually
  • Removes Descriptive Tags (essentially used for migration and does not need to be run regularly)
  • Side Effects: Removes tags - intended effect
  • Input: None
  • Output: None
  • zivaDQCleaner()

Please contact adam@ziva-tech.com with any questions.

Published by Ziva Tech Solutions

Packages by Ziva Tech Solutions

Free packages

Package details
Version2.6.0
Licensen/a
Build date3 months ago
on 22nd Nov 2023
Depends on
File namezivaDataQualityExt.pod
File size19.44 kB
MD5fc95afde08d297b430e81f1cb384535b
SHA1 ea67ae186d01916afebe4730f9b75c877fd073f9
Published by
Ziva Tech SolutionsDownload now
Also available via SkyArc Install Manager
Tags
Pod File