Interval validation rules

Compares demand value to the high threshold parameters. This rule validates data against peak limits to detect whether demand is abnormally high. The root cause may be a misread, meter issue or energy diversion.

Note: While it is possible to include both the Demand High Limit and the Demand Low Limit in the same validation set, both rules assign the reading status of LIMIT to any interval that fails the rule. To investigate the failure when both rules are included in one validation set, you must look at the interval value itself to see whether it is higher or lower than the expected range.

Compares demand value to the low threshold parameters. Validates data against peak limits to detect whether demand is abnormally low. The root cause may be a misread, meter issue or energy diversion.

Note: While it is possible to include both the Demand High Limit and the Demand Low Limit in the same validation set, both rules assign the reading status of LIMIT to any interval that fails the rule. To investigate the failure when both rules are included in one validation set, you must look at the interval value itself to see whether it is higher or lower than the expected range.

For readings processed by AMI readings import (ARI), checks for gaps in interval data only within the defined 24-hour processing period.

For readings processed by standard reading import (SRI), checks for the following types of gap scenarios in interval data:

• Gaps between the last cut of data and the incoming cut of data.

• Gaps between the incoming cut of data and any data in the database that exists after the incoming cut.

• Gaps within the incoming data itself due to intervals marked as Missing.

Configure the following parameters:

• Front Fill and Back Fill. Determines whether Gap Check fills gaps at the beginning or end of the validation period. To fill gaps at the beginning, configure the Back Fill parameter. To fill gaps at the end, configure the Front Fill parameter. The valid values are as follows.

  • None. Does not fill gaps.

  • Midnight. Fill gaps that occur between the data and the previous midnight (Back Fill) or next midnight (Front Fill) in the time zone of the meter.

  • Integer. Fill gaps that occur between the beginning or end of the data to the defined number of hours.

    Note: The Front Fill and Back Fill parameter setting only effects readings that are routed through the standard reading import (SRI). Because readings processed by ARI are based on a 24-hour day, they do not have beginning or end of validation period reading gaps, and are not effected by the Front Fill and Back Fill parameter setting.

• Route to Queue. Determines whether to route exceptions immediately to the readings validation queue or to send exceptions through automatic estimation and secondary validation. Route to Queue (Do Not Auto-Estimate) overrides automatic estimation for an individual rule. Select the Route To Queue (Do Not Auto-Estimate) checkbox to send validation exceptions to the queue immediately. Clear the checkbox to send exceptions through estimation and secondary validation. If automatic estimation is disabled in IEE, exceptions route to the queue even when Route to Queue is enabled.

To define the number of days forward and back to check for existing data in order to define gaps, configure the Gap Detect Range system setting in System Administration > System Admin > System Settings > Validation and Estimation.

Identifies unusual usage for a site by comparing the average daily usage for the current period to the average daily usage for historical periods (as available). Using this comparison, the rule looks for dropped phases, inaccurate meter constants, energy diversions, and fast/slow meters. In combination meter/recorder installations, the rule also checks for erratic pulse input to recorders.

This rule uses the average daily usage as part of its calculation. Average daily usage is the total usage (not including intervals with missing/empty statuses, and only including estimated if the Use Estimated Data parameter is set) divided by the number of minutes used (where the minutes used must be greater than the Minimum Minutes of History parameter).

The rule calculates the delta between the current average daily usage of the incoming data against the average daily usage for a historical date range. If the ratio between the delta divided by the daily usage for the historical data is above the specified threshold, then the rule fails or warns as specified. The historical date range is taken from the 30 calendar days previous to the incoming data. If that data is not available, IEE uses the data from the same period of the previous year. If that data is not available, the rule fails. When setting up this rule, you can tell IEE to use estimated data in computing the historical Average Daily Usage and you can also indicate how much data is required for this rule to run.

Identifies intervals where the kWh load is zero but the reactive load is not. Use this rule to identify meter malfunctions or power outages. This rule only runs if the channel is in a KVA Channel Set. If multiple kWh channels and/or multiple kVARh channels are in the same KVA Channel Set, IEE must aggregate (net) the data before this check can be performed.

• If any kWh channel or summed kWh channels have an interval with a zero value that is not identified as a Power Outage or Test status, this rule examines the corresponding interval in the kVARh channel or summed kVARh channel. If the corresponding interval is greater than the specified number of pulses, the interval fails the kVARh Check.

• If all the kWh data is non-zero, IEE skips this check.

Compares a service point channel's load factor against a threshold that varies by customer class. Use this rule to determine unusual usage patterns. IEE skips this validation if the UOM of the channel is neither in, nor scalable to, kWh or load factor.

• If the UOM is load factor, IEE compares the data to the defined threshold parameters.

• If the UOM is from the kVARh, kVAh, or kWh family, the rule first calculates the service point channel's load factor using the load factor formula in the Universal Calculation Engine before it is compared to the thresholds.

Typical load factors include:

• Typical residential customers range from a .1 to .5 load factor.
• Commercial customers can have load factors that range from a .4 to .65 load factor.
• The threshold for a large C&I customer would be a .8 to .9 load factor.

Compares a service point channel's load factor against a threshold that varies by customer class. Use this rule to determine unusual usage patterns. IEE skips this validation if the UOM of the channel is neither in, nor scalable to, kWh or load factor.

• If the UOM is load factor, IEE compares the data to the defined threshold parameters.

• If the UOM is from the kVARh, kVAh, or kWh family, the rule first calculates the service point channel's load factor using the load factor formula in the Universal Calculation Engine before it is compared to the thresholds.

Typical load factors include:

• Typical residential customers range from a .1 to .5 load factor.

• Commercial customers can have load factors that range from a .4 to .65 load factor.

• The threshold for a large C&I customer would be a .8 to .9 load factor.

Maximum Number of Zero Intervals counts the number of zero intervals and compares this count to the threshold parameters. If the number of zero intervals is greater than the threshold, the rule warns or fails (as appropriate). There is a parameter to specify whether or not the zero intervals should be consecutive and a parameter to exclude intervals with a status of Power Outage.

Use this rule to find energy or water theft. For example, use this rule to check active premises for unusually low usage which could indicate diversion or bypass.

Minimum Number of Zero Intervals counts the number of zero intervals and compares this count to the threshold parameters. If the number of zero intervals is less than the threshold, the rule warns or fails (as appropriate). There is a parameter to specify whether or not the zero intervals should be consecutive.

Use this count rule to monitor usage patterns where zero reads are expected. For example, zero reads are expected from inactive meters, meters during a seasonal low, or during irrigation pumping cycles. In irrigation pumping cycles, additional usage indicates something unusual is occurring, such as leakage.

Determines whether IEE already has data for the channel being validated for a specific validation time frame. If data already exists, the new data fails. IEE uses the Overlap Check only when running as part of the standard reading import process.

Use this check to find incorrect date and time settings on meters.

IEE ignores this rule when validating data imported through the AMI reading import process. AMI meters may send reads several times per day, and the requests may include overlaps. IEE also ignores this check when you manually validate or edit the data.

Note: Data imported in handheld files (HHF format) can also have overlaps. IEE does not prevent this rule from running on HHF files. If you are validating HHF data, do not include this check in validation sets that are intended for use on HHF files.

Detects unacceptably low power factor (PF) for a service point channel. This is a complex check that computes the power factor from the appropriate channels (kWh, kVARh, kVAh). If the result is below the defined threshold, the rule fails or warns as appropriate. Set the threshold as a PF value, not a percentage.

Use this rule to measure the efficiency in an electrical power system. If the customer is on a PF penalty rate, the utility wants to know when that customer breached the threshold.

This rule is almost used exclusively for large C&I customers who generate their own power, although some utility commissions may also require consistent rules for smaller customers with reactive metering. You may need to investigate abnormally low PF situations. To mitigate a low PF situation, a customer may install capacitors to reduce the reactive power, thus increasing their true power. Otherwise, that customer's rates may increase if the PF remains low.

If a service point channel's UOM is Power Factor (PF), a kVA set not required. Otherwise, this rule requires the same KVA channel set as the kVARh check. IEE skips this rule if the service point channel's (UOM) is not Watts, VARs, VAh, or PF.

Compares intervals for the main metered load channel with each corresponding interval of an existing reference channel on the same service point. If the absolute value of the difference between the two intervals exceeds the specified threshold (expressed as a percentage), the data fails or warns. This rule is also commonly known as a Redundant Meter Check or Check Meter.

IEE performs an interval-by-interval comparison between the channel being validated and the reference channel. You can require all intervals to be present on the reference channel for the given time period for the rule to execute.

Use this rule to find discrepancies that can occur if the redundant meter’s clock is not synchronized with the main meter and if the load is erratic. When those kinds of conditions occur, redundant data may not match as closely as expected.

Identifies abnormally high usage values in 24-hour subsets of the incoming data. For each 24-hour subset, IEE only runs this rule if there are any values above a specified threshold. The threshold is specified in pulses. If any values are above this limit, then IEE detects the highest value and compares it against the Nth highest value (where N is a parameter on the rule, typically 3). If the difference between these two values (expressed as a ratio to the Nth value) is larger than the specified fail or warn threshold, then the rule will either fail or warn, as appropriate.

Use this rule to detect data anomalies, engineering issues, or operational causes, such as a customer who turns on all generators at the same time, such as on a Monday morning. This information is useful for load research and capacity planning.

Counts the number of intervals that have specific interval status values within the validation time range and compares this count to the threshold parameters. Each rule checks for a specified status.

Add status checks for each status to validate. For example, add a rule to check for a certain number of Alarm statuses, and add another rule to check for Power Outages.

Configure the following parameters:

Exclude Estimates. Defines whether the rule runs against estimated data. Select this checkbox to prevent reads from failing secondary validation when IEE retains reading statuses after estimation for informational purposes.

Skip When Done After Estimation. Determines whether secondary validation occurs after estimation. Use this option to check for the estimated status as needing estimation, to improve the prior estimates when new data is imported. IEE does not run secondary validation when this option is selected, and the estimated reads are retained.

Warning Threshold. Defines the number of readings with this interval status that is required to generate a Warn validation status.

Route to Queue. Determines whether to route exceptions immediately to the readings validation queue or to send exceptions through automatic estimation and secondary validation. Route to Queue (Do Not Auto-Estimate) overrides automatic estimation for an individual rule. Select the Route To Queue (Do Not Auto-Estimate) checkbox to send validation exceptions to the queue immediately. Clear the checkbox to send exceptions through estimation and secondary validation. If automatic estimation is disabled in IEE, exceptions route to the queue even when Route to Queue is enabled.

Status Name. Defines the interval status this rule counts. Select the interval status from the dropdown menu.

Count Consecutive Intervals Only. Determines whether the rule counts only consecutive intervals with the status or whether the rule counts all intervals with the status. Select to exclude individual, non-consecutive readings with the interval status from the final count. For example, if five intervals arrive and only the second interval has the interval status this rule is counting, that interval is not included in the final status count. However, if the first three intervals have the interval status, then this rule counts those three.

Fail Threshold. Defines the number of readings with this interval status is required to generate a Fail validation status.

Determines whether incoming data is aligned on interval boundaries. The Time Tolerance Check is strictly a Service Mode check.

This check does not use historical or reference data. Rather, the check compares the duration of the time period in the pulse file (Pfile) to the duration determined by IEE. This verifies the time difference between the meter clock and the system clock.

Use this rule to find time drift in a meter, or inconsistent Daylight Savings Time settings at a meter.

Market rules normally determine the threshold values for this check. Some markets may set this as low as two to five seconds. A value between 30 and 60 seconds is more common.

This check is especially critical for older mechanical meters, which never synchronize to a head end. Clock drift is common for these meters.

Compares absolute usage values to high threshold parameters. If usage exceeds the thresholds, the rule fails or warns. This rule validates data against the parameters defined in the rule. This rule does not use historical or reference data. Another common term for usage limit checks is Maximum Service.

Use usage high limit to check service point channels that record values other than reading data. For example, you could use this rule to check temperature.

Usage legitimately varies greatly, so there is no threshold recommendation that is valid for all service point channels.

Compares absolute usage values to low threshold parameters. If usage is lower than the threshold, the rule fails or warns. This rule validates data against the parameters defined in the rule. This rule does not use historical or reference data. Another common term for usage limit checks is Maximum Service.

Use usage low limit to check service point channels that record values other than reading data. For example, you could use this rule to check temperature.

Usage legitimately varies greatly, so there is no threshold recommendation that is valid for all service point channels.

Counts the number of non-zero intervals on an inactive meter. If the number of non-zero intervals exceeds the threshold for meters where the meter state is Inactive, the rule fails or warns.

Use this rule to detect energy diversion or water diversion on meters where no usage would be expected. The threshold accounts for conditions such as meter creep (electric meters) or leakage (water meters) where small amounts of usage can still register.

Note: This rule is not available for use during IEE AMI reading import in IEE versions prior to IEE v8.1.

Compares energy usage as calculated from the start and stop meter readings from the linked consumption register to usage calculated from intervals. You must specify one or more threshold values to define tolerances. The valid values are as follows.

• Fail. The absolute difference between the two energy usage calculations is greater than the failure tolerance.

• Warn. The absolute difference is greater than the warning tolerance, but less than or equal to the failure tolerance.

• Pass. The absolute difference is less than both the warning and failure tolerance thresholds.

Checks the usage from the start and stop reads and compares it to the usage from the intervals. Applies the percent difference (UsageTolerancePerDiff) and percent meter multiplier (UsageTolerancePerMMDiff) usage tolerance calculations and returns the best result. The data only has to pass one of the usage tolerance calculations to pass this validation rule.

IEE runs the percent difference calculation first. If the data passes this calculation, the data passes the validation rule and IEE does not run the percent meter multiplier calculation. If the data does not pass the percent difference calculation, then IEE runs the percent meter multiplier calculation.

If the percent difference calculation determined the final validation status, the Validation Report shows the results as a percent difference result. If the percent meter multiplier calculation determined the final validation status, the Validation Report shows the results as a percent meter multiplier result.

The following table shows the possible results of the Dual Method Energy Tolerance validation rule and how those results are reached:

Compares energy usage measured by the consumption register to energy usage calculated from intervals. The total of the intervals should match the total usage between the start time and stop time of the consumption register reading period. However, a certain percentage of difference may be allowed. This allowed percentage is the usage tolerance.

When you add this rule to a validation set, you define the usage tolerance that triggers a WARN validation status and the usage tolerance that triggers a FAIL validation status. You must specify one or more threshold values to define tolerances for usage tolerance percent difference rule.

Configure the following parameters:

• Warning Threshold. Defines the maximum allowed difference between the total of the interval reads and the total consumption measured by the register. Any greater difference results in the data passing this rule with a validation status of WARN. For example, the total of the interval reads is 97 percent of the consumption that was measured by the register channel. The difference between the two measurements is 3 percent. If the Warning Threshold is 2 percent, the data passes with a validation status of WARN. If the Warning Threshold is 4 percent, no warning is necessary and the data passes with a validation status of PASS.
• Missing Meter Read Level. Determines how this rule handles missing reads in the data being validated. The valid values are as follows.

  • None. Ignore missing reads. If you select one of the other values, IEE does not allow this parameter to return to None. You must add the rule to the validation set again and start over.

  • Skip. Skip this validation rule if the data is missing reads.

  • Warn. Proceed with validation but include a WARN validation status that the data was missing reads.

  • Fail. Fail this rule.

• Route to Queue. Determines whether to route exceptions immediately to the readings validation queue or to send exceptions through automatic estimation and secondary validation. Route to Queue (Do Not Auto-Estimate) overrides automatic estimation for an individual rule. Select the Route To Queue (Do Not Auto-Estimate) checkbox to send validation exceptions to the queue immediately. Clear the checkbox to send exceptions through estimation and secondary validation. If automatic estimation is disabled in IEE, exceptions route to the queue even when Route to Queue is enabled.

• Fail Threshold. Defines the maximum allowed difference between the total of the interval reads and the total consumption measured by the register. Any greater difference results in the data failing this validation rule. For example, the total of the interval reads is 97 percent of the consumption that was measured by the register channel. The difference between the two measurements is 3 percent. If the Fail Threshold is 2 percent, the data fails. If the Fail Threshold is 4 percent, the data passes.

Compares the difference between register and interval usage to a defined tolerance. The tolerance is defined by multiplying the fail or warn threshold by the meter multiplier. If the usage difference is greater, the rule fails or warns.

This rule performs the following calculations:

1. Add the interval reads to find Consumption A.

2. Add the register reads to find Consumption B.

3. Determine the difference between Consumption A and Consumption B. This is the usage difference.

4. Multiply the threshold percentage defined in the rule by the meter multiplier defined in the register channel. This is the tolerance.

5. Compare the usage difference to the tolerance. If the usage difference is greater than the tolerance, the rule fails or warns.

For example, this rule has a Fail Threshold of 5 percent. The meter multiplier is 100. The total interval consumption is 200 and the total register consumption is 210.

The usage difference is 10.

The tolerance is 5 (5 percent of 100).

Because the usage difference is greater than the tolerance (10 > 5), the rule fails.

Configure the following parameters:

• Warning Threshold. Determines the maximum tolerance percentage that results in the data passing this rule with a validation status of WARN.

• Missing Meter Read Level. Determines how this rule handles missing reads in the data being validated. The valid values are as follows.

  • None. Ignore missing reads. If you select one of the other values, IEE does not allow this parameter to return to None. You must add the rule to the validation set again and start over.

  • Skip. Skip this validation rule if the data is missing reads.

  • Warn. Proceed with validation but include a WARN validation status that the data was missing reads.

• Fail. Fail this rule.

• Route to Queue. Determines whether to route exceptions immediately to the readings validation queue or to send exceptions through automatic estimation and secondary validation. Route to Queue (Do Not Auto-Estimate) overrides automatic estimation for an individual rule. Select the Route To Queue (Do Not Auto-Estimate) checkbox to send validation exceptions to the queue immediately. Clear the checkbox to send exceptions through estimation and secondary validation. If automatic estimation is disabled in IEE, exceptions route to the queue even when Route to Queue is enabled.

• Fail Threshold. Determines the maximum tolerance percentage that results in the data failing this rule. 

Runs a user-defined formula as a validation check. This allows you to evaluate an account uniquely, based on formula constants. The formula can use historical and reference data as required. It is available for interval data only.

Use this rule when no standard rule meets your business need.

Note: For performance reasons, do not include user-defined formula rules in validation sets for use with the AMI reading import process.

Use one of the following methods to create user-defined validation rules:

• Enter a new formula when you create the validation rule.

• Assign an existing named formula to a validation set. Named Formulas are identified by unique names (such as "MyNewValidationCheck").

If you create a formula, you must design the formula to return a true (1) or false (0) result. True results in a failure or warning for the validation. False always triggers a pass validation status.

You also mark which status will be marked if True. You can export this status to external systems. Typically, this rule includes a user-defined status.

Formulas must comply with the IEE Universal Calculation Engine (UCE) syntax.

User-defined validation rules use the following components:

• Simple and complex mathematical functions.

• Nested conditional statements.

• Unit conversion functions.

• Date, time, and TOU functions.

• Historical time references.

• Constants and formula constants.

• Unit conversions.

• References to additional channels.

The following sample formulas allow each service point channel to be evaluated uniquely, based on formula constants.

• @if($spc>$spc:MAX_INT_USG,1,0) fail on true

• @if($spc<$spc:MIN_INT_USG,1,0) fail on true

• @if(@sum($spc)>$spc:MAX_TOT_USG,1,0) fail on true

• @if(@sum($spc)<$spc:MIN_TOT_USG,1,0) fail on true

Counts the number of zero intervals on an active meter. If the number of zero intervals exceeds the threshold for meters whose state is Active, the rule fails or warns.

Use this rule to detect energy diversion or water diversion on meters where at least some usage would be expected based on the account or customer type and traditional usage patterns.

Besides specifying a minimum number of zero intervals that an incoming data file can contain, you can further specify that such intervals be consecutive and whether intervals with a status code of PO (Power Outage) be included in the check.