Configuring or editing ETL settings

  1. Go to System Administration > Sys Admin > System Settings.

  2. From the Section dropdown menu, select ETL.

  3. Configure the following parameters:

  • DailyReadingSummary (DRS) ETL

    Defines when days are marked as ineligible for like-day consideration because they contain excess missing or deleted data. A value of 0 signifies missing intervals: the day is considered ineligible. A value of 100 indicates that all missing data is acceptable. The default value is 0.

    Defines whether the calculation executes as a single thread or multi-threaded process. Set to True to enable parallel execution.

    Defines whether the calculation executes as a single thread or multi-threaded process. Select True to enable parallel execution.

    The default value is 1800.

    Defines whether to include the register data in the DailyReadingSummary ETL. Select True to include the data. Select False to exclude the register data. The default value is True.

    Defines an alternate LikeDayAlgorithmFamilyAltKey value (representing an alternate LikeDayFamily dataset) that is used in the LikeDayAlgorithm (LDA) or baseline algorithm. The datasets are automatically configured in the database. Never directly edit the LikeDayFamilies. The LDA Families are automatically updated by the LDA ETL. They are dynamically maintained based on the baseline algorithm definitions that are provided by Itron. If there is a need to change the definitions, then contact your Itron representative to modify the corresponding baseline algorithm attributes.

    Setting these AltKey system settings is only required when you are using an algorithm that defines a filter window that is different from the default 24-hour window. If there is no need to filter the 24-hour window from which LDA calculates daily usage and data quality summaries, then do not define or enable Like Day Algorithm Family A AltKey,or Like Day Algorithm Family B AltKey. When using these alternate settings, set the A Altkey first, and if needed, set the B Altkey.

    Caution: Like Day Algorithm parameter values reflect specific rules and are part of customer-specific customization. If any LikeDayAlgorithm parameter changes, it can invalidate any of its previously selected LikeDays. This invalidates any of the baselines generated on its behalf that reference LikeDays constructed before the change. Check the LikeDayAlgorithm database table to see what value the A key requires. If no B family is required, set the B key to 0 (zero). These values must be set prior to running the DailyReadingSummaryETL task.

    Note: The system setting value must match the key in the LikeDayAlgorithmFamily table.

    IEE administrators must be careful when constructing and assigning these AltKey system settings. Consider the following:

    • Daily Reading Summary ETL performance is significantly affected when one or more of these system settings are in use.

    • After MDM is configured, tested, and running in production mode, do not make changes to LDA and LDAFamily parameter settings.

      Caution: This is an important consideration. Any changes to LDA and LDAFamily parameter settings will require running the DRS ETL in ‘Full’ mode in to recalculate the summaries for historic periods. Depending on the size of the database, this could be a significant challenge. Because IEE does not maintain a history of LDA or LDAFamily parameter changes, historically generated baseline profiles can never be recreated.

    Defines the maximum number of days that may be missing and still perform the estimation. Enter the number of days in whole numbers. The default is 1 day. If the setting is 1, IEE will estimate missing daily consumption if the register reads are missing for 1 day, but will skip the estimation if 2 or more days’ register reads are missing. This setting affects the workload and time of DRS ETL execution. This can make a difference for meters that provide only daily register reads. It does not affect monthly register data.

    Use the following searchable attribute name settings to select the user-defined attributes to use when defining a configuration group:

    • Searchable Attribute Name 01

    • Searchable Attribute Name 02

    • Searchable Attribute Name 03

    • Searchable Attribute Name 04

    • Searchable Attribute Name 05

    • Searchable Attribute Name 06

    • Searchable Attribute Name 07

    • Searchable Attribute Name 08

    • Searchable Attribute Name 09

    • Searchable Attribute Name 10

  • FlatConfig ETL

    Defines the number of threads that the Flat Configuration Services uses to update the Flat Configuration database tables.

    Defines the maximum number of service point channels assigned to each work bin. Enter a whole number. When assigning service points to work bins, this setting defines the maximum work bin size in service point channels. The default value of 400000 might not be the recommended or an optimal setting for your specific IEE implementation. You must tune these values to achieve optimal system performance.

    Defines the minimum number of service point channels that a work bin must contain before creating a new work bin. Use this parameter to avoid creating several tiny work bins.

    Defines the number of work bins that service points are assigned to in parallel. When new service points are created, they are assigned to a work bin for processing by AMI Reading Imports (ARI). As these service points are assigned, the work bins fill up. Filling these work bins in a parallel manner helps balance the load across ARI servers.

    When new service points are created, they are assigned to the smallest work bin until that work bin reaches the Minimum Work Bin Size. Once that happens, a new work bin is created and new service points are added to it. When the number of work bins reaches the number defined in the Work Bins to Fill in Parallel, new service points are added to the existing work bins until they all reach the Maximum Work Bin Size. This ensures that the service points are uniformly assigned across the number of work bins defined in this setting. This avoids scenarios where some work bins have a large number of service points while other work bins have very few. After the number of work bins defined in this setting all reach the Maximum Work Bin Size, then new work bins are created and filled one by one in a serial manner. A new work bin is created each time the current work bin reaches the Maximum Work Bin Size.

  • Like Day Algorithm ETL

    Determines whether candidate like-days must fall completely within the link dates of the account that is current as of the target estimation day.

    Administrative users can specify, at the system level, whether the billing account linked to the given service point for the target estimation day must be the same as the candidate like-day. When this limitation is active, candidate like-days must fall completely within the link dates of the account that is current as of the target estimation day.

    Determines whether the like day algorithm (LikeDayAlgorithm ETL) selects like days in parallel, using multiple task runners. IEE creates one subtask per work bin to execute in parallel by the available task runners. When running only one task, it is not in parallel. Select True to enable parallel processing. Select False to process one chunk of channels at a time. Itron recommends setting this value to True unless instructed to do so by Itron support for a special purpose.

    Defines whether the calculation executes as a single thread or multi-threaded process. Select True to enable parallel execution.

    Defines the day IEE uses to start baseline calculations. Enter the date or use the dropdown arrow to select a date from the calendar.

    Set this value to a date that will result in a reasonable amount of baseline data. For example, the first day that production operations started at the utility.

    Do not set this value to the “beginning of data”, nor to the Beginning of Time (BOT). Collecting too much historic data results in the LDA_ETL calculating current days that will never be used by the application and therefore waste storage space and processing time.

    Defines the number of ToDo records that each LikedayAlgorithm ETL task operates within a single transaction. A smaller number means that the task divides the work into multiple smaller pieces. If the number is too small, performance degrades as many commit transactions are performed against the database. If the number is too large, one process perform all of the work while other processors sit idle. The default setting is 1000. Optimal settings depend on the server hardware and network environment.

  1. Click Save.