Creating Buffers and Using In-Memory Aggregation

Calculating aggregated values over set time periods is a fundamental practice when monitoring data-producing items in a system, and is often the starting point for creating the simplest of KPIs. When evaluating processes, aggregated data can help to present a clear overview within a time period context. For example, it is often more useful for an operator to view the average value produced by a pressure sensor, over a minute or an hour rather than a raw value produced every second. Generating real-time aggregate values offers flexibility and the ability to limit the amount of archived data without compromising performance.

AspenTech Inmation OT DataFabric allows you to generate real-time aggregates without having to make history calls. By buffering raw values in memory, aggregates can be calculated over seconds, minutes, hours and days without having to access data from the archive. The syslib.buffer() function creates a table of configurable size and duration to store real-time values from a data-producing item. A second buffer can then be created to aggregate and store values.

This section aims firstly to introduce the user to the concepts of buffering data in AspenTech Inmation OT DataFabric and then to demonstrate how to calculate different aggregates over different time periods with working examples.

Getting Started

It is recommended to create some simulation items to create data to work on the aggregation.

Buffer functionality in Lua

AspenTech Inmation OT DataFabric supports in-memory aggregation by buffering real-time values in memory temporarily. Buffers are attached to objects producing time-series data and hold a configurable number of values for a configurable length of time. The general principle of the buffer is that it fills up with values over time then when it reaches capacity, the addition of a new value causes the oldest value in the buffer to be purged. In AspenTech Inmation OT DataFabric we can create a buffer on an object using the syslib.buffer() Lua function from the AspenTech Inmation Lua library. The syslib.buffer Lua function creates a buffer attached to the object that is producing the values that you wish to aggregate. The function has various configurable arguments, some of which are optional depending on what you want to achieve.

syslib.buffer(object_or_path, “buffer name”, input, duration, size)

In many cases the first step is to create a buffer that will store all the raw values from an item over a defined period of time. So, for example if we wish to calculate the average value every minute for an item that updates every second we would at least need to create a buffer that can contain 60 values and has a duration of 1 minute. For this the buffer function requires the object (or path) supplying the data values, a name for the buffer (entered as a string), the input (relative path to the dynamic property of the object that supplies the values), the duration of the buffer (in ms) and the size (total number of values the buffer can store).

Creating a Buffer

We will create a buffer on the simulated flow item FC4711 as an example of this:

  1. First create an Action Item underneath a folder in the I/O Model by selecting the folder, right clicking and selecting Admin  New to open the Create Object dialog.

  2. In the Create Object dialog, select Data Processing > Action Item from the left-hand panel.

  3. In the Common property compound, set the Object Name to "Buffer" and in the Lua Script Body, enter the following code:

    local pathspec = "/IO-Home/Examples/Demo Data/Process Data/FC4711"
    local obj = syslib.getobject(pathspec)
    syslib.buffer(obj, "buff", ".ItemValue", 60000, 60)
  4. Click OK to confirm the Lua script then click Create to create the Action Item in the I/O Model

In this example, we first use the syslib.getobject function to assign the source FC4711 item to the variable obj. The syslib.buffer function is called on the second line with the following arguments:

  • ‘obj’ is defined as the FC4711 I/O item (from the syslib.getobject call on first line)

  • The buffer is assigned the name “buff”

  • The input is the real-time values from FC4711: we use the relative path to the dynamic property ".ItemValue"

  • Duration is set to 60000 milliseconds as we wish to buffer 1 minute of data

  • Buffer size set to 60 as the FC4711 item produces a new value every second, hence 60 values in 1 minute

Once the Action Item is created, the Lua script is executed, the buffer is created on the specified object and starts to be populated with values…​ How can we now view the items in the buffer then?

Retrieving Buffer Contents

The buffer is now created in-memory and is receiving new values. However, how do we now retrieve those values or check that our buffer has been configured correctly? Once a buffer is created on an item we can view the contents using either the syslib.peek or the syslib.tear function:

  1. syslib.peek(object_or_path, "buffer name")

    Retrieves the contents of the specified buffer (on the specified object) and returns four values:

    • Array of buffered data values

    • Array of buffered timestamps

    • Array of buffered qualities

    • Count of items in the buffer

  2. syslib.tear(object_or_path, "buffer name")

    Retrieves and clears the contents of the specified buffer (on the specified object) and returns four values:

    • Array of buffered data values

    • Array of buffered timestamps

    • Array of buffered qualities

    • Count of items in the buffer

A script that periodically retrieves the contents of a buffer has to be executed in a separate object to the one that created the buffer. A periodically run script containing the syslib.buffer() function will recreate the buffer again, emptying it of its values, meaning that any peek or tear retrieval function will only retrieve an empty buffer.
  1. First create an Generic Item under an folder in the I/O Model, by right clicking the folder and selecting Admin  New to open the Create Object dialog.

  2. In the Create Object dialog, select Data Processing > Generic Item.

  3. In the Common property compound, name the object “Buffer Reader” and click Next.

  4. In the Generation Type property compound, change the Generation Type to Lua Script Data Generator and in the Lua Script Body, enter the following code:

    local pathspec = "/IO-Home/Examples/Demo Data/Process Data/FC4711"
    local obj = syslib.getobject(pathspec)
    local values, qualities, timestamps, count = syslib.peek(obj, "buff")
    return values
  5. Click OK to confirm the Lua script and return to the Create Object wizard then click Create to create the object in the I/O Model tree.

Select the “Buffer Reader” object in the I/O Model tree and hover the mouse over the faceplate in the Object Lens. The values from the buffer will be displayed.

  • Change the return value in the script to the "timestamps" or "qualities" variables, to return an array containing the buffered timestamp and quality values.

  • Change the return value to "count" to return a count of the total number of items that have been loaded into the buffer since the buffer was created.

The buffer fills up until the size parameter is met, then the newest value is buffered and the oldest value is purged from the buffer.

If the buffer reaches the maximum number of elements (specified by the size argument) before the specified time duration of the buffer is reached then the size argument is respected and buffer size will not increase to match the duration (e.g. if a buffer is attached to an item that produces a new value every second but the duration is set to 60,000 ms and the size to 10: the buffer can only hold 10 values before the oldest value is purged, regardless of the buffer duration). The same is also true is the duration argument is met before the specified size is reached. The buffer will only cover the specified duration (e.g. if a buffer is attached to an item that produces a new value every second but the duration is set to 10,000 ms and the size to 20: the buffer will only cover the 10,000 ms duration and therefore can only hold 11 values before the oldest value is purged).