Custom Script

Some import file types or formats are not covered by the processing options provided by the Column Based and Line-by-line options. In these cases, the Custom Script option can be used to fully customize the file processing using a Lua script written to your specification.

The Custom Script option can also be used to expand the processing method to create new objects, perform calculations or anything else that is possible with Lua scripting.

Although it is not possible to cover every different file type that may be encountered, this section will demonstrate a simple custom scripting example that will give you the foundation to create your own Lua script for Dropzone.

If you are a beginner to Lua, it is helpful to read the Lua Scripting section as an introduction. This section is also the most useful reference point for using the system Lua functions.

For the Custom Script processing option, the Lua script should be entered in the Lua Script Body of the Dropzone Datasource object and must be structured as a Lua function with the processing script contained within it. For example:

return function(file)
-- processing script including IO file handling code
end

The import file that is copied into the Dropzone Disk folder becomes the input for any returned function specified in the Lua script attached to the Dropzone object. Some file handling must be included to open the file

Using the Lua I/O library (supported in AspenTech Inmation), the lines of the input file can be parsed and read into a Lua table. This library must be called using 'require'. From there, the file contents can be processed and, using the system’s Lua functions, objects can be created in the I/O Model.

Before working through the examples, a Dropzone Datasource object must be created as described here.

Line Count Example

In this first simple example, the scripting functionality is used to count the number of lines in a file and write the result to HolderItem.

Custom Dropzone scripts are structured with the whole script written as a Lua function with the file that is dropped into the Disk Folder as the input for that function. With Custom Script processing, the way that the file should be processed must be defined exactly within the function.

Unlike the other processing options where a lot of the actions occur automatically, for a Custom Script Dropzone, these must be coded manually in the Lua script. This includes opening the file, and to do that in Lua, the Lua IO library is required. The following script has a simple function but also demonstrates the basic structure of a Custom Dropzone script.

Select the Dropzone in the I/O model and expand the Object Lens panel.
In the Dropzone property compound, select the "+" icon next to the Lua Script Body to open the editor.

Copy the following script into the editor and click Ok.

return function(file)
  local ioLib = require("io")
  local name = "Line Count"
  local parentPath = syslib.getselfpath()
  local lines = {}

  local f = ioLib.open(file, "rb")
  if f ~= nil then
    lines = {}
    for line in ioLib.lines(file) do
        lines[#lines + 1] = line
    end
  end
  f:close()
  local ioitem = nil
  ioitem = syslib.getobject(parentPath .. "/" .. name)
  if ioitem == nil then
    ioitem = syslib.createobject(parentPath, "MODEL_CLASS_HOLDERITEM")
    ioitem.ObjectName = name
    ioitem.ArchiveOptions.ArchiveSelector = 2
    ioitem.ArchiveOptions.StorageStrategy = 1
    ioitem:commit()
  end
  syslib.setvalue(ioitem, #lines)
end

Explanation

The first line returns the function which takes the import file as its argument.

The rest of the script consists of roughly three parts:

The Lua IO library is loaded using require("io") and the variables name, parentPath and lines are defined.
The Lua IO library open function is called to open the file and each line is read into the lines table. The file is then closed using f:close
A HolderItem is created and its properties set to archive data. The number of entries in the lines table is then written to the value property of the HolderItem.

Then, click Apply to save the changes.
To test the script, drop any multi-line file into the Disk Folder (configured when setting up the Dropzone Datasource object). The I/O Model should look like the example below after the file has been processed, with the "Line Count" value being the same as the one in the imported file.

Although this is a simple example, it demonstrates how the Custom Script processing Dropzone can be used to read data files into a Lua table and then process them in any desired fashion. Logic cna be applied to the data and objects can be created in the I/O Model.

The next example will demonstrate how this can be used to process a non-standard file format and successfully create object and write VQT data to the newly created objects.

Non-Standard File Format

Custom Scripting is particularly useful for processing non-standard file formats i.e., file formats that are not processed correctly using the other processing options. Take the following file as an example where the co-ordinates of different objects are recorded at different timestamps:

Initially, this type of file would look like a candidate for the Column Based processing option. However, if we set up a Dropzone object with Column Based processing and drop the file into the configured disk folder, the column names are used to create I/O items beneath the Dropzone resulting in the following:

The more appropriate results would be that the Name column would be used to create separate nodes (A, B, C, D, etc.) and the x,y,z coordinates for each node would be imported to separate I/O items.

To do this, the Custom Script processing option should be used to process the file in the desired fashion. As with the previous Line Count example, we need to write a script that will do everything we need. In this case, that includes:

Opening the file and reading all file lines into a Lua table
Reading the 'Name' column to create nodes
Reading the column headers to create 'x', 'y', 'z' Holderitems
Writing the coordinate data to the HolderItems

In this example, we will first define some Lua function to perform these tasks separately. They can be housed in the same script and called in the main function at the end. Later, you might want to use or modify these separate functions to construct your own script or place them in a script library.

Firstly, a Dropzone datasource object is required. The same Dropzone can be used from the simple Line Count example or a new Dropzone object can be created as described here.

Copy each of the scripts below into a script library. This can be done in either the Dropzone script body or any object which can host scripts in the system.

To open the file and read the lines into a table, use the readFileLines Lua function:

--function to read in lines of a input file into a Lua table
readFileLines = function(file)
	local ioLib = require'io'
	local lines = {}
	local f = ioLib.open(file, "rb")
	if f ~= nil then
  		lines = {}
  		for line in ioLib.lines(file) do
    		lines[#lines + 1] = line
  		end
	end
	f:close()
	return lines
end

To put the columns in a line into a Lua table, using the line and column separator type as arguments, use the lineColumnsToTable Lua function.

--function to put columns in a line into a Lua table. arguments are the line and the column separator type
lineColumnsToTable = function(line,sep)
    local res = {}
    local pos = 1
    sep = sep or ';'
    while true do
        local c = string.sub(line,pos,pos)
        if (c == "") then break end
        if (c == '"') then
            -- quoted value (ignore separator within)
            local txt = ""
            repeat
                local startp,endp = string.find(line,'^%b""',pos)
                txt = txt..string.sub(line,startp+1,endp-1)
                pos = endp + 1
                c = string.sub(line,pos,pos)
                if (c == '"') then txt = txt..'"' end
                -- check first char AFTER quoted string, if it is another
                -- quoted string without separator, then append it
                -- this is the way to "escape" the quote char in a quote. example:
                --   value1,"blub""blip""boing",value3  will result in blub"blip"boing  for the middle
            until (c ~= '"')
            table.insert(res,txt)
            assert(c == sep or c == "")
            pos = pos + 1
        else
            -- no quotes used, just look for the first separator
            local startp,endp = string.find(line,sep,pos)
            if (startp) then
                table.insert(res,string.sub(line,pos,startp-1))
                pos = endp + 1
            else
                -- no separator found -> use rest of string and terminate
                table.insert(res,string.sub(line,pos))
                break
            end
        end
    end
    return res
end

To create Folder Items, use the createFolder Lua function, which takes the name and parent object path as arguments. This function checks if the objects already exist and if not, will create them.

--function creates a folder
function createFolder (name, parentPath)
	local parent = syslib.getobject(parentPath)
	local folder = nil
	if parent ~= nil then
		folder = syslib.getobject(parentPath .. "/" .. name)
		if folder == nil then
			folder = syslib.createobject(parentPath, "MODEL_CLASS_GENFOLDER")
			folder.ObjectName = name
			folder:commit()
		end
	end
	return folder
end

To create the HolderItems, use the createioitems Lua function, which takes the name and paretn object path as arguments. This function checks if the objects exist and creates them if they do not. In addition, this function automatically sets the items to historize data:

--creates holder items
function createioitem (name, parentPath)
	local parent = syslib.getobject(parentPath)
	local ioitem = nil
	if parent ~= nil then
		ioitem = syslib.getobject(parentPath .. "/" .. name)
		if ioitem == nil then
			ioitem = syslib.createobject(parentPath, "MODEL_CLASS_HOLDERITEM")
			ioitem.ObjectName = name
			ioitem.ArchiveOptions.ArchiveSelector = 2
			ioitem.ArchiveOptions.StorageStrategy = 1
			ioitem:commit()
		end
	end
	return folder
end

Copy the following function into the script body of the Drop Datasource object. This is the main function that calls each of the individually defined functions.
```
--main function, takes file as input, creates I/O tree under DZ from file, writes history data
return function(filepath)
    local datasourcepath = syslib.getselfpath()
    local lines = readFileLines(filepath)
    local headercolumns = lineColumnsToTable(lines[1], "\t")

    for i=2, #lines do
        local columns = lineColumnsToTable(lines[i], "\t")
        createFolder(columns[2], datasourcepath)
        local itempath = datasourcepath .. "/" .. columns[2]
        for j=3, #headercolumns do
            createioitem(headercolumns[j], itempath)
            dhpath = itempath .. "/" .. headercolumns[j]
            utc = syslib.gettime(columns[1])
            syslib.setvalue(dhpath, columns[j], 0, utc)
        end
    end
end
```
explanation

In the first lines of the function, the file is opened and each placed in a table ( using the readFileLines function) then the lineColumnsToTable function reads the header columns into another table (using tab "\t" as the separator type).

The rest of the lines are then iterated through using a for loop and the lineColumnsToTable function is used gain to separate each line into columns.

The createFolder function creates a folder using the Name column and then the 'x', 'y', 'z' IO-Items are created below it using the createioitem function in a nested for loop (the names coming from the header columns table). The VQT values are written to the objects in the same for loop using the syslib.setvalue function.
Click Apply to save the changes.
Copy the file into the configured Disk Folder. After the file is processed, the tree should look like the image below:
Click the Add Tab button at the top of the config compilation and select the "History Grid" option from the drop-down menu to open a History Grid in the compilation.
Drag any of the newly created IO-Items to the History Grid to see that the historized values have been recorded.

The Custom Script processing options allow you to use the full Lua Scripting engine of AspenTech Inmation to process files however you want. The two examples here only touch the surface of what is possible and the potential is there to completely customize the Dropzone Datasource object to match the requirements of your system.

For more information about the system Lua library and the Lua scripting engine, see the Lua Scripting section here.