Relational Database (ODBC) Datasource

The ODBC Datasource enables connection to a database to schedule periodic SQL queries. Connection can be configured through a DSN or using a connection string and it is able to execute Lua scripts to create objects and set values from the returned queries.

By default, ODBC is supported on Windows nodes. Support on Linux container nodes requires additional container configuration.

Prerequisites

  1. To create and use the ODBC Datasource, you should first have an available ODBC/SQL database to connect to and test against. In this example, a sample DemoData database is used. This can be recreated in your own ODBC database server by running an SQL command contained in the following zip file - DemoDataSQL.zip. The zipfile contains a regular SQL file that will run on for example Microsoft SQL Server and a MySQL example. A DSN (System DSN) named "DemoData" should be set up according to the ODBC/SQL database vendor’s instructions or alternatively a connection string to the database with the default database set to "DemoData". The username and password for the database is also needed to connect. A DSN can then be set up with the DemoData name.

  2. Create the DemoData example items under your Node by importing the Inmation_Examples_Mass Config.json into a Mass Config tab in the System Config app. Also download the Inmation_Examples_ODBC_MassConfig.json which will be used later to create items to read and write data in/to the example database.

ODBC Datasource Configuration

First connect to a Node that is on the same host as the Database server using the System Config App.

  1. Select a Folder in the I/O Model and right-click. Select Admin  New to open the Create Object wizard.

  2. In the left-hand panel, expand the External Interfaces section and select Datasource ODBC.

  3. In the Common property compound,, enter a name for your Datasource and, optionally, an Object Description.

    create odbc datasource common
  4. Click on the ODBC Datasource section in the left hand panel and enter a Name for this ODBC connection.

  5. In the ODBC Connection compound, first enter the Datasource Name. This is equivalent to the DSN name (if you have a DSN set up). Alternatively a connection string of the following form can be entered directly:

    Driver={SQL Server Native Client 11.0}; Server=someserver;Uid=someuser;Pwd=somepassword;

    In this example we will use a DSN that points to the DemoData database. We must also enter a valid username and password for the connection.

    create odbc datasource odbc props
  6. The Keep Open checkbox can be ticked if you wish to keep the database connection open between calls, for this example we will leave it unchecked.

  7. In the Query Execution settings, click the Query …​ button to open an editor to enter the following SQL query:

    SELECT *,GETDATE() [timestamp] FROM [DemoData].[dbo].[Tags]

    This returns all columns from the InmationDataDemo Tags table and uses the SQL GETDATE() function to get values with the current system timestamp.

    If connecting to a MySQL Database, the following SQL query can be used:

    SELECT *, NOW() AS timestamp FROM DemoData.Tags;
    create odbc datasource odbc props query
  8. Click OK to save the query.

  9. Then, in the Scheduling property compound, change the Schedule Recurrence property to each 30 seconds (this will perform the database query every 30 seconds - we will use this initially just to check the connection).

    create odbc datasource odbc props schedule
  10. Click Create to create the datasource in the I/O Model.

  11. Select the newly created OBDC datasource and, in the Object Lens panel, open the ODBC Database > Diagnostics property compound. The Process State field indicates the time until the next data query call and any errors will be displayed in the Last Error field.

    create odbc datasource odbc props diagnostics
    create odbc datasource odbc iomodel
    The Datasource object will only display 2 green lights in the I/O model when a successful connection and first query is made. If the Datasource is still in a neutral communication state (bottom light remains grey) after repeated calls then you should check your connection settings for errors.
  12. Click on the Data button to open a table of data from the last query. The data retrieved should correspond to the query entered in the Query property.

    create odbc datasource odbc props data

Adding Data Processing Lua Scripts

Lua Scripts can be used to process incoming ODBC data either on a line-by-line basis or for a complete table at once.

The ODBC Datasource is able to process the data retrieved from the database by attaching Lua scripts to either do Line By Line Processing or Table Processing which processes the whole data at once. In both examples, the data returned from the database will be used to create objects under the datasource and historize the data in the time series repository. At this point the Inmation_Examples_ODBC_MassConfig.json should be imported into a Mass Config tab in the System Config App and the objects created to start populating the database with vqt values.

Preparing the Examples

  1. Firstly we will change the SQL Query field to make it so it retrieves time series data from the 'Data' table. Click the Query …​ button in the Object Lens of the ODBC Datasource to open the SQL Query Editor and enter the following query:

    SELECT TOP 3 T.Name, D.Value, D.Quality, D.timestamp
    FROM Tags AS T
    JOIN Data AS D ON T.ID = D.TagID
    ORDER BY D.timestamp DESC

    This query retrieves the names of the Tags from the 'Tags' table (there are 3 tags) then does a JOIN with the 'Data' table to return the last VQT for each tag. Click Ok to commit the query.

    The same SQL query in MySQL format:

    SELECT T.Name, D.Value, D.Quality, D.timestamp
    FROM Tags AS T
    JOIN Data AS D ON T.ID = D.TagID
    ORDER BY D.timestamp DESC
    LIMIT 3;
  2. Once the next query cycle has completed, check the Diagnostics section to see if the query was successful (if successful the Datasource should show 2 green light in the I/O Model, check the Last Error field to get more details of any errors returned by the SQL Server). Then open up the Data table to see the result of the last query and to check that the right data is returned.

    create odbc datasource odbc data example

Line By Line Processing

Lua scripts hosted in the Row Script property process each line which is returned for the data query individually.

  1. Go to the Data Processing property compound and open the Row Script editor by clicking the …​ icon. The Row Script defines the Lua script that will be executed on every row in the returned data table. The function has two arguments "data" which is the row from the returned table and "index" which is the row number. For this script execution, we will create Variable objects below the Datasource object corresponding to the Tag names in the database. The returned VQT will then be written to the variable, and historized. To make the object creation (and subsequent upserting) easier we will use an internal ´esi-variables´ library to create the Variable and automatically configure it to historize data. To do this, enter the following script into the Row Script editor and click OK to confirm.

    local VAR = require'esi-variables'
    return function(index, data)
    	VAR:SETVARIABLE(data.NAME, data.Value, data.Quality, data.timestamp)
    end
  2. When the next query is made after 30 seconds, the script will be executed on each row of the returned data table. The Variable objects wil be created underneath the Datasource in the I/O Model.

  3. Add the Variable objects to a History Grid tab to see the historized values.

Table Processing

As an alternative to line-by-line processing done by the script hosted in the Row Script property, the script in the Final Script property processes the complete returned data table at once.

The maximum number of rows returned for the query is limited by the value configured in the Top Rows property in the Diagnostics compound of the ODBC Datasource object’s properties. Rows beyond this limit are ignored.
  1. In the Data Processing property compound, open the Final Script editor by clicking the …​.

  2. The Final Script defines the Lua script that will be executed once, on the complete returned data table. The only argument for this function is "data" - the whole table which is returned for the query. Making use of the ´esi-variables´ library again, the following simple example adds the complete query result as a JSON string in a Variable object underneath the Datasource object. Enter this script into the Final Script editor and click OK to confirm.

    local VAR = require'esi-variables'
    return function(data)
    	VAR:SET("Final/data", data)
    end
  3. Select Final / Data underneath the ODBC Datasource object and right-click onto the Faceplate in the Object Lens.

create odbc datasource final script

Using SQL Actions in Table Processing

The data returned by the Final Script property can include SQL action descriptions, so that the ODBC Datasource object can then apply these actions to the returned table data. 'EXECUTE' currently is the only supported SQL action.

Action descriptions are provided in form of a JSON of the following structure:

{
	{cmd = "EXECUTE", sql="<some SQL query>"},
	{cmd = "EXECUTE", sql="<another SQL query>"},
	...
}

The following sample function illustrates the use of an SQL action to set the value of the Datetime column in the Tags table. This column does not exist by default in the DemoData database. If you want to execute the the sample function, you have to prepare the database by creating a new column in the dbo.Tags table, with the Column Name 'Datetime' of Data Type 'datetime'.

return function(Tags)
    --IMPORTANT: data contains the DIAGNOSTIC rows only!

    --do something with the data
    local count = #Tags

    if count > 0 then
        --Prepare the actions table defining what is to be
        --executed after final script, structure example
        -- actions = {
        --              {cmd = "EXECUTE", sql="INSERT INTO ..."},
        --              {cmd = "EXECUTE", sql="UPDATE ... SET ..."}
        --          }
        local actions = {}
        --Only supported cmd type for now
        local cmd = "EXECUTE"
        --prepare a sql statement
        local sql = "UPDATE Tags SET Datetime = GETDATE() WHERE ID in (%s);"
        --get ids from the passed in data
        local ids = {}
        for i=1,#Tags do
            table.insert(ids, Tags[i]["ID"])
        end

        --adjust sql statement accordingly
        sql = string.format(sql, table.concat(ids,","))
		--sql = "UPDATE Tags SET [Processed] = 1 WHERE ID in (1,2,3)"
        --set actions table
        table.insert(actions, {cmd = cmd, sql = sql})
        --return table to ODBC data source for execution
        return actions
    end

    --no action required
    return nil
end

Dynamic Variables

When specifying the SQL query it is possible to use dynamic variables in the query that can be defined in the Dynamic Variables table property under the Query Execution property compound. The dynamic variable is accompanied by a script function that will retrieve a value for the variable at runtime.

Dynamic variables must be decorated with a double underscore at either side of the variable name. For example: *__MYVAR__*

The __STARTTIME__, __STARTTIMELOCAL__, __ENDTIME__ and __ENDTIMELOCAL__ dynamic variables are already available to be used without needing to be added to the Dynamic Variables table.

To demonstrate the use of dynamic variables, we will use the example where the user wishes to query 10 rows of the database for each cycle. A dynamic variable will be used to retrieve the last processed row for the cycle (which will be stored in a Variable object) so the next query will begin on the next row from where the last cycle finished.

  1. Firstly, create a xVariable object underneath the ODBC Datasource in the I/O model named "last" (you can choose to historize the values if you wish). This Variable will be used to write the row ID number for each cycle.

  2. Open the Dynamic Variables table property for the ODBC Datasource and enter __last__ in the Variable column (remember to use double underscores). In the Script Function column, enter the following script:

    return function()
    local ok,
    	res = pcall(function() return syslib.getvalue("./last") end)
    		if ok then return res
    		else return 0
    		end
    end

    This script will get the value from the Variable created in the first step. Click OK to save the dynamic variables table.

  3. Now open the Query property and enter the query shown below and click OK. The query utilizes the __last__ dynamic variable to pick the next set of rows for the next cycle.

    SELECT TOP 5 * FROM Data WHERE Data.ID > __last__
  4. Now open the Script property of the OBDC Datasource under the Data Processing property compound and enter the Lua script below. This takes the ID of the last returned row of the SQL query and sets the value on the Variable object created in the first step. Click OK to save the script.

    local VAR = require"esi-variables"
    return function(index, data)
    		local last = tonumber(data.ID)
    		VAR:SETVARIABLE("last", last)
    
    end
  5. Wait for the next query cycle to finish then open the Data table property of the ODBC datasource under the Diagnostics property compound.

    As you refresh the table after each cycle, the row IDs will update to show the results of the last query following on from the rows queried in the cycle before.