Subscribing to a very large number of Nodes in a Prosys OPC UA Java SDK based Server.

This one is from one of our customers, Daniel Fernandez Boada from Bergauer AG, who is describing an ingenious way to subscribe to a huge number of changing data in an aggregating OPC UA server.

The project

In our last project based on the Prosys OPC UA Java SDK, my team at Bergauer AG was in charge of providing a system containing an aggregation OPC UA server that had to connect to 6 different OPC UA servers, also developed by us.

Each of these 6 OPC UA servers is in permanent communication with the PLCs of one different section of the 35 km long Lötschberg base tunnel in Frutigen, Switzerland. They have to handle an approximate amount of 100k nodes each, with an average of 250 of them changing per second, and notify all these value changes to the aggregation OPC UA server. Here we process all this incoming data and generate alarms, events or also changing the value of other nodes configured exclusively in this server. It represents a data model containing over 600k nodes. To make this all a bit more complex, this aggregation server needs to notify also all its value changes and events to different OPC UA clients that are connected: the GUI management, the alarm management server, the event management server, the historical management server…

A quick glance to the system will tell you that we are generating millions of MonitoredDataItems and MonitoredEvenItems on both sides, server and client, with the consequent loss in performance. I have to say that the first trials subscribing all the elements one by one were not very promising. As one of the requirements of the system, for robustness reasons, was to notify always all the value changes and events from each layer to the next, it seemed that there was very little we could do about it. After several trials with different lengths of subscriptions and monitored items we gave up and realized that a new approach, but still OPC UA compliant, had to be considered.

Monitoring Events via Server Object – only

First we reduced the number of used MonitoredEventItems to one. By reading again the OPC UA specification I found out that we just need to subscribe to the EventNotifier of the Server Object. As long as the Server Object has a reference of type HasNotifier to each of the nodes that may generate events, whenever one of these nodes generates an Event, it must be also notified from the referenced node, in this case the Server Object. This may seem obvious for the experienced OPC UA programmers but I will detail it here for the beginners.

To implement this we create our own class, MyEventManager that extends com.prosysopc.ua.server.EventManager. However, in our notifyEventItem method we always check if the node has a HasNotifier reference to another node. If this is true, we call the same method again until we reach the Server Object, which is the only node we have subscribed to in the client:

public class MyEventManager extends EventManager {

public void notifyEventItem(UaNode notifierNode, EventData eventData) {

Listitems = getMySubscriptionManager().
getMonitoredEventItem(notifierNode.getNodeId());
for (MonitoredEventItem item : items) {
item.notifyEvent(eventData);
}

UaReference notifierReference = notifierNode.
getReference(Identifiers.HasNotifier, false);
if(notifierReference!=null) {
notifyEventItem(notifierReference.getTargetNode(), eventData);
}
}
...
}

The “Event Design” Applied for Data Changes

This reduction of objects for the events gave me another idea, why not just using use one MonitoredDataItem to notify all the value changes in the system? After thinking about this, I came up with the ‘LastValueNode’ concept. This is nothing but a node that contains, and notifies, always the last value that has been changed in the server, no matter which node has been changed. The only problem was to know, which node had really been changed. As the values of our Variables are all Structured Objects all I had to do was to add a new property indicating the NodeId that this value belongs to. Now, just by subscribing to this LastValueNode, the client applications would get all the value changes of all the nodes in the server.

Let’s start by looking at the subscription to this new node. We need to set the MaxNotificationsPerPublish to 0 so we accept any number of notifications per publish. Also we tune the notification buffer size to allow a big number of notifications and set the publishing interval to the minimum value:

 subscription.setMaxNotificationsPerPublish(0);
subscription.setNotificationBufferSize(1000000);
subscription.setPublishingInterval(20);

The important parameter for the MonitoredDataItem at the client side is its queue. We need to increase it, otherwise we won’t be notified of all changes and most of them will be overridden. In our case it looks like:

 NodeId lastValueNode = new NodeId(2, serverName.concat(".")
.concat(CoreIdentifiers.CORE_NODE_LAST_VALUE_NAME));
MonitoredDataItem item = new MonitoredDataItem(
lastValueNode, Attributes.Value,
MonitoringMode.Reporting);
item.setQueueSize(1000000);
item.setDataChangeFilter(dataChangeFilter);
subscription.addItem(item);
client.addSubscription(subscription);

In the server we need to remember to always change the LastValueNode immediately after the original node has been changed. A good place to do so is right before the notification of the value change. In our server this is done in our NodeManager class:

public class MyNodeManager extends NodeManager {

private NodeId lastValueNode;

public synchronized void updateValue(
NodeId notifyNode, MyNodeValue value) {

Listitems = getCoreSubscriptionManager().
getMonitoredDataItem(notifyNode);

if(items!=null) {
for(MonitoredDataItem item : items) {
item.notifyDataChange(new DataValue(
new Variant(value)));
}
}

if(hasNode(lastValueNode)) {
updateValue(lastValueNode, value);
}
}

...
}

In our case we also included some extra logic for this node. When we subscribe to this node, we trigger notifications for all the current values of all the variable nodes. This is done in our SubscriptionManager:

public class MySubscriptionManager implements
SubscriptionManagerListener {

NodeId lastNodeValue;

@Override
public void onAfterCreateMonitoredItem(
final ServiceContext serviceContext,
final Subscription subscription,
final MonitoredItem itemToCreate) {
//System.out.println(counterB++);
if(itemToCreate instanceof MonitoredDataItem) {

if(itemToCreate.getNodeId().equals(lastNodeValue)) {
notifyAllValues((MonitoredDataItem)itemToCreate);
}
}
}

Useful Parameters

Some other parameters we use in our client…

 client.getEndpointConfiguration().setMaxArrayLength(120000000);
client.getEndpointConfiguration().setMaxByteStringLength(120000000);
client.getEndpointConfiguration().setMaxMessageSize(64 * 12000000);
client.getEndpointConfiguration().setMaxStringLength(120000000);
client.getEndpointConfiguration().setMaxBufferSize(120000000);

TcpConnection.setReceiveBufferSize(1000000);
TcpConnection.setSendBufferSize(1000000);

And in the server:

 server.getSubscriptionManager().
setMaxRetransmissionQueueSize(100000000);

TcpConnection.setReceiveBufferSize(1000000);
TcpConnection.setSendBufferSize(1000000);
UATcpServer.setReceiveBufferSize(1000000);

Deployment

This control system is currently deployed at the control center of the Lötschberg base tunnel in Frutigen, Switzerland. It has been controlling the whole tunnel for more than a month and currently we are engaged in the final acceptance tests which hopefully will be passed by 7th of July.

Daniel Fernandez Boada.
Chief of Bergauer AG – Vietnam
dfb (at) bergauer.ch
http://www.bergauer.ch

Jouni Aro