11:24, EET
January 27, 2022
Hi everyone,
every now and again in our app when try to subscribe for updates from PLCs we experience Bad_Timeout errors like the one below:
“`
com.prosysopc.ua.ServiceException: Bad_Timeout (code=0x800A0000, description=”The operation timed out.”)
at com.prosysopc.ua.client.Subscription.a(SourceFile:1080)
at com.prosysopc.ua.client.Subscription.createMonitoredItems(SourceFile:1615)
at com.prosysopc.ua.client.Subscription.addItems(SourceFile:238)
Caused by: com.prosysopc.ua.stack.common.ServiceResultException: Bad_Timeout (code=0x800A0000, description=”The operation timed out.”)
at com.prosysopc.ua.stack.transport.tcp.io.SecureChannelTcp$1.run(SourceFile:227)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
“`
I tried to reproduce this issue locally by running both our app and the OPC server we use locally and I couldn’t get even a single timeout, although I connected much more PLCs and data points than we usually have in our production setup. In our production setup, though, this started happening quite often recently. There are no substantial CPU/memory/Network spikes when this operation happens. Can you perhaps point us towards what might be causing these timeouts? What are the default values for them? Is there a way for us to control these values? Thanks!
12:34, EET
April 3, 2012
Hi,
What version of the SDK you use (like in general most likely doesn’t matter here, but it is good info to give)? How many items you have in the subscription, 100?, 1000? 10000? more?
The general timeout can be set via UaClient.setTimeout(…) (milliseconds).
One possible scenario is that the PLC tries to get the individual initial values for the items while processing the CreateMonitoredItemRequest, the more there are items, the longer it can potentially take (if it is e.g. sequential operation to the true datasource), thus possibly timeouting.
You can control how many items at max (per CreateMonitoredItemRequest) the Subscription does via Subscription.setMaxMonitoredItemsPerCall(…). It is by default -1. If it is -1, then we’ll use the OperationLimits.getMaxMonitoredItemsPerCall() instead. This is calculated to a minimum of the Client and Server limits. Client will read this info when connecting. OperationLimits are an optional feature of UA (they were introduced I think in OPC UA 1.02), if they do not exist, we use a default of 10000 for each limit (which will be limited if the client used lower values for its own limits; this is also our default limit). Client limits can be controlled via UaClient.setOperationLimits(…).
Most likely the easiest is to put Subscription.setMaxMonitoredItemsPerCall to e.g. 100 (or even 10), just to first check that this fixes the problem, then you can increase it.
P.S.
At least in more recent SDK versions you should see the operationlimits being used in INFO level logs on UaClient, like e.g.:
INFO [main] com.prosysopc.ua.client.UaClient – Using OperationLimits [maxMonitoredItemsPerCall=10000, maxNodesPerBrowse=10000, maxNodesPerHistoryReadData=10000, maxNodesPerHistoryReadEvents=10000, maxNodesPerHistoryUpdateData=10000, maxNodesPerHistoryUpdateEvents=10000, maxNodesPerMethodCall=10000, maxNodesPerNodeManagement=10000, maxNodesPerRead=10000, maxNodesPerRegisterNodes=10000, maxNodesPerTranslateBrowsePathsToNodeIds=10000, maxNodesPerWrite=10000]
13:22, EET
January 27, 2022
Thanks for the reply, Bjarne!
We are using version 4.4.0-1240 of the SDK. The problem occurred in a production environment when we have 36 PLCs connected to the OPC server with a total of 111,838 data points (tags). Please note that we are using only the prosys client library and we use a different vendor for the OPC servers.
I tried to reproduce it locally, but it never happens even with significantly more PLCs and data points (I managed to get up to 85 PLCs and 278,359 data points without getting an error before I gave up trying)
In another production environment, though, the problem didn’t happen at all although we have more PLCs there as well – 40 PLCs and 160,046 data points connected to a single OPC server.
13:52, EET
April 3, 2012
OK, then I think you do not see the log, as I think it was added in 4.5.6, or at least around then we did some changes e.g. now as the first action we read the Read limit individually, as it could be lower than the number of OperationLimits nodes thus even that might be needed to split). Most likely it wont affect this however (but if you have the option please try also with the latest 4.7.2. release, just in case). Like 99.99% it is just the server side and you would be better of debugging that (like yes, you can always increase the timeouts etc, but this could also potentially be some server-side issue).
So it is just one OPC UA Server and the PLCs are “data sources” to it (e.g. like https://www.kepware.com/en-us/products/kepserverex/ or something similar)?
IF yes, does this mean you have that 111838 MonitoredItems in a single Subscription (and thus then single UaClient)?
I’ll note that it is quite … much, though in the client side we do not really care that much as pretty much everything is server-driven in the Subscriptions (the only thing like is that can the client keep up etc.).
Please try the options from the earlier post, though I’ll must note that with those amounts of items (IF it was a single subscription!), the “Subscription.setMaxMonitoredItemsPerCall to e.g. 100 (or even 10)” could cause making items to take forwever (as it is split more, latency will be multiplied by the number of calls that must be made). So maybe a … 1000 would be a good place to start (though, that would still be 100+ calls..).
Our current maximum defaults in the server side of the SDK is 10000 items per subscription, and that might also take quite much of processing from it, though it completely depends how often the items change and what are the publishing and sampling intervals etc.
Also, in a theoretical case it is possible that randomly a set of items to make fall under all PLCs, then it is possible that the Server does some sort of Read call to all of them, sequentially. If some of them are slow to respond, it could add up. Or depending on things, maybe the server is aggregating PLC’s UA servers (IF they have those), and then it could potentially be doing CreateMonitoredItemsRequests to them as well (and possibly sequentially). Though, these are just theories, you should ask the OPC UA Server vendor about these.
14:04, EET
January 27, 2022
Thanks a lot! We are using the TANI OPC server (https://tanindustrie.de/) and making one subscription for each PLC. We are using only 1 OPC-UA client connection for all the subscriptions, though.
So in the case I mentioned above where we have 36 PLCs and 111,838 data points (monitored items) this would mean 36 subscriptions. Most of the PLCs have ~1000-2000 items associated with them (some even ~100), but there are some exceptions where a PLC can have ~30,000 data points (monitored items).
I will try with Subscription.setMaxMonitoredItemsPerCall, thanks :+1: Do you think there might be some negative performance impact if I set the value to 1000?
Our subscription interval is set to 500ms
15:36, EET
April 3, 2012
Note that I said 1000 might be needed if you had the 100k items in one subscriptions, just because doing 100k items 10 or 100 at a time would take potentially ages. Though, if my theory is right then there is also an upper bound of how many items the server can make in a time, thus there is an upper limit (or then you increase the timeout, but doing 1000 items in 2 minutes should … not be an issue, though the server might be validating that those nodes exists from the PLC, which might be slow).
Basically that is the only impact, i.e. creating the items would take more calls to the server. It is sequential, i.e. we wait for the response of first before making a second call etc. Few extra calls doesn’t matter, 1000+ extra might, depending on the latency to the server (e.g. in theory 1 second latency with 1000 sequential calls would be 16.4minutes just from latency, or double, depending how it is calculated/defined; OK probably the latency is like few milliseconds etc, but just as an example).
So, basically via that setMaxMonitoredItemsPerCall you can test is it just that the server is so slow in making the items that we will timeout during it. If yes, preferably you could ask the server vendor can they do something about that.
Most Users Ever Online: 1919
Currently Online:
19 Guest(s)
Currently Browsing this Page:
1 Guest(s)
Top Posters:
Heikki Tahvanainen: 402
hbrackel: 144
rocket science: 88
pramanj: 86
Francesco Zambon: 83
Ibrahim: 78
Sabari: 62
kapsl: 57
gjevremovic: 49
Xavier: 43
Member Stats:
Guest Posters: 0
Members: 735
Moderators: 7
Admins: 1
Forum Stats:
Groups: 3
Forums: 15
Topics: 1523
Posts: 6449
Newest Members:
rust, christamcdowall, redaahern07571, nigelbdhmp, travistimmons, AnnelCib, dalenegettinger, howardkennerley, Thomassnism, biancacraft16Moderators: Jouni Aro: 1026, Pyry: 1, Petri: 0, Bjarne Boström: 1026, Jimmy Ni: 26, Matti Siponen: 346, Lusetti: 0
Administrators: admin: 1