Avatar

Please consider registering
guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —




— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed Topic RSS sp_TopicIcon
Bad_TooManySessions (0x80560000)
July 27, 2022
15:45, EEST
Avatar
rvdborn
Utrecht, Netherlands
Member
Members
Forum Posts: 14
Member Since:
November 11, 2020
sp_UserOfflineSmall Offline

Hello,

We are moving from the open source Eclipse Milo SDK to the Prosys SDK but are running into a problem. We have an application that connects to a single UA Server and reads data from it. We run many of those applications conected to all different UA servers.
The Milo SDK has no problem connecting to any of the servers.
But with the Prosys SDK some of the servers return the following error:
ServiceResult=Bad_TooManySessions (0x80560000) “The server has reached its maximum number of sessions.”.
For now the choice of SDK is configurable, so we can switch back to the Milo SDK within the same application and things work like a charm again. We can switch back and forth between SDKs, each time with the same result. So it is hard to believe that there are actually too many sessions.

When I run our application against the Prosys UA simulator I don’t see any strange behaviour like more than one session being created.

Does anyone have a suggestion in what direction to look for a cause of, or better, a solution to this problem?

Thanks, Reinier

July 27, 2022
16:42, EEST
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 1026
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

Hi,

Very odd. That kind of an error shouldn’t be related to any SDK (unless like they would use multiple sessions etc). Basically we (the UaClient) do first a secure channel to do the GetEndpoints call, then close that channel and open another (security mode is tied to the opensecurechannel call and GetEndpoints needs None mode) once we have the EndpointDescription and then open the Session on top of that. Thus there should be just a single Session.

However, you do mention “We can switch back and forth between SDKs”. Please note that we have nothing to do with Eclipse Milo. Any API there that might have the same name as we do might not do the same thing.

Do you get the error always? Is this reproduce-able? Any option to name or make such server available to us (uajava-support@prosysopc.com if needed)?

Does this happen on first connect or somewhere later? Is a reconnection situation happening i.e. does this only happen after losing connection? IF yes, do you happen to have calls to UaClient.reconnect()? There is a somewhat known issue if the AutoReconnect feature of UaClient is on (it is by default) and manual calls would be done to reconnect. Currently that would log a WARN message, but in the future we will probably start throwing instead (warn log was safer option when we did add this, but this is somewhat big of a problem). This could cause a situation where multiple sessions are opened due to race conditions, but not closed as both the manual call and SDK internals would be trying to reconnect at the same time.

July 27, 2022
16:55, EEST
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 1026
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

Additionally I should note that this is a server-generated error, i.e. basically if a server is at this state, no additional (new) clients should be able to connect (try e.g. UaExpert or our Browser) before old sessions either timeout or are closed. IF a client did connect previously and lost connection i.e. it didn’t close the session and it has not timed out it might be also possible for that client to re-activate that session even after connecting back.

July 27, 2022
18:27, EEST
Avatar
rvdborn
Utrecht, Netherlands
Member
Members
Forum Posts: 14
Member Since:
November 11, 2020
sp_UserOfflineSmall Offline

Hi,

Regarding this switching back and forth: we have two Client modules, one based on Milo, the other based on the Prosys SDK. Between the two modules there are no calls to shared code, except a piece of code that gets certificates from a keystore and code that stores the data that is read from the UA server. When starting the application we use a Spring profile to activate only one of the modules, they are never both active during the same run.

So to switch we change the profile and restart the application. From the problem-servers we never get TooManySessions errors with Milo but always with the Prosys SDK. From the start it basically never connects. There are no reconnects() in our code, we only connect() and disconnect(). When starting up the application loads the nodes to read, tries to connect and start reading. When later the nodeset is updated or if a connection error occurs when reading the application would disconnect, connect and start reading again. But that never happens: we don’t get past the first connect.

What totally confuses me is the error: TooManySessions. The SDK doesn’t invent it, so it is the server that sends it. Of course it is very well possible that I do not correctly configure the UAClient, which would cause the UA Server to refuse the connection. But TooManySessions? Or do servers exist that send this error just to obfuscate the real problem, only to keep information from intruders that might make it easier for them to break in? Like password screens no longer tell you whether the username or the password is wrong.

Unfortunately we cannot give you access to the UA server. I myself cannot even access it. But I did ask our client to increase the SDK logging to DEBUG, so I can share that with you.

July 27, 2022
18:29, EEST
Avatar
rvdborn
Utrecht, Netherlands
Member
Members
Forum Posts: 14
Member Since:
November 11, 2020
sp_UserOfflineSmall Offline

Forgot to mention: our application keeps trying to connect as long as it fails

And the server software we connect to varies, but we have two servers with the same software of which one connects and the other not.

July 28, 2022
11:20, EEST
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 1026
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

Sorry, but this will be complicated to solve with just this info. Like I can make guesses and theories, but they might be quite specific scenarios. Basically you would want to observe the ‘problem server’ about how many active sessions it has. All this is also on the assumption that the server is behaving properly i.e. that the Bad_TooManySessions really means it, nothing per se prevents servers just like returning the wrong status code.

Anyway, first on the status codes, they are defined by the OPC Foundation (https://github.com/OPCFoundation/UA-Nodeset/blob/v1.04/Schema/StatusCode.csv). Just the number is transferred (as UnsignedInteger), rest relies on look-up-tables inside SDKs i.e. the text is not given by the server, just the code (but the text comes form the CSV file then).

Then on the Sessions. It is important to differentiate Sessions and Connections, or more specifically secure channels. I’m also assuming here that opc.tcp is being used, as that is 99.999% of the cases (I’ll note that opc.https exists, but basically it should not be used). Sessions can be “live” on the server even though their channel is broken or closed. If the “owner” of that session connects again, they can use the ActivateSession to reactivate the session on top of another secure channel (https://reference.opcfoundation.org/Core/docs/Part4/5.6.3/). Sessions also have a timeout value, typically it is something like an hour. So, the session will exist on the server unless closed or it times out.

Each Session on the server will take up resources, typically servers do limit themselves to X amount of concurrent Sessions. For example in our SDK’s server side the default would be 100 concurrent Sessions, but it is impossible to say what is the limit in the ‘problem server’. However in some chip-lvl micro embedded OPC UA Servers it could be like 2 (https://reference.opcfoundation.org/Core/docs/Part7/6.6.67/).

If you connect your app (in “Milo mode”) to our Simulation Server, you can observer active sessions (Options->Switch to Expert Mode) in the ‘Sessions’ tab. IF it shows the session even after you have seemingly disconnected it then there is a CloseSession service call missing and you must consult on Milo if there is a call you must make before disconnecting. Just a very_specific_scenario_theory though, this is kinda hard to debug without having access to the server.

Assuming the ‘problem server’ is properly using the error code, it did mean we were able to open the secure channel, but creating the session on top of that failed (and per the server it has already too many). Can you try with e.g. can https://www.unified-automation.com/products/development-tools/uaexpert.html open the Session? If not, that sort of would indicate the problem is with the server itself. Or another way to ask this is, did you always “start with Milo” and only after that made the “switch”? But anyway, like typically at least normal servers will allow many sessions so it would be not that normal to reach the max limit. Though, e.g. our public test instance of the simulation server (at opc.tcp://uademo.prosysopc.com:53530/OPCUA/SimulationServer) could very will reach the limit (I believe we have 50 sessions as the limit in that application) as many test on that and we have it as a sample in our Browser application.

IF you are not using sign&encrypt, you could maybe see via https://www.prosysopc.com/blog/opc-ua-wireshark/ does a CloseSession happen with the “Milo Mode”.

Like I’m also not ruling out possible bugs in our implementation, just that it sort of feels a bit odd.

Also, since you mentioned 2 servers with the same software with one of failing, any option to compare their configurations?

July 28, 2022
19:29, EEST
Avatar
rvdborn
Utrecht, Netherlands
Member
Members
Forum Posts: 14
Member Since:
November 11, 2020
sp_UserOfflineSmall Offline

Thanks for your elaborate reply. I do understand it is difficult to analyse from a distance. It is about the same for me :-).

Our client sent us two chunks of log, one where connection is made OK, the other where it isn’t. Unfortunately the logs don’t show the start of the application or the first connections that are made, which I think are the most interesting. Now some log-entries seem to be missing. So I’ll ask for new logs, from the start of the application until it is up (connected or not) for some time. I will not get them before Monday.

If you are interested I can send you the logs, but this forum doesn’t seem the right place for that (no upload facility). Maybe we should take this offline and once we get to some conclusions we report back here for posterity?

I like the idea of looking at the server using UAExpert. If the logs don’t help that would certainly the next step. Configuration items like the max number of sessions, can that be read off the server that way, or is that server dependent?

July 29, 2022
11:23, EEST
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 1026
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

Some day we will improve this forum, but anyway for now if you want you can send them to uajava-support@prosysopc.com (zip first).

As far as I’m aware, there is no OPC UA specification-wise way to get that limit. Nothing would prevent a server e.g. making a custom node to show it, but most likely there is none. There is a defined diagnostics node(s) from where maybe the current session count could be observed, though it depends on the server to support diagnostics and also have them on (and it is a server-wide flag, so it could affect all clients i.e. it could run more slow). Anyway, our simulation server does have that in path /Root/Objects/Server/ServerDiagnostics/ServerDiagnosticsSummary/CurrentSessionCount.

August 3, 2022
13:36, EEST
Avatar
rvdborn
Utrecht, Netherlands
Member
Members
Forum Posts: 14
Member Since:
November 11, 2020
sp_UserOfflineSmall Offline

Offline we analysed the new logs taken from the moment the application was started. It proved that, different from what I claimed before, the application doesn’t get Bad_TooManySessions (0x80560000) errors from the start. I should have looked at the logs from the start earlier.

Origin of the problem proved to be that the UA server returns empty (server)-nonces. The OPC UA standard is not clear about this but the current understanding is that a proper nonce (random sequence of 32+ bytes) is needed unless an Anonymous connection is made with security policy http://opcfoundation.org/UA/Se…..olicy#None and message mode None. In our case we are using None+Noe but with UserName instead of Anonymous.

After the no-nonce error (at client-side), the session is left not-activated and an attempt is made to close it. However the server refused to close the session, returning a Bad_SessionNotActivated (0x80270000) code. Giving us a catch-22: the session cannot be activated without the nonce and the session cannot be closed because it is not activated: bad behaviour of the UA Server once more. This means that each attempt to connect will leave a session behind which after a while leads to the Bad_TooManySessions result. Actually the server should recycle non-activated sessions when running short: yet another deficiency in the server software.

Our final conclusion is that the UA server is not behaving properly, probably due to being too old (other version of the same server software has no problems), and that the Prosys SDK simply cannot be used to connect with it. Of course the server software should be updated, but in our case that may not prove to be easy.

Anyway, although we didn’t arrive at an immediate solution, I want to thank Bjarne for the excellent support he gave us.

August 4, 2022
15:14, EEST
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 1026
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

I might reply things via email as well, but anyway to note most of the things here:

Sorry, I forgot that we also had a blog post regarding the thing https://www.prosysopc.com/blog/invalid-server-nonces-vulnerability/.

So, like technically we could make a flag to UaClient which could then be used to disable the check. I’m not sure should one be made yet, but it is something that we will discuss internally and it might take a bit of time. We do have other flags for some checks, but the impact of disabling this is like a lot bigger. Though then in turn IF you are in a situation where you have a server that wouldn’t support Anonymous UserTokenPolicy and you could not change the server AND you would be in a network where everyone who ever will be able to monitor the traffic is effectively allowed to know or knows that password already it probably would be ok to skip the test. But IF there is an option to use Anonymous instead of username/password I would say it should be done instead (as that would skip the test), or basically disabling the check would “convert” the Username/password into Anonymous as everyone on the network who would have seen at least once ActivateSession would be able to replay-attack that same encrypted password again (as there would be no nonces to make the difference it would be the same each time).

August 24, 2022
17:42, EEST
Avatar
Kevin Herron
Member
Members
Forum Posts: 8
Member Since:
March 7, 2014
sp_UserOfflineSmall Offline

rvdborn said
Hello,

We are moving from the open source Eclipse Milo SDK to the Prosys SDK…

Bjarne, sorry if this is inappropriate to ask here, but –

@rvdborn, can you tell me what has prompted you to make the switch? I am not trying to change your mind, just curious because it may influence some future feature development or priority. AFAIK we have not had any email correspondence that I can refer to.

Thanks

Forum Timezone: Europe/Helsinki

Most Users Ever Online: 1919

Currently Online:
35 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

Heikki Tahvanainen: 402

hbrackel: 144

rocket science: 88

pramanj: 86

Francesco Zambon: 83

Ibrahim: 78

Sabari: 62

kapsl: 57

gjevremovic: 49

Xavier: 43

Member Stats:

Guest Posters: 0

Members: 734

Moderators: 7

Admins: 1

Forum Stats:

Groups: 3

Forums: 15

Topics: 1523

Posts: 6449

Newest Members:

christamcdowall, redaahern07571, nigelbdhmp, travistimmons, AnnelCib, dalenegettinger, howardkennerley, Thomassnism, biancacraft16, edgardo3518

Moderators: Jouni Aro: 1026, Pyry: 1, Petri: 0, Bjarne Boström: 1026, Jimmy Ni: 26, Matti Siponen: 346, Lusetti: 0

Administrators: admin: 1