Avatar

Please consider registering
guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —




— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed Topic RSS sp_TopicIcon
File operations
March 16, 2020
14:18, EET
Avatar
Hashrin
Member
Members
Forum Posts: 8
Member Since:
March 6, 2020
sp_UserOfflineSmall Offline

Hi,
I don’t know how related this is to my previous query (https://forum.prosysopc.com/forum/opc-ua-java-sdk/overriding-filedirectorytype-methods-and-massspectrometerdevicetype-methods/). Hence, I’m posting this as a new topic.
Here’s what I did. I overrode the CreateFile method of FileDirectoryType and created a file. Then I overrode FileTypNode methods and realized that the type of data to be written using the Write method is ByteString. But I don’t know how to declare a ByteString in the client’s console when it is trying to call the Write method. If I give a string instead, it gives me llegalArgumentException and ClassCastException saying that strings can’t be cast to ByteStrings. What is the right approach for this? On https://forum.prosysopc.com/forum/opc-ua-client/readwrite-files-using-opc-ua-client/, it’s implied that ByteStrings and byte arrays are the same. Still, whatever input I give will be considered as a string.
Another thing is that OPC documentation (Part 5: Information Model) says that the fileHandle is generated by the server and is unique for each session. Since I’m building both the server and the client, does this mean that I should define the fileHandle using something like a random number generator? At present, I have set it as 1 when requestFileOpen is true and 0 otherwise.

March 16, 2020
17:06, EET
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 983
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

Hi,

Please note that unlike the FileDirectoryType, FileType’s Methods are implemented by the SDK. Due to their complexity, I also recommend using those, unless you encounter any error (in which case let us know). For FileTypeNode you can call setFile(File object) that the node represents. SDK handles all filehandles etc.

For the FileTypeImpl in client side is a bit complicated, I’ll return to that in later post.

Assuming you are using the Prosys OPC UA Browser, you can r-click on a FileType node in the address space, and it has Read file and Write file options, which will then open a standard “load/save” file dialogs.

The UA Browser (same for the the old Prosys OPC UA Client as well) can only pretty much call basic types for the arguements, i.e. ByteString most likely does not yet work, please use https://www.unified-automation.com/products/development-tools/uaexpert.html instead. But generally you should not need them due to the above.

March 16, 2020
17:25, EET
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 983
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

The following might help for the client side. Note that this is based on some quite old code that once was within the SDK, but was removed (see 3.0.0 section More for more details, https://downloads.prosysopc.com/opcua/Prosys_OPC_UA_Java_SDK_3_Release_Notes.html#version-3-0-0). While it should work, you might need to change things or fill some gaps. However we’ll should add these 2 methods at some point to the FileTypeImpl since they can be still of some use.

public void readFile(File localFile, FileTypeImpl fileType, boolean append)
throws ServiceException, StatusException, IOException {
logger.debug("readFile: {} local={}", fileType.getBrowseName(), localFile);
long size = fileType.getSize().longValue();
DateTime timestamp = fileType.getTimestamp();
UnsignedInteger fileHandle = fileType.open(FileTypeOpenMode.Read);
FileOutputStream fos = new FileOutputStream(localFile, append);
try {
int actualBlockSize = getActualBlockSize();
for (long p = 0; p < size; p += actualBlockSize) {
fos.write(ByteString.asByteArray(fileType.read(fileHandle, actualBlockSize)));
}
} finally {
fos.close();
fileType.close(fileHandle);
}
localFile.setLastModified(timestamp.getTimeInMillis());
}

public void writeFile(File localFile, FileTypeImpl fileType, boolean append)
throws ServiceException, StatusException, IOException {
logger.debug("writeFile: {} local={} append={}", fileType.getBrowseName(), localFile, append);

long size = localFile.length();
EnumSet<FileTypeOpenMode> mode = EnumSet.of(FileTypeOpenMode.Write);
if (append) {
mode.add(FileTypeOpenMode.Append);
} else {
mode.add(FileTypeOpenMode.EraseExisting);
}
UnsignedInteger fileHandle = fileType.open(mode);
FileInputStream fis = new FileInputStream(localFile);
try {
int actualBlockSize = getActualBlockSize(); // (int) Math.min(size,
// getActualBlockSize());
byte[] buffer = new byte[actualBlockSize];
for (long p = 0; p < size; p += actualBlockSize) {
if (actualBlockSize > (size – p)) {
buffer = new byte[(int) (size – p)];
}
fis.read(buffer);
fileType.write(fileHandle, ByteString.valueOf(buffer));
}
} finally {
fis.close();
fileType.close(fileHandle);
}
}

The getActualBlockSize would determine how big of chuncks are fetched per Method call, sometimes we experimented with client.getEndpointConfiguration().getMaxByteStringLength() / 2

P.S.
I would be very cautious with anything related to Files within OPC UA. A simple FileType to e.g. update some configuration file or to update firmware might work, but anything bigger might end up complicated. Additionally the last time we tested the perf was abyssmal for any large file due to the overhead of multiple method calls. Additionally, unless you can pretty much guarantee that the files are only ever modified by one instance (client, server OS etc.) at a time you will need to build some kind of checksum system to ensure the read data was correct (i.e. OPC UA level can have locks for Read/Write, not all OS Filesystems have a lock concept and/or it can be ignored).

March 16, 2020
17:25, EET
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 983
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

(and seems the code tag did fail with indentation, but that is probably good enough for now….)

March 18, 2020
8:38, EET
Avatar
Hashrin
Member
Members
Forum Posts: 8
Member Since:
March 6, 2020
sp_UserOfflineSmall Offline

Hi,

What is the use of the setFile() method exactly? If I create a FileTypeNode object using the createFile() method, should I then create a File object and map it to the FileTypeNode object using setFile()? What if I want to open and read an existing file object? Or is it only possible to open and read FileTypeNode objects instead? In either case, how should opening and reading be done?

Regarding the impl classes, I saw many of them (eg, MassSpectrometerDeviceImpl) in the generated code for the client. What are their uses? Aren’t the corresponding server classes (eg, MassSpectrometerDeviceTypeNode) enough? What about the FileTypeImpl class? Since the SDK implements all FileType methods, can’t I just call them from the client, instead of defining methods (eg, readfile() and writeFile() methods given above) on my own?

March 18, 2020
11:38, EET
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 983
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

Sorry, I don’t understand.

A bit too broad question, so I’ll try to clarify it a bit, but basically some parts of your question doesn’t exactly make sense. Those parts would be covered by “we do not teach basic java programming” and “Do you know what client-server architecture means?”. It is also possible that I have misunderstood something.

I generally assume you would want to call these FileType UA Object’s UA Methods from the client side? (Since otherwise there is no point to make such objects in the server address space in the first place?). If you instead mean something else, let me know. Rest of this is assuming you mean calling them from the client side.

The pipe so to say is then:

File (Java Object, Client) -> FileTypeImpl (UA Object, Client; calling methods of it, with the code of prev post) -> Network -> FileTypeNode (UA Object, Server) -> File (Java Object, Server)

Please see the codegen manual for explanation of the generated classes and interfaces (but ask of course if something is not clear, it is quite complicated system anyway).

A FileTypeNode without a File object attached to it does nothing. So yes, you will need to create a Java File object that the node represents. (and the basic java programming part is that a File object in java is just an abstraction of a file, it might not e.g. even exist yet, basically it is just a filesystem path).

The reason it is like this is if a server would support so called node management, where a client can add Nodes, it could also add a FileType node usually. If we would just automatically create a backing file (somehow? since there isn’t direct 1:1 mapping on what it actually is, but let’s say we would assume the BrowseName is the file path) one could make it e.g. “C:\Windows\somethingimportannt” and then override it. That would not be good. Therefore instead the FileTypeNode is by default “unbound” from any File thus any method calls to it without a File attached will fail. This is also what I meant earlier that you will need to sanitize user inputs, so that e.g. something like this wont be possible.

You can chose to not use the above methods, but you would end up doing the pretty much same code by yourself. (and that would be “what client-server architecture means”). Those FileTypeImpl methods called is the OPC UA API, there is nothing else you can route to the server. Those methods are implemented inside the SDK in FileTypeNode.

March 18, 2020
15:40, EET
Avatar
Hashrin
Member
Members
Forum Posts: 8
Member Since:
March 6, 2020
sp_UserOfflineSmall Offline

Sorry for the confusion. I am familiar with Java, and I also have a basic understanding of what a client-server architecture is.
I am able to call the FIleType UA object’s UA Object’s UA Methods from the client now. SetFile() method works. The only thing which still doesn’t make sense is the following:

Bjarne Boström said
The following might help for the client side. Note that this is based on some quite old code that once was within the SDK, but was removed (see 3.0.0 section More for more details, https://downloads.prosysopc.com/opcua/Prosys_OPC_UA_Java_SDK_3_Release_Notes.html#version-3-0-0). While it should work, you might need to change things or fill some gaps. However we’ll should add these 2 methods at some point to the FileTypeImpl since they can be still of some use.

public void readFile(File localFile, FileTypeImpl fileType, boolean append)
throws ServiceException, StatusException, IOException {
logger.debug("readFile: {} local={}", fileType.getBrowseName(), localFile);
long size = fileType.getSize().longValue();
DateTime timestamp = fileType.getTimestamp();
UnsignedInteger fileHandle = fileType.open(FileTypeOpenMode.Read);
FileOutputStream fos = new FileOutputStream(localFile, append);
try {
int actualBlockSize = getActualBlockSize();
for (long p = 0; p < size; p += actualBlockSize) {
fos.write(ByteString.asByteArray(fileType.read(fileHandle, actualBlockSize)));
}
} finally {
fos.close();
fileType.close(fileHandle);
}
localFile.setLastModified(timestamp.getTimeInMillis());
}

public void writeFile(File localFile, FileTypeImpl fileType, boolean append)
throws ServiceException, StatusException, IOException {
logger.debug("writeFile: {} local={} append={}", fileType.getBrowseName(), localFile, append);

long size = localFile.length();
EnumSet<FileTypeOpenMode> mode = EnumSet.of(FileTypeOpenMode.Write);
if (append) {
mode.add(FileTypeOpenMode.Append);
} else {
mode.add(FileTypeOpenMode.EraseExisting);
}
UnsignedInteger fileHandle = fileType.open(mode);
FileInputStream fis = new FileInputStream(localFile);
try {
int actualBlockSize = getActualBlockSize(); // (int) Math.min(size,
// getActualBlockSize());
byte[] buffer = new byte[actualBlockSize];
for (long p = 0; p < size; p += actualBlockSize) {
if (actualBlockSize > (size – p)) {
buffer = new byte[(int) (size – p)];
}
fis.read(buffer);
fileType.write(fileHandle, ByteString.valueOf(buffer));
}
} finally {
fis.close();
fileType.close(fileHandle);
}
}

The getActualBlockSize would determine how big of chuncks are fetched per Method call, sometimes we experimented with client.getEndpointConfiguration().getMaxByteStringLength() / 2 

My question is, what purpose do the aforementioned methods serve if the SDK already handles the file handlers? I’ll break down what I did into the following:

1) Wrote the code for creating a FileTypeNode object using the overridden createFile() method (just like you asked me to do here : https://forum.prosysopc.com/forum/opc-ua-java-sdk/overriding-filedirectorytype-methods-and-massspectrometerdevicetype-methods/ ).

2) Created a File object and used the setFile() method to bind it to the FileTypeNode object.

3) Set UserWritable and Writable to true.

4) Ran the server and client.

5) Called the createFile() method.
Result: FileTypeNode object created successfully (I ignored the requestFileOpen part. I will check that later).

6) Called the Open() method.
Result: File opened successfully. Works for both read and write modes.

7) In write mode, I tried writing a bytestring value to the file.
Result: Works when using UaExpert. Fails when using SampleConsoleClient in the SDK. There, I’m getting an error saying that a string value cannot be converted to a bytestring value.
When you said

Bjarne Boström said The UA Browser (same for the the old Prosys OPC UA Client as well) can only pretty much call basic types for the arguements, i.e. ByteString most likely does not yet work, please use https://www.unified-automation.com/products/development-tools/uaexpert.html instead. But generally you should not need them due to the above.

, by “old Prosys OPC UA Client”, did you mean the SampleConsoleClient class? If that is the case, is there any workaround for that, instead of using other tools like UaExpert?

Now, to my queries:
1) If read(), open(), and write() methods work, then what is the need for the readFile() and writeFile() methods in the client?

2) The createFile() method should return a fileHandle. I hardcoded the fileHandle value as “1”. Is this the right approach, or is there something I’m missing?

3) Forgive me if this is a foolish question, but In general, what is the purpose of Impl classes in the client? If this is a part of client-server architecture and not OPC UA, then you don’t have to explain. I will try and figure it out.

March 18, 2020
16:51, EET
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 983
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

I’ll try to be short, but this is like 5-10 years of SDK history that I would need to explain, as we try to be backwards-compatible, where possible, or at least make it so that changes wouldn’t be too difficult to migrate (not that it always works like that).

The old name of “Prosys OPC UA Browser” is “Prosys OPC UA Client” (see the confusion vs. SDK’s UaClient class), since you linked to that application’s forum I was assuming you meant that.

Generally none of our applications/samples are on the same level as UaExpert for Write/Call operations if the data is anything other than basic numbers, strings i.e. this means that you cannot call the method from SampleConsoleClient, i.e. there is no automatic conversion logic built in (yet), that would know to convert String -> ByteString. You will need to do that manually in code yourself. Basically the only conversions that do work are the ones listed in the OPC UA Specification, version 1.04, Part 4, Table 122 within section 7.4.3 for the event filter cast operator, since that is for what it original was done. But we should probably add some extra rules eventually. But anyway, for ByteString String the option is “X” in both ways in that table, i.e. no conversion possible.

Also it might be possible, that the old “Prosys OPC UA Client” might have allowed some “0x1234567890abcd” form for ByteStrings. Large part of that application had to be rewritten so that all our java based apps share the same UI framework (and that app was our oldest one), so it is possible that we did miss something. We should regradless add something like that eventually to the SDK and to the Browser.

You will need to do a ByteString object in code, with the static factory method ByteString.valueOf(byte[]). For these cases, normally you would already have the bytes e.g. from reading the File object in the client side, like the above code does. But you can use e.g. CryptoUtil.hexToBytes(String)

I make a separate answer post for your queries, as the explanation is long.

March 18, 2020
16:52, EET
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 983
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

Query 1:

Well, nothing per se, but you pretty much end up making the same logic yourself eventually. Only if your files are so small that they could always fit in a single ByteString (there are encoding limits that can be transmitted in a single CallRequest/CallResponse) you might avoid that. The limit is normally few MBs. Also usually you have a File object in the client side you wish to copy data to or from the server. Additionally if you have large enough files they might not fit into memory at once even if there would not be any encoding limit.

March 18, 2020
17:21, EET
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 983
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

Query 2:
Some part is client-server architecture (or so I would say at least), other part of our design how the codegen classes + SDK operate (and how our SDK was before we made the codegen first place in version 2.x). This will be quite long, no way to avoid that.. this is still somewhat explained in the codegen manual in the codegen folder of the SDK zip you downloaded.

Per each OPC UA Type (using name XXXType here as example) we create 5 outputs: an interface named XXXType, 2 classes for client side XXXTypeImplBase (that implement the interface) and XXXTypeImpl (that subtypes the base) and 2 classes for server side XXXTypeNodeBase (implement the interface) and XXXTypeNode (that subtypes the base).

Java lacks partial classes that e.g. C# has. This means there is no way to modify generated code, other than writing to the file. Since we generate a lot, and sometimes we need to create more API and/or change how something internally works. For that reason we emulate partial classes by making one that we can change more freely (XXXTypeImplBase, XXXTypeNodeBase) and one we try to avoid the need to change (XXXTypeImpl, XXXTypeNode), thus you can put them in version control and keep the rest generated only during a build. Also the interface can be used to develop a component that needs to work on both sides. Also note that due to the type subtyping, it is not possible to simply develop an API where the impls would be done just be overriding, since that will not work for situations where a type is an InstanceDeclaration as part of a larger type (so typenodebase extends the supertype’s typenode etc.; is complicated…).

So both the XXXTypeImpl and XXXTypeNode are places where code can be written by the SDK user (or SDK internally for the standard model).

For server side it is more obvious, as usually the OPC UA Method implementations are written there. Additionally there is an initializer method (afterCreate) that can be used to init complex node hierarchies for initial values e.g. for state machine types. Also that allows the SDK server API to be backwards compatible better as the SDK has evolved, since we have like 20 types that add methods to the API from pre-codegen era and/or need some kind of an initialization that the OPC UA Standard model NodeSet2 XML doesn’t provide (that can be done in the afterCreate), plus UA Method implementations of course.

For the client side, well as far as the SDK goes, it is not used that much. There is few helper methods in FileTypeImpl. But there used to be a lot more, however the codegen has also evolved to create some convinience methods so pretty much internally we only need it for the FileTypeImpl. However it should still be noted that we need still an implementation for that interface, plus any of our users might have added their own custom helpers.

Additionally we could e.g. add the 2 methods that I posted to FileTypeImpl eventually. But we’ll do that once we have time and we are sure that is the best way for them to be part of the API (since they might take a really long time, since for e.g. transmitting 1GB File you will do like 100s of OPC UA Method Calls, it might not make sense to have a synchronized version of those).

Additionally due to the SDK design, even though the client and server side are both UaNodes, they pretty much need to be inherited from respective side base classes for things to work properly.

That is about the short version of it with some details skipped.

March 19, 2020
7:38, EET
Avatar
Hashrin
Member
Members
Forum Posts: 8
Member Since:
March 6, 2020
sp_UserOfflineSmall Offline

Thanks for the detailed explanation. Things are more clear now. Can I have your input for the following query too? You probably missed it.

Hashrin said

2) The createFile() method should return a fileHandle. I hardcoded the fileHandle value as “1”. Is this the right approach, or is there something I’m missing?
.  

How is fileHandle actually generated?

March 19, 2020
10:26, EET
Avatar
Bjarne Boström
Moderator
Moderators
Forum Posts: 983
Member Since:
April 3, 2012
sp_UserOfflineSmall Offline

fileHandle output for the CreateFile: (1.04, Part 5, Annex C.3.4 “CreateFile”)

“fileHandle The fileHandle is returned if the requestFileOpen is set to True.
The fileNodeId and the fileHandle can be used to access the new file through the
FileType Object representing the new file.
If requestFileOpen is set to False, the returned value shall be 0 and shall be ignored by
the caller.”

and for the requestFileOpen Boolean parameter:


requestFileOpen Flag indicating if the new file should be opened with the Write and Read bits set in the
open mode after the creation of the file. If the flag is set to True, the file is created and
opened for writing. If the flag is set to False, the file is just created.

Thus if the requestFileOpen is true, you will need to do the following:

Long fileHandleAsLong = fileTypeNode.open(serviceContext.getSession(), EnumSet.of(FileTypeOpenMode.Read, FileTypeOpenMode.Write));
UnsignedInteger fileHandle = UnsignedInteger.valueOf(fileHandleAsLong);

Note that there is also modes FileTypeOpenMode.Append and FileTypeOpenMode.EraseExisting, but the spec text only mentions Write and Read bits so thus I’m not adding Append nor EraseExisting to the open.

It is important that it is the version of the open that takes in a session, since the file handles are session-specific.

If for some reason the open call fails, it will throw a StatusException, that should then be thrown also from the CreateFile most likely.

And just in case I’ll refer this post later, the fileHandle is used in all other calls to the FileType node. Basically it is a cursor/pointer within the file. It starts at “index” 0, and its position can be changed via SetPosition Method. Note that it is automatically advanced when doing Read/Write calls, but e.g. if any calls would fail due to e.g. small temporary connection break, you could return from the last known position (the current position can be get via GetPosition). Basically the FileType API is equivalent of Java’s RandomAccessFile.

March 19, 2020
11:09, EET
Avatar
Hashrin
Member
Members
Forum Posts: 8
Member Since:
March 6, 2020
sp_UserOfflineSmall Offline

Although I had already gone through the documentation regarding File Transfer (1.04, Part 5, Annex C), I couldn’t figure out how the code for fileHandle needed to be implemented. Works perfectly now. Thank you.

Forum Timezone: Europe/Helsinki

Most Users Ever Online: 518

Currently Online:
18 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

hbrackel: 135

pramanj: 86

Francesco Zambon: 81

rocket science: 77

ibrahim: 76

Sabari: 62

kapsl: 57

gjevremovic: 49

Xavier: 43

fred: 41

Member Stats:

Guest Posters: 0

Members: 680

Moderators: 16

Admins: 1

Forum Stats:

Groups: 3

Forums: 15

Topics: 1467

Posts: 6260

Newest Members:

sagarchau, elviralangwell4, Donnavek, Eddiefauth, DonaldPooma, fidelduke938316, Jan-Pfizer, DavidROunc, fen.pang@woodside.com, aytule

Moderators: Jouni Aro: 1009, Otso Palonen: 32, Tuomas Hiltunen: 5, Pyry: 1, Petri: 0, Bjarne Boström: 983, Heikki Tahvanainen: 402, Jukka Asikainen: 1, moldzh08: 0, Jimmy Ni: 26, Teppo Uimonen: 21, Markus Johansson: 42, Niklas Nurminen: 0, Matti Siponen: 321, Lusetti: 0, Ari-Pekka Soikkeli: 5

Administrators: admin: 1