Expose save file bytes

[khalibloo]

Describe the feature requested
My use case scenario allows users to pick a location to save files and also to open files from the disk. For this, I need to have a way of specifying the save/load location. Further, I'm working on a web platform release as well which reads the bytes from a url and needs to somehow reconstruct the scene from that.

How does it help the user (you)?
Having more control over the save location or being able to load from bytes is essential to my scenario.

Is it related to a problem?
Unreal Save Game system does not allow this level of control either. So the only way is through a plugin.

[Optional] Propose an approach to the feature (the less we think the better!)
I would suggest exposing serialization/deserialization blueprint nodes that can handle any UObjects or structs. And also compress/decompress blueprint nodes to complete the flow.

Any other possible approach?
Exposing a save location input or setting is the next best thing, but it wouldn't solve my web platform use case.

[Optional] Additional context
The app in my scenario is a model viewer. Saving and loading to and from disk and URLs are absolutely essential.

[muit]

Hi @khalibloo

Thanks for the feature proposal!
That is something that could help other people as well, sounds worth it.

Just for naming convention, serialization is the process of obtaining world data while saving files is, well, saving the file. Due to the asynchronous way the plugin works this two processes are very well defined and splitted.

I assume your request is related to saving the files and how to customize this process and not directly with serialization.

The way our files are handled is that, appart from the fact that we create our own files our way, is UE4 the one who decides where and how to store this bytes depending on each platform.

I think it's possible to modularize this so that you can connect your own "file adapter" (just giving it a name atm haha)
With a few rules I guess:

• It needs to be thread-safe because saving files is optionally done without stoping the game.
• Being something of such low level, it would be one of two: A lot of control and C++, or fairly limited but in Blueprints.

Sorry for the long message haha, trying to stablish a conversation about the feature :)

muit

[khalibloo]

Haha, it's not that long of a message. About as long as mine :)

That's correct. The request is about saving the file. But since you mentioned that UE4 currently decides the save location per platform, this feature would require a way to bypass that and save to a specified location directly. One way to do this is to use the FileManager class in UE4 to just write the bytes to the file.
I'm a fan of the "fairly limited but in Blueprints" idea. Advanced C++ users already have access to many of these file access functions anyway. And by going with blueprints, you will not offload the responsibility of being thread safe to the average user as all the functions are still contained in the plugin.

Simple Use Case
For the option to choose a save/load location, the plugin could provide a variant of the Save node with location input. Then it would use FileManager/FFileHelper underneath to write to the file.

Advanced Use Case
For even more control, eg to allow saving/loading to/from other sources of bytes besides the disk, the plugin could expose a "SaveToBytes" node (technically, SerializeToBytes) with a bytes array output, and a WriteBytesToFile node with a string input for the full file path. Basically, just modularizing the serialize/save processes. The latter node can be found in the free LE Extended Standard Library, in fact. Turns out that was a custom node I made using the FFileHelper.

bool UFileOpsBPLibrary::SaveBytesToFile(FString Filepath, TArray<uint8> ByteArray) {
	return FFileHelper::SaveArrayToFile(ByteArray, *Filepath);
}

If I'm not mistaken, the compression is likely tied to the save process as well. If so, then perhaps additional nodes will be needed to compress/decompress the bytes. I'm not very familiar with how compression really works, but if it can be split off from the file reading/writing process, then it would help. So the entire flow could look like so: SaveToBytes -> Compress -> WriteBytesToFile
But if compression is tied to the serialization process, then there would be no need to expose it as a separate node. The flow could just be SaveToBytes -> WriteBytesToFile. It would be simpler in fact since the user won't have to remember to decompress when loading from bytes.
That got long pretty fast :D

[muit]

This is now scheduled to go out with 1.2 :P Sorry for the wait.

Also, notice that in 1.1, loading and saving files asynchronously was implemented.

I really like the proposals you made. Will have them in great consideration.
At the same time, because the saving and loading process is very parallelized, I was thinking of splitting it into modular parts. Each part would allow you to customize different areas of saving/loading. For example, a part that does the saving of a file, other that does the serialization.

This would also help when you need certain rules to apply while saving. Depends on the game.

Finally, if you know C++ and want to check how files get saved, it's at:
FileAdapter.h
FileAdapter.cpp

Yes, compression is tied to saving, but its optional and can be externalized.
In therms of vocabulary, I like to split saving into 3 main concepts. Serialization -> Compression -> Saving
Sometimes you only do one (like when a sublevel streams in. It only gets deserialized. It was already loaded). Other times you only do compression and saving.

Kind of a mixed up message there haha

[khalibloo]

That's awesome news! Thanks very much for your hard work. Itching to try out the next release.

Modularizing saving and loading would surely help with customization and widen the use cases for the plugin. Depending on how much modularity is introduced eventually, it may be worth considering still maintaining some nodes that aggregate the steps. Just so that the simple use cases still are able to save and load with one or two nodes.

Thanks for the insight as well. Much appreciated.

[muit]

Delayed a little bit to release 1.2 with 4.22

[khalibloo]

No worries, I understand. 4.22 is fresh out the oven. You might as well pat 2 cats with one hand.

[muit]

Will close this issue since the concept is now clear on SaveGraphs.
Thank you for the detailed feedback!

Make sure to check #40