Hi!
I was wondering how we should encrypt large data that doesn’t fit into just one ciphertext and serialize.
Assume you have a large file and want to read & encrypt.
I was doing it by 1) read some portion of data that can be encrypted with the given key and encrypt. 2) push the ciphertext into a vector structure, e.g., vector. 3) repeat 1 and 2 till all data is read. 4) serialize the whole vector at once.
I am not sure if I am doing it correctly or if there are any better ways to do.
Hi @ballb,
The typical way in OpenFHE to deal with the data that requires multiple ciphertexts is to serialize each ciphertext to file/stream, i.e., create a separate serialization file for each ciphertext. Then during deserialization you can read these files one by one and load them in a vector of ciphertexts.
At the application level, you could write some custom logic to “tar”/consolidate ciphertexts into a single serialization file, but this is not natively available in OpenFHE.
To add to what @ypolyakov said, typically it is best to have a separate serialization for each crypto object, in that it 1) makes deserialization easier, 2) the resulting buffers/streams/files are quite large as they are — even something simple like a vector of CT, which one could serialise in one shot using Cerial (the techique OpenFHE uses) will get quite big, and 3) When serializing, you usually incur large memory overhead in that the code serializes the (quite large) crypto object to a memory buffer, then sends it off to a file, or socket. Thus it is always best to serialize the objects individually, using a naming convention that makes sense, and then if needed, tar them together using system calls. Always use Binary vs Json unless you have a need for json as binary is smaller. it is also unlikely that you will get any compression from programs like zip as most crypto objects are close to random.
Dave