Cannot pickle 'openfhe.Ciphertext' object

OpenFHE ports/wrappers
, ,
1

Hello, I am trying to use multiple processes to evaluate multiple ciphertexts in the same time, by using openfhe-python (version 1.3.0.0). To do so, I tried to implement a simple toy example with BFV by using the python module concurrent.futures. Here is an excerpt from the code:

...

# Vectors to be encrypted
vec_of_ciphertexts = []
vec_of_ciphertexts.append([1, 2, 3, 4, 5])
vec_of_ciphertexts.append([12, 22, 33, 46, 15])
vec_of_ciphertexts.append([11, 12, 13, 14, 15])
vec_of_ciphertexts.append([31, 23, 33, 34, 35])

# Pack and encrypt vectors
enc_vec_of_ciphtexts = []
for vec in vec_of_ciphertexts:
    vec_packed = crypto_context.MakePackedPlaintext(vec)
    enc_vec_of_ciphtexts.append(crypto_context.Encrypt(keys.publicKey, vec_packed))

# Parallel call
with concurrent.futures.ProcessPoolExecutor() as executor:
    enc_results = list(executor.map(HE_add, enc_vec_of_ciphtexts))

...

where HE_add() is just a function that adds a ciphertext with itself to showcase the parallel execution.

When I try to execute this, the following error shows up:

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle 'openfhe.Ciphertext' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/asdasd/workspace/python_parallel/import concurrent.py", line 155, in <module>
    enc_results = list(executor.map(HE_add, enc_vec_of_ciphtexts))
  File "/usr/lib/python3.10/concurrent/futures/process.py", line 575, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle 'openfhe.Ciphertext' object

Additionally, reading from the Introduction page of a different module for multi-processing, MPI for Python:

… It supports point-to-point (sends, receives) and collective (broadcasts, scatters, gathers) communication of any picklable Python object…

The problems in both cases seem to be that ‘openfhe.Ciphertext’ object (and presumably other OpenFHE objects) cannot be pickled.

If I undstand correctly, these OpenFHE objects don’t implement the class method that pickles them, in order for concurrent.futures for example, to be able to handle it.

I am aware that you can serialize an OpenFHE object in a standalone way, following the examples here, but I don’t see how this could be used in the above example.

Could the implementation of the pickle method in the Openfhe object solve this issue? Is there any other way to circumvent this problem? My python knowledge is somewhat limited, so please correct me if I got any of this wrong.

Thanks!
Giorgos

2

@g_tasop
Hi, you’re right. In OpenFHE, most custom objects (not just ciphertexts) can’t be pickled using Python’s standard pickle module. But OpenFHE provides its own serialization and deserialization methods, which are separate from pickle.

This is similar to how OpenFHE works in C++, using internal tools to save and load data in binary or JSON formats. These methods are available in Python and are the correct way to handle objects like ciphertexts, keys, etc.

Quick example:

# Serialize
serialized_str = ciphertext.Serialize()
....................................................................................
# Deserialize
ciphertext = Ciphertext()
ciphertext.DeserializeCiphertextString(serialized_str)

For a complete example, check out this script

1 Like
3

This seems to work! I guess these multiprocessing libraries, can operate with the custom serialize function and do not have a hard requirement for pickling. So instead of providing the complex openFHE objects as input (that raises the pickling error), one can provide the serialization of them.

P.S in the linked resource you provided I only see serialization to file but a serialization to a string is require (as pointed out in your code snippet). I found some usefull examples/tests here.

Thanks a lot!