Convolutional Layer with FHE

FHE Questions
1

Hello everyone, I have some questions on doing the convolution operation on encrypted data:
Suppose we have a ciphertext encrypting a (32x32) image with ring dimension N = 4096 and a convolutional filter with size (3x3) and all weights of the filter are 1.

  • When I call cc->MakeCKKSPackedPlaintext with a 2-dimensional input, I get an error. How to do 2-dimensional packing with OpenFHE?
    *First, we take the convolution between the filter and image for the 1st pixel, which can be done with multiplication followed by rotation and sum. Then, we write the corresponding value to the 1st pixel of the plaintext result. How to make this “writing the result to the corresponding pixel” operation homomorphically in the ciphertext format?
    Thank you so much for your help!
2

What do you mean with “2-dimensional input”? Two values? An error log would help anyway :slight_smile:

Anyway, in order to store your image you would need 1024 values, assuming it is single channel (if rgb, you will need a larger ring)

3

Hello, I am assuming an input image of size 32x32x1. For an rgb image of size 32x32x3, yes I would increase the ring dimension, but my questions stay the same.
For the 2-dimensional input, I meant the vector<vector< double>> variable type as my input vector.
Here is a code snippet to output the error for a 2-dimensional input:

CCParams<CryptoContextCKKSRNS> parameters;
    parameters.SetSecurityLevel(HEStd_128_classic);
    parameters.SetRingDim(1 << 15);
    parameters.SetMultiplicativeDepth(5);

    CryptoContext<DCRTPoly> cc = GenCryptoContext(parameters);
    cc->Enable(PKE);

    vector<vector<double>> x1 = {{1,1,1,1,1,1,1,1},{1,1,1,1,1,1,1,1}};

    Ptxt pt1 = cc->MakeCKKSPackedPlaintext(x1);

The error:

error: no matching member function for call to ‘MakeCKKSPackedPlaintext’
Ptxt pt1 = cc->MakeCKKSPackedPlaintext(x1);
^~~~~~~~~~~~~~~~~~~
…/src/pke/include/cryptocontext.h:1042:15: note: candidate function not viable: no known conversion from ‘vector<vector>’ to ‘const vector<std::complex>’ for 1st argument
Plaintext MakeCKKSPackedPlaintext(const std::vector<std::complex>& value, size_t scaleDeg = 1,
^
…/src/pke/include/cryptocontext.h:1061:15: note: candidate function not viable: no known conversion from ‘vector<vector>’ to ‘const vector’ for 1st argument
Plaintext MakeCKKSPackedPlaintext(const std::vector& value, size_t scaleDeg = 1, uint32_t level = 0,
^
1 error generated.

4

The argument of the function is a vector of double, not a vector of a vector of double :-P, so you just have to reshape from 32x32 to 1024

5

Yes, I know. What I am asking is more related to the reason. The choice of the resize 2d->1d method will affect the #rotations. Some of the papers I am reading mentions sophisticated packing methods (e.g. tile packing) and I am trying to understand what OpenFHE allows.

6

Oh! I am not aware of any “custom” packing method in OpenFHE.

You might be interested in this implementation, which packs the image simply row-wise. The weighs of the filters, though, have a more complex packing that is done as part of pre-processing (so, in the end, they are encoded using the native openfhe functions).

7

I have been looking at your paper for a couple of days and was trying to figure out the structure. I could not understand how to put the convolution result into a plaintext slot.
Suppose our image is an encryption of the following input (assume row-packing):
1 3 1 4
1 2 1 1
1 5 3 2
1 2 1 3
and the filter is
1 1
1 1
After the convolution (suppose no padding) the result will be (11 + 13 + 11 + 12 = 7). This arises 2 questions:

  • What is the next step to put the convolution output to the first pixel? i.e. the final output should be
    7 7 7
    9 11 7
    9 11 9
    How to get the ciphertext encrypting this vector?

  • If we do row-packing, we need to divide the 1st convolution into 2 parts, am i right? The first 2 values 1 and 3 are in the ciphertext for the first row: 1 3 1 4. The second row values 1 and 2 are encrypted in the ciphertext for the second row: 1 2 1 1. How do we deal with this?

8

According to the convolution algorithm, that would work like this:

c = [1, 3, 1, 4, 1, 2, 1, 1, 1, 5, 3, 2, 1, 2, 1, 3]

we generate w^2-1 (w is the width of the kernel) rotations:

c_1 = [1, 3, 1, 4, 1, 2, 1, 1, 1, 5, 3, 2, 1, 2, 1, 3]
c_2 = [3, 1, 4, 1, 2, 1, 1, 1, 5, 3, 2, 1, 2, 1, 3, 1]
c_3 = [1, 2, 1, 1, 1, 5, 3, 2, 1, 2, 1, 3, 1, 3, 1, 4]
c_4 = [2, 1, 1, 1, 5, 3, 2, 1, 2, 1, 3, 1, 3, 1, 4, 1]

Notice that each “column” contains the pixels required by each convolution. For example, the first one contains 1, 3, 1, 2, that are the pixels needed for the first filter application. Then, moving to the right, we find 3 1 2 1, and so on.

The issue with your setting is that the kernel is not odd-sized, so this will lead to useless slots, for instance the fourth and the fifth. Ignoring this issue (perhaps the algorithm in the paper could be generalized for any kernel, but idk), we simply multiply each ciphertext with the corresponding kernel value, and we sum all the results up.

The main idea of this convolution is that the output is again in row-major order and can be used again, with no extra rotations or processing (indeed the idea was to minimize the number of automorphism keys). For example, the first column gives rise to the “new” first pixel, and it will be in the first slot.

I hope this clarifies a bit, but feel free to ask

1 Like
9

Thank you so much for this detailed (and very cool) answer!

1 Like