Evaluation of ReLU Activation Function through Scheme Switching

Library Questions
1

I am conducting an experiment which is basically to see whether it is possible to Evaluate an activation function through Scheme Switching in OpenFHE.

I have gone through all the exercises on this topic and I am not sure whether this is possible. Theoretically, it sounds feasible but practically, I am not getting anything close to the correct results with my implementation. Performance is not an issue for me.
I have implemented the relu function as

schemeswitch_relu(Ctext encryptedVector) {
auto ccLWE = context->GetBinCCForSchemeSwitch();
auto pLWE = ccLWE->GetMaxPlaintextSpace().ConvertToInt();

auto fp = [](NativeInteger m, NativeInteger p1) -> NativeInteger {
    if (m < 0)
        return 0;
    else
        return m;
};
auto lut = ccLWE->GenerateLUTviaFunction(fp, pLWE);

auto cTempToFHEW = context->EvalCKKStoFHEW(encryptedVector);
std::vector<LWECiphertext> cFunc(cTempToFHEW.size());
for (uint32_t i = 0; i < cTempToFHEW.size(); i++) {
    cFunc[i] = ccLWE->EvalFunc(cTempToFHEW[i], lut);
}

auto cTempToCKKS = context->EvalFHEWtoCKKS(cFunc, num_slots, num_slots, pLWE, 0, pLWE);
return cTempToCKKS;

}

And I have set my parameters like

num_slots = 1 << num_slots;
int dcrtBits               = 55;
int firstMod               = 56;

auto secretKeyDist = SPARSE_TERNARY;
parameters.SetSecretKeyDist(SPARSE_TERNARY);
parameters.SetSecurityLevel(lbcrypto::HEStd_NotSet);
parameters.SetNumLargeDigits(3);
parameters.SetRingDim(1 << ringDim);
parameters.SetBatchSize(num_slots);
level_budget = {4, 4};
ScalingTechnique rescaleTech = FLEXIBLEAUTO; 
parameters.SetScalingModSize(dcrtBits);
parameters.SetFirstModSize(firstMod);
parameters.SetScalingTechnique(rescaleTech);

uint32_t levelsAvailableAfterBootstrap = mlevelBootstrap;
circuit_depth = levelsAvailableAfterBootstrap + FHECKKSRNS::GetBootstrapDepth(level_budget, secretKeyDist);
parameters.SetMultiplicativeDepth(circuit_depth);
context = GenCryptoContext(parameters);

context->Enable(PKE);
context->Enable(KEYSWITCH);
context->Enable(LEVELEDSHE);
context->Enable(ADVANCEDSHE);
context->Enable(FHE);
context->Enable(SCHEMESWITCH);

keyPair = context->KeyGen();
context->EvalMultKeyGen(keyPair.secretKey);
context->EvalSumKeyGen(keyPair.secretKey);

std::vector<uint32_t> bsgsDim     = {0, 0};
numSlots = context->GetRingDimension() / 2;
usint halfnumSlots = numSlots/2;

context->EvalBootstrapSetup(level_budget, bsgsDim, halfnumSlots);
context->EvalBootstrapKeyGen(keyPair.secretKey, halfnumSlots);


/*** Prepare the FHEW cryptocontext and keys for FHEW and scheme switching **/
BINFHE_PARAMSET slBin = TOY;
uint32_t logQ_ccLWE   = 25;
SchSwchParams params;
params.SetSecurityLevelCKKS(HEStd_NotSet);
params.SetSecurityLevelFHEW(slBin);
params.SetArbitraryFunctionEvaluation(true);
params.SetCtxtModSizeFHEWLargePrec(logQ_ccLWE);
params.SetNumSlotsCKKS(num_slots);
params.SetNumValues(num_slots);
auto privateKeyFHEW = context->EvalSchemeSwitchingSetup(params);
auto ccLWE          = context->GetBinCCForSchemeSwitch();

context->EvalSchemeSwitchingKeyGen(keyPair, privateKeyFHEW);
ccLWE->BTKeyGen(privateKeyFHEW);
// pLWE = ccLWE->GetMaxPlaintextSpace().ConvertToInt();
double scaleCF = 1.0 / pLWE;
context->EvalCKKStoFHEWPrecompute(scaleCF);

I am not very sure of where my I am going wrong with my configurations as the output of my relu is completely wrong (or my understanding of scheme switch capability is wrong :slight_smile: ).

2

The EvalFunc only works with plaintext modulus up to p = 8 (see the paragraph here about GenerateLUTviaFunction). I don’t see what inputs you use. Also, returning from FHEW to CKKS works when the message is significantly smaller than the plaintext modulus (see the above link, as well as the example, which for p = 8 reduces the range of the message by quite a lot.

To implement ReLU, a much better option is to use scheme switching to compute the sign of you ciphertext, for instance, with EvalCompareSchemeSwitching, which returns a vector of elements encrypting zero for positive values and 1 for negative values, and then multiply your ciphertext by (1-sign). See also How to calculate ReLU function through ciphertext conversion? - #2 by andreea.alexandru.

3

Thank you for the response It was very helpful. I implemented the function as shown below and it works.

secure_schemeswitch_relu(const Ctext encryptedVector, Ctext secondCiphertext) {
auto cResult = context->EvalCompareSchemeSwitching(encryptedVector, secondCiphertext, num_slots, num_slots);
auto signResults = context->EvalSub(1, cResult);
auto results = context->EvalMult(signResults, encryptedVector);
return results;
1 Like
5

Hi great to see you managed to evaluate the ReLU function using scheme switching CKKS and FHEW. Just for your information, you can actually evaluate the ReLU function without scheme switching as well. In CKKS, you can do this by approximating the sign function, check Minimax Approximation of Sign Function by Composite Polynomial for Homomorphic Comparison and Efficient Homomorphic Comparison Methods with Optimal Complexity.

In BFV / BGV, you can do this using the method in HEBridge: Connecting Arithmetic and Logic Operations in FV-style HE Schemes | Proceedings of the 12th Workshop on Encrypted Computing & Applied Homomorphic Cryptography. They also provide open source code. Some running examples have results like:

Input: 312 127 110 -276 166 -296 100 167 -46 173
Decrypted ReLU: [312] [127] [110] [0] [166] [0] [100] [167] [0] [173]
Expected Sign: 1 1 1 0 1 0 1 1 0 1
Decrypted Sign: [1] [1] [1] [0] [1] [0] [1] [1] [0] [1]
6

Thank you for the information.
This is just an experiment I am conducting and I want I am using both approaches.

The strange thing is that with the Scheme switch technique, the minimum value of my results ciphertext does not turn to 0.

For example, I have input data of size 2^14 with all slots holding data in the range Range [ -3.6966 , 2.89697 ]

My second input ciphertext is just 0s. When I go through the evaluation, my output range is actually Range [ -0.518389 , 2.89697 ] instead of Range [ 0.0 , 2.89697 ]
The approximation approach with polynomial degree of 50 gives me an output o Range [ -0.0234161 , 2.89772 ]

I was expecting that the scheme swithcing approach will yield better precision than approximation. I don’t yet know how to resolve this issue and any ideas will help.

@andreea.alexandru

7

Unfortunately, the conversion between FHEW and CKKS is noisy, meaning that instead of 0 and 1 you will get approximations of 0 and 1. I haven’t ran an example that’s so large, but I do expect you to get a deviation from 1 at least of magnitude 0.001. This error propagates when you do (1-sign)*x, and that’s why you see that range.

Have you checked what happens with the error when you decrease the batch size from 2^{14} to smaller? The error should decrease. So one option would be to perform ReLU in parallel on smaller batches (careful how you set the arguments in EvalCompareSchemeSwitching).

Given that running 2^{14} FHEW bootstraps is very slow, we haven’t studied and optimized the precision for this case in the main code. However, there is a branch I started on improving the FHEW to CKKS conversion but didn’t have time to incorporate. Another option from there that you can try is to apply a low-depth cleaning polynomial over the result of the comparison: -2x^3 + 3x^2. This will bring the values closer to 0 and 1. Note that you can apply this to improve the output of the polynomial approximation as well!

Regarding your comment on the comparison between scheme switching and the polynomial approximation. The range of the values you are looking at in this example is very small. Given such a small range, it’s no surprise that you can find a relatively small degree polynomial which gives you very good results. The problem with direct polynomial approximation arises when the range is large, therefore it would require a polynomial of an enormous degree to get reasonable results for a discontinuous function. In that case, scheme switching is a better option, as it supports an input interval of type [0, 2^{17}-1].

8

If the computation is fixed-point, you can encode the fixed-point numbers in the BFV/ BGV plaintexts and use BFV/ BGV to compute the ReLU function. BFV/ BGV are exact and avoid the approximation erros brought by CKKS polynomial approximation, as well as the approximated scheme switching.

However, if you target real number, using CKKS (with either polynomial approximation or scheme switching) is indeed more reasonable. Please refer to Alex’s reply regarding this.