Approximation of RELU using EvalChebyshevFunction in OpenFHE

Library Questions
1

I want to implement a RELU approximation function using a vector with data within the range [-6.33978, 3.21301]

My function is

Ctext relu(const Ctext vec, double scale,  int vectorSize) {
double lowerBound = -1;
double upperBound = 1;
uint32_t polyDegree = 60;

auto masked_scale = generate_scale_mask(10, vectorSize); // function generates a mask of 0.1 of size vectorSize
auto mask_data = context->MakeCKKSPackedPlaintext(masked_scale, 1, 1);
auto innVec = context->EvalMult(vec, mask_data);
Ctext relu_result = context->EvalChebyshevFunction(
    [](double x) -> double { if (x < 0.0) return 0.0; else return x; }, 
                                        innVec,
                                        lowerBound,
                                        upperBound, 
                                        polyDegree);
return relu_result;

}

When I run this function, returned cipher does not follow the rule I have set in double { if (x < 0.0) return 0.0; else return x; } as values less than 0 are still returned in my relu_result. Some of them are turned to positive, some remain negative values while a few actually approximate to 0.

I am not sure whether I understand how EvalChebyshevFunction works in OpenFHE. I will be glad if I can get any help in debugging this function and what I am doing wrong in my implementation.

2

First, try to follow this guide.

You will get better results if you set the variables lowerBound and upperBound to the range of your data {-6.33978, 3.21301}. Try higher degrees for a more accurate approximation.

Note that approximating non-smooth functions in FHE is not straightforward. There is still ongoing research to implement that efficiently and accurately.

3

Thank you for your comment. I don’t want to make the approximation function static to data as if it is a relu approximation function for a classification problem using CNNs. I played around with the value polyDegree and left it at 60 is that still small??
My vec is of size vecorSize which is 576 elements is that too large?

4

Choosing a specific range when using polynomials to approximate functions is essential. This is because the approximation of f(x) will be most accurate within the defined interval for x. You may consult any existing reference on function approximation and you will realize this. To do this:

  1. Examine your input/intermediate data carefully.
  2. Determine a reasonable range where most of your data falls in.
  3. If needed, you can reduce the size of your numbers by scaling them down by a known scalar to keep them within this range.

If you use the polynomial approximation outside the defined range, the approximation results can be unpredictable and most likely will cause overflow/underflow.

5

Hello, Caesar.

I’d like to ask what the principle of the EvalChebyshevFunction in OpenFHE is. Can I see the source code?

6

See the derivation of Eq. (6) in https://arxiv.org/pdf/1810.04282 (Sections 2.1 and 2.2) for an introduction to the Chebyshev interpolation. OpenFHE implements this using CKKS. A more concise specification is provided in Chebyshev Approximations — GSL 2.8 documentation

1 Like