[Bug report] 128-bit CKKS plaintext encoding right-shifts by out-of-range counts for tiny inputs and misencodes results

Library Questions
1

Hi OpenFHE team,

I would like to report a reproducible 128-bit CKKS plaintext-encoding bug for extremely small finite inputs.

Summary

In the NATIVEINT == 128 CKKS packed-encoding path, the real branch computes:

int64_t re64       = std::llround(dre);
int32_t pRemaining = pCurrent + n1;

if (pRemaining < 0) {
    re = re64 >> (-pRemaining);
}

So for sufficiently small inputs, the negative branch right-shifts a signed 64-bit intermediate without bounding the shift count.

I reproduced this through the public API:

cc->MakeCKKSPackedPlaintext(values);

on a 128-bit CKKS build.

For an input vector whose entries are all -2^-103, the same preprocessing used by CKKSPackedEncoding::Encode() yields min_real_pRemaining = -64, UBSan reports:

runtime error: shift exponent 64 is too large for 64-bit type 'int64_t'

and the first decoded value comes back as approximately -1.8189894e-12 instead of a tiny or zero result.

As a control:

  • -2^-102 gives min_real_pRemaining = -63, does not trigger UBSan, and decodes near zero
  • -denorm_min drives the same branch much further (min_real_pRemaining = -1035) and also produces a spurious finite decrypted-and-decoded artifact

This same unchecked negative right-shift pattern is also present in the 128-bit FHECKKSRNS::MakeAuxPlaintext() path. I have not dynamically exercised that second path in this report.

Environment

  • OpenFHE local tested revision: ed361af22049007db2107e7c69bcff209e8c420d
  • Source links below use that exact tested revision
  • Configuration: NATIVE_SIZE=128, MATHBACKEND=4
  • OS: Linux x86_64
  • Compiler: Clang 14.0.0
  • Sanitizers: AddressSanitizer, UndefinedBehaviorSanitizer

Representative OpenFHE build:

cmake -S openfhe-development -B build-openfhe-128-asan \
  -DNATIVE_SIZE=128 \
  -DMATHBACKEND=4 \
  -DCMAKE_C_COMPILER=clang-14 \
  -DCMAKE_CXX_COMPILER=clang++-14 \
  -DCMAKE_BUILD_TYPE=Debug \
  -DBUILD_UNITTESTS=OFF \
  -DBUILD_EXAMPLES=OFF \
  -DBUILD_BENCHMARKS=OFF \
  -DWITH_OPENMP=OFF \
  -DCMAKE_CXX_FLAGS='-fsanitize=address,undefined -fno-omit-frame-pointer -g -O0' \
  -DCMAKE_C_FLAGS='-fsanitize=address,undefined -fno-omit-frame-pointer -g -O0'
cmake --build build-openfhe-128-asan -j

Minimal reproduction

The reproducer below mirrors the encoder preprocessing once to print the effective minimum real-branch pRemaining, and then uses the normal public CKKS API.

#include "math/dftransform.h"
#include "openfhe.h"

#include <algorithm>
#include <cmath>
#include <complex>
#include <cstdint>
#include <iomanip>
#include <iostream>
#include <limits>
#include <string>
#include <vector>

using namespace lbcrypto;

namespace {

int32_t MinRealPRemaining(const CryptoContext<DCRTPoly>& cc,
                          const std::vector<std::complex<double>>& values,
                          uint64_t pBits, size_t slots) {
    auto inverse = values;
    inverse.resize(slots);
    DiscreteFourierTransform::FFTSpecialInv(inverse, cc->GetRingDimension() * 2);

    const int32_t pCurrent = static_cast<int32_t>(pBits - 52);
    int32_t minValue       = std::numeric_limits<int32_t>::max();

    for (const auto& coeff : inverse) {
        int n1 = 0;
        (void)std::frexp(coeff.real(), &n1);
        minValue = std::min(minValue, pCurrent + n1);
    }

    return minValue;
}

std::vector<std::complex<double>> DecryptVec(CryptoContext<DCRTPoly> cc, const PrivateKey<DCRTPoly>& sk,
                                             const Ciphertext<DCRTPoly>& ct, size_t slots) {
    Plaintext pt;
    cc->Decrypt(sk, ct, &pt);
    pt->SetLength(slots);
    return pt->GetCKKSPackedValue();
}

void RunCase(CryptoContext<DCRTPoly> cc, const PublicKey<DCRTPoly>& pk, const PrivateKey<DCRTPoly>& sk,
             const std::string& name, const std::vector<std::complex<double>>& values) {
    constexpr size_t slots = 8;
    std::cout << "case=" << name << '\n';
    std::cout << "min_real_pRemaining=" << MinRealPRemaining(cc, values, 90, slots) << '\n';
    auto pt  = cc->MakeCKKSPackedPlaintext(values);
    auto ct  = cc->Encrypt(pk, pt);
    auto dec = DecryptVec(cc, sk, ct, values.size());
    std::cout << std::setprecision(17);
    std::cout << "decoded0_real=" << dec[0].real() << '\n';
    std::cout << "decoded0_imag=" << dec[0].imag() << '\n';
}

}  // namespace

int main() {
    std::cout << std::unitbuf;

    CCParams<CryptoContextCKKSRNS> parameters;
    parameters.SetMultiplicativeDepth(1);
    parameters.SetScalingModSize(90);
    parameters.SetFirstModSize(90);
    parameters.SetBatchSize(8);
    parameters.SetCKKSDataType(COMPLEX);

    auto cc = GenCryptoContext(parameters);
    cc->Enable(PKE);

    auto kp = cc->KeyGen();

    RunCase(cc, kp.publicKey, kp.secretKey, "neg_pow2_102",
            std::vector<std::complex<double>>(8, {-std::ldexp(1.0, -102), 0.0}));
    RunCase(cc, kp.publicKey, kp.secretKey, "neg_pow2_103",
            std::vector<std::complex<double>>(8, {-std::ldexp(1.0, -103), 0.0}));
    RunCase(cc, kp.publicKey, kp.secretKey, "neg_denorm_min",
            std::vector<std::complex<double>>(8, {-std::numeric_limits<double>::denorm_min(), 0.0}));
    return 0;
}

Representative probe build:

mkdir -p fuzz/bin

clang++-14 -std=gnu++17 -fno-omit-frame-pointer -g -O0 \
  -fsanitize=address,undefined \
  fuzz/openfhe_ckks_tiny_plaintext_probe.cpp \
  -Iopenfhe-development/src/pke/include \
  -Iopenfhe-development/src/core/include \
  -Iopenfhe-development/src/binfhe/include \
  -Iopenfhe-development/third-party/cereal/include \
  -Ibuild-openfhe-128-asan/src/core \
  -Ibuild-openfhe-128-asan/src/pke \
  build-openfhe-128-asan/lib/libOPENFHEpke.so.1.5.1 \
  build-openfhe-128-asan/lib/libOPENFHEbinfhe.so.1.5.1 \
  build-openfhe-128-asan/lib/libOPENFHEcore.so.1.5.1 \
  -lpthread -ldl -lm \
  -Wl,-rpath,build-openfhe-128-asan/lib \
  -o fuzz/bin/openfhe_ckks_tiny_plaintext_probe

Actual behavior

With:

ASAN_OPTIONS=detect_leaks=0 \
UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=0 \
./fuzz/bin/openfhe_ckks_tiny_plaintext_probe

the reproducer reports:

case=neg_pow2_102
min_real_pRemaining=-63
decoded0_real=1.7702900382008513e-24
decoded0_imag=-3.4353561946901903e-25

case=neg_pow2_103
min_real_pRemaining=-64
.../ckkspackedencoding.cpp:164:23: runtime error: shift exponent 64 is too large
for 64-bit type 'int64_t'
...
decoded0_real=-1.8189894035462794e-12
decoded0_imag=2.0226729868934959e-24

case=neg_denorm_min
min_real_pRemaining=-1035
decoded0_real=-8.8817841946443095e-16
decoded0_imag=-1.0209060619437281e-24

So once the real-branch shift count reaches 64 or more, public plaintext creation reaches invalid right-shift behavior and the decrypted-and-decoded result jumps from the adjacent control’s near-zero range to a much larger finite artifact.

The neg_denorm_min case reaches the same negative branch with an even larger derived shift count. UBSan reports the first out-of-range instance at the shared source location; the later case stays in the same regime and also decodes incorrectly.

Expected behavior

Tiny finite CKKS plaintext inputs should not cause invalid shift counts in the public encoding path.

If the input magnitude falls below what the chosen scaling path can represent, the encoder should either:

  • reject it explicitly; or
  • underflow it to zero deterministically.

It should not:

  • invoke undefined behavior through out-of-range right shifts; or
  • decode back to a much larger unrelated finite value.

Impact

This is a correctness bug in the 128-bit CKKS plaintext-encoding path, with confirmed undefined behavior.

The confirmed impact is:

  • UBSan-reported invalid right shifts in public plaintext creation; and
  • spurious decrypted-and-decoded values resulting from the malformed encoding of tiny real inputs.

The dynamically confirmed scope in this report is:

  • CryptoContextImpl::MakeCKKSPackedPlaintext(const std::vector<std::complex<double>>&, ...)

From source inspection, the same negative-branch right-shift pattern is also present in FHECKKSRNS::MakeAuxPlaintext(), so the auxiliary CKKS bootstrapping-plaintext path appears to be affected as well. I have not dynamically exercised that second path here.

Cause analysis

In the 128-bit real branch of CKKSPackedEncoding::Encode(), OpenFHE does:

int64_t re64       = std::llround(dre);
int32_t pRemaining = pCurrent + n1;
int128_t re        = 0;
if (pRemaining < 0) {
    re = re64 >> (-pRemaining);
}

The same unchecked negative right shift is also used for the int64_t im64 intermediate in both CKKSPackedEncoding::Encode() and FHECKKSRNS::MakeAuxPlaintext(), although this reproducer dynamically triggers and analyzes the real branch.

With scalingModSize = 90, the encoder uses:

pCurrent = 90 - 52 = 38

So:

  • -2^-102 gives pRemaining = -63
  • -2^-103 gives pRemaining = -64
  • -denorm_min gives pRemaining = -1035

The first case stays within the valid 64-bit shift-count range [0, 63].
The second reaches the type width, and the third exceeds it.

That matches the observed transition:

  • -63: no UBSan, near-zero decoded output
  • -64: UBSan at the real-branch right shift and a spurious decrypted-and-decoded artifact
  • -1035: the same negative-branch regime and another spurious artifact

The erroneous decoded magnitudes are also consistent with the effective masked shift counts observed on this build, although that behavior is not portable and is not guaranteed once C++ undefined behavior has been reached.

For the triggering real coefficient in the -2^-103 case:

frexp mantissa = -0.5
re64           = -0.5 * 2^52 = -2^51
pRemaining     = -64

The observed decoded value -1.8189894035462794e-12 is nearly exactly -2^-39, which matches:

(-2^51) * 2^-90 = -2^-39 = -1.8189894035458565e-12

For -denorm_min, an effective shift count of 1035 mod 64 = 11 would give:

(-2^51 >> 11) * 2^-90 = -2^40 * 2^-90 = -2^-50
                      = -8.881784197001252e-16

which closely matches the observed -8.8817841946443095e-16.

Relevant source locations

Suggested direction

To address the confirmed negative-pRemaining cases, the encoder should validate the shift count before performing the right shift. At minimum:

  • if -pRemaining is greater than or equal to the width of the intermediate type, the encoder should not perform the shift; and
  • the implementation should define whether that case means:
    • deterministic underflow to zero; or
    • explicit rejection as unsupported.

The same shift-count policy should then be applied consistently to:

  • the ordinary 128-bit CKKS packed-encoding path; and
  • the duplicated 128-bit logic in MakeAuxPlaintext()

Question

For tiny finite CKKS plaintext inputs that fall below the representable range of the current 128-bit scaling path, would you prefer OpenFHE to reject them explicitly, or to treat them as deterministic underflow-to-zero cases before any out-of-range right shift occurs?

Reported by Jiang Chao, Beijing University of Posts and Telecommunications