Overflow when encoding numpy.int64 variables

[AuthoStats]

When working with Numpy arrays, I ran into issues of Overflow errors for simple values. After testing, I found that the issue stemmed from the numpy.int64() datatype. Accessing elements in a numpy array in certain ways yields either a Python int datatype or numpy.int64 datatype. The latter value will cause Overflow errors during the Encoding step regardless of the value used.

The source of this is the change from:
int_rep = int(round(scalar * pow(BASE, -exponent)))
to
int_rep = round(fractions.Fraction(scalar)* fractions.Fraction(BASE) ** -exponent)

Although the latter produces the same numerical value, the data type remains a numpy.int64 data type, whereas the old code would convert it to a python integer, which avoids the eventual Overflow during int_rep % public_key.n

To reproduce:

import sys, math, fractions
import numpy as np
from phe import *
BASE = 16
LOG2_BASE = math.log(BASE, 2)
public_key, private_key = generate_paillier_keypair()
scalar = np.int64(0)
FLOAT_MANTISSA_BITS = sys.float_info.mant_dig
precision= 2**-12
prec_exponent = math.floor(math.log(precision, BASE))
exponent = prec_exponent
print(f"Exponent is {exponent} with type {type(exponent)}")
int_rep = round(fractions.Fraction(scalar)* fractions.Fraction(BASE) ** -exponent)
print(f"Integer Representation is {int_rep} with type {type(int_rep)}")
if abs(int_rep) > public_key.max_int:
    raise ValueError('Integer needs to be within +/- %d but got %d'
                        % (public_key.max_int, int_rep))
print(f"Boolean is {abs(int_rep) > public_key.max_int} with type {type(abs(int_rep) > public_key.max_int)}")
print(f"Message Space is {public_key.n} with type {type(public_key.n)}")
print(f"Max int is {public_key.max_int} with type {type(public_key.max_int)}")
print(f"Pre-mod integer is {int_rep} with type {type(int_rep)}")
print(f"Int representation is {int_rep % public_key.n} with type {type(int_rep % public_key.n)}")

Results in the output:

Exponent is -3 with type <class 'int'>
Integer Representation is 0 with type <class 'numpy.int64'>
Boolean is False with type <class 'numpy.bool'>
Message Space is 4093299914960611076154454533787578959581289998492125181114018178394152381143259675082188008209902629589277524113649765838989914674553123919650964754355090186061631401912263318624324659591997557708788672559568824602328027976345926757402999498477431785905261332426933383936527584684316875370279339658246077469650376555206669856341137276375169669166016379742473704839743919194618007230416266659944199011129118409143247399262217719388418100575681908486778404197758682239390515139663887743361308583944099922333000196944755144800444830909706571603073189184902547818334812327157739004621975723494689058176223158608699653041708414807064308834400223968525343654300847843980229932031740833833638237420339655168209879554389770295321696887140992207523682688296751642050754397764494146304963381200475898981702838357704214479336874785549974224778267706339243552119495008674855538122633053209545487395389320411741764062626452834900290053827 with type <class 'int'>
Max int is 1364433304986870358718151511262526319860429999497375060371339392798050793714419891694062669403300876529759174704549921946329971558184374639883654918118363395353877133970754439541441553197332519236262890853189608200776009325448642252467666499492477261968420444142311127978842528228105625123426446552748692489883458851735556618780379092125056556388672126580824568279914639731539335743472088886648066337043039469714415799754072573129472700191893969495592801399252894079796838379887962581120436194648033307444333398981585048266814943636568857201024396394967515939444937442385913001540658574498229686058741052869566551013902804935688102944800074656175114551433615947993409977343913611277879412473446551722736626518129923431773898962380330735841227562765583880683584799254831382101654460400158632993900946119234738159778958261849991408259422568779747850706498336224951846040877684403181829131796440137247254687542150944966763351274 with type <class 'int'>
Pre-mod integer is 0 with type <class 'numpy.int64'>

---

OverflowError Traceback (most recent call last)
Cell In[4], line 22
20 print(f"Max int is {public_key.max_int} with type {type(public_key.max_int)}")
21 print(f"Pre-mod integer is {int_rep} with type {type(int_rep)}")
---> 22 print(f"Int representation is {int_rep % public_key.n} with type {type(int_rep % public_key.n)}")

OverflowError: int too big to convert

Whereas:

import sys, math, fractions
import numpy as np
from phe import *
BASE = 16
LOG2_BASE = math.log(BASE, 2)
public_key, private_key = generate_paillier_keypair()
scalar = np.int64(0)
FLOAT_MANTISSA_BITS = sys.float_info.mant_dig
precision= 2**-12
prec_exponent = math.floor(math.log(precision, BASE))
exponent = prec_exponent
print(f"Exponent is {exponent} with type {type(exponent)}")
int_rep = int(round(scalar * pow(BASE, -exponent)))
print(f"Integer Representation is {int_rep} with type {type(int_rep)}")
if abs(int_rep) > public_key.max_int:
    raise ValueError('Integer needs to be within +/- %d but got %d'
                        % (public_key.max_int, int_rep))
print(f"Boolean is {abs(int_rep) > public_key.max_int} with type {type(abs(int_rep) > public_key.max_int)}")
print(f"Message Space is {public_key.n} with type {type(public_key.n)}")
print(f"Max int is {public_key.max_int} with type {type(public_key.max_int)}")
print(f"Pre-mod integer is {int_rep} with type {type(int_rep)}")
print(f"Int representation is {int_rep % public_key.n} with type {type(int_rep % public_key.n)}")

Results in the output:

Exponent is -3 with type <class 'int'>
Integer Representation is 0 with type <class 'int'>
Boolean is False with type <class 'bool'>
Message Space is 4632064407707281026303273495076274986239900031399402470241477394127551953831094191281219038334713985254889185304161653243503759185417918664119368195923259715525156625229037559004097318182716964686573978619780826614950377163034890125922697125576054790349892640400958036205926844105110567864591794189955654364071048561624817664074029218202602537898448018943343553782659064533206705898849506642470405071106707182419590486867363012636624439424863228881364444909045598227897365624132947926421182200790667982680508194894394565171205106951397193487804433878510272933308362102718591289553326209129351152085387466861256468847086147860954708068049795811665522470072358481269243089031014011336328919524507588640699013554558359868959494256742349725955028495834741794640003777873741370503372256258008908507825331012875642390732706557626968755786169246025239136979616812358792860642118776326122686655274154596895264317344200617176130890163 with type <class 'int'>
Max int is 1544021469235760342101091165025424995413300010466467490080492464709183984610364730427073012778237995084963061768053884414501253061805972888039789398641086571841718875076345853001365772727572321562191326206593608871650125721011630041974232375192018263449964213466986012068642281368370189288197264729985218121357016187208272554691343072734200845966149339647781184594219688177735568632949835547490135023702235727473196828955787670878874813141621076293788148303015199409299121874710982642140394066930222660893502731631464855057068368983799064495934811292836757644436120700906197096517775403043117050695129155620418822949028715953651569356016598603888507490024119493756414363010338003778776306508169196213566337851519453289653164752247449908651676165278247264880001259291247123501124085419336302835941777004291880796910902185875656251928723082008413045659872270786264286880706258775374228885091384865631754772448066872392043630053 with type <class 'int'>
Pre-mod integer is 0 with type <class 'int'>
Int representation is 0 with type <class 'int'>

[hardbyte]

Nice find!

Would you be willing to prepare a PR including tests?

[AuthoStats]

I'm not familiar with the PR/Test process.

[hardbyte]

You've actually done the hardest part already - working out what is wrong!

I'm happy to help if you'd like to give it a go - essentially you want to Fork this repo and add a failing test - which is just a python method (see https://github.com/data61/python-paillier/blob/master/phe/tests/paillier_test.py). Commit that. You could open a PR in GitHub with your changes at that point (with a fix or not) and either I or someone at Data61 will take a look.