fastest prime number check formulae available

Question

which is the fastest prime number check formulae available and that can be used using a scientific calculator easily. you can also refer to a patent if available. looking to use it in a software.

Answer 1

No formula (for the usual sense of that on a calculator) will do. It's needed an algorithm and a programmable calculator. I'll assume we limit integers to what the calculator can natively handle, and it has no built-in modular exponentiation or prime-testing.

We want our prime-testing algorithm to be deterministic in both possible senses for that: flawlessly tell if any given integer $n$ in some supported input range is prime or not; and internally always perform the same operations for a given input $n$ to test. We want it to be fast (if not the fastest, which would depend on many things including benchmarking conditions), and within this as simple as possible.

I propose a preliminary test handling integers with small factors, followed by strong pseudoprime test(s) with witnesses $a$ the primes up to some limit determined according to $n$ per the sequence A014233. It works for integer $n$ as large as the limit where $0≤u≤v<n$ allows $u\cdot v\bmod n$ to be computed exactly by the calculator. With the native multiplication followed by keeping the remainder of Euclidean division, that would be good for $n$ like 5 to 7 decimal digits. And at the expense of a lot of complexity that we won't detail, and loosing some speed, that can be extended to the limit $n-1$ is distinguishable from $n$, like 10 to 14 decimal digits.

Given integer $n$ with $0\le n<n_\max$ [see above for value of $n_\max$]:

1. if $n≤30$, then $n$ is prime if and only if $\lfloor545925292/2^n\rfloor$ is odd, done.

2. set $u:=n$ and $v:=6469693230$
   while $v\ne1$:

   • if $u=0$, then $n$ is not prime, done.
   • set $u:=u\bmod v$
   • exchange $u$ and $v$

3. set $d:=(n-1)/2$ and $s:=1$
   while $d$ is even:

   • set $d:=d/2$ and $s:=s+1$

   [Note: from then on it holds $n=d\cdot2^s+1\,$]

4. for $i$ incrementing from $1$

   • set $a$ to $i^\text{th}$ prime $2,3,5,7,11,13⋯\,$, see A000040
   • set $b:=a$ and $c:=a$ and $e:=d$
   • while $e\ne 1:$
     • set $e:=e-1$
     • while $e$ is even
       • set $c:=c\cdot c\bmod n$ and $e:=e/2$
     • set $b:=b\cdot c\bmod n$
   • [Note: here it holds $b=a^d\bmod n\,$]
     if $b\ne 1$
     • set $j:=0$
     • while $j<s$ and $b\ne n-1$: [Invariant: $b=a^{d\cdot2^j}\bmod n\,$]
       • set $b:=b\cdot b\bmod n$ and $j:=j+1$
     • if $b\ne n-1$ then $n$ is not prime, done.
   • if $n<m$ where $m$ is the $i^\text{th}$ term in the sequence A014233, starting in $2047,1373653,25326001,3215031751,2152302898747,3474749660383⋯$, (including if that $m$ exceeds $n_\max$), then $n$ is prime, done.

---

Rationale:

• Step 1 handles inputs up to $n=30$ using the constant $545925292=\sum 2^{p_i}$ for $p_i$ the primes up to $30$.
• Step 2 improves the average speed for random input $n$, by simultaneously testing divisibility of $n$ by the primes up to $30$. This is by using the Euclidean algorithm to compute the Greated Common Divisor of $n$ and $6469693230=\prod p_i$ for $p_i$ the primes up to $30$. This GCD is $1$ if $n$ is a prime (larger than $30$, but the test is not performed for other $n$ thanks to step 1).
• Step 3 finds the uniquely defined $d$ and $s$ such that $n=d\,2^s+1$ and $d$ is odd. Theses quantities are used at each outer iteration in step 4.
• Step 4 executes a number of strong pseudoprime tests for witnesses $a$ the $i^\text{th}$ prime for incremental $i$.
• Each such test is based on the fact that for every odd prime $n$ and every $x\in[2,n-1)$, it holds $x^{n-1}\bmod n=1$ and $x^2\bmod n\ne1$. Therefore, for every prime $n=d\cdot2^s+1$ and every $a\in[2,n-1)$, it holds $a^d\bmod n=1$ or $a^{d\cdot2^j}\bmod n=p-1$ for some $j\in[0,s)$. If we find an $a$ for which that does not hold, then $n$ is not prime.

To handle any 64-bit input $n$, it's enough to test for $a$ the first $12$ primes, the last being $37$.

In cryptographic contexts, we often manipulate much larger $n$, and we don't know enough terms of sequence A014233, or even know a lower bound for that. We tend to use the Miller-Rabin probable prime test, which uses random $a$ instead of the first primes. It can fail to identify that an input $n$ is not prime, but that's exceedingly rare for large random prime $n$ (see this), and even in the worst case of $n$ deliberately made a Carmichael number, the probability of misclassifying it as prime is less than $2^{-2i}$ after $i$ tests.