SHA-3 Indexing Clarification for Theta() C[x,z]

Question

SHA-3: http://csrc.nist.gov/publications/drafts/fips-202/fips_202_draft.pdf

I’m working on a Python implementation of SHA-3(256) and using the intermediate hash value found at http://csrc.nist.gov/groups/ST/toolkit/documents/Examples/SHA3-256_Msg30.pdf for the example 30-bit hash message. I’m having trouble with the indexing of the state array.

In the 30-bit SHA-3(256) example message, see “Xor'd state (as lanes of integers)”, before Theta() of Round #0.

[0, 0] = 00000001997b5853
[1, 0] = 0000000000000000
[2, 0] = 0000000000000000
[3, 0] = 0000000000000000
[4, 0] = 0000000000000000
[0, 1] = 0000000000000000
[1, 1] = 0000000000000000
[2, 1] = 0000000000000000
[3, 1] = 0000000000000000
[4, 1] = 0000000000000000
[0, 2] = 0000000000000000
[1, 2] = 0000000000000000
[2, 2] = 0000000000000000
[3, 2] = 0000000000000000
[4, 2] = 0000000000000000
[0, 3] = 0000000000000000
[1, 3] = 8000000000000000
[2, 3] = 0000000000000000
[3, 3] = 0000000000000000
[4, 3] = 0000000000000000
[0, 4] = 0000000000000000
[1, 4] = 0000000000000000
[2, 4] = 0000000000000000
[3, 4] = 0000000000000000
[4, 4] = 0000000000000000

I'm assuming that the lanes are labeled [x,y], and are with respect to the convention outline in SHA-3, 3.1.4 Where [0,0]=00000001997b5853 would represent the middle lane of the state array.

I can produce those 25 64-bit lanes. My understanding is that, if you concatenate those lanes top to bottom, that is to say in the order with which they appear in the example document, that you will have produced the total bit string representing the state array, that is to then be passed to Theta()? string_representing_state_array='0000000000000000000000000000000110011001011110110101100001010011000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'

Now the from 3.1.2, A[x,y,z]=S[w(5y+x)+z]

And w is constant 64.

C[x,z]=S[w(5(0)+x)+z] ? S[w(5(1)+x)+z] ? S[w(5(2)+x)+z] ? S[w(5(3)+x)+z] ? S[w(5(4)+x)+z].

Function exclusive or:

def xo(bit_string_1,bit_string_2):
    xor_list=[]
    for i in range(len(bit_string_1)):
        if bit_string_1[i]=='0' and bit_string_2[i]=='0':
            xor_list.append('0')
        if bit_string_1[i]=='1' and bit_string_2[i]=='1':
            xor_list.append('0')
        if bit_string_1[i]=='0' and bit_string_2[i]=='1':
            xor_list.append('1')
        if bit_string_1[i]=='1' and bit_string_2[i]=='0':
            xor_list.append('1')
    return(l_s(xor_list))

def L_P(SET,n):
    #A function to break SET into n length chunks
    to_return=[]
    j=0
    k=n
    while k<len(SET)+1:
        to_return.append(SET[j:k])
        j=k
        k+=n 
    return(to_return)

def beta_theta(s):
    c_xz=[]
    for x in range(5):
        for z in range(64):
            c_xz.append(xo(xo(xo(xo(s[(64*((5*4)+x))+z],s[(64*((5*3)+x))+z]),s[(64*((5*2)+x))+z]),s[(64*((5*1)+x))+z]),s[(64*((5*0)+x))+z]))
    c_xz=L_P(c_xz,64)
    return(c_xz)

Where s=string_representing_state_array.

My question now is if the function beta_theta(s) returns, c_xz, with the correct indexing for step 2 in theta()? I'm taking a 320 bit c_xz and splitting it into 5 64-bit chunks, but I don't feel like this would return the correct index to produce d_xz.

PS. SHA-3 <<<<<<< Occam's Razor.

Answer 1

I spent quite a bunch of time to read your code and I do think that this question is in the end a bit offtopic here. So please find as follow all my remarks and code review (even though I should not...)

First, even though the way you implement it is unconventional, you are not wrong with your indexes etc... for the $\theta$ operation (it gets trickier for $\pi$ and $\rho$).

The $\theta$ step need to use two auxiliary arrays : one for the parity (your c_xz) and one to calculate the theta effect. Be sure to do that so you don't read modified values!!.

An overview of $\theta$ is the following (and for now you are in the right):

Here is my review of your code and some advises to improve, continue...

First you write this:

def xo(bit_string_1,bit_string_2):
    xor_list=[]
    for i in range(len(bit_string_1)):
        if bit_string_1[i]=='0' and bit_string_2[i]=='0':
            xor_list.append('0')
        if bit_string_1[i]=='1' and bit_string_2[i]=='1':
            xor_list.append('0')
        if bit_string_1[i]=='0' and bit_string_2[i]=='1':
            xor_list.append('1')
        if bit_string_1[i]=='1' and bit_string_2[i]=='0':
            xor_list.append('1')
    return xor_list

It is hard to understand what you are doing, you should try to separate into multiple sub functions. See as follow:

# implementation of the xor between two chars (easier to read and uses the XOR operator)
def xor_string(a,b):
    return '1' if ((a == '1') ^ (b == '1')) else '0';

# the same function as yours but a bit more readable
def xor_list_string(a,b):
    xor_list = []
    for i in range(len(a)):
        xor_list.append(xor_string(a[i],b[i]))
    return xor_list

# test : should result in ['0','1','1','0']
lst1 = ['0','0','1','1']
lst2 = ['0','1','0','1']

print xo(lst1,lst2)
print xor_list_string(lst1,lst2)

On the next function you define:

def L_P(SET,n):
    #A function to break SET into n length chunks
    to_return=[]
    j=0
    k=n
    while k<len(SET)+1:
        to_return.append(SET[j:k])
        j=k
        k+=n
    return(to_return)

The use of k is superfluous, see as follow:

def L_P2(SET,n):
    to_return = [] ; j = 0
    while j + n <= len(SET):
        to_return.append(SET[j:j+n])
        j += n
    return to_return

And then your definition of beta_theta is rather obscure (what is going on in that chain of xo. You don't need to inline the expression. Also because you are accessing bits, I would advise you to define an access function:

def pos(x,y,z,w):
    return (w*((5*y)+x))+z

And thus you can compute the Parity plane or c_xz:

# get the parity plane
def theta_plane(s,w):
    c_xz = []  #this will be the parity plane
    for x in range(5):
        for z in range(w):
            aux1 = xor_string(s[pos(x,0,z,w)],s[pos(x,1,z,w)])
            aux2 = xor_string(s[pos(x,2,z,w)],s[pos(x,3,z,w)])
            aux1 = xor_string(s[pos(x,4,z,w)],aux1)
            aux1 = xor_string(aux1,aux2)
            c_xz.append(aux1)
    c_xz = L_P(c_xz,w)
    return(c_xz)

# or even better:
def theta_plane_b(s,w):
    c_xz = []  #this will be the parity plane
    for x in range(5):
        for z in range(w):
            aux = '0'
            for y in range(5): #this shows clearly that you are xoring over y.
                aux = xor_string(aux,b(s,x,y,z,w))

            c_xz.append(aux)

    c_xz = L_P(c_xz,w)
    return(c_xz)

Afterward you can compute your theta effect:

def theta_effect(c_xz,w):
    d_xz = []
    for x in range(5):
        aux = []
        for z in range(w):
            aux.append(xor_string(c_xz[(x-1)%5][z],c_xz[(x+1)%5][(z-1)%w]))
            #This needed a mod 5
        d_xz.append(aux)
    return d_xz

You can then XOR back the theta effect to the state:

def xor_back_to_state(s,d_xz,w):
    for x in range(5):
        for z in range(w):
            for y in xrange(5):
                s[pos(x,y,z,w)] = xor_string(s[pos(x,y,z,w)],d_xz[x][z])

    return s

def theta(state,w):
    c_xz = []
    d_xz = []
    s_xyz = []
    c_xz = beta_theta(state,w)
    d_xz = theta_effect(c_xz,w)
    s_xyz = xor_back_to_state(d_xz,w)
    return s_xyz

I hope this answer helped you in your coding practices and view on Keccak.