Thomas Pornin's post was useful in helping me to get started with finding the inverse of a polynomial with coefficients in GF(2^8). I thought it may be helpful to show a complete version of his worked example, using hexadecimal instead of decimal representation for the coefficients.
I am also tremendously grateful to @corpsfini in helping me to find the inverse using the Extended Euclidean algorithm here.
Let's say you have a polynomial a(x) with coefficients that are elements of GF(2^8).
a(x) = {03}x^3 + {01}x^2 + {01}x + {02}
In AES, these coefficients are elements of the finite field GF(2^8). I am representing these in hexadecimal, but if you convert them to binary each bit is a coefficient in GF(2) of another polynomial. So you could think of a(x) as a "polynomial of polynomials".
{03} = {00000011} = x + 1
{01} = {00000001} = 1
{01} = {00000001} = 1
{01} = {00000010} = x
We also have a polynomial modulus of x^4 + 1. You could write this polynomial as having the following coefficients:
p(x) = {01}x^4 + {01}
We can find the inverse of {03}x^3 + {01}x^2 + {01}x + {02} mod {01}x^4 + {01} using the Extended Euclidean Algorithm.
Euclidean Algorithm
First of all, we use the division algorithm to find the gcd(a(x), p(x)).
In these steps I will be using long division for polynomials. The coefficients for these polynomials are elements of GF(2^8), so we are not using typical integer arithmetic but Polynomial Arithmetic,
Step 0:
{f6}x + {52}
--------------------------------------------
{03}x^3 + {01}x^2 + {01}x + {02} | {01}x^4 + {00}x^3 + {00}x^2 + {00}x + {01}
{01}x^4 + {f6}x^3 + {f6}x^2 + {f7}x
------------------------------------------
{f6}x^3 + {f6}x^2 + {f7}x + {01}
{f6}x^3 + {52}x^2 + {52}x + {a4}
--------------------------------
{a4}x^2 + {a5}x + {a5}
For example, to find how many times {03} "goes in to" {01}, you need to find the multiplicative inverse of x + 1 mod x^8 + x^4 + x^3 + x + 1, which works out to be {f6}. As you may notice, you also use the Extended Euclidean algorithm for finding multiplicative inverses of polynomials, although it will be somewhat simpler than this current example as the coefficients for these polynomials are in GF(2).
Subtraction of the two polynomials is the same as adding each of the coefficients (addition in GF(2)). This is the same as XORing each of the coefficients.
On to the next step of the Euclidean algorithm.
Step 1:
{8a}x + {4f}
----------------------------------
{a4}x^2 + {a5}x + {a5} | {03}x^3 + {01}x^2 + {01}x + {02}
{03}x^3 + {89}x^2 + {89}x
--------------------------------
{88}x^2 + {88}x + {02}
{88}x^2 + {c7}x + {c7}
----------------------
{4f}x + {c5}
Similarly, to find how many times {a4} "goes in to" {03}, you start by finding the inverse of {a4}, which is {8f}. If we multiply {8f} by {03} we get {8a}, therefore {8a} * {a4} = {03}.
From here I'll continue with the rest of the division algorithm until we end up with a remainder of {00}.
Step 2:
{f3}x + {ca}
------------------------
{4f}x + {c5} | {a4}x^2 + {a5}x + {a5}
{a4}x^2 + {bf}x
----------------------
{1a}x + {a5}
{1a}x + {3f}
------------
{9a}
Step 3:
{a8}x + {9a}
--------------
{9a} | {4f}x + {c5}
{4f}x
------------
{c5}
{c5}
----
{00}
So now we know that gcd(a(x), p(x)) = {9a}, which was the last non-zero remainder after repeatedly using the division algorithm for these two polynomials.
You might think that {03}x^3 + {01}x^2 + {01}x + {02} does not have an inverse mod {01}x^4 + {01} as the gcd worked out to be {9a} (and not {01}). However, this {9a} can be converted to {01} by multiplying it by its inverse, which we will see in a moment.
Extended Euclidean Algorithm
From the Euclidean algorithm above we found that:
gcd(a(x), p(x)) = {9a}
Now that we know the gcd, the extended Euclidean algorithm allows us to find the polynomials s(x) and t(x) that satisfy:
s(x)a(x) + t(x)p(x) = {9a}
If we write this equation as a congruence mod p(x) we have:
s(x)a(x) = {9a} mod p(x)
The auxiliary calculation to find s(x) is as follows:
si = si-2 - (si-1 * qi-2)
s0 = {00}
s1 = {01}
s2 = {00} - ({01})*({f6}x + {52})
= {00} - {f6}x - {52}
= {f6}x + {52}
s3 = {01} - ({f6}x + {52})*({8a}x + {4f})
= {01} - ({8f}x^2 + {cc}x + {8c}x + {44})
= {8f}x^2 + {40}x + {45}
s4 = ({f6}x + {52}) - ({8f}x^2 + {40}x + {45})*({f3}x + {ca})
= ({f6}x + {52}) - ({09}x^3 + {ea}x^2 + {92}x^2 + {50}x + {80}x + {9f})
= {09}x^3 + {78}x^2 + {26}x + {cd}
Therefore we can now say that:
({09}x^3 + {78}x^2 + {26}x + {cd}) * a(x) = {9a} mod p(x)
However, multiplying a(x) with this polynomial results in {9a}, so it's not quite yet the multiplicative inverse.
To get {01} on the right-hand side of the equation, we can multiply both sides by the inverse of {9a}, which is {9f}:
{9f} * ({09}x^3 + {78}x^2 + {26}x + {cd}) * a(x) = {9a} * {9f} mod p(x)
Which results in:
({0b}x^3 + {0d}x^2 + {09}x + {0e}) * a(x) = {01} mod p(x)
So now we have found that {0b}x^3 + {0d}x^2 + {09}x + {0e} is the inverse of {03}x^3 + {01}x^2 + {01}x + {02} mod {01}x^4 + {01}.