I think approaching the "field with one element" from the perspective of the field axioms makes it seem a bit silly, when it really is (in my opinion) deep mathematics, so lets use the definition of a field as a ring all of whose (finitely generated) modules are free.
One can reconstruct a commutative ring $R$ from its category of $R$ modules, so this isn't losing anything, and in this sense we can look for categories that behave just like $R-Mod$ for $R$ a field. These categories are fantastic in many senses, but one important fact is that every object decomposes into a direct sum of simple objects, and there is a unique simple, up to isomorphism.
Then the observation is that the category $Set$ satisfies a lot of these desirata, in particular this last point. The category of sets carries a lot of the categorical structure of $Vect_k$, such as an internal $\otimes$ functor, and internal hom functor, satisfying the tensor Hom adjunction (these are $\times$ and functions from X to Y, respectively), but also carries more of the vector space specific functors, such as exterior and symmetric powers, mapping a set to its set of sub/multisets of size $m$. So in this sense, the category of sets strongly resembles the category of $k$ modules, aside from the slight issue that its not additive.
Hopefully this has convinced you that someone could draw parallels, but one would be right to be skeptical at this point if I asked you to believe that this is deep mathematics. So now I'll try give some examples of nontrivial parallels, that don't require much background.
The automorphism group of the sum of $n$ copies of our simple object admits a non-obvious map to an abelian group (the determinant/sign map, respectively), and the kernel is in general, a simple group ($A_n$ and $SL_n(k)$).
We can turn the set of $m$ dimensional subspaces of an $n$ dimensional vector space into a projective variety $Gr(m,n)$, and the number of points of $Gr(m,n)$ over the field with $q$ elements is a polynomial $P(q)$ in $q$. When we set $q=1$, we recover the number of subsets of an $n$ element set of size $m$. Furthermore, if we have a cyclic group of order $n$ acting transitively on our $n$ element set, it acts on the set of $m$ element subsets, and the value of that same point counting polynomial at a primitive $n$th root of unity, $P(\zeta_n)$ yields the number of $m$ element subsets that are invariant under this cyclic group action. This is the cyclic sieving phenomena, and has shown up in a variety of counting problems.
Finally, there is a machine (algebraic K-theory) that one can apply to the system of compatible groups $GL_n(R)$ for any ring $R$, and applying this to $S_n"="GL_n(\mathbb{F}_1)$ yields the stable homotopy groups of spheres, which are incredibly rich and hard to understand objects. This can be stated as the algebraic K groups of $\mathbb{F}_1$ "are" the stable homotopy groups of spheres.
These are just a few examples I know of, there are many more parallels, but hopefully this is enough to convince you that there is deep mathematics at play here. The degeneration of field-theoretic behaviour is why these analogies are termed the "field with one element", and it seems like our restrictive notion of a field (as defined by its axioms) is not sufficient to capture the full spectrum of "field theoretic behaviour".