Being a functor in argument $B$ means that we fix $A$, so the functor will be between objects of the form $X^A$ and $Y^A$. If we had said "functor in argument $A$" instead, then we'd fix $B$ and consider functors between objects $B^X$ and $B^Y$.
How do we define such functors (focusing on the first kind we mentioned)? Well if we have an arrow $f:X\to Y$ (in the category of Sets), we get an induced functor $X^A\to Y^A$ given as follows: an object $\varphi\in X^A$ (i.e. a morphism $\varphi:A\to X$) gets mapped to $f\circ\varphi:A\to Y$, which we sometimes call $\varphi_*$.
Edit: okay, so let's reevaluate what is going on. It seems we are talking about a functor $F:\bf{Set}\to\bf{Set}$ (first, fixing $A$). How will we define $F(X)$? We will define $F(X)=X^A$. What is the action of $F$ on arrows? It takes an arrow $f:X\to Y$ to the arrow $F(f):X^A\to Y^A$, which is what we discussed in the above paragraph; this arrow takes an object $\varphi\in X^A$ to $f\circ\varphi\in Y^A$, i.e. $F(f)(\varphi)=f\circ\varphi$ (although this is not a nice looking way to write it). Note that $F$ is covariant.
Now, let's do the same thing but fixing $B$. That is, if we again call the functor $F$, then $F(X)=B^X$. The value on arrows this time? Well if we have an arrow $f:X\to Y$, then we get an induced map $B^Y\to B^X$ as follows: $\varphi\in B^Y$, i.e. $\varphi:Y\to B$ gets sent to $\varphi\circ f\in B^X$. Note then that this is indeed contravariant, since $f:X\to Y$ but $F(f):B^Y\to B^X$.
Sorry about the confusion there... I hope this helps. Let me know if there's anything I can clarify further.