The correlation coefficient between X and Y is defined as follows:
$${\displaystyle \rho _{X,Y}={\frac {\operatorname {\mathbb {E} } [(X-\mu _{X})(Y-\mu _{Y})]}{\sigma _{X}\sigma _{Y}}}}$$
However, $\rho$ can be expressed in terms of uncentered moments:
$${\displaystyle \rho _{X,Y}={\frac {\operatorname {\mathbb {E} } [\,X\,Y\,]-\operatorname {\mathbb {E} } [\,X\,]\operatorname {\mathbb {E} } [\,Y\,]}{{\sqrt {\operatorname {\mathbb {E} } [\,X^{2}\,]-\left(\operatorname {\mathbb {E} } [\,X\,]\right)^{2}}}~{\sqrt {\operatorname {\mathbb {E} } [\,Y^{2}\,]-\left(\operatorname {\mathbb {E} } [\,Y\,]\right)^{2}}}}}.}$$
It seems that you are struggling with the orders of integration. It helps to recall the Law of Total Expectation, which states that
$\mathbb{E}[X] = \mathbb{E}[ \mathbb{E}[X | Y]]$ and $\mathbb{E}[Y] = \mathbb{E}[ \mathbb{E}[Y | X]]$
Then, the integrals you need to compute are:
$\mathbb{E}[X] = \mathbb{E}[ \mathbb{E}[X | Y]] = \int_0^1\int_0^y xf(x,y)\,dx\,dy $
$\mathbb{E}[X^2] = \mathbb{E}[ \mathbb{E}[X^2 | Y]] = \int_0^1\int_0^y x^2f(x,y)\,dx\,dy $
$\mathbb{E}[Y] = \mathbb{E}[ \mathbb{E}[Y| X]] =\int_0^1\int_x^1 yf(x,y)\,dy\,dx$
$\mathbb{E}[Y^2] = \mathbb{E}[ \mathbb{E}[Y^2| X]] =\int_0^1\int_x^1 y^2f(x,y)\,dy\,dx$
$\mathbb{E}[XY] =\int_0^1\int_x^1 xyf(x,y)\,dy\,dx$