I wish to bring forward an idea first noted by Giuseppe Peano that could be a part of an answer, or at least, helpful for it. Because an adequate answer should also be capable of distinguishing a mathematical statement from a non-mathematical one (for example, a theoretical statement in physics).
Before that, I should remark that such a description as "mathematical statement is a statement that is either true or false but not both," by itself, is too naïve, and not distinctive of mathematics (consider other formal scholarly disciplines). The view that a mathematical statement must ultimately (that is, even if it remains conjectural for a while) have a strictly determinate truth value is traditionally cited as one of its key characteristics (however, a topic of philosophical debate in modern times; there are a lot of fine distinctions around this, let us leave it off here), but there are a lot more to say about it. Probably, the mentioned words have been expressed wrapped in a context that focuses on the nature of mathematical discourse stressing that it has to be exact and precise (avoiding vagueness or ambiguity), express clear and distinct ideas, and the like.
No doubt a mathematical statement has to satisfy the grammatical conventions of mathematical discourse, otherwise, it could not be even meaningful, let alone exact and precise in expression.
Notice that we begin to talk about mathematical discourse in connection with the notion of mathematical statement. So, a question suggests itself: What are the constituents of mathematical discourse? Very crudely, it involves rigorous arguments built on mathematical principles, established facts, concepts and terms. Thus, we have come to Peano's idea. It is more explicitly articulated by F. P. Ramsey, so I shall refer to Ramsey. Ramsey distinguishes semantic paradoxes from the purely logical by the notions involved in their construction (see "The Foundations of Mathematics" in The Foundations of Mathematics and Other Logical Essays, pp. 20-21):
It is not sufficiently remarked, and the fact is entirely neglected in
Principia Mathematica, that these contradictions fall into two fundamentally distinct groups, which we will call A and B. The best
known ones are divided as follows :—
A. (1) The class of all classes which are not members of themselves.
(2) The relation between two relations when one does not have itself
to the other. (3) Burali Forti’s contradiction of the greatest
ordinal.
B.
(4) ‘I am lying.’
(5) The least integer not nameable in fewer than nineteen syllables.
(6) The least indefinable ordinal.
(7) Richard’s Contradiction.
(8) Weyl’s contradiction about ‘heterologisch’.
The principle according to which I have divided them is of fundamental
importance. Group A consists of contradictions which, were no
provision made against them, would occur in a logical or mathematical
system itself. They involve only logical or mathematical terms such as
class and number, and show that there must be something wrong with our
logic or mathematics. But the contradictions of Group B are not purely
logical, and cannot be stated in logical terms alone; for they all
contain some reference to thought, language, or symbolism, which are
not formal but empirical terms. So they may be due not to faulty logic
or mathematics, but to faulty ideas concerning thought and language.
If so, they would not be relevant to mathematics or to logic, if by
‘logic’ we mean a symbolic system, though of course they would be
relevant to logic in the sense of the analysis of thought.
Likewise, it appears that an indispensable feature of a mathematical statement is that it be set down in the terms and norms of mathematics (as practised by the community of mathematicians). Indeed, we witness in the history of mathematics such problems that have been originated and defined outside of mathematics, but redefined in mathematical terms and solved by mathematicians, thus amassed into the body of mathematical knowledge along with those emerged from within mathematics.
For those who want to reflect on such issues further, I recommend David Bostock's reader-friendly book Philosophy of Mathematics: An Introduction.