In general, the sets of finite and infinite elements are each closed under conjugacy but will not be subgroups. So take these sets for $H$.
The set of infinite elements will never form a subgroup. However, if you attach the identity it sometimes will and sometimes will not. For example,
Take $C_2\ast C_3\cong \langle a, b; a^2, b^3\rangle$. Then the set of elements of infinite order is closed under conjugacy, but it is not a subgroup as $ab$ and $ba$ both have infinite order (erm...I cannot think of a simple reason why - it follows from the theory of free products) while $ba\cdot ab=ba^2b=b^2$ has finite order.
Take $D_{\infty}\cong \langle a, b; a^2, b^2\rangle$. Then every element of infinte order is in the infinite cyclic subgroup $\langle ab\rangle$, and so the set of elements of infinite order along with the identity forms a normal subgroup of $G$.
For the case of torsion elements, again both cases are possible:
Take $D_{\infty}\cong \langle a, b; a^2, b^2\rangle$. Then the set of elements of finite order is closed under conjugacy, but it is not a subgroup as $ab$ has infinite order.
Take a group $G$ which is the cross product of a finite group with a torsion-free group (every element has infinite order), for example $\mathbb{Z}\times A_5$. Then the set of elements of finite order forms a normal subgroup of $G$.
As a nod to your comment on the main question, note that all the examples I have given are "matrix groups", that is, you can view then as matrices (over $\mathbb{Z}$).