I just thought I would add an example of one implementation of the algorithm that the accepted answer recommends. It only does the absolutely necessary bits, and fails quickly. (JavaScript)
Thanks to https://stackoverflow.com/a/9755252/5946596. This is essentially that code with explanations. I did not understand what it was doing when I first looked at it, but writing the code to do the full accepted answer algorithm snapped it into place.
Start the function. a, b, c implement {x:number, y:number} and are the vertices of the triangle A, B, C. point is P, the point being checked.
function triangleContains(point, a, b, c) {
Get the vector between A and the point P.
const AP = {x:point.x - a.x, y:point.y - a.y}
(b.x - a.x), (b.y - a.y) is side AB, and is "clockwise". We'll do just the third term of the crossproduct ABxAP, and then only keep its direction.
const AB = {x:(b.x - a.x), y:(b.y - a.y)}
const thirdTermABxAPisPositive = AB.x * AP.y - AB.y * AP.x > 0;
(c.x - a.x), (c.y - a.y) is side AC, and is "counter clockwise" from AB. Again we'll do just the third term of the crossproduct ACxAP, caring only about its direction. By using the "counter clockwise" AC instead of CA we have saved having to calculate CP.
const AC = {x:(c.x - a.x), y:(c.y - a.y)}
const thirdTermACxAPisPositive = AC.x * AP.y - AC.y * AP.x > 0
This result should NOT match the direction of what we kept before because AC and AB are in different directions. We can go ahead and exit the function returning false.
if (thirdTermACxAPisPositive == thirdTermABxAPisPositive) return false;
(c.x - b.x), (c.y - b.y) is sideBC and like the first side is "clockwise". (point.x - b.x),(point.y - b.y) is the vector BP. One last time, we only care about the third term of the crossproduct BCxBP, and really just its direction.
const BC = {x:(c.x - b.x), y:(c.y - b.y)}
const BP = {x:(point.x - b.x), y:(point.y - b.y)}
const thirdTermBCxBPisPositive = BC.x * BP.y - BC.y * BP.x > 0
This value should match the direction of thirdTermABxAPisPositve because BC is the same direction as AB. If it doesn't, again we can go ahead and leave the function, returning false.
if (thirdTermBCxBPisPositive != thirdTermABxAPisPositive) return false;
If we made it this far, then we're inside because all the booleans have the same value. It does not matter if they are all true or all false, they just have to be all the same.
return true; }
All together
function triangleContainsE(point, a, b, c) {
const AP = {x:point.x - a.x, y:point.y - a.y}
const AB = {x:(b.x - a.x), y:(b.y - a.y)}
const thirdTermABxAPisPositive = AB.x * AP.y - AB.y * AP.x > 0;
const AC = {x:(c.x - a.x), y:(c.y - a.y)}
const thirdTermACxAPisPositive = AC.x * AP.y - AC.y * AP.x > 0
if (thirdTermACxAPisPositive == thirdTermABxAPisPositive) return false;
const BC = {x:(c.x - b.x), y:(c.y - b.y)}
const BP = {x:(point.x - b.x), y:(point.y - b.y)}
const thirdTermBCxBPisPositive = BC.x * BP.y - BC.y * BP.x > 0
if (thirdTermBCxBPisPositive != thirdTermABxAPisPositive) return false;
return true;
}
Not Mixing Directions Example, recommend in comment by orig. author
function triangleContains_NotMixingDirections(point, a, b, c) {
const AP = { x: point.x - a.x, y: point.y - a.y };
const AB = { x: (b.x - a.x), y: (b.y - a.y) };
const thirdTermABxAPisPositive = AB.x * AP.y - AB.y * AP.x > 0;
const BC = { x: (c.x - b.x), y: (c.y - b.y) };
const BP = { x: (point.x - b.x), y: (point.y - b.y) };
const thirdTermBCxBPisPositive = BC.x * BP.y - BC.y * BP.x > 0;
if (thirdTermBCxBPisPositive != thirdTermABxAPisPositive)
return false;
const CA = { x: (a.x - c.x), y: (a.y - c.y) };
const CP = { x: (point.x - c.x), y: (point.y - c.y) };
const thirdTermCAxCPisPositive = CA.x * CP.y - CA.y * CP.x > 0;
return (thirdTermCAxCPisPositive == thirdTermABxAPisPositive);
}