If a merchant decides to accept a Monero payment the moment it appears in the memory pool (before confirmation) what risks are being assumed?
Based on the probability of these risks being exploited, might there still be a business compelling need to accept them if amounts involved fall below a certain threshold?
The risk is that this transaction will not be included in a block, ever, and thus never pays the merchant.
The risk are the same as with Bitcoin, since Monero uses the same type of PoW system, so you may want to read up on doing this with Bitcoin.
One possibility for attack is that the transaction being sent to the merchant has an input which will be spent in another transaction that is being mined, but either not relayed to the merchant, or rejected by the merchant's daemon (since it spends an output that's already being spent in the pool). This requires that the second transaction be sent after the one to the merchant (so the merchant node can't receive it before from other relays), but reach mining nodes before the one sent to the merchant. So it's a timing race.
Another possibility is that a tx is sent to the merchant only. The merchant's daemon would normally relay it to ther nodes, so this only works if the merchant is in "listen only" mode. There is no such thing in the current monero daemon, but this might be done as an optimization in the future by a merchant who wants to save bandwidth.
As to how to balance the risks vs the loss if an attack succeeds, that is a decision for the merchant to take. If quick service is part of the merchant's appeal, there is an incentive to.
Any unconfirmed/instant transaction on Monero should be assumed reversible. In practice however, a merchant needs to consider whether the goods need to be released immediately - something like shipped goods do not.
If the goods do not need to be released immediately, the merchant can give visual feedback to the user on an instant confirmation, then simply never send the goods if there was a double-spend.
If the goods are virtual and immediately released, a variable number of confirmations (depending on the price) could be done. For reference, XMR.to does immediate release for amounts worth <0.1 BTC.
