In the CLI the command is as follows:
set store-tx-info <1|0>
Where 1 as argument will store transaction info. Note that for newly created wallets it is automatically set to 1. Thus, newly created wallets will automatically save transaction info.
I am not sure if it's the actual address that is saved or the one-time public key,
It will save the transaction hash, amount, private transaction key (you can obtain this by using get_tx_key <transaction-hash>), recipient's address, and payment ID.
Note that the GUI will automatically save transaction info as well. If you go to the History page and click on Details of a particular transaction, it will show all relevant information.