Pitfalls of manual AES encryption for data transfer

Question

Context

I've used OpenSSL to encrypt some socket communications.
I am however using some functionality from the windows API that prevents me from using OpenSSL's opaque builtin sockets, so I am buffering through their BIO_s_mem interfaces - that requires 4 extra copies for all data passing through (socket->read buffer->read BIO->OpenSSL state machine->write BIO->write buffer->socket, as opposed to socket->OpenSSL state machine->socket).
This is not efficient, especially seen as the goal is (among other things) large file transfers.

Idea

As a result I'm thinking I could establish an OpenSSL session, generate some symmetric keys, and use that secure session to share them. I would then open another channel (or just use the underlying socket which I have full control over) to transfer the bulk data using something like AES256-GCM - doing that 'manually' (using OpenSSL crypto libraries) would make it incredibly fast as it would no longer imply the inefficiencies of passing it through the OpenSSL state machine.

Problem, Goal & Question

I'm not a cryptographer. I don't even know what I don't know. All I know is if people can find side-channel attacks based on compression ratio of encrypted data, then any mistakes I make with this would probably be fatal.

I'm aware key generation has to be done very carefully - I've read through this, and I was thinking of using OpenSSL's facilities for the purpose. I'm also aware that AES is by itself secure, but a wrong implementation will easily throw that away.

This does however, on the surface, seem like a manageable challenge. Generate keys -> Run data through AES256-GCM. The upside is big, and this is a personal project that likely won't have the most proficient hackers (or any hackers) trying to mess with it. I just want it secure so that if people start to use it, they're not using something could be easily broken.

---

So, is there something obvious I'm missing? Something less than obvious?

Extra considerations, perhaps a correction on something wrong I might have said, some resource I should read before the attempt? All is appreciated, but if your contribution is "just don't do it", I ask you to please add some context as to what could go wrong so I get the picture.

Answer 1

Briefly, your approach has following problems that are solved in TLS (see details here.):

1. It allows man-in-the-middle attacks (TLS doesn't)
2. It allows replay attacks (TLS doesn't)
3. It doesn't provide forward secrecy (TLD does)
4. It doesn't provide key revocation mechanism (TLS does)

Even if one is aware of the these problems and wants to address them in the own solution, a correct implementation of such solutions can require a lot of efforts.

One more potential problem is key distribution. Before any communication you need to provide each 2 parties with a key for their communication. In case one of parties lost the key or the device was compromised, you need again to provide these 2 parties with a new key. If the channel to provide the key is not secure enough, an attacker can intercept the key and can then decrypt the communication.

Some people may feel the usage of TLS "boring" or "not cool" compared to implementing their own solutions. But TLS provides solutions for many security risks.

And, as always for security topics, before making a decision, first evaluate the risks. What will a successful attack cost you and how much efforts and money are you ready to put to prevent it? For instance, if you want to communicate with a bank and a compromised communication can cost you \$100 000 000, then may be you will put a lot of money and time into implementing highly secured connection. But if you want to communicate with a video camera installed in a forest to observe birds nest, then successful attack on this communication will probably doesn't cost you much and it makes no sense to put let say \$10 000 into security.