Is this DIY remote lock protocol secure?

Question

I need your advice on following scheme of exchange protocol between remote lock and key. I'm planning to use following algorithm:

1. Key generates unique value that never repeats (in reality it's just counter that increments every time the button is pressed), let's call it "NONCE1".

2. Key sends a request to the lock which consists of NONCE1 and H1 = MD5(NONCE1 + KEY1). Second part is used only for "signing" the request, attacker wouldn't be able to generate valid request if he doesn't know KEY1.

3. Lock checks if H1 = MD5(NONCE1 + KEY1), this is used to check if incoming request is coming from the key (or at least from the one who is aware of the value of KEY1). Additional steps that were not described previously is:

   • Lock checks if NONCE1 of the previous key request less than NONCE1 received just now. This part prevents acting on previously processed requests, attacker who has such request won't be able to go any further. I'm trying to leave him in the situation where old requests are not accepted by the lock, and new requests couldn't be generated because he can't reproduce it without KEY1.

   • Lock sets NONCE1(previous request) = NONCE1(current request)

4. Lock generates completely unique value that never repeats, let's call it "NONCE2".

5. Lock sends a response to the key which consists of H2 = MD5(NONCE2 + KEY2). NONCE2 - is counter based (in order to completely exclude possible collisions), so in order to hide it's "counter - based nature" from an attacker, MD5(NONCE2 + KEY2) is used here.

6. Key does H3 = MD5(H2 + KEY3) and sends it back to the lock

7. Lock checks if H3 = MD5(H2 + KEY3) and acts on that

What could be possible vulnerabilities? Is it safe enough to use MD5 in this particular case?

I've added some additional "substeps" to clarify it a little bit

Accepted suggestion / solution is following:

• Get rid of steps 1 - 3
• Replace MD5 with HMAC-XXX

Answer 1

Even after your updates, the first part seems unnecessary. However, steps 4-5 do indeed prevent the attacker from learning future nonces they could ask the key MAC values for. So the protocol steps 4-7 would be secure with a secure MAC.

I agree with CodesInChaos that using HMAC would be better, because H(m||k) has some weaknesses, while HMAC is standard. Unless you have a good reason to use MD5, I would also prefer to use a stronger hash, though HMAC-MD5 is still secure (that is about encrypted data, but applies generally).

However, I do not think the weaknesses actually apply here, since the attacker does not have freedom to choose the values they need to calculate the keyed hash for. So even as is it should be secure. Despite that I would recommend switching to HMAC. Even better would be to use an existing mutual authentication protocol meant for this purpose, like the one descrived in Chapter 10 of HAC (see 10.2.3).

---

The below applies to the protocol as described in the original version of the question.

1. Key generates completely unique value that never repeats (some kind of a counter), let's call it "NONCE1".
2. Key sends a request to the lock which consists of NONCE1 and H1 = MD5(NONCE1 + KEY1)
3. Lock checks if H1 = MD5(NONCE1 + KEY1)

This part does not prove anything to anyone, so it is useless. An attacker who impersonates the key can reuse any previous nonce+MD5-pair because the lock does not know if the nonce is unique. An attacker who impersonates the lock can simply ignore the data received.

4. Lock generates completely unique value that never repeats, let's call it "NONCE2".
5. Lock sends a response to the key which consists of H2 = MD5(NONCE2 + KEY2)

Why not simply send NONCE2 to use as H2? The hash does not seem to help in any way, because the nonce is not used for anything else.

6. Key does H3 = MD5(H2 + KEY3) and sends it back to the lock
7. Lock checks if H3 = MD5(H2 + KEY3) and acts on that

This is the only part of the protocol that has a clear purpose: the lock is verifying an ad-hoc MAC to authenticate the key.