I am a newbie to the cryptography world. I am working on a custom algorithm that encrypts the data multiple times and results in a cipher text. It uses a different key in each round to encrypt it. I researched a lot but couldn't find any answer to this question. What do you think is the ideal number for the rounds of encryption when the keys are different in each iteration? Could you show me some math?
Alex
How many rounds cannot really be answered as it depends on the design of a (block) cipher. As many as required to withstand attacks found by crypt-analysis, plus quite few more in case the attacks are enhanced is the best we can do.
AES-128 has a mere 10 rounds, but those rounds are rather complex. Threefish has a less complex inner structure, but uses 72 rounds for the 256 bit version. There are attacks on Threefish that break 53 of 72 rounds. If the rounds could be compromised that easily for AES then it would be utterly broken. Fortunately the best known attack against AES-128 covers 8 rounds out of 10 - and because it is a key distinguishing attack it doesn't give all that much power to an attacker either.
So for your own cipher you'll have to try to find out how well it withstands all known attacks. This is very hard thing to do and it is unlikely that you will succeed without help. This is one of the reasons why it is hard to design a cipher. Another is to avoid meet-in-the-middle attacks between the rounds of the cipher.