We all know we should NOT roll our self-designed encryption algorithm, how about block cipher mode with a certificated cipher method like AES? Will a self-designed block cipher mode cause any flaws or weakness to the cipher?
I have search over the Internet but can't find a answer. Hope here is the right place to ask such a question.
Similar concerns apply as to any self-designed cryptographic algorithms. Standard block cipher modes usually have security proofs. If yours does not then even if it seems correct you may be missing some weakness.
An example of where security proofs of cipher modes are clearly important is the CCM mode (pdf). The same cipher key is used both for CBC-MAC authentication and to produce the CTR blocks used for encryption. This key-reuse is generally a bad idea and in a home-baked mode could be a weakness, but is proved safe in the particular composition used.
Very simple modes may be easy to prove secure yourself. However, I doubt those have any real advantages over using a standard mode either.
There is a fundamental difference between designing your own algorithm (e.g., block cipher) and designing your own mode of operation. In particular, it is possible to formally prove the security of a mode of operation, and this is not possible with a block cipher. Thus, if you know how to prove security, then there isn't any reason not to do this. (Of course, you should in general have a reason for doing something that isn't currently covered by any standard. But, there are reasons that come up.)
Personally, I would never try to design a block cipher or the like, since this requires many years of expertise and even then is really hard. Even block ciphers designed by experts have flaws, so it's best to just use AES etc. However, I would design a mode of operation if I had a reason to. But then again, I have years of experience writing proofs. If you don't, then I would recommend to stay clear of this as well.
