The most important take-away is that if you are asking this question, you are almost certainly not qualified to design a secure cryptographic primitive. Sounds harsh, but I mean it in all earnestness.
You wouldn't trust someone who hadn't been to medical school to do surgery on you. Similarly, we wouldn't trust someone who doesn't already know the research literature on cryptanalysis to design a new cipher. More importantly, history shows that amateurs who try to design a new cipher without having spent year studying the research literature almost inevitably invent something useless -- either a cipher that is insecure, or a cipher that is hopelessly non-competitive with the state of the art.
As Bruce Schneier once wrote, "anyone can invent a cipher they themselves can't break". The hard bit is to design a cipher that no one else can break. Usually cipher designers have blind spots for the weaknesses in their own design; by definition, they design their cipher to defend against all the attacks they can think of – but it's easy for there to be other attacks the cipher designer didn't think of, and in that case the cipher designer is unlikely to be aware that their own creation is flawed.
On top of that, the world doesn't really need new ciphers. Our current ones are pretty good. I mean, I'm not saying they're perfect, I'm not saying it is impossible to improve upon them, but I'm saying they're for the most part good enough for all engineering purposes – a modern cryptographic primitive is almost never the weakest link in the security of a system. So working on this problem is in some sense likely to be working on the wrong problem.
So the best advice I can give you on how is: give it up. Don't expect your cipher to be any good. (If you really, really want to learn how to design a new cipher, you could start by studying the research literature on cipher design and cryptanalysis – but be prepared that this will take years of dedicated study.)
As far as how to vet a new cipher, here's how it tends to work. Confidence in the security of cryptographic primitives usually comes from the fact that a lot of really smart, knowledgeable people have tried really hard to break these algorithms, without making much of a dent at all. Of course, this is not a guarantee that no clever attack exists – it is always possible there is some incredibly sneaky mathematical shortcut attack that no one has been clever enough to find, but the more people who try to find one and fail, the less likely that looks. For practical purposes, it seems unlikely that a garden-variety attacker is going to discover an attack that dozens of really smart people tried to find and failed.
It is important to understand that this process does not provide a "proof" of the security of the cryptographic primitive. We have no proof that SHA256, or AES, or RSA are secure – it is widely believed that they are secure, but I could not give you a mathematical proof of that fact, and who knows, it is always possible that widespread beliefs are wrong. There are fundamental reasons why it is incredibly difficult to produce an ironclad mathematical proof of security for most cryptographic primitives.
It is also important to understand that vetting a new primitive is extremely expensive: it takes person-decades of effort from intensely-talented specialists. This is one reason why there is little or no demand for new ciphers, particularly from amateurs who don't already have a deep understanding of the field. The extremely high cost of vetting a new cryptographic primitive is one reason that smart users of cryptography generally try to use existing, vetted primitives, rather than inventing their own. If you invent your own scheme, it is extremely unlikely you'll be able to arrange for as much analysis and vetting of your own scheme as the standard ones have already received – so don't do that. Don't "roll your own". Instead, I advise people to use existing, standard, accepted primitives, like AES, SHA256, RSA, etc.
