"We need to color $4×4$ square using $4$ black color and $12$ white color. Then, how many cases it may be? Flip is prohibited but rotating is ok"
I tried case by case (inner square and rest) anf obtained answer 389. But I don't know if it's correct. Help me please.
We apply PET (Polya Enumeration Theorem) here and this needs the cycle index. There are four rotations. The first is the identity which contributes
$$a_1^{16}.$$
There are the rotations by $90$ degrees and by $270$ degrees, which contribute
$$2 a_4^4.$$
The rotation by $180$ degrees contributes
$$a_2^8.$$
This yields the cycle index
$$Z(G) = \frac{1}{4} (a_1^{16} + 2 a_4^4 + a_2^8).$$
We thus have for the desired quantity
$$[B^4 W^{12}] Z(G; B + W) \\ = [B^4 W^{12}] \frac{1}{4} ((B+W)^{16} + 2 (B^4+W^4)^4 + (B^2+W^2)^8) \\ = \frac{1}{4} {16\choose 4} + \frac{1}{2} [B W^3] (B+W)^4 + \frac{1}{4} [B^2 W^6] (B+W)^8 \\ = \frac{1}{4} {16\choose 4} + \frac{1}{2} {4\choose 1} + \frac{1}{4} {8\choose 2}.$$
This yields
$$\bbox[5px,border:2px solid #00A000]{464}$$
confirming the data from the comment.
Assuming rotation is also prohibited, we have ${16 \choose 4} = 1820$ ways to select the black tiles. However, we are allowed to rotate the board, so we need to account for overcounting.
We have counted most colorings four times, as, in general, each rotation will be counted separately. However, symmetry can change that.
There are ${4 \choose 1} = 4$ colorings which are counted only once. These are the coloring that don't change after rotation. If we split the $4 \times 4$ table into four $2 \times 2$ tables, they will have one black tile in each of these $2 \times 2$s placed symmetrically.
There are also colorings that are counted twice: these are the ones which change from a 90-degree rotation, but are preserved after 180-degree rotation. Let's look at the top two rows of the $4 \times 4$ square. We can color any two tiles there, but then we'll need to color the two symmetrical ones (across the center, not across the horizontal line) to preserve the coloring after 180-degree rotation. There are ${8 \choose 2} = 28$ ways to do that. From these, we need to subtract the four which have 4-way symmetry, and 24 are left. Now, these are double counted, so there are 12 distinct colorings (for example, the coloring where the two first column middle cells, and two last column middle cells are black, are the same as the coloring where the two first row middle cells and the two last row middle cells are black).
The remaining are counted four times, therefore there are $\frac{1820 - 4 - 2 \times 12}{4} = 448$ colorings remaining.
This gives us a total of 448 + 12 + 4 = 464 colorings.
