1

I have to tested multiple reinforcement learning systems and record their performances [test-accuracy] based on two categories of [system-type] and [communication-level]. The system-type can be either one of the two types namely "centralized" or "decentralized". However, the second factor, communication-level, is always the same for the centralized system, whereas the decentralized system can have one of the four possible "communication-levels".

To study how these factors affect the overall performance of the system I am not sure how to proceed with the data.

should I first perform one-way Anova test with the primary independent factor being the [system-type] and see its effect and then proceed to perform another one-way Anova test with only the data from the decentralized-system (that can have either one of the four communications level) to study the affect of the communication on the decentralized system?

or

It does not matter that one of the system-types has always (strictly) one form of communication level and I can perform a two-way Anova with these two being the main factors?

Hirad Gorgoroth
  • 111
  • 1
    Assuming data are normal and variances are equal: best to do one-factor ANOVA with factor of five levels levels C1, D1, D2, D3, D4. Then (if rej H0 that all 5 have same means) use contrasts to compare C1 with avg of the Ds, and to make comparisons among the Ds. // Doing a two sample test of C vs D could go awry if Ds have different values averaging about the same as C1 (find no effect). Also, using contrasts provides a rigorous way to avoid 'false discovery' by controlling the 'family error rate' of the contrasts you want to test. – BruceET Oct 10 '18 at 00:14
  • 1
    @BruceET thanks for your reply. I have made a small change to the structure of the experiment and the factors are clearly separated now. However, your suggestion is really good and infact currently this is what I will probably do at the end of the day. – Hirad Gorgoroth Oct 11 '18 at 23:02
  • 1
    Before you see the data, you may want to specify four linearly indep contrasts among (C1, D1, D2, D3, D4). Perhaps D1&D2 are similar and D3&D4 are similar. Then coefficients for contrasts might be given in vectors (4,-1,-1,-1,-1), (0,1,1,-1,-1), (0,1,-1,0,0), and (0,0,0,1,-1). Note that each vector sums to 0, also vector products btw any two are 0; eg vector prod of 1st 2 is $(40)+(-11)+(-11)+(-1-1)+(-1*-1)=0,$ which makes them 'orthogonal'. The standard for declaring significance of orthog contrasts in such a set of four is less demanding than for contrasts you contrive based on data. – BruceET Oct 11 '18 at 23:40

0 Answers0