G-ANOVA VARIANCE COMPONENTS RESULTS
Examples from Artificial Continuous Formulation: Task Wobble, Rater Smear
Scoring 1-6 scale with category boundaries 1.5 2.5 3.5 4.5 5.5
Underlying formulation Contininuous data, following development of CLBH.
Population distribution: True scores: skew
For each individual, start with a true ability (like CLBH "true level" top p.8)
which I'll call theta. For a population (or a school) theta can have various
distributions:
Skew distribution on [.5. 6.5] yielding true category
proportions {.25 .25 .25 .10 .10 .05} constructed with
continuous Uniform between score boundaries +/- .50.
mean 2.7; variance 2.1933.
Basic rater-task misclassification. Homogeneous raters and homogeneous tasks;
error facets defined by Task Wobble,Rater Smear. NO error term besides task-wobble, rater-smear.
TASK WOBBLE: N(0, var)
var values {small, big} = {0.584888, 1.09667}
RATER SMEAR N(0, var)
var values {best rater,...., worst rater} = {0.177696, 0.355391, 0.710782, 1.42156}
pXtXR anova
variance components components given in table below
General Conclusions
Person (universe?) components: Component for p decreases as rater smear variance increases--i.e., "universe variance" gets smaller as raters get worse. Component for p
decreases as task wobble increases.
Task components: Decreases in the quality of the task (increasing task wobble) leave t component = 0, but increase personXtask interaction. Interaction component taskXperson is large (even for this homogenous, symmetric task misclassification); the only large component besides p and error.
Also magnitude of txp decreases as rater smear increases--task by person interaction gets smaller as
recorders get worse.
Rater components: In both examples the r,pr,tr components show absolutely no sensitivity to large changes in rater acuity--all components remain 0, even for the worst raters!
Continuous Formulation Examples Based on Task-wobble, Rater Smear
pXtXr, G-anova
VARIANCE COMPONENTS RESULTS
RATER SMEAR N(0, var) SMALL TASK VARIANCE "BIG" TASK VARIANCE
N(0, .5849) N(0, 1.0967)
Source Variance Source Variance
W || W component component
O || O 1 sub 1.74 1 sub 1.58
R || R 2 tsk 0.00 2 tsk 0.00
var = .177696 S || S 3 rat 0.00 3 rat 0.00
(g-coeff .742, .632) E || E 4 sub*tsk 0.421 4 sub*tsk 0.749
|| 5 sub*rat 0.003 5 sub*rat 0.001
\/ 6 tsk*rat 0.000 6 tsk*rat 0.000
R || R 7 Error 0.206 7 Error 0.202
A || A ------------------------------ ----------------------
T || T
E || E Source Variance Source Variance
R || R component component
S || S 1 sub 1.71 1 sub 1.55
var = .355391 || 2 tsk 0.00 2 tsk 0.00
|| 3 rat 0.00 3 rat 0.00
|| 4 sub*tsk 0.403 4 sub*tsk 0.718
|| 5 sub*rat 0.00 5 sub*rat 0.001
\/ 6 tsk*rat 0.00 6 tsk*rat 0.00
W || W 7 Error 0.334 7 Error 0.324
O || O ------------------------------ ---------------------
R || R
S || S Source Variance Source Variance
E || E component component
|| 1 sub 1.57 1 sub 1.48
|| 2 tsk 0.00 2 tsk 0.00
var = .710782 R || R 3 rat 0.00 3 rat 0.00
A \/ A 4 sub*tsk 0.368 4 sub*tsk 0.646
T || T 5 sub*rat 0.00 5 sub*rat 0.00
E || E 6 tsk*rat 0.00 6 tsk*rat 0.00
R || R 7 Error 0.577 7 Error 0.547
S || S ------------------------------ -------------------- -
\/
|| Source Variance Source Variance
|| component component
|| 1 sub 1.399 1 sub 1.256
|| 2 tsk 0.00 2 tsk 0.00
var = 1.42156 \/ 3 rat 0.00 3 rat 0.00
W || W 4 sub*tsk 0.328 4 sub*tsk 0.622
O || O 5 sub*rat 0.012 5 sub*rat 0.030
R || R 6 tsk*rat 0.00 6 tsk*rat 0.00
S || S 7 Error 0.986 7 Error 0.931
E || E ============================== ----------------------