THE FEASIBILITY OF APPLYING STATISTICAL MODERATION IN PRACTICAL

THE FEASIBILITY OF APPLYING STATISTICAL MODERATION IN PRACTICAL

THE FEASIBILITY OF APPLYING STATISTICAL MODERATION IN PRACTICAL COMPONENTS USING EXTERNAL EXAMINATIONS AS THE ANCHOR Advancing learning, certifying your future Background Moderation only applied to practical subjects Their syllabuses contained practical objectives Practical subjects differ in nature, e.g. HE, Agriculture, Music, PE, D&T, Art, hence different modes of assessment Different format of moderation depending on the subject Advent of Outcome-Based Assessment requires SBA in all subjects Advancing learning, certifying your future Background Contd e.g BGCSE D&T, employs statistical moderation techniques in its moderation, without necessarily linking it to External exams (Kempa, 1986) (y = -1.211 + .508*Sch Mark). In some subjects visit schools (JC Agric), in some artifacts centrally marked (BGCSE Agric) Visiting moderation is expensive, and resources are limited. Stats moderation is cheap Advancing learning, certifying your future STATISTICAL MODERATION Statistical moderation tries to adjust the level and spread of each schools assessments of its students in a particular study, to match the level and spread of the same students scores on a common external examination. Thus statistical moderation is applied to two different marks measuring the Same Learning Outcomes. Advancing learning, certifying your future Stats Mod Contd Thus students External Score should be a valid predictor of coursework

achievement. The Coursework score must itself be valid Hence strong correlation between the two!!!!. Advancing learning, certifying your future Subjects Components JC Subject 1 PP1: MC 35% PP2: SASI 40% PP3: SBA 25% JC Subject 2 PP1: MC 30% PP2: Prac Exam 45% PP3: SBA 25% JC Subject 3 PP1: Listening 25% PP2: SASI 25% PP3: Prac Exam 30% JC Subject 4 PP1: MC 20% PP2: SASI 40% PP3: SBA 40% BGCSE Subject 5 PP1: Pract Exam 50%

PP2: Pract Exam 50% PP4 or 5: SBA 50% BGCSE Subject 6 PP1: Written Exam 50% PP3: Indiv Study 30% PP2: SBA 20% Advancing learning, certifying your future PP4: SBA 20% PP3: Pract Exam 50% Methodology A mixed methodology employing Correlational study design as baseline and Design-Based Research design as developmental Purposive sampling of Subjects (3 from JC and 2 from BGCSE) and schools. Teacher coursework scores correlated with all other scores Advancing learning, certifying your future Methodology Contd Agreement (Concordance)/Inter-Rater Reliability between moderator and Teacher scores for coursework scores National descriptive statistics and Schools descriptive statistics were computed Advancing learning, certifying your future RESULTS Teacher Coursework score and Moderator Coursework score Means . Level

JC Subjects Subject Score type Mean SD 1 Teacher Mark Sample Moderation Mark Teacher Mark Sample Moderation Mark Teacher Mark Sample Moderation Mark Teacher Mark Sample Moderation Mark Teacher Mark Sample Moderation Mark Teacher Mark Sample Moderation Mark 40.17 40.22 51.88 53.59 42.68 36.52 45.74 12.91 59.01 47.55 64.22 61.57 6.40 6.53 19.49 18.23 13.89 14.34 17.26 6.37 17.22 14.52 14.06 15.17

2 3 4 BGCSE 5 6 Advancing learning, certifying your future The relationship between Teacher Coursework score and Moderator Coursework score . Level JC BGCSE Subject r r2 ICC 1 2 .93 .98 86% 96% .96 .98 3 4 .97 .65 94% 42%

.95 .20 5 .85 72% .80 6 .91 83% .93 Advancing learning, certifying your future Correlation coefficients between Teacher Coursework score and other components . JC Subject 2, JC Subject 3, and BGCSE Subject 5 Teachers Coursework scores didnt have the highest correlation with Practical Exam! Advancing learning, certifying your future Shared Variance between SBA and External Score with highest r Coefficien t Subject 1 JC BGCSE 2 3 4 5 r

.27 .45 .66 .52 .41 r2 7% 20% 44% 27% 17% 24% 24% 72% Paper with highest r Scaled Score Scaled Score Scaled Score Scaled External Score Practical Exam 6 .49 .49 .85 Indiv. study NB: JC Subjects 2, 3, and BGCSE Subjects 5, and 6 all have a Practical Exam Component. BGCSE Subject 5 has 3 papers to choose 1 from. Need to investigate such low r2 for All subjects except BGCSE Subject 6!!

Advancing learning, certifying your future The Statistical Moderation Process: School X JSS Subject 1 Descriptive Stats . Advancing learning, certifying your future Standardise the coursework scores using the Scaled Score JSS School X Scaled Score Scaled Scores Z- Teacher Coursework Teacher CourseWork . 27 16 13 52 55 32 12 20 28 49 51 28 53 40 39 35 6 32 Scores -0.31 -1.25 -1.50 1.81 2.06 0.11 -1.58 -0.91 -0.23 1.55 1.72 -0.23 1.89 0.79 0.70 0.37 -2.09

0.11 Score 38 13 42 43 34 33 40 12 41 34 47 45 49 42 47 35 30 40 Score Z-Score 0.25 -2.94 0.76 0.88 -0.26 -0.39 0.50 -3.07 0.63 -0.26 1.40 1.14 1.65 0.76 1.40 -0.14 -0.77 0.50 Advancing learning, certifying your future Standardizing the Teacher Coursework score Teacher C/W Mark Z- Stsd Teacher C/W Scaled Score Scaled Score Z- Teacher C/ . 27 16

13 52 55 32 12 20 28 49 51 28 53 40 39 35 6 32 30 Scores -0.31 -1.25 -1.50 1.81 2.06 0.11 -1.58 -0.91 -0.23 1.55 1.72 -0.23 1.89 0.79 0.70 0.37 -2.09 0.11 -0.06 W Mark 38 13 42 43 34 33 40 12 41 34 47 45

49 42 47 35 30 40 45 Score 0.25 -2.94 0.76 0.88 -0.26 -0.39 0.50 -3.07 0.63 -0.26 1.40 1.14 1.65 0.76 1.40 -0.14 -0.77 0.50 1.14 Advancing learning, certifying your future Score 33.49793746 -4.81731461 39.62837779 41.16098787 27.36749713 25.83488705 36.56315763 -6.3499247 38.09576771 27.36749713 47.29142821 44.22620804 50.35664837 39.62837779 47.29142821 28.90010721 21.2370568 36.56315763 44.22620804

Applying the VCE correction factor Then Subjecting the Teacher coursework score to the model: 2 yij a j xij b j xij c j Advancing learning, certifying your future Final Moderated . Ext. Score Ext. Score Z-Scores Teach Teacher Final er Mark Z- Moderated Mark Score Mark (orig) Stsd Teacher C/ y Work Score t_moder ated 27 16 55 32 12 20 28 28 53 40 35 6 32 30 -0.31 -1.25 2.06 0.11 -1.58 -0.91 -0.23 -0.23

1.89 0.79 0.37 -2.09 0.11 -0.06 38 13 34 33 40 12 41 45 49 42 35 30 40 45 33.49793746 -4.81731461 27.36749713 25.83488705 36.56315763 -6.3499247 38.09576771 44.22620804 50.35664837 39.62837779 28.90010721 21.2370568 36.56315763 44.22620804 16.10186 3.144243 12.82014 12.07164 17.91537 3 18.86528 22.95268 27.50045 19.84397 13.59741 9.998794 17.91537 22.95268

0.25 -2.94 -0.26 -0.39 0.50 -3.07 0.63 1.14 1.65 0.76 -0.14 -0.77 0.50 1.14 19 6.5 17 16.5 20 6 20.5 22.5 24.5 21 17.5 15 20 22.5 Advancing learning, certifying your future 32.20372 6.288486 25.64027 24.14328 35.83073 6 37.73056 45.90535 55.00091 39.68794 27.19481 19.99759 35.83073 45.90535 Agric stats moderation outcome . 1 Low

2 Low 3 Low 4 High Schools and Pass Rate 5 6 7 High Average High Min Mark Max Mark Mean SD External min Score max mean SD Standardi min sed max Teacher Mean Coursew SD ork Score 29 0 12 36 34 32 50 22

49 50 49 43.51 3.81 4 53 24.19 11.06 -18.00 43.06 24.19 11.06 13.94 4.76 6 49 25.01 10.23 -4.95 42.33 25.02 10.23 36.07 7.84 6 55 30.69 11.79 -6.35 50.36 30.54 12.01 46.99 1.89 20 65 42.29 8.96 -9.83 56.56 42.28 8.96 Stats min

moderate max d score Mean SD 1.63 53.12 24.12 11.71 6 49.01 25.00 10.55 6.00 55.00 30.68 11.60 16.21 65.01 42.28 11.08 Domain Statistic s Raw Teacher Coursew ork 8 9 Avera Average ge 10 Low 30 26 36 21 47

48 50 48 49 44.18 3.32 9 55 35.18 11.14 1 52 35.32 11.19 43.57 2.63 10 60 34.03 10.27 -11.12 47.41 34.02 10.27 42.18 3.652 7 59 35.96 12.40 -107 69.83 35.96 12.33 43.39 4.074 0 59 30.20 12.81 -24.5 50.98 30.20 12.81

44.81 2.33 12 62 30.57 10.80 -10.20 45.33 30.57 10.80 41.09 5.142 6 54 27.54 11.40 6 54 27.54 11.40 9.59 54.32 34.16 11.45 3.82 60.01 34.02 15.03 7.00 54.00 35.95 11.93 . 4.15 59.00 30.19 13.11 2.89 62.00 30.50 17.14 6 54 27.54

11.40 Advancing learning, certifying your future Development of a Statistical Model for BEC Employed Design-Based Research (DBR) methodology. DBR develops product with practitioners interactively and iteractively To ensures practicability and utility Advancing learning, certifying your future Development of the model Intervention (R Q 2) Baseline (RQ 1) Prototype Prototype 33 Design Design Specifications Specifications Prototype Prototype 22 Baseline Baseline survey survey findings findings Prototyp Prototyp ee 11 Expert Expert Review Review Define Measure Baseline survey Final Final Prototyp Prototyp ee 44 Analyse

Design Pilot Pilot Field test Try-out Try-out Develop Intervention Advancing learning, certifying your future Implement Developmental work . System-wide Approach (pathways/OBE) Moderation Observation and interviews Need to understand why some subjects teacher coursework and moderator coursework are high and others moderate What is it that the teachers or moderators or students or schools or the system does/ doesnt do Advancing learning, certifying your future Conclusion Subjects showed high correlation and agreement between Teacher Score and Moderator Score suggesting that Teachers and Moderators applied the scoring guide consistently the same. Thus the teachers scoring could be relied upon. Correlations between the Teacher score and the External scores was low to moderate for all the subjects Correlations between Teacher score and Practical test scores was found to be low to moderate Advancing learning, certifying your future Conclusion Development of the model is a systemwide endeavour hence takes long Interventionist: developing an intervention in a real world setting.

Interactive: Involves active participation of practitioners & professionals in the various stages and activities Iterative: incoporates cycles of analysis, design & development, evaluation & revision. Advancing learning, certifying your future THANK YOU!! Advancing learning, certifying your future JC Subject 1 Corr Matrix Components . PP 1 PP 2 1 .82** .94** 1 .97** 1 Teacher C/ .25** W Mark Sample .23** Mod Mark .26** .27** 1 .23** .24** .93** 1 Final Mod Mark .25**

.26** .99** .93** PP1 PP2 Ext. Score .25** Ext. Score Teacher C/ Sample Final W Mark Mod Mark ModMark Advancing learning, certifying your future 1 Corr matrix for JC Subject 2 Components Component PP 1 PP 2 ExtScor e Teacher C/ work Mark PP 1 1 .35** .73** 1 PP 2 ExtScore Teacher C/work Mark

Sample Mod Mark Final Mod Mark .27** Sample Mod Mark .26** Final Mod Mark .28** .89** .44** .42** .44** 1 .45** .44** .45** 1 .98** 1.0** 1 .99** 1 Advancing learning, certifying your future Corr matrix for JC Subject 3 Components Component PP 1 PP 2

PP 3 External Teacher Sample Score Mark Mod Mark PP 1 -Listening PP 2 (SASI) PP 3 (PRACT Exam) External Score Teacher Mark (SBA) Sample Mod Mark Final Mod Mark 1 .77** .57** 1 .60** 1 .83** .54** .87** .53** .90** .63** 1 .66** 1 .62** .62** .73** .75** .96**

1 .56** .57** .67** .70** .94** 1.0** Advancing learning, certifying your future Final Mod Mark 1 Correlation matrix for JC Subject 4 Component Paper 1 Paper 1 Paper 2 External Score 1 .769** .872** Teacher C/work .443 ** Mark Sample Mod .447** Mark Final Mod Mark .417** Paper 2 External Teache Sampl Final Score r e Mod Mod C/Work Mark Mark Mark 1 .982**

1 .511** .516** 1 .509** .512** .647** 1 .476** .482** .711** .658** 1 Advancing learning, certifying your future Corr matrix for BGCSE Subject 6 Components Component PP1 PP 2 PP3 External Teache Sample Final Score r Mark Mod Mod Mark Mark PP 1 PP2 1 .46** 1 PP3 .48** .39** 1 External Score .46** .48**

1** Teacher Mark .50** .34** .85** .50** 1 Sample Mod Mark .46** .35** .98** .46** .85** 1 Final Mod Mark .48** .39** 1** .85** .98** 1 .48** Advancing learning, certifying your future 1 Corr matrix for BGCSE Subject 5 Component PP 1 PP2 PP 3 Final Scaled Mark

Teacher Mark PP 1 1 .a .a 1.000** PP 2 .a 1 .a PP 3 .a .a 1 Final Scaled Mark Teacher Mark Sample Mod Mark Final Mod Mark .414** .495** .512** 1.000** .487** .570**

.537** .999** .492** .537** .573** 1 .482** .553** .559** 1 .906** .959** 1 .907** Sampled Mod Mark Final Mod Mark 1 Advancing learning, certifying your future

Recently Viewed Presentations

  • Christmas Songs and Poems Deck the Halls Deck

    Christmas Songs and Poems Deck the Halls Deck

    In the meadow we can build a snowman, Then pretend that he is Parson Brown He'll say: Are you married? we'll say: No man, But you can do the job when you're in town. Symbols of Christmas Christmas Songs and...
  • Online resources  Interactive resources to use as you

    Online resources Interactive resources to use as you

    There are 16 short films (on DVD or video), each introducing a very small number of key words and phrases: These are the Pack 1 contents.. Hola - greetings. Adiós - saying goodbye ¿Qué tal? - asking people how they...
  • Dark Energy Task Force http://www.nsf.gov/mps/ast/detf.jsp  Three agencies: DOE;

    Dark Energy Task Force http://www.nsf.gov/mps/ast/detf.jsp Three agencies: DOE;

    DETF Membership Andy Albrecht, Davis Gary Bernstein, Penn Bob Cahn, LBNL Wendy Freedman, OCIW Jackie Hewitt, MIT Wayne Hu, Chicago John Huth, Harvard Mark Kamionkowski, Caltech Rocky Kolb, Fermilab/Chicago Lloyd Knox, Davis John Mather, GSFC Suzanne Staggs, Princeton Nick Suntzeff,...
  • Problems with Yalta

    Problems with Yalta

    Quiz. Answer on your own paper. Remember your academic integrity! Based on your reading, answer ONEof the following questions:. What were the problems with the Yalta agreement? What was agreed and disagreed at Potsdam?
  • What's New in Quartus II Software v11.0— General Overview

    What's New in Quartus II Software v11.0— General Overview

    What's New in Quartus II Software v11.0—General Overview. Jun 2011. ... Altera recommends that you use Qsys for new designs. In version 11.0, both Qsys, which is next generation of system integration tool, as well as it's predecessor, the SOPC...
  • MarketspaceU - Chapter 2 Enhanced Lecture Slides

    MarketspaceU - Chapter 2 Enhanced Lecture Slides

    Divides the market into different geographical units Country / region / city, city size, density (urban, suburban, rural), ISP domain Demographic Divides the market on the basis of demographic variables Age, gender, income, occupation, education, nationality, family status, Internet connectivity...
  • A comprehensive model for Personalised Care September 2018

    A comprehensive model for Personalised Care September 2018

    Lyn Romeo (Chief Social Worker for Adults) DHSC, DfE, DWP, HMT, MHCLG. Also attended: EPC taskforce workshop, C4CC partners meeting, TLAP partners meeting, PHE Board, as well as April's Commissioning Committee and May's EGM. The following sets out the .
  • Ethics - chillicothe.va.gov

    Ethics - chillicothe.va.gov

    A nurse refuses to treat a patient in the ICU for reasons of conscience, and the nursing supervisor must decide whether the refusal is appropriate. A patient with decision-making capacity wants to return home, but his family and the clinical...