Hard to Flunk? SOLs Scores & More ...

A Must-read:  My friend, John R. Butcher, a/k/a "The Cranky Taxpayer" explains how RPS and VDOE conjure those SOL scores.  
VDOE admits that the accreditation scores are "adjusted."  In fact, the scores are so tortured as to be meaningless.

Note: These data embrace a spectrum of tests but VDOE boils the scores down, generally, to a single number for each subject at each school.  See 8VAC20-131-300.C.1.d.  Thus, the elementary English score is produced by averaging scores of the reading tests in grades 3 to 5.  So please be aware that the terms "test" and "score" below probably refer to results of several tests, generally at several grade levels.  The score distributions tell part of the story.  The math distribution is the less complicated so we'll start there.


The passing score is 70.  Recall that the State both designs the tests and sets the passing score so the location of the line hardly comes as a surprise.

For English, there are two passing scores: 75 for the elementary schools, 70 for all the rest.  So we have two distributions.


Interestingly, the elementary English test is harder and has a higher passing threshold.  In any event, it's hard to flunk on any of these tests. 

Richmond was up to the challenge, though.  Statewide, only 6.5% of the schools failed to be fully accredited.  30.4% of the Richmond schools failed.



How Big the Boost?

In contrast to the past, the VDOE Web site gives a fairly honest description of the accreditation calculations:
Ratings are based on the achievement of students on tests taken during the previous academic year and may also reflect a three-year average of achievement. Adjustments also may be made for students with limited English proficiency and for students who have recently transferred into a Virginia public school. Accreditation ratings also may reflect the success of a school in preparing students for retakes of SOL tests.
What it doesn't tell us is the size of these "adjustments."
At my request, the always helpful Chuck Pyle of VDOE sent along a spreadsheet showing the one-year and three-year adjusted scores of the Richmond public schools (send me an email if you'd like a copy).  With those data in hand, it's straightforward (albeit tedious) to pull the actual scores (with the high school scores already boosted by stealing the Maggie Walker scores) and compute the boosts from the accreditation adjustments.  I'll stick to the English and math tests but the data are there for anyone who is interested in the other tests.

Let's start with the high schools.  The boosted scores in English averaged 2.6 points higher than the Maggie Walker enhanced scores in English; 2.5 points higher in math.  The three year boost (obtained by using the three year average of the adjusted scores) was 1.5 points in English but 22.4 points in math.  You read that right: The average 3-year boost available to the math scores was 22.4 points.  Said otherwise, the math accreditation scores could be only distantly related to the actual 2012 SOL scores.   Here are those numbers in a table:

Boost 1Boost 3
English2.61.5
Mathematics2.522.4
Here "Boost1" is the accreditation score minus the score on the Report Card.  "Boost3" is the three year average of (boosted) accreditation scores, again minus the actual 2012 score.

The table by school shows the details.  There are two entries for each test: The "Current" accreditation score and the "3-yr" average accreditation score.  The "Pass Rate" column (named by VDOE, not by me) holds the current or 3-yr accreditation score.  Note: This terminology implies that they are reporting the actual pass rate.  But this is NOT the pass rate reported on the online School Report Card; it's the "adjusted" pass rate, i.e., the (boosted) accreditation score.  That is, they are trying to sell the fiction that they are reporting an actual pass rate, not a manipulated number.  This is the same kind of rank misrepresentation that in 2004 converted a 73.7 math score into a "perfect score" and embarrassed the Governor.
The last three columns are mine.  "Report Card" contains the actual score from the online School Report Card.  "Boost1" and "Boost3" again are the boosted 1-year and 3-year accreditation scores ("Pass Rates") minus the Report Card score, i.e., the scoring boosts available from the "adjustments."
Here is the table for the Richmond high schools. 
Sch NameSubjectAverage TypePass RateReport CardBoost1Boost3
Armstrong High EnglishCurrent817476
Armstrong High English3-yr80
Armstrong High MathematicsCurrent4134731
Armstrong High Mathematics3-yr65
Franklin Military AcademyEnglishCurrent908732
Franklin Military AcademyEnglish3-yr89
Franklin Military AcademyMathematicsCurrent595727
Franklin Military AcademyMathematics3-yr64
George Wythe HighEnglishCurrent95932-2
George Wythe HighEnglish3-yr91
George Wythe HighMathematicsCurrent5247530
George Wythe HighMathematics3-yr77
Huguenot HighEnglishCurrent908820
Huguenot HighEnglish3-yr88
Huguenot HighMathematicsCurrent4237534
Huguenot HighMathematics3-yr71
John Marshall HighEnglishCurrent918654
John Marshall HighEnglish3-yr90
John Marshall HighMathematicsCurrent4544131
John Marshall HighMathematics3-yr75
Open HighEnglishCurrent10010000
Open HighEnglish3-yr100
Open HighMathematicsCurrent919106
Open HighMathematics3-yr97
Richmond Community HighEnglishCurrent10010000
Richmond Community HighEnglish3-yr100
Richmond Community HighMathematicsCurrent7272020
Richmond Community HighMathematics3-yr92
Thomas Jefferson HighEnglishCurrent949222
Thomas Jefferson HighEnglish3-yr94
Thomas Jefferson HighMathematicsCurrent5656020
Thomas Jefferson HighMathematics3-yr76

Notice the Huguenot scores.  That school flunked the math test with a score of 37.  The VDOE spreadsheet reports an accreditation score ("Pass Rate") of 42, a 5 point boost.  That still didn't make the required 70.  The three year average, however, was 75, a 34 point boost over the actual 37 score.  So Huguenot is fully accredited in math, despite a miserable test score this year.
Here we see the two schools that failed in math: Armstrong and Franklin.  To the point of "adjustments," we also see that 3-year boosts ranging from 20 to 34 points saved Wythe, Huguenot, Marshall, and TJ.  So, in fact the passing score was not 70; it was as low as 37.

Got that?  Huguenot's dismal 37 gets boosted to a passing 71, which looks to be about the same as Community's actual, unboosted 72.  Thus this profoundly dishonest process makes a grossly failing math program look as if it's within one point of a school that earned a passing score.  Perhaps more to the point, the manipulated score at Huguenot tells the school its math program is "fully accredited," implying that a math program that has failed miserably does not need to be improved.

The middle schools enjoyed smaller one-year average boosts but a 24.3 point 3-year math boost:
Boost 1Boost 3
English1.85.9
Mathematics1.724.3
The middle school table is slightly more complicated than the high school table.  For example, AP Hill scored 57 on the math test and three points of "adjustments" left it at 60, ten points shy of accreditation.  The three year average, however, raised the actual 57 point score to 82, comfortably above the required 70.  Binford, in contrast, got a five point boost from the English adjustments, which raised it to a passing 71.  Binford did not need the 13 point boost that was available from the 3-year average so VDOE reported Binford at 71.

Get that: Binford's 66 is reported at 71; Hill's 57 is reported as 82!  Can you spell "ludicrous"?
Sch NameSubjectAverage TypePass RateReport CardBoost1Boost3
Albert Hill MiddleEnglishCurrent918924
Albert Hill MiddleEnglish3-yr93
Albert Hill MiddleMathematicsCurrent6057325
Albert Hill MiddleMathematics3-yr82
Binford MiddleEnglishCurrent7166513
Binford MiddleEnglish3-yr79
Binford MiddleMathematicsCurrent4036428
Binford MiddleMathematics3-yr64
Elkhardt MiddleEnglishCurrent767247
Elkhardt MiddleEnglish3-yr79
Elkhardt MiddleMathematicsCurrent4037322
Elkhardt MiddleMathematics3-yr59
Fred D. Thompson MiddleEnglishCurrent747313
Fred D. Thompson MiddleEnglish3-yr76
Fred D. Thompson MiddleMathematicsCurrent4440422
Fred D. Thompson MiddleMathematics3-yr62
Henderson MiddleEnglishCurrent737126
Henderson MiddleEnglish3-yr77
Henderson MiddleMathematicsCurrent2724335
Henderson MiddleMathematics3-yr59
Lucille M. Brown MiddleEnglishCurrent787628
Lucille M. Brown MiddleEnglish3-yr84
Lucille M. Brown MiddleMathematicsCurrent5150119
Lucille M. Brown MiddleMathematics3-yr69
Martin Luther King Jr. Middle EnglishCurrent6156516
Martin Luther King Jr. Middle English3-yr72
Martin Luther King Jr. Middle MathematicsCurrent1713440
Martin Luther King Jr. Middle Mathematics3-yr53
Richmond Alternative EnglishCurrent5053-31
Richmond Alternative English3-yr54
Richmond Alternative MathematicsCurrent2531-610
Richmond Alternative Mathematics3-yr41
Thomas C. Boushall MiddleEnglishCurrent8385-2-5
Thomas C. Boushall MiddleEnglish3-yr80
Thomas C. Boushall MiddleMathematicsCurrent5051-118
Thomas C. Boushall MiddleMathematics3-yr69

I don't know what to make of the negative boosts at Alternative and Boushall.
The average boosts for the elementary schools were yet smaller.

Boost 1Boost 3
Grade 3 - 5 English1.20.9
Mathematics1.619.5
But the effects were dramatic.
Sch NameSubjectAverage TypePass RateReport CardBoost1Boost3
Bellevue ElementaryGrade 3 - 5 EnglishCurrent949400
Bellevue ElementaryGrade 3 - 5 English3-yr94
Bellevue ElementaryMathematicsCurrent7373016
Bellevue ElementaryMathematics3-yr89
Blackwell ElementaryGrade 3 - 5 EnglishCurrent7476-21
Blackwell ElementaryGrade 3 - 5 English3-yr77
Blackwell ElementaryMathematicsCurrent4447-327
Blackwell ElementaryMathematics3-yr74
Broad Rock ElementaryGrade 3 - 5 EnglishCurrent949221
Broad Rock ElementaryGrade 3 - 5 English3-yr93
Broad Rock ElementaryMathematicsCurrent909005
Broad Rock ElementaryMathematics3-yr95
Chimborazo ElementaryGrade 3 - 5 EnglishCurrent868333
Chimborazo ElementaryGrade 3 - 5 English3-yr86
Chimborazo ElementaryMathematicsCurrent5251127
Chimborazo ElementaryMathematics3-yr78
Clark Springs ElementaryGrade 3 - 5 EnglishCurrent878610
Clark Springs ElementaryGrade 3 - 5 English3-yr86
Clark Springs ElementaryMathematicsCurrent908826
Clark Springs ElementaryMathematics3-yr94
E.S.H. Greene ElementaryGrade 3 - 5 EnglishCurrent928932
E.S.H. Greene ElementaryGrade 3 - 5 English3-yr91
E.S.H. Greene ElementaryMathematicsCurrent78671120
E.S.H. Greene ElementaryMathematics3-yr87
Elizabeth D. Redd ElementaryGrade 3 - 5 EnglishCurrent888710
Elizabeth D. Redd ElementaryGrade 3 - 5 English3-yr87
Elizabeth D. Redd ElementaryMathematicsCurrent5755224
Elizabeth D. Redd ElementaryMathematics3-yr79
Fairfield Court ElementaryGrade 3 - 5 EnglishCurrent939031
Fairfield Court ElementaryGrade 3 - 5 English3-yr91
Fairfield Court ElementaryMathematicsCurrent7573212
Fairfield Court ElementaryMathematics3-yr85
G.H. Reid ElementaryGrade 3 - 5 EnglishCurrent878343
G.H. Reid ElementaryGrade 3 - 5 English3-yr86
G.H. Reid ElementaryMathematicsCurrent5951829
G.H. Reid ElementaryMathematics3-yr80
George Mason ElementaryGrade 3 - 5 EnglishCurrent858234
George Mason ElementaryGrade 3 - 5 English3-yr86
George Mason ElementaryMathematicsCurrent6261115
George Mason ElementaryMathematics3-yr76
George W. Carver ElementaryGrade 3 - 5 EnglishCurrent85850-2
George W. Carver ElementaryGrade 3 - 5 English3-yr83
George W. Carver ElementaryMathematicsCurrent5350322
George W. Carver ElementaryMathematics3-yr72
Ginter Park ElementaryGrade 3 - 5 EnglishCurrent777528
Ginter Park ElementaryGrade 3 - 5 English3-yr83
Ginter Park ElementaryMathematicsCurrent3736134
Ginter Park ElementaryMathematics3-yr70
J.B. Fisher ElementaryGrade 3 - 5 EnglishCurrent898720
J.B. Fisher ElementaryGrade 3 - 5 English3-yr87
J.B. Fisher ElementaryMathematicsCurrent5857123
J.B. Fisher ElementaryMathematics3-yr80
J.E.B. Stuart ElementaryGrade 3 - 5 EnglishCurrent939210
J.E.B. Stuart ElementaryGrade 3 - 5 English3-yr92
J.E.B. Stuart ElementaryMathematicsCurrent7169216
J.E.B. Stuart ElementaryMathematics3-yr85
J.L. Francis ElementaryGrade 3 - 5 EnglishCurrent88880-4
J.L. Francis ElementaryGrade 3 - 5 English3-yr84
J.L. Francis ElementaryMathematicsCurrent5249323
J.L. Francis ElementaryMathematics3-yr72
John B. Cary ElementaryGrade 3 - 5 EnglishCurrent939212
John B. Cary ElementaryGrade 3 - 5 English3-yr94
John B. Cary ElementaryMathematicsCurrent6969016
John B. Cary ElementaryMathematics3-yr85
Linwood Holton ElementaryGrade 3 - 5 EnglishCurrent9293-1-4
Linwood Holton ElementaryGrade 3 - 5 English3-yr89
Linwood Holton ElementaryMathematicsCurrent7170111
Linwood Holton ElementaryMathematics3-yr81
Mary Munford ElementaryGrade 3 - 5 EnglishCurrent949224
Mary Munford ElementaryGrade 3 - 5 English3-yr96
Mary Munford ElementaryMathematicsCurrent8280212
Mary Munford ElementaryMathematics3-yr92
Miles Jones ElementaryGrade 3 - 5 EnglishCurrent87870-3
Miles Jones ElementaryGrade 3 - 5 English3-yr84
Miles Jones ElementaryMathematicsCurrent5047325
Miles Jones ElementaryMathematics3-yr72
Oak Grove/Bellemeade ElementaryGrade 3 - 5 EnglishCurrent787445
Oak Grove/Bellemeade ElementaryGrade 3 - 5 English3-yr79
Oak Grove/Bellemeade ElementaryMathematicsCurrent5046426
Oak Grove/Bellemeade ElementaryMathematics3-yr72
Overby-Sheppard ElementaryGrade 3 - 5 EnglishCurrent9091-1-2
Overby-Sheppard ElementaryGrade 3 - 5 English3-yr89
Overby-Sheppard ElementaryMathematicsCurrent5357-423
Overby-Sheppard ElementaryMathematics3-yr80
Patrick Henry School Of Science And ArtsGrade 3 - 5 EnglishCurrent858413
Patrick Henry School Of Science And ArtsGrade 3 - 5 English3-yr87
Patrick Henry School Of Science And ArtsMathematicsCurrent4848018
Patrick Henry School Of Science And ArtsMathematics3-yr66
Southampton ElementaryGrade 3 - 5 EnglishCurrent898722
Southampton ElementaryGrade 3 - 5 English3-yr89
Southampton ElementaryMathematicsCurrent6768-116
Southampton ElementaryMathematics3-yr84
Summer Hill/Ruffin Road ElementaryGrade 3 - 5 EnglishCurrent787355
Summer Hill/Ruffin Road ElementaryGrade 3 - 5 English3-yr78
Summer Hill/Ruffin Road ElementaryMathematicsCurrent6257521
Summer Hill/Ruffin Road ElementaryMathematics3-yr78
Swansboro ElementaryGrade 3 - 5 EnglishCurrent8587-2-3
Swansboro ElementaryGrade 3 - 5 English3-yr84
Swansboro ElementaryMathematicsCurrent5962-318
Swansboro ElementaryMathematics3-yr80
Westover Hills ElementaryGrade 3 - 5 EnglishCurrent818014
Westover Hills ElementaryGrade 3 - 5 English3-yr84
Westover Hills ElementaryMathematicsCurrent4645129
Westover Hills ElementaryMathematics3-yr74
William Fox ElementaryGrade 3 - 5 EnglishCurrent9596-1-4
William Fox ElementaryGrade 3 - 5 English3-yr92
William Fox ElementaryMathematicsCurrent868425
William Fox ElementaryMathematics3-yr89
Woodville ElementaryGrade 3 - 5 EnglishCurrent868600
Woodville ElementaryGrade 3 - 5 English3-yr86
Woodville ElementaryMathematicsCurrent5755226
Woodville ElementaryMathematics3-yr81

In all, five elementary schools (Greene, Fairfield, Stuart, Oak Grove, and Summer Hill) were saved by the one-year boosts while eighteen(!!) (Blackwell, Chimborazo, Redd, Reid, Mason, Carver, Ginter Park, Fisher, Francis, Cary, Jones, Oak Grove, Overby-Sheppard, Southampton, Summer Hill, Swansboro, Westover Hills, and Woodville) were spared by the 3-year boosts.  Oak Grove and Summer Hill are twofers, both accredited by way of one-year boosts (4 and 5 points) in English and 3-year boosts (26 and 21 points) in math.

The one-year and 3-year dichotomy again produced strange results.  Thus Holton scored 70 on the math test and was boosted to 71 (passing by one point).  Just down the street and around the corner, Ginter Park got a 36 on the math test but a 34 point 3-year boost transformed that to a passing score of 70.  Looking at the accreditation numbers, one might think that Holton and Ginter Park both did OK; in fact, Holton passed a tough test while Ginter Park failed miserably.
In terms of a graph, the English scores are seen to be boosted in most cases:
English Accreditation v. Report Card Scores
Oak Grove and Summer Hill (the rightmost red points) missed the 75 required of elementary schools but got "adjusted" into accreditation.  As we saw above, King and Binford (the leftmost red pair) also got "adjusted" above 70 and into accreditation.
The graph for the math test is more dramatic:
Math Accreditation vs. Report Card Scores
The 25 schools left of the red line and above the green line bombed the test but were accredited in math.  That's twenty-five of forty-six schools that scored below 70 but were accredited in math.
After the other subjects and the graduation requirement, nineteen Richmond schools were fully accredited despite bombing the math test and two (Oak Grove and Summer Hill) were fully accredited in spite of flunking scores on both the English and math tests.  Ginter Park leads the Parade of Horribles with a 36 math score "adjusted" to 70.

The "Adjustments"
8VAC20-131-300, in the Accreditation portion of the Standards of Quality regulation, provides, in relevant part:
C. Accreditation ratings defined. Accreditation ratings awarded in an academic year are based upon Virginia assessment program scores from the academic year immediately prior to the year to which the accreditation rating applies. Accreditation ratings are defined as follows:

1. Fully accredited.

* * *
c. With tests administered beginning in the academic year 2012-2013 for the accreditation ratings awarded for school year 2013-2014 and beyond, a school will be rated Fully Accredited when its eligible students meet the pass rate of 75% in English and the pass rate of 70% in mathematics, science, and history and social science. Additionally, each school with a graduating class shall achieve a minimum of 85 percentage points on the Board of Education's graduation and completion index, as described in 8VAC20-131-280 B 2, to be rated Fully Accredited.
A search of the Virginia Administrative Code for "pass rate" produces the following result:
8VAC20-131-300 (show hits) ..Application of the standards.. (5)
8VAC20-630-40 (show hits) ..Program evaluation.. (1)
9VAC5-91-180 (show hits) ..Exhaust emissions standards for on-road testing thr.. (1)
The first two are regulations of the Board of Education.  8VAC20-131-300, which launched this inquiry, is quoted in relevant part above.  8VAC20-630-40 provides
Each local school division shall annually evaluate and modify, as appropriate, their remediation plan based on an analysis of the percentage of students meeting their remediation goals. The pass rate on the Standards of Learning assessments shall also be a measure of the effectiveness of the remedial program.
Hmm.  That's not much help.
8VAC20-131-5 defines "Eligible Student":
“Eligible students” means the total number of students of school age enrolled in the school at a grade or course for which a Standards of Learning test is required unless excluded under the provisions of 8VAC20-131-30 F  and 8VAC20-131-280 D relative to limited English proficient (LEP) students.
The two exceptions there relate to students with disabilities, whose participation in the SOL testing is prescribed by the applicable IEP or 504 plan, and LEP students enrolled in Virginia schools for fewer than 11 semesters. 
The regulation at 8VAC20-131-280 creates the three-year average score option and allows the accreditation to be based (in unspecified part) on the "number of students who successfully complete a remediation recovery plan."
The regulation does not otherwise define the term "pass rate."  Standing alone, the regulation would seem to use the term in its usual sense of number passing divided by number taking.  Yet the VDOE Web page (quoted above) explicitly provides for "adjustments" to the pass rate that go considerable beyond the exceptions set out in the regulation.  The VDOE procedures for calculating the accreditation "pass rate" exclude the following test records from the calculation:
• Test records marked as cheating;
• Test records marked as Grade code “TT”;
• Test records of students attending non-public schools;
• Non-passing Retest;
• Non-passing 8th grade and EOC Recovery;
• Non-passing SOA LEP; and,
• Non-passing SOA Transfer students.
Under "Accreditation Benchmarks," the procedures further provide:
• Either the current year or the three-year average must meet the current year Board-approved
benchmark to be accredited in a given subject.
Rule:
• Passing Math and English Recovery tests scores for 4th through 8th grade and EOC will count twice.
• Failing recover tests scores for 4th through 7th grade count once.
When no scores are received for a given test, it will be indicated on the report as, “NS” (No score received for this subject area) and accreditation will be based on the other subject areas.
Exception
• Exclude failing 8th grade Recovery test scores, if the student has previously taken the same test in any school year (past or present).
Can you see VDOE's thumb on the scale here? 
  • Schools are encouraged to get their weaker students to cheat so those scores won't count.  Yet nothing in the regulation suggests that a cheating student is not an "eligible student."
  • Code TT is a "test taker" not in a grade and not post graduate.  Chuck Pyle tells me this could be a student who has completed the 12th grade but has to sit for the test to get the verified credits necessary to graduate.  In any case, a "TT" student can take the SOL until the computer wears out without effect on the school's accreditation.
  • Non-passing retests, apparently without any colorable authority in the regulation, can likewise be repeated without affecting the school's accreditation until the student gets a pass, whereupon the pass counts!
  • Eighth grade flunking "Recovery" and LEP failing scores don't count against the school while the 8th grade passing Recovery retests count twice!  And, to a lesser degree, in grades 4-7, flunking Recovery scores count once but passing scores count twice.  See this.
  • And, the Big One for this year's math test (albeit expressly allowed by the regulation), if the school doesn't make the "adjusted" grade on this year's test, VDOE will use a three year average (and, as we have seen above, boost a school's score by as much as 34 points).
All these "adjustments" go in one direction: Up.  All these adjustments are totally unnecessary: VDOE writes the test and can make it as easy or hard as they want.  The adjustments only contribute opacity to the process . . . and, as we see above, create opportunities for accreditation scores that are divorced from a school's actual performance.
Your tax dollars at work.