Thursday, August 27, 2015

PASSING STANDARDIZED ASSESSMENTS WITH FADING PROMPTS

Amy Marie GREENE

Clarion University
Northcentral University, USA

Recived: 27.08.2014
Accepted: 22.10.2014
Original article

Citation: Greene AM. Passing standardized assessments with fading prompts. J Spec Educ Rehab 2015; 16(1-2): 68-84. http://dx.doi.org/10.1515/JSER-2015-0005



Introduction
 
 
Through passing No Child Left Behind Act of 2001 (NCLB), the United States Federal Government mandated that states meet certain requirements and develop state assessments to evaluate whether all students are making progress to a level of proficiency. As a result of NCLB, the members of the Pennsylvania Department of Education instituted the Pennsylvania System of School Assessment (PSSA), which evaluates writing based on prompts using a rubric. According to NCLB, measurable yearly benchmarks, which are referred to as Adequate Yearly Progress (AYP), must be set to ensure that 100% of students are performing at a level of proficient by the 2013-2014 school year. The supporters of NCLB mandate that all students, including those with learning and intellectual disabilities, perform at a level of proficient on grade level state assessments with only small group settings as an acco­mmodation (1–4).These students are expected to meet the same cut score levels as their nondisabled grade level peers. (4) Students with learning and intellectual disabilities are by nature at risk for not mee­ting a level of proficient due to their com­mon characteristics of deficits in the areas of computation, strategy development, me­mory, self-regulation, motivation, and ge­ne­ra­lization to state writing assessments (5–7).
Proficiency on state assessments, such as the PSSA, is a concern for administration, teachers, parents, and students. If AYP is not met funding is reduced and government involvement is instilled within the school districts at varying levels depending on the number of consecutive years the goal is not met (2). Members of the IEP subgroup who have a disability in the area being measured are clearly at a higher risk for not performing at a level of proficiency (8). The issue of passing statewide assessments, such as the PSSA, is becoming even more important since some states already do not permit students who do not pass the test to receive a diploma (9).
Although there has historically been a controversy between advocates for cognitive and behavioral approaches to teaching, NCLB has caused it to be even more pressing. Due to NCLB, teachers need to find the most effective strategy for assisting students with disabilities in passing state assessments. Directly teaching the skills through behavioral techniques has been deemed successful for students with learning and intellectual disabilities due to their common areas of deficits (10–18). Although the research has shown behavioral approaches’ successfulness for students with learning and intellectual disabilities, the advocates for cognitive approaches continue to criticize behavioral approaches for being too rigid (11, 19).
A substantial amount of research on teaching writing over the last several decades has been based on cognitive approaches, such as product, process, and Self-Regulated Strategies Development (SRSD). A large portion of this research was conducted by Graham (20) or replicated his work. Early on Graham (20) has found difficulties in imp­lementing cognitive strategies for students with learning and intellectual disabilities, which included deficits in composition, mechanics, and motivation (20–23). A review of the 30 years of SRSD research allowed re­searchers to find differences in strategies be­ha­vior, writing skills, knowledge, and motivation that led to difficulties utilizing this cognitive approach for students with learning and intellectual disabilities (6). Of the quantity of research on SRSD, only five experi­mental or quasi-experimental designs met the criteria for being acceptable research, while only nine single-subject designs were deemed of quality (24). None of the researchers who conducted studies on cogni­tive strategies addressed generalization of these skills to standards-based state assess­ments in writing.
Through reviewing the research on both cognitive and behavioral approaches, it can be determined that a behavioral approach for instruction in the use of strategies that provide explicit, teacher-directed instruction for all levels of the writing process is an essential component in teaching students with learning and intellectual disabilities to learn to write and generalize these skills to a proficient level on state assessments (7, 25). Although there are no other studies speci­fically addressing Fading Prompts through Graphic Organizers method (FPGO), due to the author creating the program, there is substantial research on other behavioral approaches that utilize graphic organizers. Researchers have found significant gains in writing for students with learning and intellectual disabilities as measured by Correct Word Sequencing (CWS), and the standardized TOWL-3, and maintenance of these skills through the use of the behavioral approach(16–18).
Due to NCLB there is a need for researchers to further address generalization of learned skills to state assessments. Researchers found that through the behavioral approach of fading prompts in graphic organizers, three students with learning disabilities were able to advance from below basic (1) to a level of proficient (3) on the PSSA, while two did not pass the assessment; they did advanced from a score of below basic (1) to basic (2) (26). Researchers found that all the common characteristics of students with learning and intellectual disabilities needed to be addressed in order to assist them in passing state assessments (27–29). Researchers have also found that when explicit instruction, such as utilized in a behavioral approach, is provided to students with learning disabilities they can perform at the same level as their nondisabled peers, maintain these skills over time, and generalize these skills to state assessments (30). The purpose of this quantitative study was to examine the performance scores on the PSSA writing prompts assessment following FPGO as a treatment for students with learning and intellec­tual disabilities by comparing archi­ved pretest, posttest, and actual PSSA results to determine if significant differences existed. The participants’ PSSA results were also compared to the average state PSSA results for the IEP subgroup. As more is learned about the effectiveness of FPGO, schools may use this information to assist students in passing state assessments in the area of writing.

Materials and Method
The sample population was taken from a small town located in northwestern Pennsylvania. The sample size included a total of 45 students, ranging in age from 13 to 18 years old, who were placed in the learning support English setting by the IEP team in 8th or 11th grade (PSSA testing grade levels) for the 2005-2010 school years. The sampling included all students labeled with a learning disability in writing or an intellectual disability who were exposed to FPGO treatment, which was an inclusive group. In 2005-2006, the sample included 7 students in the 8th grade. During 2006-2007, 7 students in the 11th grade were included. In 2007-2008, three 8th grade students were included and 11 students in 11th grade participated. In the 2008-2009 school year 7 students in the 11th grade were included. During the 2009-2010 school year 10 students in the 11th grade were involved. The demographic characteristics of the 45 students consisted of 34 students with a learning disability, which represents 76%, and eleven with an intellectual disability, which represents 24%. Thirteen of the students were female, while 32 were male. Racially, 100% of students were Caucasian, due to 99% of the population within the school district being Caucasian.
The researcher utilized prompts selected from the PSSA writing assessment preparation book and the PSSA scoring rubric, as well as the results of the archived pretests, posttests, and the results of the actual PSSA for the 2005-2010 school years. The pretest and posttest results were determined through the use of the PSSA rubric as a measurement instrument to determine if students’ writing was at a score of 1- 4. The scale indicated below basic (1), basic (2), proficient (3), or advanced (4), as adopted in 1999 by the members of the Pennsylvania Department of Education. (31) A score of proficient or advanced is considered a passing score. Reliability for the PSSA was addressed through a stratified coefficient alpha, standard errors of measure (SEM), conditional standard errors of measure (CSEM) with the Rasch, decision consistency, and rater agreement. In regards to validity, the PSSA addressed the following: (a) test content, (b) response processes, (c) internal structure (d) the relationship between test scores and other variables, (e) the consequences of testing. (31)
Prior to the implementation of the FPGO as a treatment, students were presented with a PSSA writing prompt that was taken from a PSSA preparation book, which included narrative, informative, and persuasive prompts. Each student’s response was evaluated through the use of the rubric that was utilized within FPGO in order to provide specific feedback to students. The PSSA rubric was then utilized to derive a score of below basic (1), basic (2), proficient (3), or advanced (4). The scores were then reported as students’ baseline data due to the need to have the pretest and posttest scores reflect the same assessment tool, as the PSSA.
The teacher then utilized FPGO as classroom instruction. FPGO provided a graphic organizer that contained twenty-five boxes representing five paragraphs with at least five sentences in each. The method results in completing a writing response that consists of an introductory paragraph, three paragraphs as the body, and a closing paragraph. Students were provided with the FPGO step 1, which contains the most explicit set of prompts. As students became familiar with the process, prompts were slowly removed and the students were presented with FPGO step 2, which removes the names of the paragraphs, the prompting for the introductory sentence, and names of the topic and supporting sentences. When students reached a level of mastery, they were given FPGO step 3, which faded the prompting by removing all prompting through words, and left only numbers and letters. As skills continued to develop, students were presented with FPGO step 4, which provided them only empty boxes.
Once mastery of the skills had been met, students were provided with three pieces of blank typing paper and were expected to develop the graphic organizer independently by drawing the boxes. Having the students create their own graphic organizers was essential due to PSSA administration guidelines not allowing students any supplemental aids other than blank paper. Students progressed through the fading prompts steps at different speeds, but all students were expected to reach a level of proficiency.
One writing prompt was completed weekly to ensure retention. After exposure to FPGO and prior to the PSSA assessment, samples of the students’ written responses were evaluated by the researcher and three other trained teachers with the PSSA rubric. The teachers consistently derived the same scores. These data were reported as posttest data. A quantitative design was used to test if significant differences occurred between performance scores on the PSSA writing assessment following FPGO for students with learning and intellectual disabilities. The archived data used in this study was the result of a manipulation of the independent variable by presenting the extra stimulus of graphic organizers.
Four dichotomies for percent differences were utilized to determine if significant differences occurred in PSSA results after the implementation of FPGO. The first dichotomy compared the teacher ad­minis­ter­ed pretests to the teacher administered posttests for the 2005-2010 school years. The second dichotomy compared the teacher administered pretests to the actual PSSA results to determine if differences existed. Rater reliability and generalization of the lear­ned skills by the students were then addressed through a dichotomy of percent differences that compared the teacher administered posttests to the actual PSSA results. The fourth dichotomy compared the PSSA state administered local results for the students that received FPGO to the average PSSA pass and non-pass rates for the entire state of Pennsylvania’s IEP subgroups. Rater reliability was addressed through having the researcher re-score pretest and posttest data at a later time, having three other teachers also score the data, and comparing the archived posttest scores to the actual PSSA scores. As indicated, the PSSA rubric addresses reliability through a stratified coefficient alpha, standard errors of measure (SEM), conditional standard errors of measure (CSEM) with the Rasch, decision consistency, and rater agreement and validity through (a) test content, (b) res­ponse processes, (c) internal structure (d) the relationship between test scores and other variables, (e) the consequences of testing. (31).

Results

All 45 students received a below basic (1) on their archived pretest and a proficient (3) score on their archived posttest. Forty-three students earned a score of proficient (3) on the actual, archived PSSA assessment; while two students received a score of basic (2) (see Table 1, for archived pretest, posttest, and PSSA scores). As indicated earlier, the reliability and the validity of the actual PSSA assessment were determined by the Pennsylvania Department of Education.

Table 1.Pretest, Posttest and PSSA results 
 

Four dichotomies for percent differences were carried out. The results were calculated by subtracting the difference in percent between the two columns in either row. The outcomes of the first dichotomy for percent differences, which compared the percent of pass and non-pass rates when comparing the teacher administered pretests and posttests for the students with learning and intellectual disabilities that received FPGO for the 2005-2010 school years, resulted in FPGO making a 100% difference (100% - 0%=100%) in passing the PSSA, or not passing the PSSA (see Table 2, for dichotomy for percent differences for the pretest and posttest data).

Table 2.Dichotomy for percent differences in pretest and posttest data

Therefore, it was determined that FPGO was effective in assisting the students with learning and intellectual disabilities in passing the standards-based state assessment of the PSSA at a significance level of 0.001, or 99.99% confidence level. The second dichotomy, which compared the percent of pass and non-pass rates through comparing the teacher administered pretests to the actual state administered PSSA, analysis showed that FPGO made a 96% difference (100% - 4% = 96% and 96% - 0% = 96%) in passing the PSSA, or not (see Table 3, for dichotomy for percent differences for pretest and PSSA data).


Table 3.Dichotomy for percent differences for pretests and PSSA data

The third dichotomy, percent differences analysis, addressed the teacher administered posttests and compared them to the actual state administered PSSA results. The results found a 4% difference (4% - 0%=4% and 100% - 96%=4%), which indicated rater reliability and generalization of the learned skills by the students to the actual PSSA (see Table 4, for the dichotomy for percent differences for posttest and PSSA data).

Table 4.Dichotomy for percent differences for posttest and PSSA data

The fourth dichotomy compared the PSSA posttest state administered local results for the students that received FPGO to the average PSSA pass and non-pass rates of the entire state of Pennsylvania. The fourth dichotomy resulted in FPGO making a 53.8% (57.8% - 4%=53.8% and 96% - 42.2%=53.8%) difference for students with IEPs, which indicates learning and intellecttual disabilities, in passing the PSSA, or not (see Table 5, for dichotomy for percent differences for local and state PSSA data).

Table 5.Dichotomy for percent differences for the local and state PSSA data  

The PSSA data that were utilized to calculate the overall non-passing percent for the entire state of Pennsylvania when addressing the IEP subgroup was utilized in order to be the most reflective of the sample addressed in this study. It is important to indicate that the state’s IEP subgroup included all students with an IEP. Therefore, students that did not have deficits in writing were included; while this study’s sample addresses only students with an IEP reflecting deficits in written expression. Of the 188.212 students with an IEP in the state of Pennsylvania, 57.8% did not pass the PSSA (see Table 6, for PSSA data utilized to calculate overall average non-passing percent).

Table 6. PSSA data utilized to calculate the overall average of non-passing percent  


The PSSA data that were utilized to calculate the overall passing percent for the entire state of Pennsylvania when addressing the IEP subgroup, again reports all students with an IEP, not just students with disabilities in written expression. Although this included students who did not have deficits in writing, only 42.2% passed the PSSA (see Table 7, for PSSA data utilized to calculate overall average passing percent).

\Table 7. PSSA data utilized to calculate the overall average of passing percent  


Based on the outcomes of the four dichotomies for percent, it was determined that significant differences in performance scores on the PSSA writing prompts assessment existed following FPGO treatment for students with learning and intellectual disabilities through comparing archived pretest, posttest, and actual PSSA results. It was further determined that the outcomes of FPGO generalized to the PSSA.

Discussion


Although the supporters of NCLB have placed a great deal of emphasis on performing at a level of proficient on state assessments, no studies address the effectiveness of cognitive approaches generalizing to state assessments, few studies have addressed the effectiveness of behavioral approaches generalizing to state assessments, such as the PSSA, and no other studies have addressed the use of FPGO. This research provides additional insight into the effectiveness of behavioral approaches to teaching writing, as well as addresses how effectively these skills generalize to state assessments, such as the PSSA. Only 2 of the 45 students in this study did not pass the PSSA. However, both students advanced from a score of below basic (1) to a score of basic (2). It is also important to indicate that the 2 students who did not pass were identified as having an intellectual disability. The current review of literature excluded these students.
The major limitations within this study focus around the sampling procedures and the research design. The sampling procedures were based on accessibility and convenience and did not include random sampling. True random sampling did not occur since the testing group was established based on students being identified with a learning or intellectual disability and placed in the learning support setting for their English instruction by the IEP team and parent consent. Convenience sampling led to including only students in one school district, which resulted in all of the students within the sample population being Caucasian. Given that the PSSA is only administered to middle and high school students in 8th and 11th grades, the results of the FPGO were not tested on any other grade levels of students.
The structure of the study resulted in only one teacher implementing FPGO treatment for teaching writing. Although this controlled for treatment fidelity, which refers to following the exact procedures specified by the researcher, this also leads to questioning whether another teacher would have the same success with FPGO. It would be beneficial to generalize the outcomes of this study to the entire state of Pennsylvania in order to increase school districts’ AYP to comply with NCLB; however, the lack of random sampling makes generalization difficult. Comparing the local state administered PSSA results to the entire state of Pennsylvania’s IEP subgroups’ non-pass and pass percent also led to comparing students with disabilities in writing at the local level to students that may not have a deficit in writing in the state’s IEP subgroup.
There were further limitations due to the research design. In the quasi-experimental research design, the researcher is attempting to identify cause and effect by controlling the independent variable. Although the research was collected over a span of several PSSA testing years, limitations of not having a control group or returning to baseline as a means of comparison make it difficult to insure that the independent variable was controlled. The research design did not allow for return to baseline due to the fact that the students would automatically utilize the learned writing skills that were developed during the treatment phase.
Further studies should address sampling limitations by conducting a larger scale study to include several school districts from other demographic areas. The inclusion of numerous school districts may also be needed to create a stratified sample to represent other ethnic groups. Additional studies should also address the effect of FPGO on students in 7th, 9th, and 10th grades.
Further studies should also be conducted to address any structural limitations within this study. Larger scaled studies should evaluate the effectiveness of FPGO treatment when different teachers with varying backgrounds implement the treatment. In order to further explore the amount of control that was exhibited over the independent variable of adding an extra stimulus of graphic organizers to the prompts, additional research should be conducted to reflect the use of a control group. The results of this program should be compared to other strategies for teaching writing, such as cognitive approaches that are currently being utilized.

Conclusion

Given that the only two students who did not pass the PSSA with a score of proficient (3) were identified as having intellectual disabilities, additional studies that compare students with learning disabilities versus students with intellectual disabilities may provide further insight into the effectiveness of the program. Additional research should also be conducted on the effectiveness of FPGO treatment with students who do not have disabilities. Based on the outcomes of this study, which indicate that FPGO treatment led to significant differences between performance scores on the PSSA writing assessment for students with learning and intellectual disabilities, it is highly recommended that this program be utilized at least for students with learning and intellectual disabilities until further research can be done.

Conflict of interests
Author declare that have no conflict of interests.

References

 
  1. No Child Left Behind Act of 2001, Pub. L. No. 107-110, § 115, Stat. 1425 (2002).
  2. Yell ML. The law and special education. Upper Saddle River, NJ: Prentice Hall; 2006.
  3. PSSA Coach. New York, NY: Triumph Learning; 2007.
  4. Pennsylvania Department of Education. Accommodations guidelines: PSSA [online]. 2010 [cited 2011 Mar 1]. Retrieved from: URL:http://www. education.state.pa.us/
  5. Eisenmajer N, Ross N, Pratt C. Specificity and Characteristics of learning disabilities. Journal of Child Psychology 2005; 46(10):1108–1115.
  6. Graham S, Harris KR. Almost 30 years of writing research: making sense of it all with the wrath of khan. Learning Disabilities Research & Practice 2009; 24(2):58–68.
  7. Hallahan DP, Kauffman JM. Exceptional learners: An introduction to special education. Boston, MA: Pearson; 2006.
  8. Swain KD. Students with disabilities meeting the challenge of high-stakes writing assessments. Education 2006; 4(126):660–665.
  9. Yell ML, Katsiyannas A, Shiner JG. The no child left behind act, adequate yearly progress, and students with disabilities. Teaching Exceptional Children 2006; 4(38):32–39.
  10. Ganz J, Flores M. The effectiveness of direct instruction for teaching language to children with autism spectrum disorder: Identifying materials. Journal of Autism and Developmental Disorders 2009;39:75−83.
  11. Hang-Li Z. Investigation and analysis of current writing teaching mode among English majors in normal universities in China. US-China Education Review 2010; 8(7):22–27.
  12. Kauffman JM, Hallahan DP. Special education: What it is and why we need it. Boston, MA: Pearson; 2005.
  13. Martella R, Waldron-Soler K. Language for writing program evaluation. Journal of Direct Instruction 2005; 5(1):81–96.
  14. Stebbins LB, Pierre RG, Proper EC, Anderson RB, Cerva TR. Education as experimentation: A planned variation model. Cambridge: Abt Associative; 1977.
  15. Vitale M, Joseph B. Broadening the institutional value of direct instruction implemented in a low-SES elementary school: Implications for scale-up and school reform formal. Journal of Direct Instruction 2008; 8(1):1–18.
  16. Walker BD, Shippen ME, Alberto P, Houchins DE, Cihak DF. Using the expressive writing program to improve the writing skills of high school students with learning disabilities. Learning Disabilities Research & Practice 2005; 3(20);175–183.
  17. Walker BD, Shippen ME, Alberto P, Houchins DE, Cihak DF. Using the expressive writing program to improve the writing skills of high school students with learning disabilities. Journal of Direct Instruction 2006; 6(1):35–47.

  1. Walker BD, Shippen ME, Houchins DE, Cihak DF. Improving the writing skills of high school students with learning disabilities using the expressive writing program. International Journal of Special Education 2007; 2(22):66–76.
  2. Yi Y. Toward an eclectic framework for teaching EFL writing in a chinese context. US-China Education Review 2010; 3(7):29–33.
  3. Graham S. The role of production factors in learning disabled students’ composition. Journal of Educational Psychology 1990; 82:781–791.
  4. Graham S, Harris KR, MacArthur C, Schwartz S. Writing and writing instruction with participants with learning disabilities: A review of a program of research. Learning Disabilities Quarterly 1991; 14:89–114.
  5. Morris NT, Crump DT. Syntactic and vocabulary development in the written language of learning disabled and non-disabled participants at four age levels. Learning Disabilities Quarterly 1982; 5:63–172.
  6. Thomas C, Englert C, Gregg S. An analysis of errors and strategies in the expository writing of learning disabled participants. Remedial and Special Education 1987; 8(46):21–30.
  7. Baker SK, Chard DJ, Ketterlin-Geller LR, Apichatabutra C, Doabler C. Teaching writing to at-risk students: The quality of evidence for self-regulated strategy development. Exceptional Children 2009; 3(75):303–319.
  8. Troia G, Graham S. The effectiveness of a highly explicit, teacher-directed strategy instruction routine: Changing the writing performance of participants with learning disabilities. Journal of Learning Disabilities 2002; 35:290–305.
  9. Deatline-Buchman A, Jitendra AK. Enhancing argumentative essay writing of fourth-grade students with learning disabilities. Learning Disabilities Quarterly 2006; (29): 39–54.
  10. Garcia-Sanchez JN, Fidalgo-Redondo R. Effects of two types of self-regulatory instruction programs on students with learning disabilities in writing products, processes, and self-efficacy. Learning Disabilities Quarterly 2006; 3(29):181–211.
  11. Graham S, Perin D. A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology 2007; 99:445–476.
  12. Monroe BW, Troia GA. Teaching writing strategies to middle school students with disabilities. Journal of Educational Research 2006; 1(100):21–33.
  13. Schumaker JB, Deshler DD. Adolescents with learning disabilities as writers: Are we selling them short? Learning Disabilities Research & Practice 2009; 24(2):81–92.
  14. Pennsylvania Department of Education. The 2009 PSSA writing state level proficiency results. [online] 2009 [cited 2010 June 1] Retrieved from: URL:http://www.education. state.pa.us/

 


No comments:

Post a Comment