DEVELOPING SCORING RUBRIC : DO THE STUDENTS NEED IT ?

For many reasons, the use of scoring rubrics has acquired more attention. However, the number of studies investigating the perception and the writing performance as well as the relationship between those two variables in the use of rubric is limited. Therefore, by involving 27 students in Writing -class of an English Language Education Department at a particular university in Indonesia, the present study aimed to explore the students’ perceptions through surveys and their writing performance in the three different situations in which they had no scoring rubric, generate their own scoring rubric and had teachermade scoring rubric prior to the writing tasks. Also, the study was intended to explore how the students perceived the three situations related to their writing performance. After analyzing the data, it was found that the most helpful situation for the students was the second situation. It also helped the students to achieve the higher score compared to the other two situations. However, there was a contradictory finding in which the students’ perception and their performance were negatively correlated.


BACKGROUND
A scoring rubric is generally defined as an instrument that a teacher needs to assess students' performance.It is an assessment guide that describes evaluation criteria (or grading standards) based on the expected outcomes and performances of the students.Naturally, rubrics are utilized to advance the reliable use of learning outlook, learning goals, or learning models in the classroom, or to quantify their accomplishment against a steady arrangement of criteria (Hidden Curriculum, 2014).
In addition, rubrics are commonly used in scoring or grading written assignments or oral presentations and also provide a source for doing self-assessment, reflection, and peer review.As confirmed by Office of Graduate Studies Teaching at UNL (2017) scoring rubrics are especially appropriate for evaluating complex tasks or assignments such as: written work (e.g., assignments, essay tests, papers, portfolios); presentations (e.g., debates, role plays); group work; or other types of work products or performances (e.g., artistic works, portfolios).
Scoring rubrics are usually presented in a table which contains some aspects to be rated, quality descriptions for each aspect and a rating scale for those descriptions.Each rubric comprises of an arrangement of scoring criteria and point esteems related with these criteria.In many rubrics, the criteria are assembled into classes so the teacher and the students can separate among the classifications by the level of execution (Hricko & Howell, 2006).A rubric isolates the assigned work into categories and gives clear portrayals of the work related to every aspect, at different levels of performance (Karkehabadi, 2013).
Rubrics can be classified as either holistic or analytical.Holistic rubrics can be defined as scoring rubrics that provide a single score based on an overall impression of a student's performance on a task.A holistic scoring rubric comprises of a single scale with all criteria to be incorporated into the assessment being viewed together (e.g., content, organization, language use, and mechanics).By using this rubric, the rater doles out a single score (as a rule on a 1 to 4 or 1 to 6 point scale) in a view of a general judgment of the students' work (Jackson, 2017).It benefits the raters as it provides an overview of students' achievement (Lane, 1996).From a practicality viewpoint, it is faster (Alaei, Ahmadi, & Zadeh, 2014) and absolutely saves time by decreasing the number of judgment the raters make (Jackson, 2017).
In contrast, an analytical rubric measures various categories of the work independently in which its process of assessment tends to be more closed, restricted and detailed.Consequently, it possibly occurs a phenomenon which is one dimension of the work could be excellent, but on one or more other dimensions of the work might be poor to average.As noticed by (Çetin, 2011) and Lane (1996), analytical rubrics benefits both the raters and the students by giving more detailed feedback and access to self-assess in evaluating their strengths and weaknesses of their performance.However, it is very time-consuming to score compared to holistic.Additionally, based on the research conducted by Rezaei & Lovorn (2010) raters using an analytic rubric for writing assessment tends to concern on the mechanical characteristics of students' writing rather than the content itself.
The use of rubrics as the core instrument in assessment has been very popular today as they promote objectivity.However, the existence of the rubric itself in evaluating writing proficiency has been a challenge and a controversy among scholars (East, 2009;Rakedzon & Baram-Tsabari, 2017;& Sundeen, 2014).A number of previous studies suggest that scoring rubrics are essential since they bring a number of benefits for both students and teachers.Ene & Kosobucki (2016) state that criteria shared with the students in advance becomes integrated into instruction as a formative teaching tool.They can assist the evaluator to achieve intra-rater and inter-rater reliability in assessing students' writing as well.Further, rubrics are important since they make it clear for students about the qualities their work should be.It is often conveyed in terms of students' understanding of the learning target and criteria for success.As confirmed by Allen & Tanner (2006), rubrics are not only intended to formulate guidelines on what to achieve in a certain performance but additionally, they can be used to make these guidelines clear and specific to the students.For this reason, rubrics help teachers teach and manage instruction and assessment, and they support students to learn as well.
On the contrary, some argue that scoring rubrics are not crucial for students' success in terms of their writing.As stated by Matsuda & Jeffery (2012), rubrics cannot capture or encourage the development of voice.The rubrics can also restrict the student's mind power since they will feel that they need to complete the assignment strictly to the rubric instead of taking the initiative to explore their learning.Additionally, developing a rubric can be time intensive because there is a need to ensure that the students are aware of the expectations and to guarantee the intra-rater and inter-rater reliability (Labs, 2017).
While many studies have been completed to show the benefits and weaknesses of rubrics use in the instructional context especially in writing classes, very few investigated the effect of involving the students in developing scoring rubric prior to writing tasks.In the era in which student-centered approach is highly promoted, the students are not only expected to evaluate their performance, but they also need to decide the criteria as well as the tools to determine the score of their work.Andrade & Du (2005)investigate the students' perspectives regarding the regular use of rubrics, including co-creating rubrics in class, formal rubric-referenced selfassessment, and teacher feedback.The students stated that using rubrics helped them focus on their efforts, produce better quality work, get a better grade, and feel less anxious about an assignment.Additionally, Neill (2007) and Yoshina & Harada (2007) supported that students who are involved in the process of creating a rubric have a better understanding of what must be done to reach expectations.Similarly, Becker (2016)examines the effectiveness of developing and/or applying a rubric on the writing performance of adult English as second language learners.Using a pretest-posttest, control group design, his study resulted in a way that the holistic scores on the post-test summary writing task were significantly higher for those who participated in the development and/or application of the scoring rubric.
Regarding this shift of attention and the emphasis placed on the writing assessment, the research in the area of involving students in developing scoring rubrics compared to not using as well as using scoring rubrics in the instructional context is still relevant to be conducted.Given the findings mentioned previously, further research on students' perception and students' performance on the presence of scoring rubric in writing class is warranted.Therefore, this study incorporated three situations prior to the students' writing tasks: 1. having no rubric, 2. generating scoring rubric, 3. doing peer assessment by using a teacher-made scoring rubric.Furthermore, the study attempted to reveal the students' perception as well as their writing performance in those settings.Finally, the correlation between the students' perceptions and performances would be explored.
Accordingly, the present study sought to answer the following research questions: 1.How do the students perceive the situation in which they have no rubric?2. How do the students perceive the situation in which they are involved in developing a scoring rubric?3. How do the students perceive the condition in which they are involved in using a rubric to do peer-assessment?4. What are the effects of: (a) having no rubric, (b) generating scoring rubric, and (c) participating in peer-assessing by using a rubric on the students' writing scores? 5. How do the students' perceptions of the three situations correlate with their writing scores?

METHOD
The 27 students (5 males and 22 females) of Writing II in Class A participated in this current study.They are the 4 th semester students of English Language Education Department at Universitas Muhammadiyah Malang who joined Writing II class twice a week (on Wednesday and Friday Data collection took 11 days in total.The study employed surveys and a task that assigned the students to write a paragraph on three different situations that were (a) having no rubric, (b) generating a scoring rubric, and (c) participating in peerassessing by using a rubric from the lecturer.The survey was done by distributing a questionnaire with five-5-point Likert scale-type questions.The questionnaires were to reveal the students' perceptions associated with three different situation treatments prior to the writing task.The questions in the questionnaire covered the aspects of writing quality, expectations on the writing ability, writing process, feedback, and motivation.
In addition, to get the data of the students' writing score, the students were assigned to write a paragraph on each situation created by the writing lecturer.The paragraph was assessed by using scoring rubric made by the lecturer.
After all the data from the questionnaire and students' score had been collected, they were inputted to the spreadsheet.To answer the first, second and third research questions which were asking the students' perceptions of each situation, the study combined the five-5-point Likert-type scale questionnaire scores for a maximum of 25 points and displayed the data using bar charts.The survey data were also used to rank the three situations in order of preference.Later, the data were analyzed statistically by using Statistical Package for the Social Sciences Version 21.0 (SPSS).To determine the differences and to test all pairwise comparisons (situation 1&2; 1&3; 2&3), the study used a within-subjects ANOVA with Bonferroni's post hoc.
The study also used an experimental design for research question four as it was deemed necessary whether those three situations affected the students'writing performance.The statistical analysis by using ANOVA with Bonferroni's post hoc was used to examine the differences among the students' score on those different situations.Meanwhile, to obtain the answer for the research question five in terms of whether or not there was a relationship between students' perceptions and their writing performance, the study used Spearman's Rho correlation coefficient.Later, it determined a positive, negative, or zero correlation between those variables.

FINDINGS a. Research Questions 1, 2 and 3
To answer the research questions one to three, the descriptive statistics was presented as it provides the measurement of the students' perceptions in the situations of having no rubric, being participated in designing a rubric and using a rubric from the lecturer to peer-assessing prior to their writing tasks.The students' score was calculated with maximum points of 25 and the result was displayed using bar charts (figure 1,2 and3).To interpret the data, the score range of 1-5 is defined as strongly disagreement, 6-10 as disagreement, 11-15 as a neutral response, 16-20 as agreement and 21-25 as strong agreement.
Figure 1 illustratesthat17 students agreed and 5 students strongly agreed that although they did not know how their work would be evaluated, they could present how much they knew about writing a good paragraph.Only 5 outof 27 students positioned themselves in a neutral area.Furthermore, there were no students had a disagreement or strong disagreement with this condition.Figure 2 indicates that after being participated in designing rubric, the number of students who agreed that they can compose a better writing was increasing to be 22 students and 4 students gave strongly agree responses.Additionally, only one participant gave a neutral response and there was no respondent who felt disagreed and strongly disagreed.In Figure3, it can be found the students' response toward the third situation in which the students were peer-assessed by using scoring rubric from the teacher.It shows that there was still one participant in the neutral position.18 out of 27 students agreed and 8 students strongly agreed that this setting helped them to perform better.Overall, the response of disagreement and strong disagreement could not be found in this setting.To ensure the students' preference on those settings in rubrics uses, the statistical analysis using within-subject ANOVA was used.Then, the means of the students' perceptions in relation to having no rubric, generating rubric and using the teacher-made rubric to do peer-assessment were compared.Using Bonferonni's post hoc, it was found that the students' perceptions of being participated in designing rubric differed significantly from having no rubric in prior to the writing task (p=.027).However, when comparing the students' perceptions of having no rubric to using the teacher-made rubric, there was no statistically significant (p=.330).Similarly, the students' perceptions of the condition of having no rubric did not differ greatly from using the rubric in peer-assessing (p=.176).

b. Research Question 4
The purpose of research question 4 was to reveal the impact of those conditions in using the rubric to the students' writing performance.All the students' writings in those settings were assessed by using the scoring rubric from the teacher by two raters by a maximum point of 6.In order to guarantee the reliability of the students' writing score, Cronbach's alpha coefficient was calculated.The resulting scores were all at .984 showing that the reliability level of this research was high.
Further, the means of the students' score of the three different situations were calculated.The highest mean score was for the second situation in which the students were involved in generating scoring rubric before they were assigned to write that was 4.07.Meanwhile, the mean score of the students' writing when they did not know how their work would be evaluated was 3.59.This information is visually illustrated in figure 4.Although it was found that the highest score obtained by the students was from the second setting, a process of statistical analysis was still needed.The SPSS calculation using ANOVA Bonferroni's post hoc when investigating the students' perception was done similarly to analyze the students' performance to ensure whether or not those three conditions affect their writing performance.The result showed that there was no significant different (p=.341) in the score of the students when they had no rubric and participated in designing rubric.In the same way, the comparison between the students' score in having the absence of rubric and the rubric from the teacher before they composed a paragraph also showed no meaningful difference (p=.191).The other insignificant difference (p=.790) was also indicated from the students' score in the second setting which they were involved to make rubric compared to the third setting which they were given a rubric to peerassessing.These results suggest that the students had comparable writing proficiency although they were on different situations.

c. Research Questions 5
This research question was about the correlation between the students' perceptions of each alternative use of rubrics and their writing performance on each situation.In order to reveal the possible relationships, a Spearman's Rho correlation coefficient was calculated to assess.The visual illustrations on the students' perception and their writing score of a particular condition later were presented in figures 5,6, and 7.
After being analyzed, in the first situation in which the students did not know what considerations used by the lecturer to evaluate their writing performance, there was a positive correlation between the students' perception and their scores, p=.095 with a medium effect size of .328.As the score increased, positive perceptions of having no rubric also increased and so did the reverse.The result is displayed in Figure 5 with a scatter plot.With regard to the students' involvement in designing a scoring rubric prior to their writing task, there was a weak negative correlation between the students' score and their perceptions with a small effect size -.147.Some students positively perceived their participation in generating a rubric but had average or low scores while the others viewed it negatively but had high scores.The following, Figure 6, summarizes the results with a scatter plot.In the last setting in which the students' were involved in doing peer assessment using a rubric designed by the lecturer, it showed a weak positive relationship of the students' writing score and their performance with a small effect size of .274. Figure 6 indicates the relationship existed between those aspects.From the illustration through the scatter plot, it can be inferred that some students positively perceived the use of a rubric from the lecturer and achieved high scores, while the others felt the same way but had average or low scores.

DISCUSSION
The purpose of the current study was to investigate the students' perceptions and performance as well as the correlations between those variables about three different conditions prior to the writing tasks.Those conditions were having no rubric, being involved in generating a rubric and using a lecturer made rubric in doing peer-assessment.
After answering the research question 1, 2 and 3, it was found that there was a significant difference between the students' perception of the first and second situation.The result also showed that there was no great difference between the first and the third condition as well as the second and the third.It indicated that among those situations made by the lecturer before assigning the students to write a paragraph, the best-perceived setting was when the students participated in developing scoring rubric to evaluate their writing performance.
Furthermore, it was also found that although there was no significant difference among the students' writing performance in those three different situations.Despite the fact that there was no statistically significance, the students' writing performance in the second setting obtained the highest score.When inviting students to create rubrics at the beginning of an activity, it helps them plan to write better (Yoshina &Harada, 2007 andNeill, 2007) as well as enhance their motivation and interest (Vision, 2017).When students are involved in rubric construction, the assignment itself becomes more meaningful to the students (Lane, 1996).In addition, Becker (2016)also revealed that there is a positive effect on the writing performance of ESL learners when they are involved to design and apply a scoring rubric.When comparing the students' perceptions about their performances on each situation, a moderate relationship existed between the students' perception of having no rubric and their performance.As positive perceptions increased, so did the students' scores.Meanwhile, a negative correlation between the students' perceptions and their scores occurred in the situation of the students' involvement in designing a rubric.It means that the students positively perceived this condition but they had low scores.The correlation analysis also revealed that the relationship between the students' perceptions and their score in the third condition had a weak positive relationship.
One thing that becomes a concern in this study is that the situation in which the lecturer invited the students to design the rubric was the best-perceived situation for the students.Also, this condition showed that it enhanced the students to achieve the higher score compared to the other two conditions.However, there was a contradictory fact in terms of their correlation.A negative relationship between the students' perceptions and the students' writing performance occurred.It happened as there were some students positively perceived their participation in designing a rubric but they had average or low scores.Additionally, the other students thought that this situation could not help them to write better but they could achieve high scores.

CONCLUSION
The findings led the conclusion that the best-perceived situation for the students was when the students were involved in designing a rubric before the lecturer assigned them to write a paragraph.Although there was no significant number of difference in terms of students' performance in those three different situations, the highest score the students achieved were in the second condition when they experienced themselves in deciding criteria to evaluate their performance.However, there was a fact that both variables (the students' performance and their score) are negatively connected.It occurs as those who positively believed that this activity can enhance them to make a better writing had low or average scores.Moreover, the students who could achieve the high scores were those who were not confident that they could improve their writing after they participated in generating a rubric.

Figure 1 .
Figure 1.The Students' Perception of Having No Rubric.

Figure 2 .
Figure 2. The Students' Perception of Generating Rubric

Figure 3 .
Figure 3.The Students' Perception of Using Rubric

Figure 4 .
Figure 4.The Comparison of the Students' Mean Score Figure 5.The Correlation between the Students' Score and the Students' Perceptions of Having no Scoring Rubric

Figure 6 .
Figure 6.The Correlation between the Students' Score and the Students' Perceptions of Developing a Scoring Rubric Figure 6.The Correlation between the Students' Score and the Students' Perceptions of Using a Scoring Rubric