COSSET has undergone a system trial in March 1998 and used extensively in 3 subsequent course evaluation exercises. The figures in the table below summarized the results of the trials.
 
Response rate
The overall response rate has been consistently at around 75%. This is slightly higher than the average response rate for that of the paper-based system ( around 70%).
 
Students' views about COSSET
Ever since COSSET was first put to trial in March of 1998, effort has been made to collect students' views on the system. Every time a student completes an evaluation online, his/her opinions on the system will be asked. The data collected so far indicated that students in general preferred the online system to the paper-based system.

Student Evaluation of Teaching in HKUST using COSSET	Number of classes involved	Number of student-evaluations involved	Response rate	Percentage of responses in favour of COSSET
March 1998	11	1140	88%	72%
May 1998	16	1025	73%	-
October 1998	15	704	80%	73%
December 1998	234	12136	75%	56%(66%)*

 
In the table above, the number of student evaluations refers to the possible number of responses from students. Since students could be asked to evaluate more than one class, this figure is always greater than the number of students participating. The last column of the table indicates the percentage of received responses which are in favour of COSSET. The figures clearly show that students are in general quite positive about the system.

The above method to collect students' views is restricted to those who responded to online evaluation. A more in-depth survey was conducted after the first trial in May 1997 to collect students' feedback about the system using paper questionnaire. This allowed the project team to collect the views of students who did not respond to COSSET. Results^** from the survey basically corroborated the findings collected online. Some of the salient findings are as follows:

• 72% preferred the on-line system to the paper-based system.
• 65% found the log-on method, which requires them to use their usernames and passwords, acceptable. Only 7% objected to the arrangement.
• 64% said the log-on method would not affect the way they responded to the evaluation. Only 4% said they would not do the evaluation because of the log-on method.
• 72% found the system easy to use and only about 1% said they had great difficulty.

Instructors' views about COSSET
Two surveys were conducted to solicit instructors' views about COSSET. The first one involved 21 instructors who participated in the first two trials. In the second one, the questionnaire was sent to all instructors who participated in the December 1998 exercise (about 120 instructors). 47 of them responded. Since results from both surveys are very similar, only the results from the later one is reported here.

There is unanimous agreement among instructors surveyed that COSSET is a very convenient method to conduct teaching evaluation. 67% of them indicated that they would like to conduct all of their teaching evaluation using COSSET. However that does not mean that they have no reservation about the system. Their major concern is that a web-based system allows any student enrolled in a class to do the evaluation, even if s/he has never attended it. Such students would have little information on which to base their opinion about the class. 65% of them expressed such a concern. Another concern expressed by some instructors (25%) is that online teaching evaluation will give them lower ratings.
 
Comparison with paper-based system
Whether teaching evaluation data collected using an online system like COSSET is comparable to data collected using paper forms in classes is a question of practical importance and theoretical interest. Data collection using COSSET is different from that using paper forms in classrooms in at least three aspects.

• The working environment is different. Students used to complete questionnaire in classrooms under supervision and in the company of their classmates. When students use COSSET to respond, they could be anywhere, in any kind of groupings and most probably without supervision.
• COSSET requires students to go through a login procedure prior to doing the evaluation. This as suggested above could pose a threat to students' anonymity.
• COSSET does not discriminate against students who never turned up in classes. All students enrolled in a course have equal chance of making a response. That probably would not be the case when paper questionnaires are used to collect data in class.

In March, 1998 when COSSET was put to its first trial, students in 14 classes were split into two random halves, one group doing the evaluation using COSSET, the other using paper questionnaires in class. Using the data thus collected, a comparison was made for each class separately.

To examine if the distributions of answer options which students chose ( A, B, C, .. ) for each question for the two groups are comparable, Chi Square test was used. The results in fig. 3 below shows that no significant appears in any of the questions in 6 of the 14 classes. In the remaining 8 classes, significant difference appears in only 1 to 2 questions.

For some questions (e.g. Likert type questions), the answers are often converted to numerical values using pre-specified numerical weights. These values are then averaged to give numerical ratings. For these questions, a t-test was used to compare the ratings calculated based on data from the two sources. The results of the t-test again showed that the mean ratings are quite comparable.

Classes	Number of responses received using COSSET	Number of responses received using paper questionnaires	Total number of questions used in questionnaire	Number of questions with significant difference in response distribution (Chi-Square test)	Number of questions with significant difference in the mean ratings (t-test)
A	34	25	10	0	1
B	27	17	10	0	2
C	27	18	27	1	2
D	26	24	27	2	0
E	19	24	4	1	1
F	22	21	14	2	2
G	50	23	12	0	2
H	75	58	12	1	2
I	66	24	12	2	1
J	5	4	19	1	1
K	22	12	12	1	1
L	23	12	12	0	0
M	23	19	12	0	0
N	18	12	12	0	0

Comparison results using Chi-square test and t-test (level of significance = 0.05 )

A similar kind of study was conducted last December. Students from 3 classes were involved. The results once again showed that the evaluation results obtained using data from online are very similar to those from paper questionnaires. Significant difference was found only in 1 out of 20 cases (level of significance = 0.05).

*Of the 9063 responses received, in about 14% of them, the students did not answer the question to express their preferences. In previous trial, such percentage is very low.  The reason for the high percentage is probably because some of the students who have to do more than one evaluation online, chose to ignore the questions since they had already responded to them in an earlier evaluation.  The figure in the bracket is the percentage calculated if these cases were excluded.