FormativeEvaluation

=Formative Evaluation and Revisions Report=

I. Introduction The goal of this project was to create a training that would cause EMIS coordinators to correctly obtain a State Student IDentification (SSID) number for each student. Part of the problem was that proper, thorough training did not exist which resulted in a considerable amount of incorrect SSIDs being assigned. Incorrect SSID assignment resulted in lost funding and increased staff hours spent in resolution of incorrect SSIDs. This project saw the creation of a training that will meet the needs of the EMIS community. It considered the stage the learners - new EMIS coordinators - were in their understanding of the SSID process. It also considered exactly what needed to be included in the training; how that training would capture the interest of the EMIS coordinators and most importantly - be information that they retained and were able to recreate at their workplace. I chose to do this project because I wanted to use something that was meaningful to me in my current position. I have been frustrated by the lack of training available to new EMIS coordinators - this project allowed me to create a solid lesson plan that will prove valuable to the districts I serve as well as other districts across the state. It was much more authentic for me to create something to fill a real-life need than to create a ficticious lesson that I would never put to use.

II. Methods To begin my tryouts I asked three EMIS coordinators who are in their first year to be my test group. These three women graciously accepted the invitation to be part of my experiment and promised to provide honest feedback. Each was asked to complete the pre-lesson survey online. Since I knew I would not be able to do a hands-on demonstration for the larger group, I chose the same sort of setting for the tryouts. We used a training room at my place of employment, TRECA, but I was the only person with access to the web and the SSID site. I kept a timer on hand to keep track of how the lesson was progressing to determine if the one hour time frame would be sufficient.

The first order of business was displaying a PowerPoint slide that contained the agenda for the session - this gave everyone a time frame and understanding of how the next hour would be spent. Each person was given two index cards and asked jot down new things they learned throughout the session on one card. The second card was to be used to record the results of the pre and post assessments.

The session began with a pre-assessment - each person was given five sets of information for ficticious students (a birth certificate, registration form, and blank SSID form). The learners were asked to use the information to complete the student attributes required on the SSID form (actually a screen shot of the online SSID entry form) for each of the five sample students. I had expected this exercise to take about ten minutes, but the women were still working diligently at fifteen minutes and did not finish until twenty minutes had passed. Once they finished we discussed the correct answers and each determined how many errors they had.

Next I gave all three women sets of cards to use for the game. Since there were only three people we had to improvise and each person received multiple cards - when this training is done for larger groups each person will only receive one card. They were asked to assemble sets of student attributes by exchanging cards amongst themselves. This activity took ten minutes and had the desired effect of proving to be a good opportunity to talk to each other while learning what a full set of attributes contained.

Once we had full sets of attributes I began a demonstration of the SSID site, asking each woman to provide attributes for one of their students. As I entered information into the SSIDs site I emphasized the need for accuracy in typing and in determining the correct attribute. We worked our way through the three examples the cards provided. During this time I also displayed PowerPoint slides that highlighted the cost of inaccurate SSIDs and the time lost in resolving errors. This demonstration took approximately 15 minutes to complete.

Upon agreeing to participate each person was asked to bring the SSID validation report for their district. At this point we reviewed each district's report and discussed how to resolve the problems. Discussion was held regarding how to best handle issues with other districts having the same SSID and other issues. This discussion took 12 minutes in order to give each person a chance to ask questions and relate their issues.

Now that a demonstration had been done and discussion held, we did a second round of demonstrations for three students from the pre-assessment forms. This provided reinforcement for using the SSID site. This demonstration went quickly due to the previous practice and took eight minutes to complete.

Finally a post-assessment was done. Each person received a new packet containing forms for five new students. Learners were asked to complete the post-assessment the same way they did the pre-assessment. This went faster than the pre-assessment due to the learners' familiarity with the procedure and took about thirteen minutes for all to finish. I went through the correct answers and each recorded their own score. The learners were then asked to tell the group a few of the things they learned - from what they had recorded on their cards. Learners were also asked to anonymously record the results from their pre and post assessments on the other card and leave it with me. These activities took another ten minutes.

III. Results and Discussion

The pre-lesson survey provided some information for me, but it was not really any "news" to me as these people are all known to me. I would expect that use of the survey for a larger group would prove more fruitful since I will not personally know the background of all participants. The participants felt the survey was short enough to not be a burden and asked appropriate questions - they were able to see my intention - get to know their skill level.

Overall the results of the actual lesson were very positive for this small group. The learners all had a variety of "I learned..." comments on their cards to share and all showed improvement from the pre and post assessments. Specific comments included "I learned that I had been recording an incorrect value for "place of birth", I didn' t know you could record the hospital if that's what the birth document listed."; "I learned that I have to use an asterisk if the child has no middle name."; "I learned that there is a difference between a dash and a blank in a name." These responses tell me that the learners picked up new and, more importantly, accurate information.

The pre and post assessments showed improvement in this small sampling. The pre-assessment had the learners missing 3, 7 and 8 out of a possible 40 attributes. The post-assessment had the same learners missing 0, 2, and 1 attributes respectively. A percentage improvement from 15% missed on the pre-assessment to only 2% missed on the post-assessment. The most frequently missed answers were in the hyphenated name, no middle name, and date of birth attributes.

The inclusion of the learners' SSID validation form brought up a number of district-specific questions. As the support person for the tryout group I was able to answer these questions and help resolve the issues shown.

The group asked several questions regarding the authenticity of my responses - mostly due to their being trained by virtue of hand-me-down information. I had to emphasize that the Ohio Department of Education had reviewed the attributes and agreed the answers I gave were accurate.

I believe the needs assessment was accurate based on the fact that this small group missed 15% in the pre-assessment. This low score reflects a lack of available training for SSID assignment. The time I took to really do the analysis of the learners gave me some true insight into who I would be teaching. This small tryout group was exactly what I found in the analysis phase - two of the three have college degrees and a high level of technical knowledge - they asked several questions about the "how" behind the SSID site. Their questions were more probing than that of a person who just accepts that technology works, they wanted to know HOW it works. They wanted to be engaged and enjoyed the short game - albeit amended for the smaller group. They appreciated being able to ask questions along the way in an informal manner and saw the value in making the "I learned..." notes.

IV. Revisions to be Made

The most important revision to be made is in the length of the training - we went over the one hour time frame. I think it is important to stick to the one hour time frame because this training can be used at conferences - and most conference sessions are in the one hour range. A good revision to reduce the time without sacrificing content would be to use only three samples for the pre and post assessments. It took much too long for the pre-assessment to be completed. Recognizing that since several more samples are shown during the actual demonstration, there is no need for five samples in the pre and post assessments. It would be important to be sure to include some common errors in the three samples to be sure all problematic areas are covered in the assessments.

I found that several people inaccurately recorded the date of birth from the sample because they used the date the birth certificate was recorded rather than the birth date. I never realized this as a source of error - knowing this now I would revise the training to include a PowerPoint slide emphasizing this error and how to avoid it. The slide would show a sample birth certficate with a recorded date and a birth date. The slide would also state the best way to avoid the error - compare the birth date on the registration form to the date on the birth document as a sort of check and balance. Most parents will correctly record their child's birthdate - but it's best to check between the birth document and the registration document for a final check.

As the analysis of the learners had indicated - these people rely on the Ohio Department of Education (ODE) as THE source for answers. Even though I emphasized that my answers had been vetted by ODE, I still got questions on that front. As a revision, I would ask ODE to allow me to provide a written statement that the answers are correct - a sort of imprimatur that I speak with their blessing.

The discussion section where learners provided their own SSID reports may have to be revised in order to be successful. In the tryout group I had people for whom I provide support, a larger group will include people from districts for whom I do not provide support - it might prove awkward, or even inaccurate for me (or anyone providing this training) to try to answer these district specific questions that might involve more than meets the eye. As a revision, I would bring a copy of the SSID validation report and discuss the various error codes in generalities rather than specifics.

The goal was to teach new EMIS coordinators how to accurately obtain an SSID - without error. I believe the training reaches this goal even though the tryout group had a 2% error rate on the post-assessment. Each person had significant gains between the pre and post assessment and with some revisions to the training to emphasize problem areas - such as birth date - the goal can be achieved with the larger group.