Frequently Asked Questions
Training
How do you develop and maintain reliability in a system of assessment?
Who is authorized by the authors to provide training on the environment rating scales?
How do I become "certified" on the scales?
When will you next offer training on the [FCCERS-R, SACERS, etc.]?
What are the various types of training ERSI offers?
Can you put me in touch with a person in my area that is trained on the scales?
Q: How do you develop and maintain reliability in a system of assessment?
A: Many states and other groups that are using the Environment Rating Scales (ERS) need to train large numbers of people to reliability. However, training a large group to reliability is a substantial task, requiring days of work, and a commitment of staff as well as resources. The following represents the plan to be used in “high stakes” observations (those observations where ERS scores are used to determine a program’s status or funding).
We suggest training a core group to reliability with the authors of the scales and their associates. Often this might be a group of 6-8 people, but in some cases, is a larger group of up to 25 people. Other less costly (in terms of commitment and resources) training, for very large groups of people who do not do high stakes observations, for example, those who deliver technical assistance on the scales, can also be provided, but this can not substitute for the training to reliability procedure that is described below.
Note: This plan must be implemented separately for each of the different scales in the ERS series. Reliability on one scale is not an indication of reliability on any other of the scales.
Training to Reliability
1. Initial training of a core of assessors, who will later be able to train others
A period of 5 days training by at least one author, with the author’s associates, is required. The first day of training is classroom lecture in which the scale is introduced, and includes the following topics:
- What the scale is measuring
- How the scale measure the three basic components of quality
- The meaning of the scale scores
- Understanding reliability and validity of the scales
- Accurate scoring procedures
- Introduction to basic interpretations of terms used throughout the scale
- Observation procedures that are required for reliable use of the scale
Days 2 through 5 consist of mornings spent in doing independent observations with a small group of participants, led by a group leader with proven reliability and experience in training others to reliability. The afternoons are spent in debriefing sessions in which participants compare scores, and work to come to a correct consensus score through explanations of interpretation by the group leader as well as consideration of all evidence observed in the morning’s observation.
As a result of the five-day training, all participants are expected to have improved reliability scores, with at least one person usually found to be reliable.
- Acceptable reliability is defined as having 85% agreement (within one point) with the consensus scores.
- Reliability for any one observation is calculated by dividing the number of correct (within one point) scores by the number of items completed during the observation. Scores of N/A are counted in the calculation of reliability.
- An individual’s reliability is based on the average of the three most recent reliability scores received. For example, if a person scored reliabilities of 75, 85, and 95, across the last three observations, the reliability would be the average of the three, or 85%. As a newer reliability score is added to the person’s reliability history, the oldest score is dropped, and the newest added, so that the last three reliability scores are always used.
No official assessments should be completed by an observer who does not have a reliability average of 85%. However, new people can be trained (or a non-reliable assessor can be retrained) by a reliable assessor during official assessments.
2. Completing reliability training for all participants
Since acceptable reliability will generally not have been attained by all participants in the 5-day training, it is necessary to continue training. This is done by having the observer (or observers) who reached the acceptable reliability level continue to do observations and debriefing sessions with the remaining observers. This is continued until all participants have reached the acceptable level of reliability.
3. Establishing one or more “state anchors” for establishing statewide reliability and completing reliability checks across the state
Usually, the state anchor for reliability is the most consistently reliable observer. It is the responsibility of the state anchor to:
- Communicate with the ERS authors for clarification on interpretation when needed
- Communicate clarifications to all assessors in a state
- Complete reliability checks on observers throughout the state. One check is completed every 10th observation on a scale, until the observer being checked has reliability scores consistently, of 90% or above. High reliability observers require less frequent checks. Those who fall below the level of 85% require more frequent checks, until their reliability is consistently at or above 85%.
4. Maintaining reliability, expanding the reliability check system, and training new observers to meet acceptable reliability standards (See figure 1)
- Observers who have consistently high reliability scores of 90% or above are considered to be Level 1 observers. Level 1 observers can take on the responsibility of checking the reliability of others who have lower reliability scores. Those with lower reliability scores or those being trained ate called Level 2 observers. In addition, they can provide the training during practice observations followed by debriefing.
- The anchor then takes on the task of checking Level 1 Observers, and checking a sample of level 2 observers. .
Q: Who is Authorized by the authors to provide training on the environment rating scales?
A: Information on Authorized Trainers for the Environment Rating Scales (ERS)
In the past years there has been an extraordinary increase in the use of the Environment Rating Scales (ERS)—ECERS-R, ITERS, FDCRS and SACERS. This has been greatly due to continuing research that ties higher environment rating scales scores to better child development in a broad range of areas that are considered necessary for success in our society. Throughout the United States, and internationally as well, the Environment Rating Scales are now heavily used for both program quality assessment and improvement. This has resulted in an increase in training on the scales. Training is provided to early childhood and school-aged care practitioners, professionals who deliver technical assistance to encourage quality improvement, and to assessors who evaluate program quality for licensing, quality rating systems, or research purposes.
Training on the scales has always varied in scope and accuracy, which until the late 1990s caused few concerns. Researchers using the scales were never held accountable for individual scores, and the ERS are robust as research instruments, allowing a certain amount of error while still providing meaningful results. When used in program improvement efforts, or informal assessments, the exactness of scores was not considered important. Under these conditions, many people provided training on the scales, but too often training has provided incorrect information and procedures, weakening the relationship between scale scores and child outcomes.
In the late 1990s, with the advent of “high-stakes” scale use in quality rating and improvement systems, tiered reimbursement systems, rated licensing systems, and other consumer awareness efforts, the necessity for extremely reliable and accurate scale use has increased dramatically. Official assessors in high-stakes systems must be trained correctly to use the scales according to the authors’ interpretations and procedures required to maintain high levels of reliability to ensure fair and accurate observations across the system.
Since accurate and correct training on the ERS is a necessity under high-stakes conditions, the authors are the only people authorized to provide training on the scales for these purposes. We realize that others may indicate that they are appropriate trainers under these conditions, but in too many cases agencies who have used unauthorized trainers, have run into difficulties because the scales have not been used properly resulting in practitioner confusion, unrealistic scores for programs, lack of fairness across a system, and no relationship between scale scores and desirable outcomes for children. The ERS authors have had to retrain too many improperly trained assessors, after the fact, and have also needed to give advice on how to help ineffective quality rating system to be improved.
The authors (specifically Thelma Harms and Debby Cryer, and their associates of Environment Rating Scales Institute, Inc.) are the only authorized trainers on the ERS.
Q. How do I become "certified" on the scales?
A. To be considered “reliable” in the use of a scale, the authors provide training through the Frank Porter Graham Child Development Institute at the University of North Carolina (primarily for participants who come to Chapel Hill, NC), and through the Environment Rating Scales Institute (primarily when trainers visit locations other than Chapel Hill). Varying degrees of training on the scales are provided- everything from a four-hour introduction (for example, at the NAEYC conference) to a five-day course of training towards reliability and an 8-day In-Depth course offered in Chapel Hill and in other cities sponsored by various agencies. Ongoing training and reliability checking are available under special arrangements.
Although records have always been kept about who attended trainings, and more recently the levels of reliability attained by participants, there is as yet no credential that goes with these levels of training that would testify to certain knowledge levels or proficiency in scoring. In the past we did not test the participants who complete our 3 day and 5 day courses that include 2 supervised field practice sessions to familiarize them with trying out a scale in a classroom. We have begun keeping records of reliability scores attained. However, we do not follow up with participants over time to see that their scoring is consistently reliable over time or to ensure minimal observer drift. In some circumstances, groups requiring high stakes assessments do contract with ERSI do ensure that reliability is maintained after participants attain initial reliability.
The 8 day course does include a pre- and post-test but does not include field practice observations. Hence, there are few people outside of a small circle that work closely with the authors at FPG who could be considered "certified" for training purposes.
We do award letters of completion for training provided in Chapel Hill. Usually agencies contracting with ERSI will provide letters or certificates of attendance. Participants who completed the 8-day In-depth course receive either a certificate of attendance or a certificate that shows the requirements for passing the course have been met.
Q. When will you next offer training on the [FCCERS-R, SACERS, etc.]?
A. All the courses we have planned are listed at the top of this site’s training page. If you do not see the scale in which you would like training, or it is not on a date that works for you, you have two options. You can wait until we schedule training on the scale you need, or you can look into contracting with one of the authors to provide a separate training customized for your purposes.
- If you are going to wait, you should join the Rating Scales listserv so you will be notified when we schedule new courses and start to accept applications. You should check the site periodically, as well, and perhaps send us an e-mail just to let us know you’d like that scale to be offered again.
- If you are interested in finding out what is involved in contracting to arrange your own training course with the authors, please e-mail Cathy Riley at riley.ersi@gmail.com.
Q. Is training ever held outside North Carolina? Is it possible to have ERSI staff come to my area to conduct training?
A. Courses where the authors or their associates provide training, outside of North Carolina, generally are sponsored by other agencies, and we do not publicize those on our website. We leave it up to the sponsoring agency to decide whether the event is open to the public and to handle publicity and registration.
If you are interested in finding out what is involved in having our trainers come to your area to conduct a training session, please e-mail Cathy Riley at riley.ersi@gmail.com.
Q. What are the various types of training ERSI offers?
A. Levels of Training on the Environment Rating Scales
(ECERS-R, ITERS-R, FCCERS-R, or SACERS)
Levels of training usually provided are found here, but many training programs are tailored to a sponsor’s specific needs.
All levels are offered through ERSI in Chapel Hill, or in other locations by special arrangement with the authors.
If you wish to arrange training for a specific group, either in Chapel Hill or at another desired location, please contact Vanessa McCullough.
Q. Can you put me in touch with a person in my area that is trained on the scales?
A. Generally we do not share the names of people we have trained because it implies an endorsement of their qualifications that we cannot make. At this point the only people we are able to recommend are those that have been trained to reliability with the authors of the scales, and presently most of those people are our colleagues here at FPG.
You can always check to see who might be using the scales in your area by looking through our Directory of Projects that Use the Rating Scales (click "In Practice" on the menu at the top).
Q. Will completing the 3 day ERS training course in Chapel Hill make me a reliable user of one of the scales?
A. No. Only two practice observations are completed, and at least three are required to show consistent reliability. It should be noted that the authors of the scale design and conduct training to reliability in North Carolina and at various sites upon request, and have done so in a variety of communities across the US and Canada. These trainings are generally no shorter than 5 days.
- Content of Scales
- Training
- Products
- Website
