The Board, working with various sub-committees, worked for several hours on the proposed New Ratings System to develop something new and improved from the system previously used. While we believed it was comprehensive and will be effective, we appreciate the feedback that you, our members, provided. Please see the feedback received and the Board responses below.
Having been a part of an association with public ratings I’ve observed that is has been more of a ‘interesting thing to look at’ for 100-200 level referees and can be frustrating or embarrassing for 400-500 level referees. It led to some tense conversations with members who believe they should be rated higher. It might also be worth considering having 2 ratings for each referee one for CTR and one for AR as people’s skill levels can be vastly different between these two positions.
Board Response: As can be seen by the summary of responses received, the membership is not in favor of publishing individual ratings. Instead we will publish an aggregation to share the distribution of number of referees per rating band, e.g., 100, 200, etc. we will also publish the monthly changes in ratings from old to new rating but without identifying each subject referee. At this time we are not contemplating AR ratings different to the CR rating but it is a valid point. We will look at expanding the ratings to include AR ratings once all CR ratings have been updated.
If the situation with referee shortages continues I believe it will be difficult to get fair evaluations of most of our very active referees. If it’s the first game on a Saturday chances are you’ll get a good idea of the referee’s ability. If it’s a Sunday of a 10-game weekend I am not so sure.
Board Response: Valid points, care will be taken to schedule the observations to minimize the concerns expressed.
If possible allow the referee to be observed to select two names from the pool of observers.
Board Response: The assessors are certified professionals who take their jobs seriously. This recommendation would over complicate the process and may impair the independence of observers we will be striving for.
Using only certified assessors seems like it will take a long time to assess the entire association.
Board Response: We believe we have sufficient observers to complete the whole membership within a year, possibly faster.
I appreciate everyone’s work on this new rating system. The details that you have provided for the criteria used for each category are good.
Board Response: Thank you, this is a good starting point, it may evolve further as time goes by.
Ratings are for the purpose of assigning games of the appropriate level for the referee and the Assignor and should be considered personal information and not for public disclosure
Board Response: This is the opinion of the majority of the respondents. Individual ratings will remain private for each individual referee and the Assignor’s information only.
This only works IF the assignor follows the New Rating System.
Board Response: True. Implementing this robust and transparent system as we have designed it to be, will enable each referee to determine if he/she is receiving the appropriate assignments. However, there are several factors that go into assigning, not just the ratings, primarily the mix of games we have on any given day.
This system is highly dependent on the quality and consistency of the assessors. Do we have the appropriate qualified assessors and are they consistent from one to the next? How will assessments be normalized” across a range of game levels? “
Board Response: All observers will be USSF-certified assessors. They will meet for SBSRA training before implementing the system to ensure a consistent approach. They will meet on a regular basis to discuss observations already completed. Observation scores will be adjusted based on each game level.
I strenuously disagree with posting all members’ ratings for all to see. I don’t see the value and it will likely breed quite a bit of discontent.
Board Response: This is the opinion of the majority of the respondents. We will share information on a macro level only on our website.
I actually think instead of posting everyone’s score (with an opt out) we should anonymously publish ALL the scores and the distribution of the scores (maybe even the trajectory of the scores – i.e. ref #X went from a 300 to a 225 etc.). I think it is important to know where each referee fits in the scheme of the Association but it’s not important to know exactly that Jonny Joseph is a 225 and Bruce Ashton is a 125 or any of that. We don’t need to be arguing about I’m better than X or Y.
Board Response: Good ideas. We will share information on a macro level only on our website.This is the opinion of the majority of the respondents.
1 assessment can’t provide enough information to change a referee’s ratings. Cal South requires multiple two for upgrades etc. The best referee can have 1 bad day or the weakest referee could have1 good day & this proposal that could greatly effect the referee’s ratings.Observers should be paid at least the same pay as the CR. SBSRA paying so little compared to Cal South. SBSRA is going to pay an Observer $21 @ U12 observation. A flat $100 min.
Board Response: We will begin with one observation per referee. The Observers will meet to discuss the results of each observation. If someone has a bad day, it is likely that the other Observers will realize this. The observed referee is able to request a second observation at his/her own cost. Observations will continue to be performed on an on-going basis, this is not intended to be a ‘one-and-done’ deal. Comment noted and discussed by the Board. The Observers will be assigned games to referee that are compatible with their observation assignments. They are in agreement with the fee decided upon. Many responses were received, of course we could not force everyone to respond to the survey or the respondents to answer every question.
The key variable is will each assessor have a fair and consistent approach. Having been involved with multiple sports (baseball soccer football basketball) as a coach referee player assessor and board member I know ratings by each individual vary and often times significantly from each evaluator. I have seen this at all levels and think that really should be looked at in depth before rolling this out.
Board Response: All observers will be USSF-certified assessors who are certified and experienced in performing USSF assessments. They will meet for SBSRA training before implementing the system to ensure a consistent approach. They will meet on a regular basis to discuss observations already completed. The Observers have the full confidence of the Board to perform a fine job.
It’s hard to be prepared for our games when we have NO idea which teams or even the game level (Flights or silver etc) we will be doing until we get there.
Board Response: Noted, we are all in the same situation. However, for many of our games, one can go to the leagues’ websites to see prior results and team standings.
Our referee association rocked for 25+ years. We brought an outstanding product.Several outstanding members left after arrivals of Gabriel Goldsman and Michael Hinz For 4-5 years experienced members categorically excluded from CR’ing any games extending past 30 minute halftimes.If assigned a AR for a 35 minute+ halftime game we watched someone calling a game from the halfway line wearing a hat and making calls from 35 yards away from play.
Board Response: Agreed that we have rocked for 25+ years and that we offer an outstanding service! Members leave from time to time for various reasons. Gabriel and Michael assumed their positions at different times, years apart. Conspiracy theories are often just that, theories. Both of these members have selflessly given many hours of their time to serve us all. This Revised Rating System is designed to ensure all referees are assigned to the appropriate games given their ratings. However, there are several factors that go into assigning, not just the ratings, primarily the mix of games we have on any given day. This Revised Rating System is designed to ensure all referees are assigned to the appropriate games given their ratings. If you have concerns about other referees, please communicate that to a Board Member.
We seem to be making this real complicated for a group of people that have a life and just want to referee a few games
Board Response: Really not complicated at all, we have just taken the trouble to explain it fully. We recognize that we have a very diverse group of referees. We have an obligation to our Leagues (Customers) to assign the appropriate referees to our games. We have an obligation to our referees to provide them with meaningful feedback. This New Ratings System is designed to accomplish both objectives and will be worth the effort.
Game, coach, player and spectator management should be top of the list during assessments.
Board Response: Noted and agreed!
I don’t think much will change. I think people are assigned because of the relationship and not because of how well they Referee.
Board Response: Implementing this robust and transparent system as we have designed it to be, will enable each referee to determine if he/she is receiving the appropriate assignments. However, there are several factors that go into assigning, not just the ratings, primarily the mix of games we have on any given day.
The proposed ranking system is a valuable achievement by Jonny Joseph. I provided a lot of comments and suggestion after reviewing the proposed process in detail.
Board Response: This has been a team effort by the Board and other volunteers.
Just two comments on what I think is overall a very good (and needed) process: 1. To make the rating process work to objective really only the ref and the assigner need to know the exact score/rating. I’m afraid that if it’s public it will encourage focus on the wrong things. E.G. ‘I know I’m better than that person so why did he/she get rated higher’. 2. Three strengths three weaknesses in the assessment? OK but have assessors focus on the top two areas that make a REAL difference.
Board Response: 1. This is the opinion of the majority of the respondents. We will share information on a macro level only on our website. 2. Noted.
I think you are asking for trouble if all referee’s ratings are posted. I predict you will get a lot of argument about the accuracy and the subject referee’s perceived rating.
Board Response: This is the opinion of the majority of the respondents. We will share information on a macro level only on our website.
I am parent of players and not just a referee. I have observed all kind of referees and I think this may become more or an issue than a solution. I think if an observation would be more beneficial if followed up with training with small groups that are having the same issue. If we start saying you can only do certain games because of feting the. We get ask to do a higher rated game because you are short that level not cool. When do not get any games with open schedules the. Get high level games.
Board Response: The observation concludes with a debrief by the Observer. If we identify common areas for improvement, group training will be designed to address these areas.The primary purpose of ratings is to identify referees best suited for the various level of games we officiate. There is no limitation of ‘certain games’ but it is important that we create assignments designed for our referees to be successful. However, there are several factors that go into assigning, not just the ratings, primarily the mix of games we have on any given day.
I don’t need to be contacted, I am 100% supportive of this process.
Board Response: Thank you.
How will you determine which age and gender to conduct an assessment as some referees are only comfortable at a certain level, for instance I am slowing down and prefer to officiate girls games up to U16, I can do boys games up to U14, but I can officiate at all levels if needed. Some refs only want AR. How do you plan to address referees that are ranked high right now and after assessment are demoted, and the only way the referee will know is by the quality of games assigned.
Board Response: Observations will be performed for each referee at the highest level that each of us has been officiating at most recently. Referees who only wish to be Assistant Referees need to communicate that to the Assignor. Each referee’s preliminary rating score will be shared with the referee during the post-match debrief. It is intended to be a transparent system whereby we all know our rating and how it was determined using the objective criteria. All of this is set out in the January 2022 Flag and Whistle.
Assessments can be challenging. I wonder what level of game I will be assessed at. I’ve only AR’d 48 games so far. I am comfortable at 14U but not sure I would be comfortable at single referee games. Still lots to learn about the association and various organizations we referee for. Is there any thought to assign mentors separately from the assessments?
Board Response: Observations will be performed for each referee at the highest level that each of us has been officiating at most recently. The observation concludes with a debrief by the Observer. If we identify common areas for improvement, group training will be designed to address these areas.Mentors will be assigned to all new members. Additional mentoring is available on request to the Assignor.