• Categories

  • Recent Posts

  • Archives

  • Copyright Notice

    Copyright © Nancy Hidy Wilson, 2010-2013. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Nancy Hidy Wilson and nancyhidywilson.wordpress.com with appropriate and specific direction to the original content.

Improving Speaker Rating Evaluations


I finished compiling all the individual session speaker ratings from SQLSaturday #107 held in Houston on April 21, 2012 and sent the results to the speakers this past weekend.  We used the default form provided on the SQLSaturday Admin site, which had two basic inputs.

1) Expectations:  Did Not Meet \ Met \ Exceeded. 

2) Overall Quality of the Presentation: rate 1-5 where 5=great.

Plus there was a request to write any other comments on the back of the form.  A few people did provide some constructive criticism which the speakers can use to improve; many people provided positive encouragement; but most wrote nothing.

Anybody have a problem with this form?  I do; and I’d like to make it more useful for both organizers and speakers.  But, how?

We’ve been conditioned since early childhood in school to receive a grade for our performance. Consequently, we tend to provide evaluations with number ratings so that we can come up with an average rating for each speaker.  The problem with this type of rating for speakers, in my opinion, is that there are no defined criteria for the students (e.g. graders) to use in their evaluation – it is purely subjective and purely based on the individual’s experiences which could vary wildly at an event like SQLSaturday.     

The “Expectations” rating to me is really not very useful – it is too general.  There are two things that I, personally, am gauging from an “expectations” basis when I attend a session: 1) the content to be delivered based on the abstract provided; 2) the quality of the speaker based on the presenter’s professional credentials.  However, what if I’m so new to a topic area that I really do not comprehend what the abstract means (although I think I do) and therefore my expectations of what I’m about to experience are very different than the reality?  Or, what if I have really high expectations of a well-known speaker and I feel like their presentation is just average? How do I handle this in my evaluation? Based on what I saw in our SQLSaturday evaluations, it seems to me that most people whose expectations were not met also rated the overall quality of the presentation low.  But, how can that be, when the majority of the other attendees at the same session believed that the session met or exceeded their expectations and gave a high rating on the overall quality?  I saw this anomaly several times.

One of the goals of SQLSaturday is to grow the speaker base. I don’t know about you, but “grow” to me doesn’t mean just numbers, it means providing experience and maturity. How do we get the necessary feedback to the speakers to enable them to improve their presentation skills in order to better train us?

Speakers, what input would you like from your audience?  If you were to redesign this simple evaluation form, what 2 or 3 questions would you ask? Do you derive value from a subjective numeric rating?

SQLSaturday organizers and User Group leaders, how would you like to see speakers evaluated to help you in selecting sessions for your next event?

Please post your comments here.

About these ads

13 Responses

    • Mike,
      I knew I wasn’t the first one with these thoughts! It seems this is an issue that we all as speakers\organizers would like to improve. I read your linked post and wonder if you’ve had a chance to see or implement any of the ideas generated by your post a year ago?

      Thanks for commenting!
      Nancy

  1. I touched on this with a blog post I wrote about the SQLSaturday in OKC last year. It’s short sweat and simple, but totally worthless if you ask me. We have always used our own in Dallas. I’ve often thought about having one without numbers or ratings at all. That may affect how many people fill them out though. I know for evaluations that have both ratings and write in questions, I’ve been guilty of just doing the ratings and leaving the questions blank.

    • Ryan,
      We didn’t do a paper eval at all for our first SQLSaturday – we tried to get the attendees to use SpeakerRate, but that flopped. So, this year, like I mentioned, we just used the default form, but I never really liked it – just ran out of time to put something else together. After compiling all of them, however, I grew to really hate that particular form – thus this blog entry! I’d love to see the form Dallas uses.

      See you next week at SQLRally!
      Nancy

  2. I used to work for a survey company, and I can’t overstate the value of having a survey professional help with the question and answer design. Everything from the order of the questions to the wording on the scale has a tremendous impact on the answers you get.

    As a speaker, I want venue-related questions NOT COUNTED AGAINST ME. Holy cow, I’m so frustrated when the venue has a crappy projector in my room or the microphone doesn’t work or it’s hot as an oven. If there’s not a separate question for it, attendees take it out on my scores, and that sucks rocks.

    Here’s where the question order starts to come in – you have to ask that question before you ask speaker performance questions. If you let responders get unrelated stuff out of their mind first, then they tend to give more accurate responses to the core issue, which is how did the speaker perform.

    • Brent,
      Very good point about the order of the questions. For our overall event evaluation form, I did have one that I used as a template and I reordered/added items with the very same thoughts you expressed in mind. I totally agree that you need to pull out the attendee’s frustration over items out of the speaker’s control before they start rating the speaker and his/her content!

      But, as a speaker – what questions do you want your audience to answer to help you improve your session for the next time? I know your presentations are already perfect – but just in case! ;-)

      Thanks for commenting!
      Nancy

      • I like two of Connections’ questions a lot: how much of the information was new to the attendee, and do they have something they’re going to do as soon as they get back to the office. I’d also like to know if the attendee would like to see me present again.

      • Thanks Brent! I like those questions – they work for both the speaker and the organizer, in my opinion.
        Nancy

  3. Buck Woody did a blog on this, and agreed to let me edit and add to the SQLSaturday Wiki site. I like his suggestion a lot, plan to use it at Pensacola’s event: http://wiki.sqlsaturday.com/Evaluations-Speaker-and-Event.ashx

  4. From an attendee perspective, I liked the evals as being simple and quick to fill out. If sticking with two questions, I would rather have seen the Expectations/Overall replaced with Speaker/Content.

    I expect that diverse ratings will always be around when you have a diverse audience. A novice speaker may impress a room full of developers new to SQL, where an experienced DBA might be turned off by technical issues in the presentation.

  5. Hi Nancy,

    I just wrote a blog post about this exact topic.
    http://johnsterrett.com/2012/10/30/is-sqlpass-helping-their-speakers/

    Regards,
    John

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 415 other followers

%d bloggers like this: