IDEA SRI - Powered by Campus Labs

  

IDEA has wealth of resources available to support the successful implementation of the Student Ratings of Instruction at your institution.  

This online "handbook" is broken down by section - getting started, administering your survey, using your feedback and using Instant Feedback.  Within each section there are helpful links and researched-based suggestions to guide you.

Faculty resources

Getting Started

Getting Started

IDEA’s Student Ratings of Instruction are designed to provide feedback you can use—formative feedback that gives you suggestions for improvement—as well as summative feedback that can be used as a component of a more comprehensive system of faculty evaluation. 

When you partner with IDEA, all of the tools are available to use.

  • Diagnostic Feedback - A comprehensive tool that provides both summative and formative feedback about student progress on relevant course objectives, instructor teaching methods, and overall impressions of the instructor and course.
  • Learning Essentials - Provides summative feedback about average student progress on relevant learning objectives and overall impressions of the instructor and course.
  • Teaching Essentials - Provides formative feedback about teaching methods highly correlated with instructor and course excellence.
  • Instant Feedback- A short instrument that gathers immediate student responses after each class or unit; weekly, or whenever desired, so faculty can see the impact of changes they have made in their class.   

    How do faculty choose which instrument to use?

Some institutions allow faculty to choose the instrument they wish to use for particular courses while others may make this decision for them based on policies for departments or other units. If you are allowed to choose your own instrument, your institution should have a way for you to communicate that choice because faculty cannot make the technical selection of an instrument--only the administrator can do this when setting up administrations. Contact your on-campus coordinator to find out how to make your choice.

See also this advice for selecting an instrument: Choosing an Instrument.

What the ratings will tell you

IDEA’s Student Ratings of Instruction provide feedback you can use to improve instruction with the focus of the instrument on student learning. The student ratings provide evidence of progress on learning objectives and teaching methods used in a course.

One of the most important of the pieces of evidence is student ratings of how well they achieved learning objectives in your course. Students are asked to rate their achievement of learning objectives in both the comprehensive Diagnostic Feedback instrument and Learning Essentials instrument. In your report, you will be able to see their rating and how you compare to other groups.


About adjusted scores

One of the unique things about the SRI instruments is that they control for course difficulty beyond the influence of the instructor. In the Diagnostic Feedback instrument, the system adjusts your Progress on Relevant Objectives scores, as well as your “excellent teacher” and “excellent course” ratings, through regression equations, to account for the influence of student background preparation, work habits, motivation, course difficulty, and class size. In Teaching Essentials, average ratings on both the “excellent teacher” and “excellent course” items are adjusted for student work habits, motivation, background, as well as class size.

These adjusted scores may raise your scores some to account for these influences. For instance, in a course where a large number of students report low motivation for the course, your scores are adjusted upward to account for the low motivation. The purpose of adjusting scores is to level the playing field when comparisons need to be made between instructors teaching different kinds of classes. So, for instance, scores of instructors teaching large, required general education classes can be compared more authentically to those teaching smaller, high level, in-major courses.  

Adjusted Scores at a Glance [PDF]

The default view you are shown in your reports is this adjusted score, but you can chose to view your Raw Averages if you wish by simply toggling the View option.

What students see and how they access the survey

Students access the survey for all their courses from one place—the Campus Labs portal for your campus.  Your On Campus Coordinator may have more directions and information for you about how students can access the portal. 

  • Once students are logged in, they will see a list of current and upcoming course evaluations to be completed.
  • Clicking on the course starts their rating of your course.  Once they are in the instrument, it is a simple matter of selecting ratings for the items presented.   
  • Students can access the instrument through desktop computers, tablets or mobile devices.  

To learn more, see Accessing Campus Labs Course Evaluations for students.

What are the questions on the instruments?

Sample versions of each instrument can be found by clicking the links below

Diagnostic Feedback Instrument
Learning Essentials Instrument
Teaching Essentials Instrument
Instant Feedback

With all instruments except Instant Feedback, it is possible to add custom questions if this option is available at your institution.  

See Adding Custom Questions to learn more. 


Administering Your Survey

Administering your Survey

In many cases, institutions choose to use only the Diagnostic Feedback instrument for all courses, or specific departments may alternate instruments based on decisions of that unit. In these cases, you as a faculty member, do not need to make any decisions about choosing an instrument. If, however, your institution allows you to select which instruments to use in each of your courses, your first task is to communicate with your IDEA Campus Coordinator about which instrument you will use in each course. Your institution should have its own internal method of communicating about this.

If you are using the Diagnostic Feedback or Learning Essentials instruments, you will need to choose the Essential or Important learning objectives for your course before the administration of the survey begins. See the Choosing Learning Objectives video for more about this. For Teaching Essentials, there is nothing you need to do ahead of time. 

ABOUT LEARNING OBJECTIVES

In the Diagnostic Feedback and Learning Essentials instruments, the main indicator of teaching effectiveness is student progress on learning objectives. The instruments are designed around 13 learning objectives that encompass all objectives you would have in a course. These learning objectives are broader objectives that precede your more specific learning outcomes but are subsequent to your major course goals. 

Here’s an example for a child development class:

 

Course Goal

Understand characteristics of each stage of child development

IDEA Learning Objective

Gaining a basic understanding of the subject (e.g., factual knowledge, methods, principles, generalizations, theories)

 

 

 

 

 

 

 

IDEA Learning Objective

Learning to apply course material (to improve thinking, problem solving, and decisions)

Learning Outcomes

1. Students will describe each of the physical, cognitive, and psychological traits of each stage of development

2. Students will differentiate the stages of child development when presented with examples.

 

 

 

1. When presented with cases, students will explain child behavior based on characteristics of the child’s stage of development

2. When presented with common childhood problems, students will suggest solutions based on child development principles.


In this example, the course goal has two IDEA Learning Objectives (though it could have more), and each learning objective has two learning outcomes (though there could more be more as well). Organizing your course learning outcomes through IDEA Learning Objectives can help you think through your course more thoroughly and ensure you are covering concepts and applications as thoroughly as needed. You might also consider organizing your course syllabus this way so that students see clearly how your learning outcomes fit with the objectives. 

 

Whether or not you organize all your course learning outcomes by IDEA objectives, those objectives in your course that are Essential or Important must be identified before administering the survey. IDEA instruments double-weight Essential items and single-weight Important items while those of minor or no importance are not used in calculating the ratings. This enables you to focus the survey on those learning objectives that are vital in your individual course. 

Choosing Learning Objectives

If you are using the Diagnostic Feedback or Learning Essentials instruments, the Essential or Important learning objectives of your course must be identified before administering the survey. Your institution may prescribe which learning objectives are to be rated in your course and pre-select them for you, or it may be left to you to choose. If they are pre-selected, you can upgrade an item (for instance, make an “Important” item “Essential), but you cannot downgrade an item. Check with your On-Campus Coordinator if you do not know the options for this at your institution.

If you are to choose your own, there are few things to consider. As a general rule, limit your selection to between 3-5 objectives by prioritizing those objectives that are Important or Essential in this course. Although all the learning objectives may be addressed to some extent in your course, it is unrealistic to expect that students could make significant progress on all or even most of them in one course. So again, try to identify those 3 to 5 that are most important. But be true to your course. If you are teaching a course where you are targeting only one objective (such as a lab experience), then it would be appropriate to select only one learning objective. On the other hand, if you are teaching a capstone class in your major, and you have targeted many learning objectives, it would be appropriate for you to select more than five.   

Selection is more obvious in some courses than others. To help, consider these three strategies:  

1. Create a chart that aligns goals, IDEA learning objectives, and your learning outcomes. Ask these questions:  

  • Is it a significant part of the course?
  • Do you do something specific to help students accomplish the objective? 
  • Does the student’s progress on the objective influence his or her grade? 

2. Use the Teaching Goals Inventory to help narrow important objectives 

3. Use relevant learning theory such as Bloom's Taxonomy to narrow the focus

To choose objectives, login to your dashboard in the Campus Labs system and choose the Objective Selection Form 

Adding Custom Questions

Individual faculty are able to add custom questions to specific courses as needed (provided this is allowed by your institution) on all instruments except Instant Feedback. For example, you may want to collect student feedback about a particular resource used in the class or their feedback about the usefulness of a grad assistant. Custom questions must be added before the administration period of the survey begins. Institutions often do not want to use custom questions so that surveys between classes are the same length and do not over-burden students, so this option may not be available to you. 

Take a look at the following guides for suggestions related to Using Additional Questions and Using Additional Questions for Online Courses.

Increasing Response Rates

While any feedback from students can be useful to the individual instructor for improving a course, higher response rates help make comparing one course to another more meaningful. To get responses from as many students as possible, there are several things you can do. 

  • Create Value for Student Feedback.  This is the single most important factor to elicit good response rates. If instructors encourage student feedback, students will be more inclined to participate. Faculty can help students learn to value the feedback process by using techniques such as the following: 
  • Use your syllabus. At the beginning of the course, put the IDEA Learning Objectives that you have selected for this course in the course syllabus alongside the specific course learning outcomes. This can demonstrate for students how the course outcomes relate to the general IDEA learning objectives and how your instructional decisions are informed by student feedback. 
  • Mid-point reminders. At the midterm point of the course, remind students what the goals of the course are and how they can enhance their learning experience. 
  • At the end of the course. When the survey has been opened for students to respond, encourage them to complete the form by telling them that throughout the course you have been giving them feedback with their best interest in mind. 
  • Assure students that their responses are confidential. When their responses are accessed by you or administrators, no identifying information is associated with the responses. Students are more likely to complete a survey if they are assured of the confidentiality of their responses.   
  • Administer Surveys During Class.  Emulate the “captive audience” nature of in-class paper ratings by asking students to login and complete the survey during class. Students can use their mobile phones, laptops, or desktop computers in a lab. 
Consult with your On-Campus Coordinator about whether or not your particular institution has specific policies about when/how to administer surveys and the use of other resources such as Email Notification Templates.







Using Your Feedback

Using your Feedback

Viewing Your Results

Login to the Campus Labs portal for your institution where you will see a list of your courses and options. When the course evaluation period has ended, click “View Reports” to see results for individual courses.

COURSE AND STUDENT CHARACTERISTICS

The three end-of-term instruments (Diagnostic Feedback, Teaching Essentials, and Learning Essentials) include a summary of student characteristics, and Diagnostic Feedback also includes student perceptions of course characteristics. 

On the first page of your report, you will see a representation of these characteristics. The Diagnostic Feedback Report provides you with a summary of how students perceived the difficulty and workload of the course as well as indicators of their motivation and effort in the course. These factors are used in calculating the adjusted scores for the other ratings in the instruments including the excellence of the instructor and excellence of the course items. In a course with unmotivated students, for instance, the adjusted scores will be raised slightly. And reviewing this information yourself will help you interpret the rest of the report in light of the characteristics of the students.

Learning Essentials and Teaching Essentials provide a summary of student motivational characteristics. These are used in calculating adjusted scores as described above, and likewise, can be used to interpret the rest of the report. 


INTERPRETING THE DIAGNOSTIC FEEDBACK REPORT

The Diagnostic Feedback instrument is a comprehensive tool that provides summary and detailed, formative feedback for your course. The summative information for your course gives a quick overview of all the ratings of your course, and the formative feedback provides detail about average student progress on relevant course objectives, instructor teaching methods, and overall impressions of the instructor and course. The Diagnostic Feedback instrument controls for extraneous factors (e.g., student work habits and motivation) beyond the instructor’s control.

Summative Information

The Summative tab provides you with a quick overview of “how you did” in the course according to the ratings.

Progress on Relevant Learning Objectives (PRO)

One of the best ways to infer teaching effectiveness is to examine student ratings of progress on relevant objectives that have been chosen as Important or Essential. The average of these ratings provides a good indication of how successfully objectives were reached in the course. From the Summative tab, simply click on “Progress on Relevant Objectives” to open the details for each learning objective. 

When you enter the Learning Objectives details, you can see how students rated their learning on that objective and make comparisons to other groups.

Interpreting Averages


Your Average

This displays the course average for the item in the course.

Converted Average Comparison

    This score compares the course relative to the overall mean in the comparison group by telling you if your score is similar, lower, or higher. [The comparison score converts the raw mean score for a class to a scale having a mean of 50 and a standard deviation of 10.] The overall comparison group is all users in the IDEA system. The comparison for disciplines includes only classes in the IDEA database where the instructor selected a specific discipline code during the most recent five years, and for the institution includes only classes at the specific institution during the previous five years. In all types of comparisons, comparison scores are only provided if there are at least 400 classes in the respective comparison group.   See Technical Reports 12 and 18 for statistical details.

    Formative Feedback

      Formative feedback provides you with student observations about how often they observed the Teaching Methods shown to be effective for achieving the learning objectives identified in your course.  When you enter the Formative Feedback area, you can see each item from the instrument and whether your rating results in the suggestion that this is a “strength to retain,” or that you should “retain current use or consider increasing,” or that you should “consider increasing use.”

    INTERPRETING THE TEACHING ESSENTIALS INSTRUMENT 

    Teaching Essentials is a short form (seven items plus a few overall items) that provides formative feedback about teaching methods highly correlated with instructor and course excellence. No information is collected about progress on course objectives as in the Diagnostic Feedback or Learning Essentials instruments. Like other IDEA instruments, the report controls for extraneous factors beyond the instructor’s control and provides converted and comparative scores on excellence of the instructor and the course.  

    Students rate their observations of the seven teaching methods that are highly correlated with ”course and instructor excellence.” Instructors who are consistently observed by students as not practicing these methods should consider adding them to their instructional practice. For each method, there is a teaching “Note,” which will help you better understand the teaching method and how to use it (see all the Teaching Notes on Instruction).   

    The seven Teaching Essentials teaching method ratings

    • Displayed a personal interest in students and their learning
    • Found ways to help students answer their own questions
    • Demonstrated the importance and significance of the subject matter
    • Made it clear how each topic fit into the course
    • Explained course material clearly and concisely
    • Introduced stimulating ideas about the subject
    • Inspired students to set and achieve goals which really challenged them   
    Learn more about the Teaching Essentials instrument.

    INTERPRETING THE LEARNING ESSENTIALS INSTRUMENT

    The IDEA survey’s chief indicator of teaching effectiveness is how well students rate their progress on the types of learning that faculty target. These ratings are part of the Diagnostic Feedback instrument and are the primary component of the Learning Essentials instrument.    

     

    Before administering Learning Essentials, you or someone on your campus, selected the Essential or Important learning objectives for your particular course.  So presumably, these learning objectives were targeted in your course by your course design and instructional practices. Students, then, rate how well they believe they achieved these objectives. When your collective rating on these items is low, it indicates that students believe they did not achieve those objectives very well, and that perhaps, you should consider how you have designed learning around those objectives.   

    For each objective identified as Important or Essential, you get the following report:  

    In addition, Learning Essentials includes the “overall” or summative items included with all IDEA instruments: “Overall, I rate this instructor an excellent teacher, and “Overall, I rate this course as excellent.” The ratings on these items may be used to monitor overall student impressions of a course and instructor. 

    INTERPRETING THE UNIT SUMMARY REPORT

    The USR uses data from the Diagnostic Feedback and Learning Essentials instruments
    The Interpretive Guide will walk through through the various components.

    CREATING A PLAN FOR IMPROVING INSTRUCTION

    Each of the IDEA Student Ratings of Instruction instruments provide a number of opportunities for you to work toward improving instruction. Whether you simply use the results to improve your teaching on your own, or as part of a collaborative effort with a colleague, or more formally, as part of an annual professional development plan, the results from each instrument can help guide you toward where you might focus your efforts.    

    For instance, if students do not rate you very highly on the method, “Made it clear how each topic fit into the course,”  (as indicated in the results of your Diagnostic Feedback or Teaching Essentials instruments), you might target this as an area for learning more about and increasing its use in your courses. Document this targeted area for improvement in your annual review report, and develop a plan for it. This may include workshops, conferences, working with your campus Center for Teaching Learning, and as well as the resources provided within your report.

    Both the Diagnostic Feedback and Teaching Essentials instruments include links to the Teaching Methods Notes pages for each teaching method. Clicking on that link will take you to a summary of each method, suggestions for implementing it, and references. These notes are a good place to start as you work to increase your use of any particular teaching method.   

    In the Diagnostic Feedback and Learning Essentials instruments, you get feedback on how well students think they achieved the learning objectives identified as Essential or Important in your course. Research has demonstrated that low ratings on a learning objective is correlated with poor performance on that objective in the course. So consistent low ratings on an item likely means there is room for improvement in how that objective is taught.   

    When you have identified objectives with low ratings, consider the factors involved. If you have aligned the objectives with your specific course learning outcomes (See About Learning Objectives), look at other evidence you have for student performance on those learning outcomes. Are students indeed scoring poorly on those outcomes? If so, the combined evidence suggests that students are not mastering that objective as well as they should and instructional changes are warranted. Be sure to read the Notes on Learning for each of the learning objectives to learn more about methods of teaching that are most effective for each.     

    Of course, you have to interpret the reports in light of your course. For instance, just because a particular teaching method is not observed much, it does not necessarily mean you should increase your use of it. Some methods are more appropriate for some disciplines and courses than others. But do not dismiss a low rating without seriously considering it. Work with your instructional lead (department chair, dean, etc.), or contact IDEA, to identify the findings that are more relevant for you and work toward improvement.       

    Using Instant Feedback

    Using Instant Feedback

    Instant Feedback is a tool you can use at any time during a course to gather formative feedback. That is, feedback that can help you make adjustments to the course during the course rather than after the fact when it is too late for the students in the course. 

    So you might, for example, ask students to complete the feedback at the end of a class that was particularly challenging, or after using a new instructional method, or teaching a topic for the first time. Using six of the items from Teaching Essentials instrument and the global question, “You understood the material covered today—yes or no” the feedback will help you decide if you need to backup and revisit some parts of the unit or can continue.    

    You can administer Instant Feedback as often as you wish. The results are only seen by you, not by administrators or anyone else. You get results emailed to you almost immediately after the survey ends.  

    The quick Instant Feedback Instrument:


    How to use Instant Feedback

    The Instant Feedback tool must be enabled by your On-Campus Coordinator. Once enabled, you can use it as many times as you wish during the semester. You can set the survey to end automatically two hours after it begins, or you can manually end it after a class. Either way, your results are emailed to you almost immediately.   Keep in mind that student enrollments must be setup in your campus system before students can access the survey. Be sure to communicate with your On-Campus Coordinator about this so that you know when enrollments are complete and students are able to access the survey. 

    You send students to the survey just like you do the end-of-term surveys. You direct them to your campus portal for surveys.

    From their course list, they will see the Instant Feedback link for your course. The survey can be completed in less than a minute.   

    It is recommended that when you use Instant Feedback in face-to-face courses, you ask students to complete it before leaving the classroom so as to capture the feedback immediately for your use. In online courses, students could complete Instant Feedback at any time after a unit or lesson.      

    Sample Instruments

    Objectives Selection Form
    Diagnostic Feedback - sample student survey
    Learning Essentials - sample student survey
    Teaching Essentials - sample student survey
    Instant Feedback  - sample student survey

    Copyright Notice: The survey instruments listed below are copyrighted by The IDEA Center, Inc. No part of the survey instruments or content may be adapted, duplicated or reprinted without prior written consent from The IDEA Center. Any such use will be viewed as copyright infringement and a violation of academic integrity. All rights reserved.

    Campus coordinator resources

    Institutional Access to Data

    In addition to online access to individual course reports and unit summary reports, IDEA provides several ways for institutions to access raw and aggregate data for customized analyses. If after reviewing this information you have questions about how to access or use IDEA data, email us at data@IDEAedu.org.  

    Raw Data (CSV File)
    Overview: Course Evaluation Administrators have the ability to export raw data from the course evaluation surveys per term. The export pulls all data into a single export file. It shows the result for each student’s responses in a row.  Sample Raw Data Export File [.xlxs]

    Data included: The export files include raw data with one question per column; no calculations are provided. The file includes:  

    • Unique student identifier (generically assigned and does not relate back to the real user’s ID)
    • Course Section External ID
    • Course Subject/Catalog/Section/Title
    • Assigned Institutional Unit
    • Administration Name
    • Administration Dates
    • Instructor and Instructor External ID
    • Course Section Attributes
    • Process: The ability to export data is limited to Course Evaluation Administrators. Refer to the following link for instructions. http://courseevaluationsupport.campuslabs.com/hc/en-us/articles/204024748-Exporting-Raw-Data
      Aggregate Data File (ADF) (XLS file) 
      Overview: A Designated Administrator (determined by the institution) may request an Aggregate Data file (ADF) that includes course data for a specified period of time.  The file provides aggregated and calculated data on a course-by-course basis for the entire institution. Sample Aggregate Data File [.xlxs]

      Data Included:
      Aggregate course data are provided in an Excel spreadsheet for the entire institution for the period of time defined on the Aggregate Data Request Form. Each row provides aggregate data for each individual survey. In addition to the aggregate data fields, the spreadsheet includes several tabs to further break down the data into consumable bites, which include, but are not limited to:
      • Relevant Objectives - Learning objectives that were selected as important or essential for each course
      • Means - the means and T-scores (raw and adjusted)  
      • Frequencies  - the frequency of each response option for each question
      Process: To protect institutional data, the data files are encrypted, therefore requests for Aggregate Data files (ADF) will only be accepted from authorized individuals. Requests can be made by completing a simple online form.  The institutional designee will be contacted by an IDEA representative following the initial request to set up the process for receiving the encrypted data files. It is advised that the request for aggregate data files be made after the survey close date for the last course in the term to ensure that all requested course data are included in the file. The turnaround time is typically 48 hours, however there may be exceptions during heavy processing time.

      IDEA Data Portal (JSON data via REST API)
      The IDEA Data Portal enables programmatic access to your institution’s IDEA survey and report data, on demand 24/7. This can be used to build integrations with on-campus archives, research, dashboards, and/or learning management systems. This is the preferred mechanism for accessing data, particularly if there is software development personnel on staff familiar with REST APIs.  

      Data included:
      The Data Portal provides access to data from the Student Ratings of Instruction system as well as IDEA Administrator and Chair Feedback Systems. More information about the REST API and JSON data can be found https://rest.ideasystem.org/IDEA-REST-SERVER/

      Process:
      The Data Portal is a new product that will be available as a subscription service. While it is in “beta” testing, there is no charge for this service. Requests for information on the Data Portal can be sent to data.portal@IDEAedu.org


     

    Campus Labs Support

    Our technology partner, Campus Labs has additional resources available.

    Share

    IDEA

    301 South Fourth Street, Suite 200, Manhattan, KS 66502
    Toll-Free: (800) 255-2757   Office: (785) 320-2400   Email Us

    Connect
    GuideStar Gold Participant