By Steve Benton
A little over two years ago during IDEA research staff meetings and at coffee breaks my colleagues and I began to have conversations about which IDEA learning objectives should be dropped or retained in the soon-to-be updated instrument—IDEA Diagnostic Feedback 2016
. Should we keep both Objectives 1 (factual knowledge) and 2 (principles and theories) even if student ratings of progress are highly correlated? How can we modify Objective 6 (creative capacities) to make it more likely STEM faculty will identify it as essential or important? Is Objective 10 (values development) outdated due to its emphasis on values rather than ethics? And what do we do with Objective 12 (interest in learning), which seems redundant to Objective 9 (find, use resources)? Finally, we all wondered whether there were any essential learning outcomes missing in the 12 IDEA objectives?
These questions hounded us and, I must admit, sometimes made it difficult for me to sleep at night. My colleagues and I knew that any changes we made to the longstanding 47-item IDEA Diagnostic Form
could have significant impact on faculty, students, administrators, and their institutions. How would you—the users of IDEA—respond to any changes? So, it has been with great care and much thought that we have arrived at the revised set of 13 IDEA learning objectives. This “baker’s dozen” is supported by extensive analysis of IDEA’s longitudinal database, as well as input from focus groups and expert panels, which were comprised of users and non-users of IDEA, faculty, students, administrators, item developers, psychometricians, and statisticians.
Probably the most noticeable difference is in the four objectives dropped from the existing instrument. The first two objectives—Gaining factual knowledge (terminology, classifications, methods, trends)
and Learning fundamental principles, generalizations, or theories—
will be replaced by a single item: Gaining a basic understanding of the subject (e.g., factual knowledge, methods, principles, generalizations, theories)
. We had several reasons for making this change. First, faculty ratings of importance and student ratings of progress on the first two objectives are highly correlated. We were not surprised to learn, then, that faculty and students reported they found the two items redundant. In the pilot study conducted last spring we learned that the existing two items and the new item had almost identical means and standard deviations on student ratings of progress. So, we are confident the single new item will provide a valid measure of student comprehension of the subject matter.
Objective #10—Developing a clearer understanding of, and commitment to, personal values
—will also be replaced by a new item: Developing ethical reasoning and/or ethical decision making
. Faculty and staff expressed concerns that the existing learning objective did not specify the ethics behind one’s personal values. In keeping with changes in the field we also recognized that several higher-education organizations (e.g., AAC&U, DQP) now specify ethical reasoning as an essential global student learning outcome.
We also received feedback that Objective 9—Learning how to find and use resources for answering questions or solving problems
—and Objective 12—Acquiring an interest in learning more by asking my own questions and seeking answers
shared some common elements. The latter item also had the lowest correlation among all the objectives between faculty ratings of importance and student ratings of progress. We decided, therefore, to create a new item that replaces Objective 12 and modifies #9: Learning how to find, evaluate, and use resources to explore a topic in depth
While we were at it we also modified Objective 6 by rearranging the parenthetical content: Developing creative capacities (inventing; designing; writing; performing in art, music, drama, etc.)
. With this change we hope STEM faculty will no longer view this as the “creative writing objective.”
In addition to dropping and modifying existing items, we created three new items to assist institutions in assessing student progress on other essential learning outcomes defined by professional organizations:
- Developing knowledge and understanding of diverse perspectives, global awareness, or other cultures (diversity and global awareness)
- Learning to apply knowledge and skills to benefit others or serve the public good (civic engagement)
- Learning appropriate methods for collecting, analyzing, and interpreting numerical information (quantitative literacy)
We hope the 13 learning objectives in IDEA 2016
align with the student learning outcomes emphasized in your academic unit. We are excited about the changes and hope you will be too. If you have any questions or concerns, let us know. We are here to help.