• Student Engagement: Attempting Better Measures of a Subjective Experience (11/9/2015) by Phillip Posillico

    with Ben Keeler

    In some of our previous blog posts we have discussed topics such as School Climate, Social-Emotional Learning, Digital Literacy, and Blended Learning. In future posts we plan on delving into the debates surrounding 21st Century learning, Parent-Teacher Relationships, Teamwork/Collaboration and Academic Motivation. Today’s post is about Student Engagement.

    The first step in measurement is always to agree on a clear definition. After sorting through the best academic literature on the topic, we’ve arrived at this one as a starting point:

    Students are engaged with learning when they are involved in their work, persist despite challenges and obstacles, and take visible delight in accomplishing their work. Student engagement also refers to a student’s willingness, need, desire and compulsion to participate in, and be successful at, the learning process.

    It is not always easy to notice students who are engaged. It’s often easier to point out the ones who aren’t. Students texting during class, interrupting the teacher or refusing to take notes are some easy signs of disengagement. This poses some problems for measurement, and as a result, many studies have fallen back on measuring engagement indirectly through more traditional metrics.

    For instance, in a research article by Leah Taylor and Jim Parsons at Arizona State University, they point out that traditional strategies for measuring student engagement “have predominantly focused on quantitative data such as attendance, standardized test scores, and truancy or graduation rates. The majority of these measures track levels of achievement (outcomes such as high scores, full attendance for the year) but not levels of student engagement in learning (interest, time on task, enjoyment in learning).”

    Attempting to improve on this issue, The Regional Educational Laboratory at the Institute for Education Sciences (IES) suggests utilizing “classroom observations of one to four hours occurring two to five times a year” as well as a 121-item student self-report questionnaire taken a handful of times a year.

    We wanted to take a much different approach, especially since we’re in the business of providing better measurement for schools while minimizing disruption to the daily routine. We couldn’t exactly convince a school to have us come in and do classroom observations for them. We also think a 121-item questionnaire in one sitting would be trying on an adult, let alone a teenager.

    So we’re starting with self-report data, much like the IES questionnaire, sent multiple times throughout the year, but instead of 121 questions at once, we’re sending a couple a week, right to mobile devices. We understand that academic engagement can come and go, therefore our data collection is constant so that we can track the ebb and flow of responses over time.

    Some sample questions from our Student Engagement module include:

    • Today I was interested in the things I learned in school.
    • Today, being at school was important for achieving my future goals.
    • Today, I liked it when school subjects made me think hard and figure things out.

    By asking questions such as these more than once over the course of a year, we end up being able to report back to schools very nuanced graphs showing student engagement over time. This is literally impossible to do with a once- or twice-yearly survey of student engagement, such as the IES questionnaire mentioned above.

    Yes, engagement is a subjective experience, something that’s sometimes hard to articulate, but it’s much more accurately measured if we ask an individual these questions multiple times. We can find both an average of all of these questions (which will provide a better measurement than a one-time survey could) but also a nuanced look at the changes in student engagement throughout the year. This is where we need to head if we ever want to illuminate what’s really going on in students’ lives day-to-day in schools.

Tweet about this on TwitterShare on Facebook1Share on LinkedIn0Share on Reddit0Share on Tumblr0Print this pageEmail this to someone
Breakthrough II campus

Oct 22, 2015 How Digitizing a Student Character Grading Rubric Changed A Magnet School

EduMetrics’ product iNOTED is a flexible platform that is helping schools collect information that helps their mission and curricular goals. The EduMetrics team works with each school to help determine which areas would help advance the goals of their school based on our available modules. However, at times there is something unique that a school wants to assess. In those cases, the research team at EduMetrics works with staff at the school to create an assessment to meet the school’s needs. Here is an example of what this looks like at one of our schools. One of our partnering schools, Breakthrough...

Read More

Aug 28, 2015 The Enduring Impact of Trauma in Schools, How Social-Emotional Learning Could Help

In the last post Phil talked about social-emotional learning. In this blog post Liz will be sharing the impact of Hurricane Katrina in the work that she does at EduMetrics, and in particular the importance of social-emotional learning in schools. This time of year always brings up a lot of things for Liz, in particular, the stories her former students told her about Hurricane Katrina. These are her opinions and experiences. Liz is a research associate working on the Templeton Study. You can read more about the study here. Five years ago in the beginning of August I was in an...

Read More

Jul 15, 2015 Social-Emotional Learning: Success Through Character

This blog post summarizes the discussion surrounding Social-Emotional learning, how we measure it, why it’s important and how it impacts other aspects of the student experience. We are looking to initiate a constructive dialog by contributing our methods of measurement and evaluation to the conversation in order to develop more effective research strategies. Social-Emotional learning (SEL) is broadly defined as the process in which students acquire knowledge and skills that help them positively manage their emotions and navigate difficult social situations. Educators and researchers are giving social-emotional learning more and more significance as a factor of influence on the holistic...

Read More

Jun 18, 2015 School Climate Research – A Trivial Pursuit or a Catalyst for Change?

What educators need to better understand the condition of their school’s climate. When it comes to educational research and reform, a schism of perspectives has been created within the field that divides school climate in two: School climate research as a necessity for progress and school climate research as a waste of time. There seems to be very little middle ground. Carolyn S. Anderson of the American Educational Research Association recognized that researchers fall into two categories when it comes to the search for school climate: Pessimists and Optimists. Pessimistic researchers view school climate as a Unicorn: a mythological creature that can be...

Read More

Mar 18, 2015 Character In Schools Part I: What Exactly Is “Character?”

This is a first in a series of three posts about student character in schools. In previous posts, we’ve described why we should measure character and how we are measuring character. This post will provide the definition of character as we see it here at EduMetrics. In future posts, we’ll be diving into what character traits are essential to measure in schools and expand upon our previous posts about how to most accurately measure these traits. To paraphrase a saying I heard a professor once say, you can’t improve what you can’t measure, and you can’t measure what you can’t precisely...

Read More

Mar 09, 2015 What I Learned At The Highlander Institute’s Blended Learning Conference

This weekend I attended The Highlander Institute’s 4th Annual Blended Learning Conference in Providence, Rhode Island at the University of Rhode Island Providence Feinstein Campus. The Highlander Institute uses research-based solutions to help improve opportunities for educators, students, and families. Their work includes blended learning, expanded learning time, and meeting the needs of diverse learners. The conference was designed to bring together, teachers, administrators, and experts in the field to discuss current trends and practices occurring to integrate technology into schools. Guest speakers included Rhode Island’s Commissioner of Education, Deborah Gist, First Gentleman, Andy Moffit, and keynote speaker author and former...

Read More

Feb 10, 2015 Our Contribution To The Grit Debate: Better Data

"Grit; that certainly is a hot topic." Last week, I was talking to a colleague about the research I’m working on. As announced previously in our blog, we’re currently conducting a year-long study using experience sampling methods to assess student character. We’re looking at three overarching themes of character development: diligence/grit, compassion, and honesty. After explaining this to my colleague he simply responded with the statement above. This is usually how conversations go down when I talk about the study.  While most people can agree that compassion and honesty are key components of a student’s character development, controversy surrounds the topic of diligence...

Read More

Jan 29, 2015 To Legitimize Character Education Programs, We Must Measure Student Character Better

A major obstacle standing in the way of comprehensive character education is that current measures of program effectiveness lack scientific accuracy. This makes it difficult to evaluate whether specific curricula are actually improving kids’ capacities. It’s also really hard for school leaders interested in implementing character education programs to compare commercially available options because each program uses different methods to measure effectiveness. But the experience sampling approach to data collection, pushing a few questions a day to devices carried around by students and teachers, could blow the door open on this field and change how we approach making sure these character education programs...

Read More

Jan 21, 2015 How “Experience Sampling” Could Revolutionize Teacher Grade Reports

Think of a typical progress or grade report a teacher writes each quarter. Most of them require a teacher to fill out a rubric or checklist to indicate progress on a variety of behaviors, like active listening, respecting others and staying on task. But what are teachers actually basing these ratings off of after several weeks of school have passed? If we're honest with ourselves (and I spent eight years as a middle and high school history teacher, so I'm included here!), we'd admit that most of us  base them off of vague recollections of past behavior that was either exceptional...

Read More

Dec 01, 2014 EduMetrics Awarded $250,000 from the John Templeton Foundation

We are excited to announce that we have been awarded a grant from the John Templeton Foundation in response to their funding competition entitled, "New Assessments and Measures of Virtue." We are honored to gain recognition from such a renowned organization that has consistently funded innovative projects to answer life's Big Questions. EduMetrics has been selected from a pool of over 100 highly competitive proposals, nine of which were selected to receive funding. The $250,000 grant funds a 12-month longitudinal study of 500 middle school students at five schools throughout New England. The main approach will be to send a small...

Read More