A slice of PSA

 

I was exploring for new resources for my Fall grad class and I came across this McKinsey and Company. The pdf contained a number of findings based on an analysis of the 2015 PISA data. Educators will recognize PISA as the Programme for International Student Assessment an internal assessment of student achievement often used to question the effectiveness of U.S. K12 education. That perspective aside, PISA does more than compare the achievement nation to nation because it also collects questionnaire data from students and educators allowing researchers to relate these data to the achievement data.

The researchers from McKinsey had used these data to identify multiple "drivers" of achievement outcomes. I was particularly interested in their conclusions regarding relationhips between science achievement and whether student classroom experiences had emphasized teacher-directed or inquiry-based experiences. Because the focus of the work was on science achievement and I imagined science to be a subject matter often supported by laboratory experiences, I thought the results relating these learning experiences to achievement would be particularly interesting.

Without getting too far into the methodological or statistical weeds, the conclusions were explained in the following chart. The various survey questions allowed the categorization of the classroom experiences students had experienced into levels of teacher-directed or inquiry-based experiences (few, some, many) with direct instruction showing a more positive relationship to achievement overall. The researchers concluded there was a sweet spot with some inquiry-based experiences combined with mostly teacher-directed experiences when it came to the best outcomes.

These findings likely frustrated the many constructivists who constantly berate the textbook and presentation-based approaches that limit student-centered activities, but the findings are consistent with the major reviews I have read. The researchers used individual items (e.g., students design their own experiments, students are asked to argue about investigations) to suggest that when inquiry activities are used structured tasks are likely to be most effective. One hypothesis proposed by the researchers was that students don't have sufficient background in many cases to benefit from inquiry activities.

The specifics of the survey questions used by PISA interested me because I always find the methods section of research students to be helpful and I could not find the specifics I was looking for in the McKinsey study. I also tend to value research studies in journals over "think tank" analyses so I did some searches for PISA 2015 research. I came across a study (Cairns, 2019) more to my liking.

Cairns focused specifically on the inquiry tasks and what differentiated effective from ineffective experiences. If you looked at the individual items making up the multiple survey items what could you discover? The author noted that the term "inquiry task" is not clearly conceptualized in the literature making this approach of potential value. Cairns also noted that more experimental investigations of inquiry tasks have been more successful in supporting inquiry methods than correlational methods (the PISA study would be considered correlational as it evaluated existing instructional conditions and the relationship to achievement rather than controlling the research setting). The results resembled the findings of the McKinsey study and emphasized the importance of scaffolded experimentation (not student-designed experiments) with instructors explaining the implications of the core principles demonstrated by the experiments.

I was able to find examples from the survey questions elsewhere.

Teacher-directed items.

Inquiry items

Bryant, J., Dorn, E., Krawitz, M., Kihn, P., Mourshed, M., & Sarakatsannis, J. (2017). Drivers of student performance: Insights from North America. New York: McKinsey & Company.

Cairns, D (2019) Investigating the relationship between instructional practices and science achievement in an inquiry-based learning environment, International Journal of Science Education, 41:15, 2113-2135