Determining how and when empirical results are generalizable is critical to increase the impact of academic research. It is also a valuable thinking skill for non-academics. Hence, we need to build this skill into our educational offerings at all levels.
Since doctoral students are academics in training, it shouldn’t be hard to persuade them that performing a replication is good practice which can help them learn why researchers made specific design choices. Additionally, performing a quasi-replication might be a straightforward way to satisfy the doctoral program’s research paper requirement. Strategic Management Journal already solicits replication studies “…that accord with prior findings as well as those that do not.“ In other cases, quasi-replications that generate different results offer researchers the opportunity to theorize about the differences across the original and reconstituted studies, suggesting a moderation effect across contexts.
How can we teach generalizability to undergraduate students? I teach the course “Culture and Institutions of the Tech Sector: Bridging Research and Practice.” Each session we read an academic article that demonstrates a particular phenomenon via systematic evidence. As an example, for serial entrepreneurship, we read the classic “Entrepreneurship and Performance Persistence” (Gompers et al., 2009), which finds that founders with prior founding experience are more likely to experience successful exits than first-time founders. Then we invite guest speaker panels to push our critical thinking. Serial founders with investment experience like Joseph Ansanelli W’92 (now at Gladly), Amy Errett WG ’88 (now at Madison Reed), and Andy Rachleff W’80 (now at Wealthfront) join the class to provide “color commentary” – perspective informed from their experiences both as serial founders and venture capitalists (VCs). Students have asked them:
- Since the Gompers conclusion was drawn from startups that received VC funding before 2007, would it still be applicable today?
- Gompers defined a successful exit as an IPO or an acquisition over $50 million. What would VCs consider as successful today?
- How has prior founding experience affected your view of investment opportunities?
Then, for their group projects, students choose one of the assigned academic articles and assess whether its findings would generalize to a new context. This context might reflect a different time, industry, region, practice or subject pool. Students utilize a combination of academic papers, expert interviews, and data sources to make a logical argument as to whether and how original findings might generalize to their chosen context. So while younger students may not have the econometric skills or the time horizon to undertake a full quasi-replication, they are quite capable of developing working hypotheses about how contextual differences might affect the generalizability of the original results. To demonstrate the impact of repeated extensions of these key studies, we’ve created a generalizability map spanning projects over all semesters to ensure that each student group chooses a new context. After five iterations of the class and over forty project teams, students can see both how findings might generalize across multiple contexts, as well as the limits of generalizability. It’s been particularly rewarding to see how multiple groups choosing the same target paper are so interested in how others approached the same paper with a different question!
How are you encouraging students at all levels to think critically about the implications of research findings in our field?
Congratulations, Lori, on this important initiative. Im excited to hear that undergraduates as well as doctoral students are thinking about this. Questions about generalizability of some of my empirical findings have led me to think much more carefully about contingencies and context. It’s a terrific lens on what’s really going on in a situation. Bravo.
I love this project idea, Lori! For the MBA students in my core OB class, I have teams of students frame a novel organizational problem facing a specific organization. They need to relate the problem to a core topic in the course, do additional library research on the topic, and then develop solutions for the organization. The generalization aspect of this assignment proves challenging and we spend a lot of time framing and reframing the problem together to figure out how to apply the research findings in specific ways to their problems. I am now wondering how I can extend this project to incorporate a generalizability map, which would potentially be very useful.
This semester I started off an undergrad second-year course on Management of Technology with an individual assignment to choose three technologies: one they were interested in (A), a second that was similar (B), and third that was quite different (C). Then for each of the three pairs (A&B, A&C, B&C), use chatGPT to compare and contrast the technologies. Then elevate beyond the three chatGPT scripts to identify characteristics of technologies that could be used to dimensionalize a technology space – where was chatGPT strong and what did it miss? This pre-work provided the fodder for a great in-class discussion about generalizing (and not generalizing) across technologies.
I like the idea of a generalizability map! Even before that, I find it compelling to raise UGs’ awareness about their research endeavours having meaningful implications; at the early stage of HE, they may still be quite naive about this aspect!
Discussing the practical use of a technology that is close to their heart (beyond a smartphone!) is terrific! thanks for sharing that!
This is great and I appreciate all of the comments. From the standpoint of qualitative research (but I think this comment applies more broadly), it is important to remember that all research is contextual and all future contexts may be different. Thus, when thinking about conclusions from research, we might use a more “open systems” approach in which the ground underneath knowledge claims is always moving. We might think of conclusions from research as building up a stock of principles which can be useful in a repertoire of tentative theories that can help understand new conditions. Thus, theories would be useful, not as “generalizable”or absolute knowledge but instead as something that either is or is not useful for the new context. If the theory doesn’t fit, it is not necessarily a failed theory but simply one that does not help in that particular context but should remain part of the repertoire as circumstances may evolve and change. Mike Pratt and Richard Whittington and I wrote more about this here: Pratt, M. G., Kaplan, S., & Whittington, R. (2020). Editorial Essay: The Tumult over Transparency: Decoupling Transparency from Replication in Establishing Trustworthy Qualitative Research. Administrative Science Quarterly, 65(1), 1–19. https://doi.org/10.1177/0001839219887663