Determining how and when empirical results are generalizable is critical to increase the impact of academic research. It is also a valuable thinking skill for non-academics. Hence, we need to build this skill into our educational offerings at all levels.
Since doctoral students are academics in training, it shouldn’t be hard to persuade them that performing a replication is good practice which can help them learn why researchers made specific design choices. Additionally, performing a quasi-replication might be a straightforward way to satisfy the doctoral program’s research paper requirement. Strategic Management Journal already solicits replication studies “…that accord with prior findings as well as those that do not.“ In other cases, quasi-replications that generate different results offer researchers the opportunity to theorize about the differences across the original and reconstituted studies, suggesting a moderation effect across contexts.
How can we teach generalizability to undergraduate students? I teach the course “Culture and Institutions of the Tech Sector: Bridging Research and Practice.” Each session we read an academic article that demonstrates a particular phenomenon via systematic evidence. As an example, for serial entrepreneurship, we read the classic “Entrepreneurship and Performance Persistence” (Gompers et al., 2009), which finds that founders with prior founding experience are more likely to experience successful exits than first-time founders. Then we invite guest speaker panels to push our critical thinking. Serial founders with investment experience like Joseph Ansanelli W’92 (now at Gladly), Amy Errett WG ’88 (now at Madison Reed), and Andy Rachleff W’80 (now at Wealthfront) join the class to provide “color commentary” – perspective informed from their experiences both as serial founders and venture capitalists (VCs). Students have asked them:
- Since the Gompers conclusion was drawn from startups that received VC funding before 2007, would it still be applicable today?
- Gompers defined a successful exit as an IPO or an acquisition over $50 million. What would VCs consider as successful today?
- How has prior founding experience affected your view of investment opportunities?
Then, for their group projects, students choose one of the assigned academic articles and assess whether its findings would generalize to a new context. This context might reflect a different time, industry, region, practice or subject pool. Students utilize a combination of academic papers, expert interviews, and data sources to make a logical argument as to whether and how original findings might generalize to their chosen context. So while younger students may not have the econometric skills or the time horizon to undertake a full quasi-replication, they are quite capable of developing working hypotheses about how contextual differences might affect the generalizability of the original results. To demonstrate the impact of repeated extensions of these key studies, we’ve created a generalizability map spanning projects over all semesters to ensure that each student group chooses a new context. After five iterations of the class and over forty project teams, students can see both how findings might generalize across multiple contexts, as well as the limits of generalizability. It’s been particularly rewarding to see how multiple groups choosing the same target paper are so interested in how others approached the same paper with a different question!
How are you encouraging students at all levels to think critically about the implications of research findings in our field?