Proposal Development
The VIP program at Georgia Tech has compiled and summarized the studies below, which may be useful to VIP team leaders who are writing proposals for extramural funding or other engagement. Additional information on the VIP model's efficacy can be found in the Publications page.
Meaningful gains for VIP students, GA Tech exit surveys
Takeaway: There were statistically significant and meaningful gains for VIP students (compared to non-VIP students) in three areas: ability to work in a multidisciplinary team, ability to work with individuals from diverse backgrounds, and understanding of technology applications relevant to your field of study.
Scope: Compared GT exit surveys for VIP participants (N=233) and non-participants (N=1,781)
Georgia Tech administers exit surveys to undergraduates prior to graduation. The survey includes questions on how the students’ experiences at Georgia Tech contributed to specific skills and abilities. VIP staff identified 17 questions related to typical VIP program activities. The Georgia Tech VIP program was established in Spring 2009. At the time of the analysis (autumn of 2015), 384 VIP participants had graduated, and 233 had completed the exit survey. The GT Office of Assessment created a comparison group of non-VIP participants using stratified random sampling and performed a t-test to compare survey results for the two groups. For the 17 questions, VIP participants had generally higher ratings than non-VIP participants. The difference was statistically significant on six questions:
- Understanding of technology applications relevant to your field of study
- Ability to work with individuals from diverse backgrounds
- Ability to work in a multidisciplinary team
- Ability to carry out projects independently
- Development of leadership skills
- Overall career training
Of the six questions, there was a meaningful effect size (Cohen’s d of .2 or greater) on three questions: To what degree did your Georgia Tech education contribute to your:
- Ability to work in a multidisciplinary team (t(1982) = 4.437, p < 0.001, d = 0.313);
- Ability to work with individuals from diverse backgrounds (t(1987) = 3.271, p = 0.001, d = 0.231);
- Understanding of technology applications relevant to your field of study (t(2002) = 3.19, p = 0.001, d = 0.224).
These results motivated the more recent analysis of peer evaluations to look at student interaction by discipline and background (race/ethnicity).
Citation (IEEE): J. Ludlum, “Georgia Tech Exit Survey Research Brief: VIP Program Comparisons,” Georgia Institute of Technology, 2015.
Student interactions within VIP by major and by race/ethnicity
Takeaway: Within VIP, students interact more often with students from other races/ethnicities than their own and more often with students from majors other than their own.
Scope: GA Tech peer evaluations, Spring 2016 (N=355)
Summary: Seeing value in Melkers’ social network analysis of VIP teams, the GT VIP program included social network questions in peer evaluations. We conducted an analysis of Spring 2016 peer evaluations. External-Internal (EI) indexes indicated that on average, students interacted more often with students of other races/ethnicities than their own and more often with students from other majors. Quadratic Assignment Procedure (QAP) analysis showed that of the 24 teams: one had significant correlation between race and student interaction, and clumping was was clear on visual inspection; six had significant correlation between student major and interaction (clumping by major was not unexpected). The clumping on one team by race/ethnicity led to the development of additional professional development for instructors.
Citation (IEEE): J. Sonnenberg-Klein, R. T. Abler, E. J. Coyle, and H. H. Ai, “Multidisciplinary Vertically Integrated Teams: Social Network Analysis of Peer Evaluations for Vertically Integrated Projects (VIP) Program Teams,” presented at the 2017 ASEE Annual Conference & Exposition, 2017.
Peer help and peer management by major, academic rank, and status as new/returning VIP student
Takeaway: Within most VIP teams (85%), there is no correlation between major and who students go to for help. Returning students provide more peer help and peer management regardless of their academic rank. This confirms that returning students take on leadership roles within their teams, both in terms of providing help and team management.
Scope: GA Tech peer evaluations, Spring 2017 (N=484)
Summary: Analysis of Spring 2017 peer evaluations looked at peer support (giving help) and peer management by major, academic rank and status as new or returning VIP students. External-Internal (EI) indexes showed more peer support between students of the same major, but when team composition was accounted for in Quadratic Assignment Procedure (QAP) analysis, much of this correlation fell away. There was significant correlation between major and giving/receiving help (clumping by major) on five (15%) of the 34 teams studied. For 29 (85%) of the 34 teams, major was not correlated with giving/receiving help, representing a balance of peer helping across disciplinary lines.
A UCINET version of analysis of variance was used to examine student rank with respect to giving help and management ratings, as well as student status as new or returning students. Student rank did not show statistical significance in who students went to for help or how students rated teammates on management ability. In contrast, student status as new or returning VIP students was statistically significant on both measures, with returning students rated higher. This shows that the social capital within teams is tied to students’ prior experience on the teams, and not necessarily their academic rank. The findings also confirm that returning students take on leadership roles within their teams, both in terms of providing help and in team management.
Citation (IEEE): J. Sonnenberg-Klein, R. T. Abler, and E. J. Coyle, “Social Network Analysis: Peer Support and Peer Management in Multidisciplinary, Vertically Integrated Teams,” presented at the 2018 ASEE Annual Conference & Exposition, 2018.
Diversity and student persistence in the VIP course sequence
Takeaway: Nationally, Black/African American and Hispanic/Latinx students (historically underserved, HU) participate in undergraduate research at lower rates than their peers. The GT VIP Program enrolls representative proportions of HU students with respect to the campus population. After their first semester of VIP, students enroll for second (and more) semesters of VIP at the same rate, regardless of race/ethnicity, representing equity in student persistence in the program.
Scope: Analysis of student enrollment and number of semesters in VIP with respect to race/ethnicity (N=1,251)
While historically underserved (Black/African American and Hispanic/Latinx) students derive greater benefits from undergraduate research experiences with faculty, across the nation, historically underserved students participate at lower rates than their peers. In contrast to the national trend, Georgia Tech’s VIP Program enrolls representative proportions of Black/African American students and Hispanic/Latinx students with respect to the campus population. An analysis of enrollments over time looked at the number of semesters students enrolled in VIP with respect to race/ethnicity. Findings indicate that students of different races and ethnicities persist in the VIP course sequence at equal rates.
Citation (IEEE): J. Sonnenberg-Klein, R. T. Abler, and E. J. Coyle, “Diversity and Student Persistence in the Vertically Integrated Project (VIP) Course Sequence,” presented at the 2018 Collaborative Network for Engineering and Computing Diversity (CoNECD), 2018.
Academic credit-use policies and number of semesters in VIP
Takeaway: There is a correlation between department policies on how VIP credits count toward degree requirements and how many semesters students participate in VIP. Two policy elements correlate with higher persistence: (1) allowing all VIP credits to count as in-major electives after a minimum number are earned and (2) allowing students to fulfill culminating design requirements through VIP, with no additional planning/requirements beyond the normal design sequence. Matches and mismatches between student major and instructor home department are not correlated with student persistence in the program.
Scope: Analysis of Persistence in the VIP Course Sequence with Respect to Major (two analyses, N=869, N=45)
A key to launching and maintaining productive VIP teams is having students participate for multiple semesters. This enables deeper learning experiences and provides the time for students to develop deeper expertise and take on increasing levels of responsibility. Academic departments have established a range of credit-use policies to incentivize their students' multi-semester participation, with different incentives and varying thresholds for incentives. Other factors that may affect student persistence are matches/mismatches between students and their instructors’ departments, and student academic rank in their first semester of VIP. This study profiles policies for the four largest academic units participating at Georgia Tech and examines student persistence (N=869) by department, by academic rank, and by matches-mismatches between student and instructor departments. In a secondary analysis, persistence rates are compared for a degree program before and after an incentivizing credit-use policy was established (N=45). Results indicate two elements correlate with higher persistence: (1) allowing all VIP credits to count as in-major electives after a minimum number are earned and (2) allowing students to fulfill culminating design requirements through VIP, with no additional planning/requirements beyond the normal design sequence. Matches and mismatches between student major and instructor home department are not correlated with student persistence in the program. This indicates that instructors leading these multidisciplinary teams are equally engaging students from outside their own departments.
Citation (IEEE): J. Sonnenberg-Klein, R. T. Abler, and E. J. Coyle, “Correlation between Academic Credit-use Policies and Student Persistence in Vertically Integrated Project (VIP) Courses,” presented at the 2018 ASEE Annual Conference & Exposition, 2018.
Learning gains for new and returning VIP students
Takeaway: Students who sought help from other students reported higher learning gains. Returning students reported greater gains in technical skills, team skills, and some communication skills. Higher enthusiasm was correlated with greater gains.
Scope: GA Tech Purdue VIP Programs; N=98 first semester; N=180 second semester. Another campus was initially included, but their numbers were too small to include in published results.
Summary: In Fall 2010 and Spring 2011, an evaluation of VIP was conducted by Julia Melkers on social networks within teams and between teams, and on students' self-reported learning. The study looked at social networks and examined self-reported learning outcomes for students who provided help to teammates; students who sought help from teammates; new students and returning students; and student levels of enthusiasm. The analysis included social network analysis and regression analysis. The social network analysis showed students were interacting within teams, and seeking/receiving advice in different areas (technical, managerial, other). In self-reported learning gains, outcomes for students who provided help to others did not differ, but learning outcomes for students who sought help from others was statistically higher, both in technical skills and managerial skills. In comparing learning gains for new students and returning students, results indicated greater gains for returning students in technical skills, team skills, and some communication skills. For students with higher levels of enthusiasm, results indicated greater gains across various skills, as would be expected.
Citation (IEEE): J. Melkers et al., “The Social Web of Engineering Education: Knowledge Exchange in Integrated Project Teams,” presented at the ASEE Annual Conference and Exposition, 2012.