KCTCS Research Study on Starfish Reveals Positive Impact on Student Success and Retention

In a 2019 report from the Kentucky General Assembly’s Legislative Research Commission (LRC), KCTCS was challenged to conduct research to better understand five prominent nonacademic barriers to student success and the services and supports available to assist in overcoming them. As open-access institutions, students at KCTCS Colleges are more likely to be the first in their family to attend college, academically underprepared, older, care for dependents, and from low-income households than are students at the state’s public four-year institutions. Figure 1 shows a comparison between KCTCS and the state’s public four-year institutions on key social and demographic characteristics. Because of these characteristics, KCTCS students face nonacademic barriers at a higher frequency and intensity than other postsecondary education students in Kentucky.

The nonacademic barriers identified in the LRC report include:

• Inability to navigate college Financial instability
• Competing time constraints
• Personal health issues
• Disengagement

KCTCS utilized a multi-faceted approach to systematically investigate nonacademic barriers facing students and to develop and execute a strategy for conducting research to respond to the recommendations made by the LRC.

The full KCTCS response to the LRC report will be available in Summer 2021. This research brief will focus on the KCTCS response to Recommendation 3 from the LRC report: The KCTCS System Office should study the relationship between programs that decrease nonacademic barriers, student success, and retention. Results should be shared with the colleges so that they can improve existing programs and more effectively implement future programs.

Starfish and Nonacademic Barriers

KCTCS administers over 140 programs designed to address the five nonacademic barriers identified in the LRC report. Administration of these programs requires significant resources, and current assessments are insufficient to determine effectiveness and inform strategic decision[1]making and return on investment. Given the large number and complexity of existing programs, KCTCS made the decision to conduct an in-depth investigation on a single program common across colleges that is used (in part) to address nonacademic barriers, the Starfish Enterprise Student Success Platform (Starfish).

Starfish is an online-based program that leverages the entire campus community to engage and motivate students through increased communications, feedback, and intervention (both academic and nonacademic). College faculty and staff use Starfish to provide encouragement and positive reinforcement to students, to provide helpful reminders about upcoming tasks and deadlines, and to provide early alert notifications that identify students who are underachieving academically. Early alert notifications can be used by advisors and Student Services staff to further engage with students and connect them to useful resources.

A survey of KCTCS Colleges was administered to better understand the use of Starfish at each institution to address nonacademic barriers. Survey results indicated that colleges vary widely in their use of Starfish, ranging from using the program to identify/address very few nonacademic barriers to using the program to identify/address all nonacademic barriers identified in the LRC report. For example, results showed that Bluegrass, Gateway, and Jefferson use Starfish to a moderate or great extent to identify students facing all nonacademic barriers and to connect these students with the appropriate services. Results from the other colleges showed that, other than engagement, Starfish is used only sparingly (or not at all) to assist students in addressing nonacademic barriers.

Data and Research Design:

Indicators of success and retention (dependent variables) were determined based on recommendations from the LRC report. Grade point average (GPA), credits earned, and credential completion were identified as the most appropriate measures of student success. GPA is a commonly accepted measure of academic performance, credits earned serve as an indicator of both academic performance and progress to completion, and credential completion is the goal of the majority of KCTCS students. Credential completion was defined as earning a credential within three years of students’ initial enrollment term. To gauge retention, a fall-to-fall retention rate was used. The operationalizations of credential completion and retention are consistent with performance measures in the KCTCS Strategic Plan.

The predictor or independent variables in this study were based on the common uses and functionality of Starfish. First, as an overall indication of any Starfish interaction, a yes/no flag was created. If a student received any type of communication or was in contact with a KCTCS institution in any way through Starfish, they were counted as a “yes.” All other students were counted as a “no.” The next independent variable was interaction type and involved the creation of yes/no indicator flags based on the types of interactions with students that occurred. The interaction types evaluated included kudos, flags, referrals, to-do items, and appointments. The final independent variable assessed also involved interaction types. However, this time a count by interaction type was used to evaluate the intensity of Starfish usage on student success and retention.

The research design included the use of statistical controls to account for other factors that may influence the relationships between the key variables of interest. Controlling for demographic and academic differences in students increases the ability to form causal inferences between the variables of interest by removing other possible explanations for findings. In other words, this adds confidence to the accuracy of the results obtained. Further, institutional differences (local practices, structure, geography, etc.) were addressed through standard error adjustments, which decreases the chances that the models would produce a significant effect when they should not. However, it should be noted that ORPA was not able to control for every potential factor. For example, KCTCS does not currently possess data on many nonacademic barriers that may impact student success and retention. Table 1 summarizes the dependent, independent, and control variables used to evaluate student success outcomes.

Additionally, the samples for each analysis were disaggregated by academic load (full-time or part-time), race/ethnicity (URM or non-URM), socioeconomic status (low-income or not low[1]income), and gender (male or female). This disaggregation allowed for comparisons (using Wald tests) between the attributes of each subpopulation (e.g., male vs. female) to determine if differences were statistically significant. When performing the regression models on the disaggregated samples, the control variable for the subgroups was omitted from the analysis (e.g., when analyzing male and female students separately, gender was removed as a control variable).

The GPA and credits earned models were estimated using multivariate regression. The models for retention and credential completion were estimated using logistic regression. The logistic regression model is a generalization of the linear regression model that allows for the use of dichotomous dependent variables. For each set of analyses, model assumptions were evaluated, and when a violation occurred, the models were corrected.

Results:

Research on the impact of Starfish on student success and retention yielded the following Key Findings:

1. Starfish interactions can be used to predict which students are struggling academically. Leveraging these results with real-time data can better inform support strategies and provide direction for how to determine the effectiveness of implemented interventions.

 a. Results from this study showed that specific types of Starfish interactions can be used to predict semester GPA, a common indicator of academic performance. For example, students who received one or more flags had a GPA that is over one-half of a letter grade lower (on average) than students who did not receive a flag. Additionally, students who received one or more referrals had a GPA that is 0.19 points lower (on average) than students who did not.

 b. Students who received a flag or referral were significantly more likely to be struggling academically. Using the real-time data available, KCTCS Colleges can identify these students and provide timely interventions to help them succeed.

 

2. Using Starfish to provide kudos improves student success and retention, but repeated use of kudos is necessary to maximize benefits.

a. Students who received one or more kudos increased credits earned by 0.94 each semester (on average) compared to students who did not receive a kudo.

b. Students who received one or more kudos through Starfish were 15.2% more likely to be retained the following fall than students who did not receive at least one kudo.

c. No significant difference was noted in the odds of credential completion for students that received no kudos compared to students who received one or more kudos. However, analysis of kudo intensity (i.e., number of kudos received) showed that for each additional kudo a student received, their odds of completing a credential within three years increased by 1.3%. Combined, these findings indicate that consistent and repeated application of kudos is necessary to enhance the likelihood of credential completion.

 

3. Student flags in Starfish provide clear signals for student retention and success, even when accounting for differences in student characteristics and academic performance. Closer examination and assessment of flag usage and support strategies is needed to improve outcomes for this group of students.

a. Students who received one or more flags saw decreases in credits earned of 0.71 credits each semester (on average) compared to students who did not receive a flag.

b. For students who received at least one flag in Starfish, the odds of being retained the following fall decreased by 23.1%, and the odds of earning a credential within three years decreased by 29.7%, compared to students who did not receive a flag. Additionally, increased frequency of flags applied strengthened these negative relationships. 6

 

4. Significant findings between Starfish interactions and student success and retention were often more pronounced for specific subgroups. These results can help inform ongoing efforts to reduce achievement gaps for target populations.

 a. Part-time students:

i. Compared to full-time students who received a kudo, part-time students who received a kudo saw larger lifts in credits earned each semester.

 ii. Part-time students who received one or more kudos were 21.6% more likely to be retained the following fall than part-time students that did not receive at least one kudo. Further, part-time students saw their odds of retention increase at higher rates than full-time students with each successive kudo received.

iii. Part-time students that received a flag saw significantly larger decreases in their likelihood of being retained than did full-time students that received a flag. Odds continue to decrease at more substantial rates for part-time students with each additional flag received.

b. URM students (American Indian or Alaska Native, Black/African American, Native Hawaiian or Other Pacific Islander, Hispanic/Latinx, Two or More Races):

i. Compared to non-URM students who received a kudo, URM students who received a kudo saw larger lifts in credits earned each semester.

 ii. For every additional kudo received, the odds of completion increased for URM students at a rate that was twice as high as the rate for non-URM students.

iii. URM students that received one or more flags earned significantly fewer credit hours each semester than did non-URM students that received one or more flags.

 c. Low-income students:

i. For every additional kudo received, the odds of retention increased for low-income students at a significantly higher rate than the odds of completion for students who are not low-income.

ii. Low-income students who received one or more flags in their first year were 30.7% less likely to complete a credential within three years than low[1]incomes students that did not receive a flag in their first year.

d. First-year students:

i. For every one-unit increase in the number of kudos a student received in their first year, the likelihood of completing a credential within three years increased by 1.6%.

ii. For every one-unit increase in the number of flags awarded during students’ first year, the odds of graduating within three years decreased by 4.0%.

Conclusions and Recommendations:

The results presented in this research brief indicate clear benefits of Starfish. Even when accounting for differences in student characteristics, college, and academic performance, the analysis of specific interactions within Starfish revealed findings with significant implications for student success and retention. The results of the study showed that Starfish can be used to identify which students are struggling academically, which can lead to earlier interventions using real-time data. Colleges should explore approaches for utilizing real-time data from Starfish to better identify and support students who may need additional resources to succeed. This may allow institutions to be more proactive, and less reactive, in supporting students who need assistance meeting the demands of their classes.

Results from this research also indicate that colleges should continue to use Starfish as a tool to improve student engagement. The results indicate that when addressing the disengagement nonacademic barrier, one of the most useful means of engaging students through Starfish is providing encouragement via kudos. The results suggest that kudos positively impact credits earned, retention rates, and credential completion, even when controlling for differences in academic performance. However, consistent and repeated use of kudos is necessary to maximize benefits for students. Therefore, colleges should encourage more widespread implementation and use of kudos across all courses.

The results of the study also indicate that colleges should focus on providing additional support to students who are flagged in Starfish. Flags were consistently associated with lower academic performance and reduced odds of retention and credential completion. Additionally, these results were often more pronounced for specific subgroups. Significantly lower student success and retention outcomes show that, while students who are flagged may be receiving additional supports currently, more intensive intervention is required to better assist these students in achieving success. Further, additional research is needed to explore the use of flags across colleges so better intervention strategies can be established and assessed.

Finally, the results of the study revealed Starfish has different effects across subpopulations of KCTCS students. In many instances, the positive impact of Starfish on student outcomes is enhanced for traditionally underserved and other target populations. For example, the positive effect of kudos on student outcomes was more pronounced for part-time, URM, and first-year students. Conversely, the negative associations between student success and retention were often more pronounced for specific subpopulations. These findings provide direction for future support efforts and may serve as a tool to further reduce achievement gaps for underserved and targeted populations.