Advancing Instrument Validation in Social Sciences: An AI-Powered Chatbot and Interactive Website based on Research Instrument Validation Framework (RIVF)
Description of Resource
Background: In social sciences, ensuring a high level of instrument validation is crucial for upholding the principles of scientific rigor and maintaining the overall quality of research.
Objectives: To develop and evaluate an AI chatbot and website for instrument validation, assess their impact on instrument validity improvement, and analyze user perceptions.
Methods: Adopting a quantitative design, the study was anchored on the developed Research Instrument Validation Framework (RIVF) of Villarino (2024). Moreover, it was evaluated through users' perceptions (n=100) by administering an online survey, whereby the employment of paired t-tests used contrasting instrument validity-pre-vs post-RIVF scores, and one-way ANOVA was used to determine if a relationship existed between users' perceptions and overall improvement in instrument validity. A G*Power analysis indicated that there was sufficient statistical power for the analyses: for paired t-tests, it was 99.73% (n = 100, dz = 0.5, α = 0.05), and for one-way ANOVA, 80.95% (n = 100, f = 0.25, α=0.05, four groups). All data were analyzed using IBM SPSS version 26.
Results: Post-RIVF use, all the validity domains showed significant improvements (p<0.001), but the primary considerable improvement was in construct validity [Mean difference=1.20±0.60, t(49)=14.14]. Participants perceived the AI chatbot as more useful [4.30±0.70 vs. 3.80±0.80, p<0.001] compared to the RIFV website.
Conclusion: This AI-powered milieu indicates a potential for increasing the validity of research instruments in RIVF, while an AI chatbot efficiently increments the construct validity. These findings would infer that using AI technologies potentially enhances the quality of research instruments in the social sciences alongside traditional validation methods.