The Effect of Vibe Coding on the Programming Skills of Beginner and Advanced Students at Telkom University Purwokerto

The growing use of generative AI has introduced vibe coding as a new programming approach in higher education, but its varying impacts on students’ experience levels have not yet been extensively explored in the Indonesian context. This study investigates whether vibe coding affects programming competencies differently between early-stage and advanced-stage students in the Software Engineering Program at Telkom University Purwokerto. A comparative design with a mixed-methods approach was used, involving 30 respondents: 15 early-stage students (Semester 4) and 15 advanced-stage students (Semester 6). Data were collected via a five-point Likert-scale questionnaire measuring the intensity of vibe coding (Section B) and perceptions of programming competence (Section C), supplemented by a prompt quality rubric validated by faculty members based on a case study of a circular single-linked list. Quantitative analysis employed Spearman’s rank correlation and the Mann-Whitney U test, while qualitative findings from the rubric were triangulated with questionnaire results. The results indicate that advanced-level students have a significantly higher coding vibe intensity (U = 57.5; p = 0.023), while beginner-level students report a significantly higher perceived ability (U = 39.0; p = 0.002). The Spearman test yielded a non-significant negative correlation in both groups, with the final-year group showing a moderate trend (rs = −0.50). No significant differences were found in prompt output quality between groups (U = 92.0; p = 0.400), and triangulation revealed inconsistencies between perceived ability and actual output quality. These findings indicate that vibe coding without structured instructional guidance has the potential to widen the gap between perceived and actual programming competence, underscoring the need for the explicit integration of prompting literacy into the programming curriculum.

Keywords: Vibe Coding, Programming Competency, Prompting Literacy, Cognitive Load Theory, Mixed Methods, Software Engineering

Copyright (c) 2026 Journal of Software Engineering and Multimedia (JASMED)

This work is licensed under a Creative Commons Attribution 4.0 International License.

[1] A. Sarkar and I. Drosos, “Vibe coding: Programming through conversation with artificial intelligence,” in Proc. 36th Annu. Conf. Psychol. Program. Interest Group (PPIG 2025), Sep. 2025, doi: 10.48550/arXiv.2506.23253.

[2] H.-F. Chang, M. Shokrolah, S. E. S. Witchger, L. Cao, and S. Koolmanojwong Mobasser, “Coding with AI: From a reflection on industrial practices to future computer science and software engineering education,” arXiv preprint arXiv:2512.23982, Dec. 2025.

[3] M. Kazemitabaar, J. Chow, C. K. T. Ma, B. J. Ericson, D. Weintrop, and T. Grossman, “Studying the effect of AI code generators on supporting novice learners in introductory programming,” in Proc. 2023 CHI Conf. Human Factors Comput. Syst. (CHI ’23), Hamburg, Germany, Apr. 2023, doi: 10.1145/3544548.3580919.

[4] P. Denny et al., “Computing education in the era of generative AI,” Commun. ACM, vol. 67, no. 2, pp. 56–67, Feb. 2024, doi: 10.1145/3624720.

[5] E. Kasneci et al., “ChatGPT for good? On opportunities and challenges of large language models for education,” Learn. Individ. Differ., vol. 103, p. 102274, Apr. 2023, doi: 10.1016/j.lindif.2023.102274.

[6] J. Prather et al., "It's weird that it knows what I want: Usability and interactions with Copilot for novice programmers," ACM Transactions on Computer-Human Interaction, vol. 31, no. 1, Article 4, Nov. 2023, doi: 10.1145/3617367.

[7] I. Samsyudin, “Vibe coding and AI-led conversational programming: Emerging trends in software development,” SSRN Electron. J., Aug. 2025, doi: 10.2139/SSRN.5469367.

[8] J. Prather et al., “The widening gap: The benefits and harms of generative AI for novice programmers,” in Proc. ACM Conf. Int. Comput. Educ. Res. (ICER ’24), Melbourne, Australia, Aug. 2024, pp. 469–486, doi: 10.1145/3632620.3671116.

[9] F. Lucchetti, Z. Wu, A. Guha, M. Q. Feldman, and C. J. Anderson, “Substance beats style: Why beginning students fail to code with LLMs,” in Proc. 2025 Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Lang. Technol. (NAACL), Albuquerque, NM, USA, Apr. 2025, pp. 8541–8610, doi: 10.18653/v1/2025.naacl-long.433.

[10] F. Geng et al., “Exploring student-AI interactions in vibe coding,” arXiv preprint arXiv:2507.22614, Jul. 2025.

[11] N. V. Ivankova, J. W. Creswell, and S. L. Stick, “Using mixed-methods sequential explanatory design: From theory to practice,” Field Methods, vol. 18, no. 1, pp. 3–20, Feb. 2006, doi: 10.1177/1525822X05282260.

[12] M. N. Alam, M. A. Islam, M. O. A. Babiker, M. S. Siddiqui, M. Bin Amin, and J. Oláh, “AI-assisted learning tools and student learning outcomes: A cognitive load theory perspective,” Comput. Hum. Behav. Rep., vol. 21, p. 100986, Mar. 2026, doi: 10.1016/j.chbr.2026.100986.

[13] G. J. Bhattacherjee, Social Science Research: Principles, Methods, and Practices, 2nd ed. Tampa, FL, USA: University of South Florida, 2012. [Online]. Available: https://scholarcommons.usf.edu/oa_textbooks/3. Accessed: Jun. 2026.

[14] M. A. Bujang, E. D. Omar, D. H. P. Foo, and Y. K. Hon, “Sample size determination for conducting a pilot study to assess reliability of a questionnaire,” Restor. Dent. Endod., vol. 49, no. 1, p. e3, 2024, doi: 10.5395/rde.2024.49.e3.

[15] S. S. Sullivan and A. R. Artino Jr., “Analyzing and interpreting data from Likert-type scales,” J. Grad. Med. Educ., vol. 5, no. 4, pp. 541–542, Dec. 2013, doi: 10.4300/JGME-5-4-18.

[16] P. Denny, J. Leinonen, J. Prather, A. Luxton-Reilly, T. Amarouche, B. A. Becker, and B. N. Reeves, “Prompt problems: A new programming exercise for the generative AI era,” in Proc. 55th ACM Tech. Symp. Comput. Sci. Educ. (SIGCSE), Portland, OR, USA, 2024, pp. 296–302, doi: 10.1145/3626252.3630909.

[17] A. Pathak et al., “Rubric is all you need: Improving LLM-based code evaluation with question-specific rubrics,” in Proc. ACM Int. Conf. Comput. Educ. Res. (ICER), 2025, doi: 10.1145/3702652.3744220.

[18] D. F. Polit and C. T. Beck, Nursing Research: Generating and Assessing Evidence for Nursing Practice, 10th ed. Philadelphia, PA, USA: Wolters Kluwer, 2017.

[19] K. S. Taber, “The use of Cronbach’s alpha when developing and reporting research instruments in science education,” Res. Sci. Educ., vol. 48, no. 6, pp. 1273–1296, Dec. 2018, doi: 10.1007/s11165-016-9602-2.