Abstract

Background Large language models (LLMs) have shown remarkable advancements in natural language processing, with increasing interest in their ability to handle tasks requiring expert-level knowledge. This study evaluates the capabilities of LLMs in a high-stakes professional setting by examining their performance on the Korean Pharmacist Licensing Examination (KPLE), a comprehensive test essential for pharmacist certification in South Korea.

Methods We assessed 27 LLMs, including proprietary models and open-source models using both the original Korean and English-translated versions of the KPLE from 2019 to 2024. Exam questions were translated, formatted, and analyzed using accuracy-based and score-based metrics. Models were grouped by size and type, and evaluated for subject-specific performance, error rates, and progression over time.

Results Seven models passed all six years of both the English and Korean exams, including five proprietary and two open-source models. Proprietary models generally outperformed open-source counterparts, though the performance gap narrowed substantially over time. The best-performing proprietary model, Claude 3.5 Sonnet, scored in the top 12% of human examinees. Larger models achieved higher accuracy overall, but recent smaller models also showed strong performance due to architectural and training improvements. Notably, LLMs struggled in topics requiring complex calculations and highly localized knowledge indicating the future improvement direction for the pharmaceutical use of LLMs through domain-specific fine-tuning.

Conclusion LLMs can pass the KPLE, demonstrating their growing potential as tools in professional domains. Their strengths currently lie in memorization and language comprehension, though weaknesses remain in complex reasoning and region-specific knowledge. While not substitutes for human pharmacists, LLMs may support and elevate pharmacists’ professional expertise and efficiency. They hold promise as assistants in education, decision support, and administrative tasks. Continued improvements through fine-tuning, domain-specific training, and architectural advances will be key to ensuring their safe and effective use in pharmacy practice.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

This work was supported by Seoul National University (370C-20220109 and AI-Bio Research Grant 0413-20230053), the National Research Foundation of Korea (Grant 2020M3A9G7103933, 2022M3E5F3081268, RS-2023-00256320, and 2022R1C1C1005080) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2023-00220628, Artificial intelligence for prediction of structure-based protein interaction reflecting physicochemical principles).

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

The study used stastical data of human examinees of the Korean Pharmacist Licensing Examination that is openly available originally located at Korea Health Personnel Licensing Examination Institute online platform. https://www.kuksiwon.or.kr/analysis/brd/m_91/list.do?seq=13

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes

6 List of abbreviationsAIArtificial IntelligenceLLMsLarge Language ModelsKPLEKorea Health Personnel Licensing ExaminationKHPLEIKorea Health Personnel Licensing Examination Institute