TY - GEN
T1 - DataDecon
T2 - 19th International Joint Symposium on Artificial Intelligence and Natural Language Processing, iSAI-NLP 2024
AU - Yuenyong, Sumeth
AU - Buppodom, Norapat
AU - Sangkaew, Koravich
AU - Boonmeeprakob, Konthee
AU - Boonkwan, Prachya
AU - Jaroenkantasima, Jillaphat
AU - Khlaisamniang, Pitikorn
AU - Lertpiya, Anuruth
AU - Piyatumrong, Apivadee
AU - Rojratchadakorn, Peerawat
AU - Rugsujarit, Thaweewat
AU - Saengsukhiran, Teerapol
AU - Saetan, Kriangkrai
AU - Sukprapa, Isada
AU - Thavornmongkol, Thanachot
AU - Thongthungwong, Nucharee
AU - Triamamornwooth, Patteera
AU - Utupon, Chanon
AU - Viriyayudhakorn, Kobkrit
AU - Witchutanon, Phoochit
AU - Wongprayon, Sadit
AU - Supnithi, Thepchai
N1 - Publisher Copyright:
© 2024 IEEE.
PY - 2024
Y1 - 2024
N2 - Large language models (LLMs) play an important role in modern NLP technology as they are versatile for a wide array of NLP tasks. However, constructing an LLM is challenging due to concealed construction pipelines, the lack of cleansed datasets, and hyperparameter settings, making it almost irreproducible. This paper presents an efficient pipeline for constructing an LLM tailored to a low-to-medium-sourced language with a high level of data contamination and tools to cleanse the dataset. Following our pipeline, we constructed OpenThaiGPT, an LLM for Thai, with only open-sourced datasets such as CC100, OSCAR, and mC4, and achieved the state-of-the-art accuracies on our downstream tasks. Here, we disclosed the data statistics and all hyperparameter settings for reproducibility.
AB - Large language models (LLMs) play an important role in modern NLP technology as they are versatile for a wide array of NLP tasks. However, constructing an LLM is challenging due to concealed construction pipelines, the lack of cleansed datasets, and hyperparameter settings, making it almost irreproducible. This paper presents an efficient pipeline for constructing an LLM tailored to a low-to-medium-sourced language with a high level of data contamination and tools to cleanse the dataset. Following our pipeline, we constructed OpenThaiGPT, an LLM for Thai, with only open-sourced datasets such as CC100, OSCAR, and mC4, and achieved the state-of-the-art accuracies on our downstream tasks. Here, we disclosed the data statistics and all hyperparameter settings for reproducibility.
KW - Thai language
KW - corpus construction
KW - data cleansing
KW - large language models
KW - text deduplication
UR - https://www.scopus.com/pages/publications/85216587816
U2 - 10.1109/iSAI-NLP64410.2024.10799278
DO - 10.1109/iSAI-NLP64410.2024.10799278
M3 - Conference contribution
AN - SCOPUS:85216587816
T3 - 19th International Joint Symposium on Artificial Intelligence and Natural Language Processing, iSAI-NLP 2024
BT - 19th International Joint Symposium on Artificial Intelligence and Natural Language Processing, iSAI-NLP 2024
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 11 November 2024 through 15 November 2024
ER -