Skip to main navigation Skip to search Skip to main content

DataDecon: Data Cleansing Tools for Large Language Model with Efficient Decontamination Techniques

  • Sumeth Yuenyong
  • , Norapat Buppodom
  • , Koravich Sangkaew
  • , Konthee Boonmeeprakob
  • , Prachya Boonkwan
  • , Jillaphat Jaroenkantasima
  • , Pitikorn Khlaisamniang
  • , Anuruth Lertpiya
  • , Apivadee Piyatumrong
  • , Peerawat Rojratchadakorn
  • , Thaweewat Rugsujarit
  • , Teerapol Saengsukhiran
  • , Kriangkrai Saetan
  • , Isada Sukprapa
  • , Thanachot Thavornmongkol
  • , Nucharee Thongthungwong
  • , Patteera Triamamornwooth
  • , Chanon Utupon
  • , Kobkrit Viriyayudhakorn
  • , Phoochit Witchutanon
  • Sadit Wongprayon, Thepchai Supnithi
  • National Electronics and Computer Technology Center
  • Artificial Intelligence Asssociation of Thailand
  • Big Data Institute (Public Organization)
  • Kasetsart University
  • Kasikorn Business Technology Group
  • King Mongkut's Institute of Technology Ladkrabang
  • Sirindhorn International Institute of Technology, Thammasat University
  • Faculty of Engineering
  • Mahidol University
  • University of Liverpool
  • King Mongkut's University of Technology North Bangkok
  • Chulalongkorn University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Large language models (LLMs) play an important role in modern NLP technology as they are versatile for a wide array of NLP tasks. However, constructing an LLM is challenging due to concealed construction pipelines, the lack of cleansed datasets, and hyperparameter settings, making it almost irreproducible. This paper presents an efficient pipeline for constructing an LLM tailored to a low-to-medium-sourced language with a high level of data contamination and tools to cleanse the dataset. Following our pipeline, we constructed OpenThaiGPT, an LLM for Thai, with only open-sourced datasets such as CC100, OSCAR, and mC4, and achieved the state-of-the-art accuracies on our downstream tasks. Here, we disclosed the data statistics and all hyperparameter settings for reproducibility.

Original languageEnglish
Title of host publication19th International Joint Symposium on Artificial Intelligence and Natural Language Processing, iSAI-NLP 2024
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9798331509910
DOIs
Publication statusPublished - 2024
Event19th International Joint Symposium on Artificial Intelligence and Natural Language Processing, iSAI-NLP 2024 - Chonburi, Thailand
Duration: 11 Nov 2024 → 15 Nov 2024

Publication series

Name19th International Joint Symposium on Artificial Intelligence and Natural Language Processing, iSAI-NLP 2024

Conference

Conference19th International Joint Symposium on Artificial Intelligence and Natural Language Processing, iSAI-NLP 2024
Country/TerritoryThailand
CityChonburi
Period11/11/24 → 15/11/24

Keywords

  • Thai language
  • corpus construction
  • data cleansing
  • large language models
  • text deduplication

Fingerprint

Dive into the research topics of 'DataDecon: Data Cleansing Tools for Large Language Model with Efficient Decontamination Techniques'. Together they form a unique fingerprint.

Cite this