1-Diffractor: Efficient and Utility-Preserving Text Obfuscation Leveraging Word-Level Metric Differential Privacy

S Meisenbacher, M Chevli, F Matthes - arXiv preprint arXiv:2405.01678, 2024

The study of privacy-preserving Natural Language Processing (NLP) has gained rising attention in recent years. One promising avenue studies the integration of Differential Privacy in NLP, which has brought about innovative methods in a variety …


101 Billion Arabic Tokens Dataset

M Aloui, H Chouikhi, G Chaabane, H Kchaou…

… , we heavily depend on Common Crawl (1) as our primary source of web-scraped content. Common Crawl provides an accessible web archive … a Dataset Pipeline expressed in the following figure (1) aimed at cleaning up the web-extracted text …


101 Billion Arabic Words Dataset

M Aloui, H Chouikhi, G Chaabane, H Kchaou… - arXiv preprint arXiv …, 2024

… We undertook a large-scale data mining project, extracting a substantial volume of text from the Common Crawl WET files, specifically targeting Arabic content. The extracted data underwent a rigorous cleaning and deduplication process, using …


14 Deep Field of Learning Artificial Impacts Intelligence in the

R Gulwani, M Aggarwal - Deep Learning Concepts in Operations Research, 2024

… These datasets, such as ImageNet for computer vision or the Common Crawl for natural language processing, have played a crucial role in training deep learning models and pushing the boundaries of AI performance. …


2 Text Preprocessing Method

This paper uses easy data augmentation (EDA) and back translation for text enhancement. After text enhancement, the number of samples in the dataset can be significantly increased, which is helpful for model parameters through training and …


4 CHATGPT IN ACADEMIA: AN IN-DEPTH EXPLORATION OF STUDENT VIEWS–PROS AND CONS

D Rajkumar, P Murugeswari, M Karthigaieswari - TRANSDISCIPLINARY THREADS

… GPT-3’s training data came from five existing datasets: • Common Crawl: A collection of text pulled from billions of web pages containing trillions of words. OpenAI filtered it for high-quality reference material only. • WebText2: OpenAI …


5 LLM Pretraining Methods

A Velu, R Ramamoorthy, SM Manasa, A Prasanth - Generative AI and LLMs

… – Websites: This feature provides a vast array of information from the internet offering a variety of linguistic expertise, eg, Common Crawl. The major drawback of such data is that the text on the web can be of two different quality levels: spam and high-quality …


6 Performance Analysis

A Safi, S Singh, G Kaur, T Kaur - Intelligent Security Solutions for Cyber-Physical …, 2024

… The dataset used in this research was gathered from Alexa and the Common Crawl archive. … from a variety of sources, including Common Crawl, Alexa, and PhishTank for the phishing … They used three datasets, including real URLs …


" Deep Learning for de-identification of clinical documents

The amount of unstructured medical documents increases each year, presenting an opportunity to extract valuable insights that could significantly improve healthcare. However, to take advantage of this potential, it is crucial to de-identify these …


" Previously on..." From Recaps to Story Summarization

AK Singh, D Srivastava, M Tapaswi - arXiv preprint arXiv:2405.11487, 2024

We introduce multimodal story summarization by leveraging TV episode recaps - short video sequences interweaving key story moments from previous episodes to bring viewers up to speed. We propose PlotSnap, a dataset featuring two crime …


“More than Words”: A Legal Approach to the Risks of Commercial Chatbots Powered by Generative Artificial Intelligence

S Migliorini - European Journal of Risk Regulation, 2024

… In the training of ChatGPT, this “clean”19 version of the Common Crawl has been complemented and filtered with the addition of a few … also that the training gave more prominence to the “clean” Common Crawl.Although this explanation may be …


A Balancing Act: Data Protection Compliance of Artificial Intelligence

Neural network based generative artificial intelligence (GenAI) systems, in particular large language models (LLMs), hold huge potential for economic, scientific, and artistic progress. However, they are also the subject of ongoing fierce data protection …


A Beginner's Guide to Large Language Models

Page 1 Page 2 Page 3 “LLMs are like Swiss Army knives for digital communication. Their versatility and adaptability enable them to perform many tasks, each with precision and finesse. Whether you're a writer seeking creative inspiration, a developer …


A Benchmark for Relevance-based Headline Classification and Generation

The task of news headline generation deals with generating a concise summary for a given news article. It is a crucial task in increasing productivity for both the readers and producers of news. Significant progress has been made in automatically …


A bilingual benchmark for evaluating large language models

M Alkaoud - PeerJ Computer Science, 2024

This work introduces a new benchmark for the bilingual evaluation of large language models (LLMs) in English and Arabic. While LLMs have transformed various fields, their evaluation in Arabic remains limited. This work addresses this …


A Blinded Comparison of Three Generative Artificial Intelligence Chatbots for Orthopaedic Surgery Therapeutic Questions

V Arora, J Silburt, M Phillips, M Khan, B Petrisor… - Cureus, 2024

… Current generative large-language models (LLMs) have been trained on massive corpora of text such as common crawl - a dataset of 250 billion webpages - and thus have both a general knowledge of the world and the capacity to recapitulate human …


A Canary in the AI Coal Mine: American Jews May Be Disproportionately Harmed by Intellectual Property Dispossession in Large Language Model Training

H Precel, A McDonald, B Hecht, N Vincent - arXiv preprint arXiv:2403.13073, 2024

Systemic property dispossession from minority groups has often been carried out in the name of technological progress. In this paper, we identify evidence that the current paradigm of large language models (LLMs) likely continues this long history …


A Collocation-based Method for Addressing Challenges in Word-level Metric Differential Privacy

S Meisenbacher, M Chevli, F Matthes - arXiv preprint arXiv:2407.00638, 2024

Applications of Differential Privacy (DP) in NLP must distinguish between the syntactic level on which a proposed mechanism operates, often taking the form of $\textit{word-level}$ or $\textit{document-level}$ privatization. Recently, several word-level $\textit{Metric} …


A Comparative Analysis of Text Embedding Models for Bug Report Semantic Similarity

A Patil, K Han, A Jadon - 2024 11th International Conference on Signal …, 2024

… For Fasttext, the ”crawl-300d-2M-subword” model was employed, which consisted of 2 million word vectors trained with subword information on the Common Crawl dataset, encompassing 600 billion tokens. In the case of Doc2Vec, the ”GoogleNews-vectors-negative300” …


A Comparative Analysis of Word Embeddings Techniques for Italian News Categorization

F Rollo, G Bonisoli, L Po - IEEE Access, 2024

Text categorization remains a formidable challenge in information retrieval, requiring effective strategies, especially when applied to low-resource languages such as Italian. This paper delves into the intricacies of categorizing Italian news articles …


A Comparative Study of ChatGPT and Seq2Seq Chatbot for Effective Students Advising

SK Assayed, M Alkhatib, K Shaalan - ChatGPT and Global Higher Education: Using …, 2024

Artificial Intelligence (AI) chatbots are increasingly becoming a part of students’ lives, from school learning to university admissions and orientations. AI chatbots are finding ways to facilitate and improve learning experiences. Moreover, AI chatbots …


A comparative study of cross-lingual sentiment analysis

P Přibáň, J Šmíd, J Steinberger, A Mištera - Expert Systems with Applications, 2024

This paper presents a detailed comparative study of the zero-shot cross-lingual sentiment analysis. Namely, we use modern multilingual Transformer-based models and linear transformations combined with CNN and LSTM neural networks. We …


A Comparison of Pretrained Models for Classifying Issue Reports

J Heo, G Kwon, C Kwak, S Lee - IEEE Access, 2024

Issues are evolving requirements which are the main factor that increase the cost of software evolution. To help developers manage issues, GitHub provides issue labeling mechanisms in issue management systems. However, manually labeling …


A Complete Survey on LLM-based AI Chatbots

SK Dam, CS Hong, Y Qiao, C Zhang - arXiv preprint arXiv:2406.16937, 2024

The past few decades have witnessed an upsurge in data, forming the foundation for data-hungry, learning-based AI technology. Conversational agents, often referred to as AI chatbots, rely heavily on such data to train large language models (LLMs) and …


A comprehensive evaluation of large language models in mining gene relations and pathway knowledge

M Azam, Y Chen, MO Arowolo, H Liu, M Popescu, D Xu - Quantitative Biology, 2024

… As LLMs typically used the Common Crawl dataset [17], any web materials open to the public, such as PubMed abstracts [18] and publications at the PMC Open Access [19], may be included in the training text of LLMs. The result in Figure 3 may …


A comprehensive literature review on phishing URL detection using deep learning techniques

E Kritika - Journal of Cyber Security Technology, 2024

… The selected papers conducted research based on the data curated from open sources common to most such as Phishtank, Openphish, ISCX 2016, Kaggle, github and phishstorm for collecting malicious urls and Alexa, domcop, common crawl, and …


A Comprehensive Review on Large Language Models

A Yadav - … Software Engineering Through AI, Federated Learning …, 2024

In the realm of computer science and language, large language models (LLMs) stand out as remarkable tools of artificial intelligence (AI). Proficient in deciphering intricate language nuances, LLMs offer sensible responses and find applications in …


A Computational Analysis of Transcribed Speech of People Living with Dementia: The Anchise 2022 Corpus

F Sigona, DP Radicioni, BG Fivela, D Colla… - Computer Speech & …, 2024

Introduction Automatic linguistic analysis can provide cost-effective, valuable clues to the diagnosis of cognitive difficulties and to therapeutic practice, and hence impact positively on wellbeing. In this work, we analyzed transcribed conversations …


A contemporary review on chatbots, AI-powered virtual conversational agents, ChatGPT: Applications, open challenges and future research directions

A Casheekar, A Lahiri, K Rath, KS Prabhakar… - Computer Science Review, 2024

This review paper offers an in-depth analysis of AI-powered virtual conversational agents, specifically focusing on OpenAI’s ChatGPT. The main contributions of this paper are threefold: (i) an exhaustive review of prior literature on chatbots, (ii) a …


A Copious Void: Rhetoric as Artificial Intelligence 1.0

A Hallsby - Rhetoric Society Quarterly, 2024

Rhetoric is a trace retained in and by artificial intelligence (AI) technologies. This concept illuminates how rhetoric and AI have faced issues related to information abundance, entrenched social inequalities, discriminatory biases, and the …


A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl

S Baack - The 2024 ACM Conference on Fairness, Accountability …, 2024

… among LLM builders about the implications of using Common Crawl’s data. This paper discusses what Common Crawl’s popularity for LLM development means … Our qualitative analysis is based on indepth interviews with Common Crawl staffers …


A critical examination of document-level machine translation systems

The need for accurate and effective translation cannot be overstated in an increasingly globalised world where communication is paramount. Bridging language barriers is important for promoting understanding and cooperation among …


A Cross-Domain Benchmark for Active Learning

T Werner, J Burchert, M Stubbemann… - arXiv preprint arXiv …, 2024

Active Learning (AL) deals with identifying the most informative samples for labeling to reduce data annotation costs for supervised learning tasks. AL research suffers from the fact that lifts from literature generalize poorly and that only a small number …


A CURATEd CATalog: Rethinking the Extraction of Pretraining Corpora for Mid-Resourced Languages

J Palomar-Giner, JJ Saiz, F Espuña, M Mina, S Da Dalt… - Proceedings of the 2024 …, 2024

We present and describe two language resources in this paper: CATalog 1.0, the largest text corpus in Catalan to date, and CURATE (Corpus Utility for RAting TExt), a modular, parallelizable pipeline used for processing and scoring documents …


A Dataset for Pharmacovigilance in German, French, and Japanese: Annotating Adverse Drug Reactions across Languages

L Raithel, HS Yeh, S Yada, C Grouin, T Lavergne… - arXiv preprint arXiv …, 2024

User-generated data sources have gained significance in uncovering Adverse Drug Reactions (ADRs), with an increasing number of discussions occurring in the digital world. However, the existing clinical corpora predominantly revolve around scientific …


A Dataset for the Detection of Dehumanizing Language

P Engelmann, PB Trolle, C Hardmeier - arXiv preprint arXiv:2402.08764, 2024

… The Common Crawl is an open repository of web crawled data, which features crawls from all … As the Common Crawl includes several petabytes of data in total, we have selectively ex… As we limit ourselves to English, the Common Crawl data …


A Decade's Battle on Dataset Bias: Are We There Yet?

Z Liu, K He - arXiv preprint arXiv:2403.08632, 2024

We revisit the "dataset classification" experiment suggested by Torralba and Efros a decade ago, in the new era with large-scale, diverse, and hopefully less biased datasets as well as more capable neural network architectures. Surprisingly, we …


A Deep Learning Approach to Fine-Grained Political Ideology Classification on Social Media Texts

ER Kovacs, LA Cotfas, C Delcea - International Conference on Computational …, 2024

Starting with the mid-2010s, the growing impact of social media on politics has been felt worldwide. This impact became even more decisive in instances such as the 2020 US Presidential Election, where a bitter division between the two main US …


A field guide to using LLMs for online conflict analysis

E Borra, P Bollini, L Caroleo, B Cerigo, R Dubèl, F Gaw…

… Two ways in which we studied LLMs was by looking at some of their open-source datasets, which one can sift through in WebText2, CommonCrawl and Wikipedia. Another is by studying autocompletions as perpetuations of standardized knowledge …


A Framework for Applying Copyright Law to the Training of Textual Generative Artificial Intelligence.

A Neill, J Thomas, E Lee - Texas Intellectual Property Law Journal, 2024

… Websites are not notified when Common Crawl scrapes their data, but they can opt out by configuring a specific site to block the crawler.46 The second corpora used, WebText2, is an expanded version of WebText and utilizes a similar parameter for …


A Framework for Embedding Entities in a Textual Narrative: a Case Study on Les Misérables

In this article, we propose a general and flexible framework in order to study narrative entities found in a literary work. This framework is exposed starting from a broad perspective, consisting in how to segment the work into textual units and …


A Framework for Preparing a Balanced and Comprehensive Phishing Dataset

… • Choon Lin Tan (2018) [19] - dataset contains 5000 legitimate records sourced from Alexa and Common Crawl and 5000 phishing records from PhishTank and … Common Crawl (commoncrawl.org)- is a humongous web archive collected by …


A Framework to Construct Financial Causality Knowledge Graph from Text

Z Xu, H Takamura, R Ichise - 2024 IEEE 18th International Conference on Semantic …, 2024

Causality analysis holds a prominent role in finance, and the presentation of causality could offer valuable insights for risk mitigation, investment decisions, and portfolio optimization. Recent research has extensively investigated the identification …


A Galician Corpus for Misogyny Detection Online

LM Álvarez-Crespo, LM Castro - Proceedings of the 16th International Conference on …, 2024

Social networks are virtual spaces where millions of people share ideas, opinions, and experiences. However, this broad social interaction also exposes negative and harmful behaviors, such as harassment and misogyny. Misogyny, particularly, is a …


A Generalize Hardware Debugging Approach for Large Language Models Semi-Syntectic Datasets

W Fu, S Li, Y Zhao, K Yang, X Zhang, Y Jin, X Guo

… C4 specifically employs the Common Crawl [44] dataset, focusing on deduplication, removing non-English content, and filtering out offensive language. Redpajama combines sources from Wikipedia and StackExchange, similar to C4 …


A global matching model of choice and response times in the Deese-Roediger-McDermott semantic and perceptual false recognition paradigms

… We used the pre-trained vectors from the fasttext library that were trained on the Common Crawl corpus (Grave, Bojanowski, Gupta, Joulin, & Mikolov, 2018)3. The word2vec model is considered the state-of-the-art semantic space model in its ability …


A Hierarchical Context Augmentation Method to Improve Retrieval-Augmented LLMs on Scientific Papers

TY Che, XL Mao, T Lan, H Huang - Proceedings of the 30th ACM SIGKDD Conference …, 2024

… C𝑝𝑟𝑒𝑡𝑟𝑎𝑖𝑛 = C𝑠𝑐𝑖 ∪ C𝑎𝑢𝑔 ∪ C𝑔𝑒𝑛𝑒𝑟𝑎𝑙 (2) where C𝑠𝑐𝑖 is the previous scientific papers and C𝑔𝑒𝑛𝑒𝑟𝑎𝑙 is the general corpus such as CommonCrawl. The mixed dataset will be used to continue pretraining the language …


A Holistic Review on Detection of Malicious Browser Extensions and Links using Deep Learning

T Zonta, M Sathiyanarayanan - 2024 IEEE 3rd International Conference on AI in …, 2024

… Common Crawl [36] is a website which crawls over the web and suggest many datasets of legitimate website and their metadata namely … Available: https://commoncrawl.or g/2021/10/september-2021-crawlarchive-now-available/ [37] AC Bahnsen, EC …


A Hybrid Deep BiLSTM-CNN for Hate Speech Detection in Multi-social media

A Kumar, S Kumar, K Passi, A Mahanti - ACM Transactions on Asian and Low …, 2024

Nowadays, ways of communication among people have changed due to advancements in information technology and the rise of online multi-social media. Many people express their feelings, ideas, and emotions on social media sites such …


A Japanese-Chinese Parallel Corpus Using Crowdsourcing for Web Mining

M Nagata, M Morishita, K Chousa, N Yasuda - arXiv preprint arXiv:2405.09017, 2024

… In ParaCrawl, they determine which websites to crawl by analyzing the Common Crawl … In this study, we analyzed 12 sets of Common Crawl archives (104TB in total) published from … and bilingual website URLs obtained from Common Crawl …


A Language Model Trained on Uruguayan Spanish News Text

JP Filevich, G Marco, S Castro, L Chiruzzo, A Rosá - LREC-COLING 2024, 2024

This paper presents a language model trained from scratch exclusively on a brand new corpus consisting of about 6 GiB of Uruguayan newspaper text. We trained the model for 30 days on a single Nvidia P100 using the RoBERTa-base architecture …


A Large-Scale Exploration of $\mu $-Transfer

L Lingle - arXiv preprint arXiv:2404.05728, 2024

Large neural network models have become a mainstay of natural language processing and computer vision, yet their initialization and learning rates are set in a largely heuristic fashion, potentially varying from paper to paper and one model size …


A Legal Framework for Natural Language Model Training in Portugal

R Almeida, E Amorim - Proceedings of the Workshop on Legal and Ethical …, 2024

Recent advances in deep learning have promoted the advent of many computational systems capable of performing intelligent actions that, until then, were restricted to the human intellect. In the particular case of human languages, these …


A Legal Framework for Natural Language Processing Model Training in Portugal

R Almeida, E Amorim - arXiv preprint arXiv:2405.00536, 2024


A Little Leak Will Sink a Great Ship: Survey of Transparency for Large Language Models from Start to Finish

M Kaneko, T Baldwin - arXiv preprint arXiv:2403.16139, 2024

… The most common sources included in all LLMs are web page sources such as C4, CommonCrawl, and the Pile. Because they are collected from various web pages, there is a risk that they may contain personal information, copyrighted texts …


A Longitudinal Study of Content Control Mechanisms

M Dinzinger, M Granitzer - Companion Proceedings of the ACM on Web …, 2024

… Both resources, robots.txt and regular pages, were collected from the Common Crawl web archive 7 and processed through a customized … As Common Crawl is furhtermore not focused on particular web domains, but randomly samples the …


A methodology for using players' chat content for dynamic difficulty adjustment in metaverse multiplayer games

MM Rezapour, A Fatemi, MA Nematbakhsh - Applied Soft Computing, 2024

… RoBERTa, on the other hand, was trained on a much larger corpus of text that includes not only Wikipedia and BookCorpus but also Common Crawl, a dataset that contains billions of web pages. This larger and more diverse corpus of text allows …


A multi-level multi-label text classification dataset of 19th century Ottoman and Russian literary and critical texts

G Gokceoglu, D Cavusoglu, E Akbas, ÖN Dolcerocca - arXiv preprint arXiv …, 2024

This paper introduces a multi-level, multi-label text classification dataset comprising over 3000 documents. The dataset features literary and critical texts from 19th-century Ottoman Turkish and Russian. It is the first study to apply large language models (LLMs) …


A Multilingual Dataset for Investigating Stereotypes and Negative Attitudes Towards Migrant Groups in Large Language Models

D Sorato, CC Ventura, D Zavala-Rojas - … of the 16th International Conference on …, 2024

… 2020), which is a cleaned version of the CommonCrawl corpus. … where stereotypes are presented in more subtle and/or strategic ways (when compared to social media/CommonCrawl texts) and the explicit discrimination of migrant groups …


A Multilingual Perspective on Probing Gender Bias

K Stańczak - arXiv preprint arXiv:2403.10699, 2024

Gender bias represents a form of systematic negative treatment that targets individuals based on their gender. This discrimination can range from subtle sexist remarks and gendered stereotypes to outright hate speech. Prior research has …


A New Massive Multilingual Dataset for High-Performance Language Technologies

O de Gibert, G Nail, N Arefyev, M Bañón… - arXiv preprint arXiv …, 2024

We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls …


A New Method for Cross-Lingual-based Semantic Role Labeling

M Ebrahimi, BM Bidgoli, N Khozouei - arXiv preprint arXiv:2408.15896, 2024

… It has received specialized training using 2.5 terabytes of meticulously processed CommonCrawl data, which includes text from 100 different languages. RoBERTa itself is a transformer-based model that has been pre-trained in a self-supervised …


A New Type of Foundation Model Based on Recordings of People's Emotions and Physiology

D Gamez, D Barcari, A Grig - arXiv preprint arXiv:2408.00030, 2024

… These models are trained on large amounts of text and image data scraped from the Internet - often accessed through the Common Crawl repository. Researchers and technology companies are starting to realize that this data source could become …


A Novel Hashcode-based Duplication Reduction via Thresholding Approach for Large-scale Web Documents

Modern search engines encounter a significant challenge when it comes to handling duplicate and nearly identical web pages, particularly during the indexing process for vast amounts of web content. This issue can lead to slow search results and …


A Permutaion Importance Based feature selection method and Deep Learning Model to Detect Phishing Websites

Phishing attacks pose a significant and escalating threat to cybersecurity in recent times. This deceptive scam aims to trick naive users, luring them into visiting harmful websites and sharing sensitive information, including credentials, credit card …


A phenomenology and epistemology of large language models: Transparency, trust, and trustworthiness

R Heersmink, B de Rooij, MJC Vázquez, M Colombo

… We know that its dataset includes the Common Crawl, which is a publicly available corpus of Webpages, including billions of Webpages … whether sources in the Common Crawl or Wikipedia are prioritised. This is again a serious issue to do …


A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)

… For low-resource languages, XLM models trained on CommonCrawl-100 did better than the ones trained using Wikipedia. Another example is mT5 [32], which uses multilingual corpus mC4 to pretrain the model. When dealing with multilingual …


A Review of Multi-Modal Large Language and Vision Models

K Carolan, L Fennelly, AF Smeaton - arXiv preprint arXiv:2404.01322, 2024

… The Falcon LLM team created the RefinedWed dataset, which is a massive web dataset based on CommonCrawl. TII focused on scaling and improving the quality of web data by employing large-scale de-duplication and strict filtering, resulting in …


A review of multimodal-based emotion recognition techniques for cyberbullying detection in online social media platforms

S Wang, AS Shibghatullah, TJ Iqbal, KH Keoy - Neural Computing and Applications, 2024

Cyberbullying is a serious issue in online social media platforms (OSMP), which requires effective detection and intervention systems. Multimodal emotion recognition (MER) technology can help prevent cyberbullying by analyzing emotions …


A Review of the Challenges with Massive Web-mined Corpora Used in Large Language Models Pre-Training

M Perełkiewicz, R Poświata - arXiv preprint arXiv:2407.07630, 2024

… Common Crawl is an open repository of web crawl data collected over years. The Common Crawl repository provides around 20TB of scraped … Due to the wide coverage of diverse domains and languages, as well as the size of the collected data …


A review on emotion detection by using deep learning techniques

T Chutia, N Baruah - Artificial Intelligence Review, 2024

Along with the growth of Internet with its numerous potential applications and diverse fields, artificial intelligence (AI) and sentiment analysis (SA) have become significant and popular research areas. Additionally, it was a key technology that …


A Robust Approach to E-Banking Phishing Detection using Ensemble Methods and LSTM

NVS Reddy, ARR Saai, TV Ramanujan, NRK Reddy… - … International Conference on …, 2024

… range of datasets from reputable community platforms like VirusTotal, Common Crawl, and PhishTank. This extensive dataset, spanning … Integration of Common Crawl further enriched the dataset, ensuring a diverse set of websites for effective …


A robust hybrid approach with product context-aware learning and explainable AI for sentiment analysis in Amazon user reviews

E Hashmi, SY Yayilgan - Electronic Commerce Research, 2024

… FastText, Footnote 5 a word representation library developed by the Facebook research team, encompasses a repository of 2 million common crawl words with 300 dimensions, resulting in a vast collection of 600 billion word vectors. Distinguished …


A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism

B Thompson, MP Dhaliwal, P Frisch, T Domhan… - arXiv preprint arXiv …, 2024

… 2021), which is in turn based on Common CrawlCommon Crawl is a long running web-scraping project which maintains a free, open source repository of web-scraped data. ccMatrix is created by embedding Common Crawl sentences into a …


A Short Commentary on Trinh & Le (2018)

… corpus (trained on the language model LM-1-Billion, CommonCrawl, and SQuAD), the probabilities of s1 and s2 appearing in a large corpus. The substitution that turns out to be more probable, is considered to be the more probable referent of “it”. There …


A Study of Aging Through Speech and Language Analysis

Speech is a promising biomarker for cognitive impairment and dementing illness. Compared to traditional biomarkers, digital biomarkers are often less invasive, cheaper to measure, and require less instruction. They also allow continuous and …


A support system for the detection of abusive clauses in B2C contracts

S Dadas, M Kozłowski, R Poświata, M Perełkiewicz… - Artificial Intelligence and …, 2024

… To better balance our dataset, we retrieved an additional 7547 safe clauses from Polish CommonCrawl Footnote 15 dumps using weak supervision techniques. Our approach involved scanning approximately 100 GB of Polish web pages and …


A Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterization

T Kumarage, G Agrawal, P Sheth, R Moraffah… - arXiv preprint arXiv …, 2024

We have witnessed lately a rapid proliferation of advanced Large Language Models (LLMs) capable of generating high-quality text. While these LLMs have revolutionized text generation across various domains, they also pose significant …


A Survey of Conversational Styles and Systems

R Nagpal, S Fatty, DG Brizan - 2024 16th International Conference on Human …, 2024

Conversational systems are an emerging area in natural language processing. This survey paper provides a comprehensive overview of architectures, methods, and datasets used to design the development of conversational agents. We have …


A Survey of Large Language Models for European Languages

… The OSCAR [122] corpus is a large multilingual text dataset extracted from Common Crawl (CC). The German subset of OSCAR is particularly noteworthy, measuring 496.7GB in size and containing approximately 7 billion documents and …


A Survey of Multimodal Large Language Model from A Data-centric Perspective

T Bai, H Liang, B Wan, L Yang, B Li, Y Wang, B Cui… - arXiv preprint arXiv …, 2024

… CommonCrawl project serves as the most commonly … , CommonCrawl’s vast archive of web pages, which includes numerous image-text pairs, has also become a vital resource for constructing multimodal pre-training datasets such as LAION-5B [236] …


A Survey of Neural Machine Translation based on Knowledge Distillation

MA Chang, T Yonghong, Z Xiaoli, SUN Kangkang - Journal of Frontiers of Computer Science …

Abstract: Machine Translation (MT) is the process of using a computer to convert a language into another language with the same semantics. With the introduction of neural network, neural machine translation (NMT), as a powerful machine …


A Survey of Resource-efficient LLM and Multimodal Foundation Models

M Xu, W Yin, D Cai, R Yi, D Xu, Q Wang, B Wu, Y Zhao… - arXiv preprint arXiv …, 2024

Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine learning lifecycle, from training to deployment …


A Survey of Web Content Control for Generative AI

M Dinzinger, F Heß, M Granitzer - arXiv preprint arXiv:2404.02309, 2024

… The documents were sourced from the web archive of Common Crawl,16 offering a broad and randomly selected cross-section of the internet. These files were all collected in November and December 2023. Further details on the experimental …


A Survey on Data Selection for Language Models

A Albalak, Y Elazar, SM Xie, S Longpre, N Lambert… - arXiv preprint arXiv …, 2024

… They train a classifier using the “high-quality” reference corpora as the positive class and unfiltered Common Crawl documents as the negative class. This classifier then defines the utility metric for selecting data that follows the desired distribution …


A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers

K Huang, F Mo, H Li, Y Li, Y Zhang, W Yi, Y Mao, J Liu… - arXiv preprint arXiv …, 2024

The rapid development of Large Language Models (LLMs) demonstrates remarkable multilingual capabilities in natural language processing, attracting global attention in both academia and industry. To mitigate potential discrimination …


A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias

Y Xu, L Hu, J Zhao, Z Qiu, Y Ye, H Gu - arXiv preprint arXiv:2404.00929, 2024

… A significant portion of these training data originates from multilingual repositories like Common Crawl, WikiPedia and Web documents, encompassing a broad range of languages. These multilingual repositories are crucial for enhancing the cross-lingual …


A survey on recent advances in named entity recognition

I Keraghel, S Morbieu, M Nadif - arXiv preprint arXiv:2401.10825, 2024

Named Entity Recognition seeks to extract substrings within a text that name real-world objects and to determine their type (for example, whether they refer to persons or organizations). In this survey, we first present an overview of recent popular …


A Survey on the Use of Large Language Models (LLMs) in Fake News

E Papageorgiou, C Chronis, I Varlamis, Y Himeur - Future Internet, 2024

… It was trained on the Colossal Clean Crawled Corpus (C4) dataset, a 750 GB dataset created from Common Crawl’s web-extracted text. The model architecture is similar to the original … It was trained on the RealNews dataset, created from …


A systematic review of multidimensional relevance estimation in information retrieval

G Peikos, G Pasi - Wiley Interdisciplinary Reviews: Data Mining and …

In information retrieval, relevance is perceived as a multidimensional and dynamic concept influenced by user, task, and domain factors. Relying on this perspective, researchers have introduced multidimensional relevance models addressing …


A Systematic Study of the Role of Data Quality and Alignment for Fine-tuning LLMs for Enhanced Autoformalization

K Chawla, A Sahai, M DePavia, B Miranda - The Second Tiny Papers Track at ICLR 2024

This study explores the role of data quality, particularly alignment, in fine-tuning Large Language Models (LLMs) for the task of autoformalization. Contrary to the conventional emphasis on dataset size, our research highlights the importance of …


A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models

H Zhang, PS Yu, J Zhang - arXiv preprint arXiv:2406.11289, 2024

… WCEP [66] is built based on news events from Wikipedia Current Events Portal (WCEP) and similar articles in the CommonCrawl News dataset. It features high alignment with several realworld industrial use cases and has 235 articles per cluster on …


A Taxonomy of Stereotype Content in Large Language Models

G Nicolas, A Caliskan - arXiv preprint arXiv:2408.00162, 2024

This study introduces a taxonomy of stereotype content in contemporary large language models (LLMs). We prompt ChatGPT 3.5, Llama 3, and Mixtral 8x7B, three powerful and widely used LLMs, for the characteristics associated with 87 social …


A Trustworthy Automated Short-Answer Scoring System Using a New Dataset and Hybrid Transfer Learning Method

M Maslim, HC Wang, CD Putra, YD Prabowo - 2024

To measure the quality of student learning, teachers must conduct evaluations. One of the most efficient modes of evaluation is the short answer question. However, there can be inconsistencies in teacher-performed manual evaluations due to an …


ABCF: An Adaptive Balanced Multimodal Website Classification Framework

Z Liu, W Liu, X Tong, Q Shen, Q Zheng, H Hu, H Liu… - 2024 27th International …, 2024

Websites are a crucial medium for conveying multimedia information. However, malicious websites can lurk among them, posing a threat to the security of users’ sensitive data and personal privacy. Hence, it is significant to identify and categorize …


AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarking

W Li, C Qu, X Chen, PRAS Bassi, Y Shi, Y Lai, Q Yu… - arXiv preprint arXiv …, 2024

We introduce the largest abdominal CT dataset (termed AbdomenAtlas) of 20,460 three-dimensional CT volumes sourced from 112 hospitals across diverse populations, geographies, and facilities. AbdomenAtlas provides 673K high-quality …


AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters

L Lucy, S Gururangan, L Soldaini, E Strubell… - arXiv preprint arXiv …, 2024

… From this Common Crawl data, we identify website hostnames that include an ABOUT page, which we define as URL paths containing about, aboutme, about-us, or bio (Appendix A). We then pair each ABOUT page with a random page on the …


Abusive Language Detection in Khasi Social Media Comments

A Baruah, L Wahlang, F Jyrwa, F Shadap, F Barbhuiya… - ACM Transactions on Asian …, 2024

This paper describes the work performed for automated abusive language detection in the Khasi language, a low-resource language spoken primarily in the state of Meghalaya, India. A dataset named Khasi Abusive Language Dataset (KALD) was …


Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review

N Prakriya, JN Yen, CJ Hsieh, J Cong - arXiv preprint arXiv:2409.06131, 2024

Large Language Model (LLM) pretraining traditionally relies on autoregressive language modeling on randomly sampled data blocks from web-scale datasets. We take inspiration from human learning techniques like spaced repetition to …


Accelerating Spiking Neural Networks with Parallelizable Leaky Integrate-and-Fire Neurons

… To achieve this, these models undergo training on massive datasets, such as the Common Crawl corpus1, which contains petabytes of textual data. Training on such extensive datasets necessitates the use of fast machine learning models like the …


ACCEPTANCE OF GENERATIVE AI IN KNOWLEDGE WORK

… The GPT-3 was trained with Corpus, a large and structured collection of texts or language data, called Common Crawl including nearly a trillion words. (Brown et al., 2020) In addition, the training set was also expanded with known high-quality …


Accessible Foundation Models: Systems, Algorithms, and Science

… [Raffel et al., 2019] which is a subset of the Common Crawl corpus.We use NVIDIA A40 GPUs for this evaluation. To measure degradation in … -shot experiments, the Pearson correlation coefficient between The Pile Common Crawl …


ACCSAMS: Automatic Conversion of Exam Documents to Accessible Learning Material for Blind and Visually Impaired

D Wilkening, O Moured, T Schwarz, K Muller… - arXiv preprint arXiv …, 2024

… We collected exam materials from the Common Crawl project4, a public archive of web data. We analyzed eight datasets, released between mid 2022 and the end of 2023, extracting PDFs, which include “exam” or “klausur” as a keyword in the URL …


Accuracy of ChatGPT in Neurolocalization

WF Dabbas, YM Odeibat, M Alhazaimeh, MY Hiasat… - Cureus, 2024

… The datasets used for pre-training included Common Crawl dataset, an expanded version of the WebText dataset, two internet-based books corpora (Books1 and Books2), and English-language Wikipedia [13]. …


Active Learning for Identifying Disaster-Related Tweets: A Comparison with Keyword Filtering and Generic Fine-Tuning

D Hanny, S Schmidt, B Resch - Intelligent Systems Conference, 2024

… [4] was utilised as it is pre-trained on approximately 198 million Tweets and has been shown to outperform the similar XLM-RoBERTa model trained using the more general CommonCrawl corpus [11] for the multilingual classification of Tweets. The …


Adam-mini: Use Fewer Learning Rates To Gain More

Y Zhang, C Chen, Z Li, T Ding, C Wu, Y Ye, ZQ Luo… - arXiv preprint arXiv …, 2024

We propose Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint. Adam-mini reduces memory by cutting down the number of learning rates in Adam: Instead of assigning an …


Adaptation of Large Language Models for Streaming Machine Translation

[EN] Machine translation (MT) stands as a pivotal domain within machine learning, where the rise of neural networks has sparked significant advancements, propelling MT into a highly researched field. This has been further fostered by the proliferation …


Adapting LLMs to Downstream Applications

… The model was fed with text that generally precedes private information in a dataset used to train GPT-2—specifically the “Common Crawl” dataset to achieve a better recall. 2 With this approach, authors were not only able to extract extensive …


Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detection

Z Lai, X Zhang, S Chen - arXiv preprint arXiv:2403.13335, 2024

… 5) XLMRoberta: XLMRoberta is a large-scale multilingual transformer-based LM pretrained on a diverse set of one hundred languages using extensive CommonCrawl data [68]. We use the “xlm roberta base multi” as pretrained backbone weight. …


Addressing Both Statistical and Causal Gender Fairness in NLP Models

H Chen, Y Ji, D Evans - arXiv preprint arXiv:2404.00463, 2024

… 2019) comprising nearly 400,000 online biographies of 28 unique occupations scraped from the CommonCrawl. The task is to predict the occupation given in the biography with the occupation title removed. Each biography includes the name and …


Addressing the Challenge of Online Health Misinformation: Detection, Retrieval, and Explainability

Nell’odierna epoca digitale, le piattaforme online costituiscono uno dei mezzi principali utilizzati dalle persone per cercare informazioni relative alla propria salute. Nonostante il web sia un vasto repository di conoscenze in tale ambito, è affetto dal …


Adjusting Interpretable Dimensions in Embedding Space with Human Judgments

K Erk, M Apidianaki - arXiv preprint arXiv:2404.02619, 2024

… The GLoVe embeddings used were trained on Common Crawl (42B tokens, 1.9M vocab, uncased, 300d vectors), downloaded from https: //nlp.stanford.edu/projects/glove/. In order to generate embeddings for contextualized instances of words in our …


AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising

P Zhang, Y Sakai, M Mita, H Ouchi, T Watanabe - arXiv preprint arXiv:2408.05906, 2024

With the increase in the more fluent ad texts automatically created by natural language generation technology, it is in the high demand to verify the quality of these creatives in a real-world setting. We propose AdTEC, the first public …


Adult Content Detection on Indonesian Tweets by Fine-tuning Transformer-based Models

AF Hidayatullah, RA Apong, DTC Lai, A Qazi - 2023 6th International Conference on …, 2023

The prevalence of adult content on social media has harmful effects on the moral values of young individuals. Therefore, effectively filtering inappropriate content on social media like Twitter is essential. Researchers have utilized machine learning …


Advances in Multimodal Information Retrieval and Generation

Writing systems emerged simultaneously and independently in many ancient civilizations across the globe, including Mesopotamia, Egypt, China, India, and Mesoamerica. The invention of writing and scripts has had a tremendous impact on …


Advancing Fake News Detection: Hybrid Deep Learning with FastText and Explainable AI

E Hashmi, SY Yayilgan, MM Yamin, S Ali, M Abomhara - IEEE Access, 2024

… FastText, a word representation tool developed by Facebook’s research division, provides both unsupervised and supervised modes, featuring an extensive lexicon of 2 million words sourced from Common Crawl. Each word is represented in a 300-dimensional …


Advancing Internet Viewpoint Diversity: A Novel Algorithm and a Corpus Creation Tool

… To achieve this, we first develop a big data processing architecture for creating indexed corpora from the Common Crawl web archives. The … Utilizing this tool, we processed approximately 1.2 billion web pages from the Common Crawl dataset …


Advancing language models through domain knowledge integration: a comprehensive approach to training, evaluation, and optimization of social scientific neural …

F Stöhr - Journal of Computational Social Science, 2024

This article proposes a comprehensive strategy for training, evaluating, and optimizing domain-specific word2vec-based word embeddings, using social science literature as an example. Our primary objectives are:(1) to train the embeddings …


Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation

K Uehara, N Goswami, H Wang, T Baba, K Tanaka… - arXiv preprint arXiv …, 2024

The increasing demand for intelligent systems capable of interpreting and reasoning about visual content requires the development of Large Multi-Modal Models (LMMs) that are not only accurate but also have explicit reasoning capabilities. This paper …


Advancing medical imaging with language models: featuring a spotlight on ChatGPT

M Hu, JY Qian, S Pan, Y Li, RLJ Qiu, X Yang - Physics in Medicine and Biology, 2024

This review paper aims to serve as a comprehensive guide and instructional resource for researchers seeking to effectively implement language models in medical imaging research. First, we presented the fundamental principles and …


Advancing Real-time Pandemic Forecasting Using Large Language Models: A COVID-19 Case Study

H Du, J Zhao, Y Zhao, S Xu, X Lin, Y Chen, LM Gardner - arXiv preprint arXiv …, 2024

HF Yang, H Du, J Zhao, Y Zhao, S Xu, X Lin, Y Chen… - 2024

Forecasting the short-term spread of an ongoing disease outbreak is a formidable challenge due to the complexity of contributing factors, some of which can be characterized through interlinked, multi-modality variables such as epidemiological …


Advancing scientific writing with artificial intelligence: expanding the research toolkit

AG Atkinson, H Lia, SM Navarro - Global Surgical Education-Journal of the …, 2024

Generative artificial intelligence (AI) is revolutionizing scientific writing by offering novel tools that integrate into various stages of the research process. This paper explores the application of AI tools in scientific writing, focusing on literature review …


Afrikaans Literary Genre Recognition using Embeddings and Pre-Trained Multilingual Language Models

E Kotzé, BA Senekal - 2024 International Conference on Artificial Intelligence …, 2024

… XLM-R (or XLM-RoBERTa) is a multilingual version of RoBERTa with 279M parameters and was trained on a 2.5TB of CommonCrawl data containing 100 languages. Afrikaans is included in all three of these multilingual models. …


AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation

M Hu, P Zhao, C Xu, Q Sun, J Lou, Q Lin, P Luo… - arXiv preprint arXiv …, 2024

… The inspiration corpus can be implemented in various ways, such as using a large-scale pre-trained corpus like Common Crawl. Alternatively, a domain-specific corpus, such as a code generation dataset [25, 7], can be used to generate environments for a …


Aggression and Misogyny in Hindi and Bangla: A Study of YouTube Comments

R Kumar, B Lahiri - Studies in Pragmatics, 2024

… The pre-trained models are typically trained on web-scale data, ie text collected from all over the web such as the whole of Wikipedia and a common crawl of the websites currently on the internet amounting to billions (or trillions) of tokens and …


AgXQA: A benchmark for advanced Agricultural Extension question answering

J Kpodo, P Kordjamshidi, AP Nejadhashemi - Computers and Electronics in …, 2024

Large language models (LLMs) have revolutionized various scientific fields in the past few years, thanks to their generative and extractive abilities. However, their applications in the Agricultural Extension (AE) domain remain sparse and limited …


AI and The European Union's Approach to Data Protection: The Case of Chat GPT

Artificial Intelligence (AI) is advancing rapidly, with generative models like ChatGPT revolutionizing numerous industries. However, these advancements present significant challenges in adhering to data protection regulations such as the General …


AI chatbots show promise but limitations on UK medical exam questions: a comparative performance study

MA Sadeq, RMF Ghorab, MH Ashry, AM Abozaid… - Scientific Reports, 2024

… All chatbots scored higher on questions leaked to the C4 database except for ClaudeInstant which performed worse on the seven questions leaked to the common crawl database (0.571 ± 0.535) than other questions (0.631 ± 0.483). Bard …


AI Literacy for the Common Good

S Ullrich, R Messerschmidt - Weizenbaum Journal of the Digital Society, 2024

… A large part of the training data (almost two-thirds) is the Common Crawl dataset, which comprises web pages collected between 2017 and 2020 provided for free by a non-profit organization. Wikipedia articles from the summer of 2022 still flow in at …


AI unveiled personalities: Profiling optimistic and pessimistic attitudes in Hindi dataset using transformer‐based models

Both optimism and pessimism are intricately intertwined with an individual's inherent personality traits and people of all personality types can exhibit a wide range of attitudes and behaviours, including levels of optimism and pessimism. This paper …


AI vs. Human: Decoding Text Authenticity with Transformers

In an era where the proliferation of large language models blurs the lines between human and machine-generated content, discerning text authenticity is paramount. This study investigates transformer-based language models—BERT, RoBERTa, and …


AI's Secret Weapon in Education. ChatGPT–The Future of Personalized Learning

A Popescu - Bulletin of the Transilvania University of Brasov. Series …, 2023

This article delves into the fascinating intersection of AI and education, with a focus on ChatGPT, a conversational AI developed by OpenAI. Noted for its human-like responses, ChatGPT is positioned as a game-changer in personalized learning. The …


AI, Robots. txt

… A significant portion of the training data for LLMs comes from large datasets like "The Pile" and "Common Crawl," which are aggregates of smaller datasets [5, 17]. These datasets, often hundreds of GBs in size, are filtered and processed into plaintext …


AI-Generated Text Detector for Arabic Language

The rise of AI-generated texts (AIGTs), particularly with the arrival of advanced language models like ChatGPT, has spurred a growing need for effective detection methods. While these models offer various beneficial applications, their potential for …


AI-Powered Code Review Assistant for Streamlining Pull Request Merging

C Adapa, SS Avulamanda, ARK Anjana, A Victor - 2024 IEEE International …, 2024

… The model's reliance on Refined Web, primarily sourced from Common Crawl, underscores its commitment to exceptional quality through advanced data augmentation techniques, notably largescale deduplication and rigorous filtering. Unlike conventional …


AI-Proof Your Assignments

ChatGPT and other AI tools are powerful, free, and seemingly everywhere. Faculty love it, fear it, and everything in between. Students also love it, fear it, and everything in between; All the while, they are using it. In this session, we’ll cover a bit about how …


AI-Replicas as Ethical Practice: Introducing an Alternative to Traditional Anonymization Techniques in Image-Based Research

This paper introduces the use of AI-replicas as an alternative to traditional anonymization methods in image-based qualitative research. It emphasizes the ethical and practical dilemmas posed by current anonymization methods, such as …


Algorithmic amplification of biases on Google Search

H Habib, R Stoldt, A High, B Ekdale, A Peterson… - arXiv preprint arXiv …, 2024

The evolution of information-seeking processes, driven by search engines like Google, has transformed the access to information people have. This paper investigates how individuals' preexisting attitudes influence the modern information-seeking …


Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs

AM Kassem, O Mahmoud, N Mireshghallah, H Kim… - arXiv preprint arXiv …, 2024

In this paper, we introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent, compared to what is revealed by prompting the target model with the training data …


ALT-GEN: Benchmarking Table Union Search using Large Language Models

K Pal, A Khatiwada, R Shraga, RJ Miller - Proceedings of the VLDB Endowment. ISSN

David Paterson states that when a eld has good benchmarks, we settle debates and the eld makes rapid progress [33]. Traditionally, benchmark generation is done using synthetic data generators that precisely control parameters such as data size …


Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages

M Andersland - arXiv preprint arXiv:2403.06354, 2024

… Less than 0.1% of CommonCrawl2 is Amharic, and even when combining open source datasets without deduplication, we find that less than 500 million tokens of Amharic are available. In addition, the content of this data tends to be biased …


An Ambiguous Technique for Nonvisual Text Entry

Text entry is a common daily task for many people, but it can be a challenge for people with visual impairments when using virtual touchscreen keyboards that lack physical key boundaries. In this thesis, we investigate using a small number of …


An Analysis of Langauge Frequency and Error Correction for Esperanto

J Liang - arXiv preprint arXiv:2402.09696, 2024

… Goyal builds Commoncrawl Corpus in 100 languages and then uses a language identification model to select Esperanto contexts[8], which … OSCAR [11] provides a collection of unannotated raw data and devises a method to improve the quality of …


An Analysis of Multilingual FActScore

KT Vu, M Krumdick, V Reddy, F Dernoncourt, VD Lai - arXiv preprint arXiv …, 2024

FActScore has gained popularity as a metric to estimate the factuality of long-form texts generated by Large Language Models (LLMs) in English. However, there has not been any work in studying the behavior of FActScore in other languages. This …


AN ANALYSIS ON SHORT-FORM TEXT AND DERIVED ENGAGEMENT

Short text has historically proven challenging to work with in many Natural LanguageProcessing (NLP) applications. Traditional tasks such as authorship attribution benefitfrom having longer samples of work to derive features from. Even …


An Assessment on Comprehending Mental Health through Large Language Models

M Arcan, PD Niland, F Delahunty - arXiv preprint arXiv:2401.04592, 2024

… The training data for GPT-3 is primarily sourced from a filtered version of Common Crawl, contributing to 60% of the weighted pre-training dataset, comprising 410 billion bytepair-encoded tokens. Other data sources include 19 billion tokens from …


An efficient parallel optimization method for large model based on cloud computing

Z Xu - … Conference on Electronic Information Engineering and …, 2024

… Subsequently, we employ a similar methodology to create Pile-CC, involving the download and refinement of two complete Common Crawl (CC) snapshots. The overall process for handling CC data entails extracting text from the raw HTML …


An Extensive Survey on Investigation Methodologies for Text Summarization

A Saklecha, P Uplavdiya, MPS Chawla - Indian Journal of Signal Processing (IJSP), 2023

Natural language processing (NLP) is a fastexpanding field, and text summarization has recently gained a lot of research interest. The necessity for automatic summarizing approaches to effectively digest massive amounts of textual data has …


An In-Depth Analysis of the Adoption of Large Language Models in Clinical Settings: A Fuzzy Multi-Criteria Decision-Making Approach

The growing capabilities of large language models (LLMs) in the medical field hold promising transformational change. The evolution of LLMs, such as BioBERT and MedGPT, has created new opportunities for enhancing the quality of healthcare …


An Integrated Data Processing Framework for Pretraining Foundation Models

Y Sun, F Wang, Y Zhu, WX Zhao, J Mao - arXiv preprint arXiv:2402.16358, 2024

… In the end-to-end evaluation, we train two GPT-2 models using CommonCrawl before and after processing, respectively. The model trained on the refined data demonstrates remarkable performance enhancement across downstream tasks of …


An integrated model based on deep learning classifiers and pre-trained transformer for phishing URL detection

NQ Do, A Selamat, H Fujita, O Krejcar - Future Generation Computer Systems, 2024

The unique nature of website URLs has made phishing detection a challenging task. Unlike natural language, URLs have an unstructured nature with non-linear and sophisticated correlations. Therefore, they should be handled as both natural …


An Introduction to Vision-Language Modeling

F Bordes, RY Pang, A Ajay, AC Li, A Bardes, S Petryk… - arXiv preprint arXiv …, 2024

… Multiple curation steps are used to curate this dataset where English data is collected from common crawl and deduplicated followed by pre-processing HTML document where useful DOM nodes are identified and retained, then for each DOM …


Analysing Slow Thinking Capabilities in Large Language Model Agent-Agent Dialogue

… LLMs can be given data from books, CommonCrawl, Reddit links and Wikipedia, where CommonCrawl consists of petabytes of web archives [113]. Having large datasets makes it more difficult to curate the data. Documentation of the data can …


Analysing The Impact of Sequence Composition on Language Model Pre-Training

Y Zhao, Y Qu, K Staniszewski, S Tworkowski, W Liu… - arXiv preprint arXiv …, 2024

Most language model pre-training frameworks concatenate multiple documents into fixed-length sequences and use causal masking to compute the likelihood of each token given its context; this strategy is widely adopted due to its simplicity and …


Analysing the landscape of Deep Fake Detection: A Survey

K Vyas, P Pareek, R Jayaswal, S Patil - … of Intelligent Systems and Applications in …, 2024

… Grover utilizes a test set created by the free and open-source web crawler and archive Common Crawl [35]. As an example, a team of academics from Harvard and the MIT-IBM Watson laboratory developed the Giant language model test room, a …


ANALYSIS OF CROSS-LINGUAL MODELS FOR ASPECT-BASED SENTIMENT ANALYSIS: A CASE STUDY IN DUTCH AS A LOW-RESOURCE LANGUAGE.

This study addressed the limited availability of well-labeled data and research resources for end-to-end ABSA in low-resource languages, particularly Dutch. The study aims to tackle this challenge by investigating the performance of state-of-the-art …


AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets

P Lesci, A Vlachos - arXiv preprint arXiv:2404.05623, 2024

Active learning for imbalanced classification tasks is challenging as the minority classes naturally occur rarely. Gathering a large pool of unlabelled data is thus essential to capture minority instances. Standard pool-based active learning is …


Ancient Greek's New Technological Muse: Extracting Topoi in the Anacreontea with LLMs

RO Nunes, JG Zandoná, JV Maia, A Spritzer… - Seminário Integrado de …, 2024

… The pre-training corpora include texts from Wikipedia, the European Parliament Proceedings Parallel Corpus, and OSCAR, a cleansed version of Common Crawl Researchers have also introduced a pilot study focused on the automatic linguistic …


and S. Abirami

RS Gulecha, SKNR Subramanian - Speech and Language Technologies for Low …

In recent years, the Internet and social media have sparked a revolution in the way information is exchanged. The growth of social media and micro-blogging sites not only provides platforms for empowering freedom of expression and individual voices …


Answer Agnostic Question Generation in Bangla Language

AR Fahad, N Al Nahian, MA Islam, RM Rahman - International Journal of Networked …, 2024

… mT5 is a massively multilingual model pre-trained using fresh Common Crawl-based data that includes 101 languages [14]. Another model, BanglaT5, a Transformer model for the Bangla language, was trained using a 27.5 GB clean corpus of Bangla …


AntiPhishStack: LSTM-based Stacked Generalization Model for Optimized Phishing URLs Detection

S Aslam, H Aslam, A Manzoor, C Hui, A Rasool - 2024

The escalating reliance on revolutionary online web services has introduced heightened security risks, with persistent challenges posed by phishing despite extensive security measures. Traditional phishing systems, reliant on machine …


AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

J Zhan, J Dai, J Ye, Y Zhou, D Zhang, Z Liu, X Zhang… - arXiv preprint arXiv …, 2024

We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. AnyGPT can be trained stably without any …


APLIKASI CHATGPT DALAM BAHASA MELAYU: ISU DAN CADANGAN PENYELESAIAN: OPERATING CHATGPT IN MALAY: ISSUES AND PROPOSED …

H Nomoto, D Moeljadi, FAA Razak - RENTAS: Jurnal Bahasa, Sastera dan Budaya, 2024

… bekerjasama dengan kerajaan Brunei dan Singapura, meminta OpenAI dan Common Crawl, iaitu sumber utama set data latihan ChatGPT, … .my yang disenaraikan dalam 500 laman web teratas yang berdaftar di Common Crawl.Untuk …


Application of an Improved Convolutional Neural Network Algorithm in Text Classification

J Peng, S Huo - Journal of Web Engineering, 2024

This paper proposes a text classification model based on a combination of a convolutional neural network (CNN) and a support vector machine (SVM) using Amazon review polarity, TREC, and Kaggle as experimental data. By adding an …


Application of Open-source Large Language Model (LLM) for Simulation of a Vulnerable IoT System and Cybersecurity Best Practices Assistance

This paper explores the role of open-source large language models in IoT cybersecurity world. The threats of malicious activity on the Internet and the loss of private information are very real and lead to serious consequences. The purpose of …


Applications of AI in Computer Vision and NLP

SB Rajasekaran - Journal of Artificial Intelligence & Cloud Computing …, 2023

… In language AI, there are several commonly used datasets such as Common Crawl, Wikipedia, and OpenWebText. However, creating and maintaining these datasets can also be a challenging task, as it needs a big amount of work and it can …


Applying German word vectors to assess flexibility performance in the associative fluency task.

M Camenzind, M Single, SM Gerber, T Nef, CL Bassetti… - Psychology of Aesthetics …, 2024

… One model was trained on common crawl, a nonprofit organization crawling the web and making resulting data publicly available, and the other was trained on Wikipedia. The vectors were trained using the CBOW algorithm with a special focus …


Applying Named Entity Recognition and Graph Networks to Extract Common Interests from Thematic Subfora on Reddit

J Sawicki, M Ganzha, M Paprzycki, Y Watanobe - Applied Sciences, 2024

… , accessed on 13 February 2024) (pre-trained BERT large models fine-tuned on the CoNLL-2003 dataset) and flair/ner-english-large (https://huggingface.co/flair/ner-english-large, accessed on 13 February 2024) (an XLM-RoBERTa [89] pre-trained on a cleaned …


Applying Transfer Learning to German Metaphor Prediction

M Berger, SM Reimann, NM Kiwitt - Proceedings of the 2024 Joint International …, 2024

… -scores using CommonCrawl embeddings (F1 60%) and News Commentary embeddings (F1 58%) while CommonCrawl contains over 2 m… 4), we can see that the bilingual embeddings approach, especially using Europarl and CommonCrawl …


Arabic sarcasm detection: An enhanced fine-tuned language model approach

MA Galal, AH Yousef, HH Zayed, W Medhat - Ain Shams Engineering Journal, 2024

Sarcasm is a complex linguistic phenomenon involving humor, criticism, or phrases that convey the opposite meaning, mask true feelings, and play pivotal roles in various aspects of communication. Therefore, identifying sarcasm is essential for …


Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Models

L Merrick, D Xu, G Nuti, D Campos - arXiv preprint arXiv:2405.05374, 2024

This report describes the training dataset creation and recipe behind the family of \texttt{arctic-embed} text embedding models (a set of five models ranging from 22 to 334 million parameters with weights open-sourced under an Apache-2 license). At the time of …


Are Human Conversations Special? A Large Language Model Perspective

T Jawale, C Animesh, S Vallath, K Talamadupula… - arXiv preprint arXiv …, 2024

… We analyze the web data from CommonCrawl[21] dumps for human conversation data and the types of conversations in Table 1. We randomly sample a subset of the dump and deduplicate it so that it can be used to approximate the data distribution …


Are LLMs Robust for Spoken Dialogues?

SM Mousavi, G Roccabruna, S Alghisi, M Rizzoli… - arXiv preprint arXiv …, 2024

… The LLM finetuned for this task is T5 Small [21], (12 layers, 60M parameters), a transformer-based encoder-decoder model, pre-trained on the Common Crawl dataset with 750GB of web page text. The fine-tuning was performed following the …


Artificial Intelligence and the Law of Machine-Readability: A Review of Human-to-Machine Communication Protocols and their (In) Compatibility with Article 4 (3) of the …

H Hamann - JIPITEC–Journal of Intellectual Property, Information …, 2024

Many legal scholars critique the supposed ineffectiveness of European copyright regulation regarding commercial text and data mining. At the same time, tech-savvy entrepreneurs keep proposing new standards to effectuate them at a rate that has …


Artificial Intelligence as the Next Front in the Class War

For many years, artificial intelligence has been confined to the realm of science fiction, and while the technology has been in development, predicting the effects AI will have on our society has been a challenging endeavor. The release of ChatGPT …


Artificial Intelligence for Anesthesiology Board-style Examination Questions: Role of Large Language Models

AA Khan, R Yunus, M Sohail, TA Rehman, S Saeed… - Journal of Cardiothoracic …, 2024

Background : New artificial intelligence tools have been developed that have implications for medical usage. Large language models such as the widely used ChatGPT, developed by OpenAI, have not been explored in the context of …


Springer MRW: [AU:, IDX:] Page 1 Artificial Intelligence in the Media Economy: A Systematic Review of Use Cases, Application Potentials, and Challenges of Generative Language Models Tim Prien and Klaus Goldhammer Contents 1 …


Artificial intelligence in the news: how AI retools, rationalizes, and reshapes journalism and the public arena

Despite growing interest, the effects of AI on the news industry and our information environment — the public arena — remain poorly understood. Insufficient attention has also been paid to the implications of the news industry’s dependence on …


Artificial Intelligence Integration, Concerns, Benefits, and the Need for Ethical Policies for Community Foundations and Nonprofit Organizations

This study describes the nature and extent of Artificial Intelligence (AI) integration in philanthropic organizations and assesses the capacity and readiness of nonprofits in adopting AI for philanthropic use. Additionally, the study explores the nature and …


Artificial Intelligence Supporting Independent Student Learning: An Evaluative Case Study of ChatGPT and Learning to Code

K Hartley, M Hayak, UH Ko - Education Sciences, 2024

Artificial intelligence (AI) tools like ChatGPT demonstrate the potential to support personalized and adaptive learning experiences. This study explores how ChatGPT can facilitate self-regulated learning processes and learning computer programming …


Artificial Intelligence, Large Language Models, and Design Thinking in TPC Classrooms

C Masters-Wheeler, J Bay, P Sullivan - Programmatic Perspectives, 2023

… Common Crawl, for instance, is an open repository of web data that is accessible to anyone and can be used to train AI models. Large language models (LLMs) use the textual data from sources like Common Crawl and other freely accessible data …


Asking GPT for the Ordinary Meaning of Statutory Terms

C Engel, RH McAdams - MPI Collective Goods Discussion Paper, 2024

We report on our test of the Large Language Model (LLM) ChatGPT (GPT) as a tool for generating evidence of the ordinary meaning of statutory terms. We explain why the most useful evidence for interpretation involves a distribution of replies rather …


Asking Questions Framework for Oral History Archives

J Švec, M Bulín, A Frémund, F Polák - European Conference on Information Retrieval, 2024

… The resulting Wav2Vec 2.0 end-to-end model was combined with a lower-case four-gram language model estimated from CommonCrawl data. To decrease the model’s … The model was pre-trained on a self-supervised text-restoration task …


Assessing and Optimizing Large Language Models on Spondyloarthritis Multi-Choice Question Answering: Protocol for Enhancement and Assessment

A Wang, Y Wu, X Ji, X Wang, J Hu, F Zhang, Z Zhang… - JMIR Research Protocols, 2024

Background Spondyloarthritis (SpA), a chronic inflammatory disorder, predominantly impacts the sacroiliac joints and spine, significantly escalating the risk of disability. SpA’s complexity, as evidenced by its diverse clinical presentations and symptoms …


Assessing the potential of leveraging LLaMA-2 to create an institute-specific online chatbot assistant.

With the increasing popularity of chatbots such as ChatGPT, when faced with questions, people are getting more and more inclined to opt for a chatbot instead of browsing, calling, or travelling and waiting in line at service points. 8 To address this …


Assessing the research landscape and clinical utility of large language models: a scoping

YJ Park, A Pillai, J Deng, E Guo, M Gupta, M Paget… - 2024

… Introduced in November 2022, ChatGPT was trained using a large corpora of unlabelled text, including CommonCrawl, WebText, and Wikipedia, as well as internet-based book corpora spanning multiple languages [3]. GPT, along with other …


ASSESSMENT OF HARMFULNESS OF INSTRUCTION-TUNED LARGE LANGUAGE MODEL LLAMA 2 BY SELF-EVALUATION

The relatively recent introduction of public large language models like ChatGPT has raised numerous concerns regarding the ethical issues surrounding their application. Their general proficiency in performing natural language tasks, rivaling that of …


Asymmetric Polysemous Reasoning for Image-Text Matching

H Zhang, M Yang - 2023 IEEE International Conference on Data Mining …, 2023

Image-text matching has received growing interest since it bridges vision and language. The key challenge lies in how to learn correspondence between image and text. Upon observation, we find existing works suffer from two limitations. Firstly …


Asynchronous Local-SGD Training for Language Modeling

B Liu, R Chhaparia, A Douillard, S Kale, AA Rusu… - arXiv preprint arXiv …, 2024

Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is an approach to distributed optimization where each device performs more than one SGD update per communication. This work presents an empirical …


Atmospheric Limitations for High-frequency Ground-based Very Long Baseline Interferometry

DW Pesce, L Blackburn, R Chaves, SS Doeleman… - The Astrophysical Journal, 2024

Very long baseline interferometry (VLBI) provides the highest-resolution images in astronomy. The sharpest resolution is nominally achieved at the highest frequencies, but as the observing frequency increases, so too does the atmospheric contribution …


AtomGPT: Atomistic Generative Pre-trained Transformer for Forward and Inverse Materials Design

K Choudhary - arXiv preprint arXiv:2405.03680, 2024

Large language models (LLMs) such as generative pretrained transformers (GPTs) have shown potential for various commercial applications, but their applicability for materials design remains underexplored. In this article, we introduce AtomGPT, a …


Auditing Large Language Models for Enhanced Text-Based Stereotype Detection and Probing-Based Bias Evaluation

Z Wu, S Bulathwela, M Perez-Ortiz, AS Koshiyama - arXiv preprint arXiv:2404.01768, 2024

Recent advancements in Large Language Models (LLMs) have significantly increased their presence in human-facing Artificial Intelligence (AI) applications. However, LLMs could reproduce and even exacerbate stereotypical outputs from …


Augmenting a Spanish clinical dataset for transformer-based linking of negations and their out-of-scope references

AJ Tamayo-Herrera, DA Burgos, A Gelbukh - Natural Language Processing

A negated statement consists of three main components: the negation cue, the negation scope, and the negation reference. The negation cue is the indicator of negation, while the negation scope defines the extent of the negation. The negation …


Augmenting Knowledge-Based Conversational Search Systems With Large Language Models

Conversational interfaces are increasingly used by websites and apps, including office software as well as creativity tools. This trend reflects a shift towards natural language in human-computer interaction, which is also facilitated by utilizing Large …


Automated Risk Analysis for Construction Contracts Using Neural Networks

K Hamdy, I AbdelRasheed, YAS Essawy… - Journal of Legal Affairs and …, 2024

… for requiring less training time compared to previous recurrent neural architectures, such as long short-term memory (LSTM), and has been prevalently adopted for training large language models on large (language) data sets, such as the …


Automated Text Identification on Languages of the Iberian Peninsula: LLM and BERT-based Models Aggregation

This paper describes our solution approach for the IberAuTexTification (Automated Text Identification on Languages of the Iberian Peninsula) competition held as part of the IberLEF 2024 conference. Machinegenerated text fragments can be spotted in …


AutoMathText: Autonomous Data Selection with Language Models for Mathematical Texts

Y Zhang, Y Luo, Y Yuan, ACC Yao - arXiv preprint arXiv:2402.07625, 2024

To improve language models' proficiency in mathematical reasoning via continual pretraining, we introduce a novel strategy that leverages base language models for autonomous data selection. Departing from conventional supervised fine-tuning or …


Automatic Coding of Contingency in Child-Caregiver Conversations

A Agrawal, M Nikolaus, B Favre, A Fourtassi - 2024

One of the most important communicative skills children have to learn is to engage in meaningful conversations with people around them. At the heart of this learning lies the mastery of contingency, ie, the ability to contribute to an ongoing exchange …


Automatic Control With Human-Like Reasoning: Exploring Language Model Embodied Air Traffic Agents

J Andriuškevičius, J Sun - arXiv preprint arXiv:2409.09717, 2024

… Most of the data used to train leading-edge large language models, such as the latest Common Crawl dataset [5], which comprises over 250 billion web pages, sources information from publicly accessible internet sites. This extensive training …


Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach

HV Vo, V Khalidov, T Darcet, T Moutakanni… - arXiv preprint arXiv …, 2024

… By filtering Common Crawl data, the authors managed to obtain large datasets to train reliable … To this end, we apply our curation pipeline to two text pools based on Common Crawl. The first … Doing so keeps the original data distribution closer to …


Automatic detection of hate speech in code-mixed Indian languages in twitter social media interaction using DConvBLSTM-MuRIL ensemble method

P Kakati, D Dandotiya - Social Network Analysis and Mining, 2024

… During the pre-training process, 2.5 terabytes of Common Crawl data in 100 languages are used. This model gives a much better performance than multilingual BERT when code-switching, especially for languages with little resources, because …


Automatic Ellipsis Reconstruction in Coordinated German Sentences Based on Text-to-Text Transfer Transformers

M Schmidt, K Harbusch, D Memmesheimer - International Conference on Text …, 2024

Ellipsis reconstruction, ie, revealing omitted syntactically obligatory words in a sentence, is still a challenging task in Natural Language Processing (NLP) technologies, even though this information is essential for advanced human-computer …


Automatic Extraction of User-Centric Aspects for Tourist Spot Recommender Systems Using Reviews in Japanese

F Uwano, R Kobayashi, M Ohta - International Conference on Human-Computer …, 2024

In tourist reviews, various pieces of information are described to confirm the characteristics of tourist spots. This paper proposes a method to extract tourist spot aspects using Japanese tourist reviews automatically. The aspects of tourist spots …


Automatic language ability assessment method based on natural language processing

N Nnamoko, T Karaminis, J Procter, J Barrowclough… - Natural Language …, 2024

Background and Objectives: The Wechsler Abbreviated Scales of Intelligence second edition (WASI-II) is a standardised assessment tool that is widely used to assess cognitive ability in clinical, research, and educational settings. In one of the …


Automatic Personality Recognition via XLNet with Refined Highway and Switching Module for Chatbot

OTC Chen, CH Tsai, MH Ha - 2024 IEEE International Symposium on Circuits and …, 2024

This study introduces an Automatic Personality Recognition (APR) model, named XLNet refined-highway-switch network, comprising XLNet, refined highway units, a switching module, and a fully-connected layer. The pretrained XLNet is employed to …


Automatic Pull Request Description Generation Using LLMs: A T5 Model Approach

MN Sakib, MA Islam, MM Arifin - arXiv preprint arXiv:2408.00921, 2024

… One reason T5 excels at producing high-quality PR descriptions is its training on the diverse ”Colossal Clean Crawled Corpus” (C4), a cleaned and structured version of the Common Crawl web dump. The C4 dataset, which encompasses 250 …


Automatické vytváření souhrnů historických dokumentů

… The dataset is comprised of a large amount of cleaned English text extracted from Common Crawl … The results show that the pre-training has 1https://commoncrawl.org/ … It uses the same pretraining objective as BART and it uses a multilingual Common …


Automating Code Adaptation for MLOps--A Benchmarking Study on LLMs

H Patel, BA Ramanan, MA Khan, T Williams… - arXiv preprint arXiv …, 2024

This paper explores the possibilities of the current generation of Large Language Models for incorporating Machine Learning Operations (MLOps) functionalities into ML training code bases. We evaluate the performance of OpenAI (gpt-3.5-turbo) and …


Automating String Encoding in AutoML

Automated Machine Learning (AutoML) systems often struggle with data sets with string data. Due to their unstructured nature and dependence on domain knowledge, string data are a frequent cause of errors and coming up with generalized ways of …


Automation and Optimisation of Seo Content Gap Analysis Task with Semantic Clustering of Portuguese Search Keywords

In the field of digital marketing, efficient content gap analysis is crucial for developing effective SEO strategies. Traditional approaches to this task are time-consuming, representing a challenge for Organic Performance teams, who must process large …


Autonomous Data Selection with Language Models for Mathematical Texts

Y Zhang, Y Luo, Y Yuan, AC Yao - ICLR 2024 Workshop on Navigating and …, 2024

To improve language models’ proficiency in mathematical reasoning via continual pretraining, we introduce a novel strategy that leverages base language models for autonomous data selection. Departing from conventional supervised fine-tuning or …


AutoScale: Automatic Prediction of Compute-optimal Data Composition for Training LLMs

F Kang, Y Sun, B Wen, S Chen, D Song, R Mahmood… - arXiv preprint arXiv …, 2024

… With more compute, data sources with diverse examples, such as CommonCrawl, demonstrate continued reductions in training loss even at … With more compute, data sources with diverse examples, such as CommonCrawl, continue to reduce …


BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

J Xue, M Zheng, Y Hu, F Liu, X Chen, Q Lou - arXiv preprint arXiv:2406.00083, 2024

Large Language Models (LLMs) are constrained by outdated information and a tendency to generate incorrect data, commonly referred to as "hallucinations." Retrieval-Augmented Generation (RAG) addresses these limitations by combining …


BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline

G Dong, D Pan, Y Sun, S Zhang, Z Liang, X Wu… - arXiv preprint arXiv …, 2024

… We collect 94 publicly available batches of CommonCrawl spanning the past decade. We construct our WARC extraction and processing pipeline, considering the effectiveness and cost of the WET and WARC formats. … https://commoncrawl.org …


Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embedding

LC Chen, ZR Li - arXiv preprint arXiv:2404.00862, 2024

… Owing to computational constraints, we further conducted additional sampling of the OSCAR dataset and the Common Crawl dataset to mitigate the issue related to dataset size. In the long run, there are around 13 billion tokens left in the corpus. The …


Balanced Data Sampling for Language Model Training with Clustering

Y Shao, L Li, Z Fei, H Yan, D Lin, X Qiu - arXiv preprint arXiv:2402.14526, 2024

Data plays a fundamental role in the training of Large Language Models (LLMs). While attention has been paid to the collection and composition of datasets, determining the data sampling strategy in training remains an open question. Most …


Balancing Multilingual Model Training Data Using Exponential Smoothing

… The XLM-R [9] model is pre-trained over a larger common crawl corpus similarly to MBERT, using language sampling according to exponentially-smoothed language probabilities. Again, performance on low-resources tasks seems to benefit …


BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts

Q Zhang, N Gritsch, D Gnaneshwar, S Guo, D Cairuz… - Workshop on Efficient Systems for …

Training Mixture of Experts (MoEs) from scratch in a large-scale regime is expensive. Previous work addresses this challenge by independently training multiple dense expert models and using them to initialize an MoE. In particular, initializing MoE …


Bangla Emergency Post Classification on Social Media using Transformer Based BERT Models

AA Nabil, D Das, MS Salim, S Arifeen, HMA Fattah - 2023 6th International …, 2023

… It is a large multi-lingual language model trained on 2.5TB of filtered Common Crawl data. XLMRoBERTa is a multilingual model trained in 100 different languages. Unlike some XLM multilingual models, it does not require lang tensors to understand …


BeeManc at the PLABA Track of TAC-2023: Investigating LLMs and Controllable Attributes for Improving Biomedical Text Readability

Z Li, S Belkadi, N Micheletti, L Han, M Shardlow… - arXiv preprint arXiv …, 2024

In this system report, we describe the models and methods we used for our participation in the PLABA2023 task on biomedical abstract simplification, part of the TAC 2023 tracks. The system outputs we submitted come from the following three …


Bella Turca: A Large-Scale Dataset of Diverse Text Sources for Turkish Language Modeling

D Altinok - International Conference on Text, Speech, and …, 2024

… To meet the data requirements for language modeling, earlier large language models (LLMs) heavily relied on the Common Crawl dataset as their primary or sole source of training data [5, 26]. While training on the Common Crawl has proven …


Benchmarking PathCLIP for Pathology Image Analysis

S Zheng, X Cui, Y Sun, J Li, H Li, Y Zhang, P Chen… - arXiv preprint arXiv …, 2024

… All these versions are trained using masked language modeling and next-word prediction techniques with training data sourced from publicly available repositories like Wikipedia, Common Crawl. The results of LLaMA demonstrate that cuttingedge …


Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance

R Aiyappa, S Senthilmani, J An, H Kwak, YY Ahn - arXiv preprint arXiv:2403.00236, 2024

We investigate the performance of LLM-based zero-shot stance detection on tweets. Using FlanT5-XXL, an instruction-tuned open-source LLM, with the SemEval 2016 Tasks 6A, 6B, and P-Stance datasets, we study the performance and its variations …


BengaliLCP: A Dataset for Lexical Complexity Prediction in the Bengali Texts

N Ayman, MA Hossain, A Aziz, RU Faruqui, AN Chy - Proceedings of the 2024 Joint …, 2024

Encountering intricate or ambiguous terms within a sentence produces distress for the reader during comprehension. Lexical Complexity Prediction (LCP) deals with predicting the complexity score of a word or a phrase considering its context. This …


BERGEN: A Benchmarking Library for Retrieval-Augmented Generation

D Rau, H Déjean, N Chirkova, T Formal, S Wang… - arXiv preprint arXiv …, 2024

Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. In response to the recent popularity of generative LLMs, many RAG approaches have been proposed, which involve an intricate number of …


BERT, RoBERTa or DeBERTa? Comparing Performance Across Transformer Models in Political Science Text

Transformer models such as BERT, RoBERTa, and DeBERTa have revolutionized the field of Natural Language Processing in recent years with substantial improvements in the contextual understanding of text. While political scientists have …


BERT, RoBERTa or DeBERTa? Comparing Performance Across Transformers Models in Political Science Text


BERTicelli at HaSpeeDe 3: Fine-tuning and Cross-validating Large Language Models for Hate Speech Detection Leonardo Grotti¹², Patrick Quick¹ ¹Universiteit …

L Grotti¹ - EVALITA Proceedings of the Eighth Evaluation …, 2024

The present paper describes the results from the experiments carried out for the HaSpeeDe 3 shared task, an Italian-language Hate Speech (HS) detection task, at EVALITA 2023. Two BERT-based language models were selected: UmBERTO (cased) …


BERTs are Generative In-Context Learners

D Samuel - arXiv preprint arXiv:2406.04823, 2024

This paper explores the in-context learning capabilities of masked language models, challenging the common view that this ability does not 'emerge' in them. We present an embarrassingly simple inference technique that enables DeBERTa to operate as …


Better Alignment with Instruction Back-and-Forth Translation

T Nguyen, J Li, S Oh, L Schmidt, J Weston… - arXiv preprint arXiv …, 2024

We propose a new method, instruction back-and-forth translation, to construct high-quality synthetic data grounded in world knowledge for aligning large language models (LLMs). Given documents from a web corpus, we generate and curate synthetic instructions …


Beyond Lexical Boundaries: LLM-Generated Text Detection for Romanian Digital Libraries

M Nitu, M Dascalu - Future Internet, 2024

Machine-generated content reshapes the landscape of digital information; hence, ensuring the authenticity of texts within digital libraries has become a paramount concern. This work introduces a corpus of approximately 60 k Romanian documents …


Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios

M Ochieng, V Gumma, S Sitaram, J Wang, K Ronen… - arXiv preprint arXiv …, 2024

The deployment of Large Language Models (LLMs) in real-world applications presents both opportunities and challenges, particularly in multilingual and code-mixed communication settings. This research evaluates the performance of seven leading …


Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression

Z Xu, A Gupta, T Li, O Bentham, V Srikumar - arXiv preprint arXiv:2407.04965, 2024

Large language models (LLMs) are increasingly deployed in real-world scenarios with the help of recent model compression techniques. Such momentum towards local deployment means the use of compressed LLMs will widely impact a large …


Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Data

A Lee, B Miranda, S Sundar, A Casasola, S Koyeyo

… Since many datasets are publicly available (eg Common Crawl, Wikipedia), data used to train new models may be curated from such … We evaluate on Pile-CC (Pile Common Crawl) and OpenWebText2 since those datasets align with intuitively …


Beyond the Black Box: Optimization Within Latent Spaces

… We compute b using Common Crawl monolingual datasets.For each language and contextual embedding model, we create 1M candidate-reference pairs by grouping two random sentences. Because of the random pairing and the corpus …


BharatBhasaNet-A unified framework to identify Indian code mix Languages

S Dey, S Thakur, A Kandwal, R Kumar, S Dasgupta… - IEEE Access, 2024

In the rapidly globalizing digital communication sphere, the imperative for advanced multilingual text recognition and identification is increasingly evident. Contrasting the previous works, which were predominantly constrained to 2-3 languages, this …


Bi-directional GRU-Based Approach for Multi-Class Text Error Identification System

S Reeha, BV Nikith, GMP Reddy, PB Pati - 2024 IEEE 9th International Conference …, 2024

… The C4 dataset, utilized in this study, is an extensively cleaned and curated version of Common Crawl's web crawl corpus. Common Crawl is a publicly available dataset accessible at commoncrawl.org. and boasting over 700 million …


BIAS ASSESSMENT IN LARGE LANGUAGE MODELS

The introduction of Large Language Models (LLMs) such as Chat-GPT has transformed the field of natural language processing, providing exceptional capabilities in text generation and comprehension. However, these models have …


Bias-Aware Low-Rank Adaptation: Mitigating Catastrophic Inheritance of Large Language Models

Y Chang, Y Chang, Y Wu - arXiv preprint arXiv:2408.04556, 2024

Large language models (LLMs) have exhibited remarkable proficiency across a diverse array of natural language processing (NLP) tasks. However, adapting LLMs to downstream applications typically necessitates computationally intensive and …


Big City Bias: Evaluating the Impact of Metropolitan Size on Computational Job Market Abilities of Language Models

C Campanella, R van der Goot - arXiv preprint arXiv:2403.08046, 2024

Large language models (LLMs) have emerged as a useful technology for job matching, for both candidates and employers. Job matching is often based on a particular geographic location, such as a city or region. However, LLMs have known …


Bilingual Adaptation of Monolingual Foundation Models

G Gosal, Y Xu, G Ramakrishnan, R Joshi, A Sheinin… - … on Foundation Models in the Wild

We present an efficient method for adapting a monolingual Large Language Model (LLM) to another language, addressing challenges of catastrophic forgetting and tokenizer limitations. We focus this study on adapting Llama 2 to Arabic. Our two-stage …


Bilingual Propaganda Detection in Diplomats' Tweets Using Language Models and Linguistic Features

A Modzelewski, P Golik, A Wierzbicki - IberLEF@ SEPLN, 2024

Our study presents an approach to a shared task of propaganda identification and characterization at the DIPROMATS 2024 hosted by the Iberian Languages Evaluation Forum. As the DSHacker team, we participated in the propaganda …


Binary Battle: Leveraging Machine Learning and Transfer Learning Models to Distinguish between Conspiracy Theories and Critical Thinking

S Mahesh, S Divakaran, K Girish, HL Shashirekha - 2024

In the context of automatic content moderation Natural Language Processing (NLP) has a complex task when it comes to distinguishing between conspiracy theories and critical thinking. While conspiracy theories present complex narratives …


Bioeconomy firms and where to find them

… This dataset uses the open-source web repository CommonCrawl to identify German company websites and has proven to be a valuable database for spatial research. From this data, we identify bioeconomy firms using a combination of …


BioMedLM: A 2.7 B Parameter Language Model Trained On Biomedical Text

E Bolton, A Venigalla, M Yasunaga, D Hall, B Xiong… - arXiv preprint arXiv …, 2024

… The Pile contains multiple sub-corpora, including general content from Common Crawl and specialized content like PubMed, Github, and FreeLaw. This model serves as an important baseline for comparison with our domain-specific models; …


BIS Papers

I Aldasoro, S Doerr, L Gambacorta, S Notra, T Oliviero… - 2024

Generative artificial intelligence (gen AI) introduces novel opportunities to strengthen central banks’ cyber security but also presents new risks. We use data from a unique survey among cyber security experts at major central banks to shed …


BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages

J Myung, N Lee, Y Zhou, J Jin, RA Putri, D Antypas… - arXiv preprint arXiv …, 2024

… [21] introduce a methodology to extract large-scale cultural commonsense knowledge from the Common Crawl corpus on geography, religion, and occupations. CREHate [16] is a cross-cultural English hate speech dataset covering annotations …


Blind Baselines Beat Membership Inference Attacks for Foundation Models

D Das, J Zhang, F Tramèr - arXiv preprint arXiv:2406.16201, 2024

… More precisely, these benchmarks introduce two datasets, CommonPool and DCLM-POOL, which contain respectively 12.8 billion image-text pairs and 240 trillion text tokens sampled from Common Crawl. To enable experiments at …


BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blocks

AV Ramesh, V Ganapathiraman, IH Laradji, M Schmidt - arXiv preprint arXiv …, 2024

… The C4 dataset is a large-scale, cleaned version of the Common Crawl web corpus used for pre-training language models, featuring diverse and high-quality text from the internet. Our experiment setup is similar to [34], following the setup from …


Boost Large Language Model Performance through Self-Training with Reward Guided Tree Search

J Hargreaves, E Fairweather, O Bellingham - 2024

Large language models have changed the field of natural language processing by enabling sophisticated language generation tasks, yet there remains a persistent challenge in enhancing their performance through autonomous learning …


Botlitica: A generative AI-based tool to assist journalists in navigating political propaganda campaigns

E Musi, EEG Aguilar, L Federico - Studies in Communication Sciences, 2024

The hype on generative AI has raised concerns about the spread of disinformation but also opened up new opportunities for hybrid journalism. The proliferation of political propaganda campaigns spread across digital media during election periods …


BOTS-LM: Training Large Language Models for Setswana

N Brown, V Marivate - arXiv preprint arXiv:2408.02239, 2024

In this work we present BOTS-LM, a series of bilingual language models proficient in both Setswana and English. Leveraging recent advancements in data availability and efficient fine-tuning, BOTS-LM achieves performance similar to models …


Brain Rush

… LLM developers could tap free data libraries – such as the Common Crawl repository of web data – supplemented by licensing foreign-language …


Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM

S Sukhbaatar, O Golovneva, V Sharma, H Xu, XV Lin… - arXiv preprint arXiv …, 2024

We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts …


Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling

YC Yu, CC Kuo, Z Ye, YC Chang, YS Li - arXiv preprint arXiv:2406.12585, 2024

Ensembling multiple models has always been an effective approach to push the limits of existing performance and is widely used in classification tasks by simply averaging the classification probability vectors from multiple classifiers to achieve …


Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models

T Blevins, T Limisiewicz, S Gururangan, M Li, H Gonen… - arXiv preprint arXiv …, 2024

Despite their popularity in non-English NLP, multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. We propose Cross-lingual Expert Language Models (X-ELM), which …


Brevity is the soul of wit: Pruning long files for code generation

AK Singh, Y Yang, K Tirumala, M Elhoushi, AS Morcos - arXiv preprint arXiv …, 2024

Data curation is commonly considered a "secret-sauce" for LLM training, with higher quality data usually leading to better LLM performance. Given the scale of internet-scraped corpora, data pruning has become a larger and larger focus. Specifically, many have …


Bridging large language model disparities: Skill tagging of multilingual educational content

Y Kwak, ZA Pardos - British Journal of Educational Technology, 2024

The adoption of large language models (LLMs) in education holds much promise. However, like many technological innovations before them, adoption and access can often be inequitable from the outset, creating more divides than they bridge. In …


Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking

EC Acikgoz, M Erdogan, D Yuret - arXiv preprint arXiv:2405.04685, 2024

… This dataset was generated by extracting content from 71 monthly snapshots of the internet via Common Crawl (CC). CulturaX contains version … However, the diversity of these datasets should be sufficient, including common crawl, book …


Browsing and Searching Metadata of TREC

T Breuer, EM Voorhees, I Soboroff - Proceedings of the 47th International ACM SIGIR …, 2024

Information Retrieval (IR) research is deeply rooted in experimentation and evaluation, and the Text REtrieval Conference (TREC) has been playing a central role in making that possible since its inauguration in 1992. TREC's mission centers …