Near to Mid-term Risks and Opportunities of Open Source Generative AI

F Eiras, A Petrov, B Vidgen, CS de Witt, F Pizzati… - arXiv preprint arXiv …, 2024

In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about potential …


Need a Programming Exercise Generated in Your Native Language? ChatGPT's Got Your Back: Automatic Generation of Non-English Programming Exercises Using …

M Jordan, K Ly, AG Soosai Raj - Proceedings of the 55th ACM Technical Symposium …, 2024

Large language models (LLMs) like ChatGPT are changing computing education and may create additional barriers to those already faced by non-native English speakers (NNES) learning computing. We investigate an opportunity for a positive …


Negotiating becoming: a Nietzschean critique of large language models

SWS Fischer, B Boer - Ethics and Information Technology, 2024

Large language models (LLMs) structure the linguistic landscape by reflecting certain beliefs and assumptions. In this paper, we address the risk of people unthinkingly adopting and being determined by the values or worldviews embedded …


Nemotron-4 15B Technical Report

J Parmar, S Prabhumoye, J Jennings, M Patwary… - arXiv preprint arXiv …, 2024

We introduce Nemotron-4 15B, a 15-billion-parameter large multilingual language model trained on 8 trillion text tokens. Nemotron-4 15B demonstrates strong performance when assessed on English, multilingual, and coding tasks: it …


Neural Approaches to Decode Semantic Similarities in Spanish Song Lyrics for Enhanced Recommendation Systems

… In this model, it was found that traditional methods of pre-training, which rely on large amounts of data from sources such as Common Crawl… The mC4 dataset is a multilingual variant of C4, containing natural text in 101 languages from the public …


NEURAL MACHINE TRANSLATION BETWEEN MYANMAR AND ENGLISH LANGUAGES

This study aims to present a thorough overview of cutting-edge machine translation solutions and evaluate their effectiveness in translating between Myanmar and English language pairs. Translating Myanmar poses challenges due to its unique …


NEURAL MODELS FOR ONTOLOGY ANNOTATIONS-NEMO

Words cannot express my gratitude to my advisors Dr. Somya D. Mohanty and Dr. Prashanti Manda for their invaluable supervision and guidance. I am extremely grateful to them for their continuous support, constructive suggestions, and …


Neural Network-based Language Modelling for Code-Switching in South African Languages

Language models, specifically those based on neural network architectures, have become an attractive alternative to classical statistical language models for natural language processing and speech recognition. Naturally, therefore, the field of neural …


Neural Networks with Model Compression

B Zhang, T Wang, S Xu, D Doermann - 2024

… Firstly, the availability of large-scale datasets, such as ImageNet for computer vision or the Common Crawl dataset for natural language processing, has enabled the training of deep neural networks with millions or even billions of parameters …


Next word prediction for newspaper article formulation

Next word prediction describes the process of predicting the next word after every written word in the authoring of texts. It refers to machine learning techniques, which are part of the language modelling domain, a sub-domain of natural language …


NLP for The Greek Language: A Longer Survey

K Papantoniou, Y Tzitzikas - arXiv preprint arXiv:2408.10962, 2024

English language is in the spotlight of the Natural Language Processing (NLP) community with other languages, like Greek, lagging behind in terms of offered methods, tools and resources. Due to the increasing interest in NLP, in this paper we …


No Filter: Cultural and Socioeconomic Diversityin Contrastive Vision-Language Models

A Pouget, L Beyer, E Bugliarello, X Wang, AP Steiner… - arXiv preprint arXiv …, 2024

We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training …


Non-governmental Governance of Trust on the Internet: WebPKI as Public Good

… The population of sites we sampled from was indexed by the non-profit foundation Common Crawl. We then employed a script to pull certificate information from our list of URLs. Slightly more than half the sample did not return certificate …


Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models

This paper identifies a cultural dominance issue within large language models (LLMs) due to the predominant use of English data in model training (eg, ChatGPT). LLMs often provide inappropriate English-culture-related answers that are not relevant to …


Notes towards infrastructure governance for large language models

This paper draws on information infrastructures (IIs) in science and technology studies (STS), as well as on feminist STS scholarship and contemporary critical accounts of digital technologies, to build an initial mapping of the infrastructural …


nowhash at SemEval-2024 Task 4: Exploiting Fusion of Transformers for Detecting Persuasion Techniques in Multilingual Memes

AN Chowdhury, M Ptaszynski - Proceedings of the 18th International Workshop on …, 2024

Nowadays, memes are considered one of the most prominent forms of medium to disseminate information on social media. Memes are typically constructed in multilingual settings using visuals with texts. Sometimes people use memes to …


NRK at SemEval-2024 Task 1: Semantic Textual Relatedness through Domain Adaptation and Ensemble Learning on BERT-based models

NT Kiet, D Van Thin - Proceedings of the 18th International Workshop on …, 2024

This paper describes the system of the team NRK for Task A in the SemEval-2024 Task 1: Semantic Textual Relatedness (STR). We focus on exploring the performance of ensemble architectures based on the voting technique and different …


NSINA: A News Corpus for Sinhala

H Hettiarachchi, D Premasiri, L Uyangodage… - arXiv preprint arXiv …, 2024

The introduction of large language models (LLMs) has advanced natural language processing (NLP), but their effectiveness is largely dependent on pre-training resources. This is especially evident in low-resource languages, such as Sinhala …


Numerical Claim Detection in Finance: A New Financial Dataset, Weak-Supervision Model, and Market Analysis

A Shah, A Hiray, P Shah, A Banerjee, A Singh… - arXiv preprint arXiv …, 2024

In this paper, we investigate the influence of claims in analyst reports and earnings calls on financial market returns, considering them as significant quarterly events for publicly traded companies. To facilitate a comprehensive analysis, we construct a …


NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural

W Wongso, DS Setiawan, S Limcorn, A Joyoadikusumo - arXiv preprint arXiv …, 2024

… We also note that NusaBERT was pre-trained on Wikipedia and common crawl corpora, which explains its effectiveness on and closeness to NusaX and NusaTranslation source domains, but not so for NusaParagraph. Thus, due to their high linguistic and …


Nyonic Technical Report

J Tian, R Wang, C Li, Y Zhou, J Liu, J Wang - arXiv preprint arXiv:2404.15702, 2024

… Our curated dataset is specifically designed to meet these criteria, incorporating sources such as the Common Crawl, books, Wikipedia, code, and … Our data sources encompass Common Crawl, Wiki, and various coding repositories. The final …


Ocean Decade Vision 2030 White Papers–Challenge 8: Create a digital representation of the ocean.

JB Calewaert, PC Sierra-Correa, O McMeel… - 2024

… The Common crawl and WebImageText are examples, with communities such as Hugging Face and KAGGLE creating new communities of practice16F … 17 https://commoncrawl.org, https://github.com/google-research-datasets/wit, https://huggingface.co, https://www.kaggle.com …


Offensive language detection in low resource languages: A use case of Persian language

M Mozafari, K Mnassri, R Farahbakhsh, N Crespi - PLOS ONE, 2024

… XLM-RoBERTa [27]: is a transformer-based multilingual masked language model pre-trained on 100 languages, including Persian, using more than two terabytes of filtered CommonCrawl data. To fine-tune XLM-RoBERTa model on our target …


OLMo: Accelerating the Science of Language Models

D Groeneveld, I Beltagy, P Walsh, A Bhagia, R Kinney… - arXiv preprint arXiv …, 2024

… well on evaluations predominated by Common Crawl, such as C4, though different ways of postprocessing Common Crawl are best fit by … The RedPajama evaluation shows a similar pattern, perhaps as only 2 of its 7 domains are from …


OLMoE: Open Mixture-of-Experts Language Models

N Muennighoff, L Soldaini, D Groeneveld, K Lo… - arXiv preprint arXiv …, 2024

We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to …


Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models

S Ruan, Y Dong, H Liu, Y Huang, H Su, X Wei - arXiv preprint arXiv:2404.12139, 2024

Vision-Language Pre-training (VLP) models like CLIP have achieved remarkable success in computer vision and particularly demonstrated superior robustness to distribution shifts of 2D images. However, their robustness under 3D viewpoint …


On humans'(explicit) intuitions about the meaning of novel words

D Gatti, F Rodio, L Rinaldi, M Marelli - Cognition, 2024

Pseudowords offer a unique opportunity to investigate how humans deal with new (verbal) information. Within this framework, previous studies have shown that, at the implicit level, humans exploit systematic associations in the form-meaning interface to …


On Initializing Transformers with Pre-trained Embeddings

HY Kim, N Balasubramanian, B Kang - arXiv preprint arXiv:2407.12514, 2024

It has become common practice now to use random initialization schemes, rather than the pre-trained embeddings, when training transformer based models from scratch. Indeed, we find that pre-trained word embeddings from GloVe, and some …


On Labs and Fabs: Mapping How Alliances, Acquisitions, and Antitrust are Shaping the Frontier AI Industry

T Aguirre - arXiv preprint arXiv:2406.01722, 2024

As frontier AI models progress, policy proposals for safe AI development are gaining increasing attention from researchers and policymakers. This paper evaluates the present landscape of integration within the AI supply chain, emphasizing vertical …


On Protecting the Data Privacy of Large Language Models (LLMs): A Survey

B Yan, K Li, M Xu, Y Dong, Y Zhang, Z Ren, X Cheng - arXiv preprint arXiv …, 2024

… For example, GPT-3, developed by OpenAI, was pre-trained using CommonCrawl, constituting 45TB of compressed plaintext before filtering [37]. Regarding multimodal LLMs, CLIP’s training dataset encompasses 400 million pairs of images and text …


On the data scarcity problem of neural-based named entity recognition

… The initial release trained on Wikipedia includes word representations for nine languages, namely Arabic, Czech, German, English, Spanish, French, Italian, Romanian and Russian, and has quickly expanded to 157 languages by training on larger …


On the evaluation and application of neural language models for grammatical error detection

Neural language models (NLM) have become a core component in many downstream applications within the field of natural language processing, including the task of data-driven automatic grammatical error detection (GED). This thesis …


On the importance of Data Scale in Pretraining Arabic Language Models

A Ghaddar, P Langlais, M Rezagholizadeh, B Chen - arXiv preprint arXiv:2401.07760, 2024

Pretraining monolingual language models have been proven to be vital for performance in Arabic Natural Language Processing (NLP) tasks. In this paper, we conduct a comprehensive study on the role of data in Arabic Pretrained Language …


On the Mathematical Relationship Between Contextual Probability and N400 Amplitude

JA Michaelov, BK Bergen - Open Mind, 2024

Accounts of human language comprehension propose different mathematical relationships between the contextual probability of a word and how difficult it is to process, including linear, logarithmic, and super-logarithmic ones. However, the …


On the Question of Authorship in Large Language Models

C Soos, L Haroutunian - KO KNOWLEDGE ORGANIZATION, 2024

… CommonCrawl exemplifies the problem of documentation debt. The dataset is an effort by The Common Crawl Foundation to “[democratize] access to web information by producing and maintaining an open repository of web crawl data” (Common …


On the Robustness of Neural Networks Quantization against Data Poisoning Attacks

Y Lu, Y Wang, G Zhang, Y Yu - ICML 2024 Next Generation of AI Safety Workshop

… For example, practitioners may continuously collect user data online (eg, using Common Crawl1), which may be maliciously contaminated by an attacker. Such a threat is called data poisoning attacks (Biggio et al. 2012), … https://commoncrawl.org/ …


On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe

N Xu, Q Zhang, M Zhang, P Qian, X Huang - arXiv preprint arXiv:2402.14404, 2024

Probing and enhancing large language models' reasoning capacity remains a crucial open question. Here we re-purpose the reverse dictionary task as a case study to probe LLMs' capacity for conceptual inference. We use in-context learning …


One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support

V Rasiah, R Stern, V Matoshi, M Stürmer, I Chalkidis…

Recent strides in Large Language Models (LLMs) have saturated many NLP benchmarks (even professional domain-specific ones), emphasizing the need for more challenging ones to properly assess LLM capabilities. Domain-specific and …


Online health search via multi-dimensional information quality assessment based on deep language models

B Zhang, N Naderi, R Mishra, D Teodoro - 2024

Background Widespread misinformation in Web resources can lead to serious implications for individuals seeking health advice. Despite that, information retrieval models are often focused only on the query-document relevance dimension to rank …


Online Health Search Via Multidimensional Information Quality Assessment Based on Deep Language Models: Algorithm Development and Validation

B Zhang, N Naderi, R Mishra, D Teodoro - JMIR AI, 2024

… In this study, we simulated online health information search scenarios with a topic set of 32 different health-related inquiries and a corpus containing 1 billion web documents from the April 2019 snapshot of Common Crawl. Using state-of-the-art …


OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameters

Z Chen, C Li, X Xie, P Dube - arXiv preprint arXiv:2409.00286, 2024

This paper explores the potential of a small, domain-specific language model trained exclusively on sports-related data. We investigate whether extensive training data with specially designed small model structures can overcome model size …


Ontology Completion with Natural Language Inference and Concept Embeddings: An Analysis

N Li, T Bailleux, Z Bouraoui, S Schockaert - arXiv preprint arXiv:2403.17216, 2024

We consider the problem of finding plausible knowledge that is missing from a given ontology, as a generalisation of the well-studied taxonomy expansion task. One line of work treats this task as a Natural Language Inference (NLI) problem, thus relying …


Open Assistant Toolkit--version 2

S Fischer, F Rossetto, C Gemmell, A Ramsay, I Mackie… - arXiv preprint arXiv …, 2024

… To source knowledge and tasks, we release a new offline pipeline to parse and augment task data from CommonCrawl. The offline pipeline supports using LLMs and additional open-source multimodal data sources to enhance tasks and make …


Open foundation models for Azerbaijani language

J Isbarov, K Huseynova, E Mammadov, M Hajili - arXiv preprint arXiv:2407.02337, 2024

The emergence of multilingual large language models has enabled the development of language understanding and generation systems in Azerbaijani. However, most of the production-grade systems rely on cloud solutions, such as GPT-4 …


Open Science at the Generative AI Turn: An Exploratory Analysis of Challenges and Opportunities

Technology influences Open Science (OS) practices, because conducting science in transparent, accessible, and participatory ways requires tools/platforms for collaborative research and sharing results. Due to this direct relationship …


Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

Q Xie, D Li, M Xiao, Z Jiang, R Xiang, X Zhang, Z Chen… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have advanced financial applications, yet they often lack sufficient financial knowledge and struggle with tasks involving multi-modal inputs like tables and time series data. To address these limitations, we introduce \textit{Open-FinLLMs} …


OpenAI ChatGPT and Biased Information in Higher Education

Motivated by the appearance of large language models and their sudden societal impacts—both beneficial and harmful, realized and potential—we evaluated several of them with respect to bias in its myriad forms. Bias in machine-learning models …


OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

F Xue, Z Zheng, Y Fu, J Ni, Z Zheng, W Zhou, Y You - arXiv preprint arXiv:2402.01739, 2024

To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-sourced and reproducible decoder-only MoE LLMs, ranging from …


OptiComm-GPT: a GPT-based versatile research assistant for optical fiber communication systems

X Jiang, M Zhang, Y Song, Y Zhang, Y Wang, C Ju… - Optics Express, 2024

… Generally, LLMs are pre-trained on publicly available datasets such as BooksCorpus, Common Crawl, and Wikipedia, thereby ensuring comprehensive coverage of knowledge. Meanwhile, with the support of powerful analysis, reasoning …


Optimization and Evaluation in Machine Learning Challenges

To develop new machine learning methods, it is necessary to evaluate them reliably. This doctoral thesis discusses some aspects of preparing machine learning challenges and techniques for developing their solutions. The work consists of …


Optimization Hyper-parameter Laws for Large Language Models

X Xie, K Ding, S Yan, KC Toh, T Wei - arXiv preprint arXiv:2409.04777, 2024

Large Language Models have driven significant AI advancements, yet their training is resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws provide valuable guidance on model size and data requirements, they …


Optimization of news dissemination push mode by intelligent edge computing technology for deep learning

JL DeGe, S Sang - Scientific Reports, 2024

… Realnew (https://paperswithcode.com/dataset/realnews) is a large corpus of news articles from CommonCrawl. The data is crawled from CommonCrawl, which is limited to 5000 news fields of Google News Index. The author uses the …


Optimization of Unsupervised Neural Machine Translation Based on Syntactic Knowledge Improvement

… Common Crawl is a multilingual aligned dataset based on web crawling. Para Crawl is a multilingual parallel corpus for large-scale web crawling. As shown in Table III, the detection accuracies of LSTM in Newstest2020, Newstest2021 …


Optimizing Data I/O for LLM Datasets on Remote Storage

Training large language models (LLMs) demands increasingly larger datasets for optimal performance [13]. In practice, these datasets may include hundreds of terabytes (TB) or even petabytes of data. For example, Common Crawl [3] needs …


Optimizing Knowledge Extraction in Large Language Models Using Dynamic Tokenization Dictionaries

Tokenization methods have long been a critical component in the performance of language models, yet traditional static approaches often fall short in capturing the dynamic nature of language. The novel concept of implementing a dynamic …


Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean

CS Choi, Y Jeong, S Park, IH Won, HS Lim, SM Kim… - arXiv preprint arXiv …, 2024

… Llama2 is a multilingual language model trained using largescale publicly available data (including CommonCrawl, Github, Wikipedia, and ArXiv) for more than 28 languages. Thus, it possesses cross-language understanding capabilities …


Optimizing Large Language Models with Multi-Degree Low-Rank Approximations

… The use of datasets such as Common Crawl and Wikipedia facilitated a broad coverage of language variations, enhancing the generalizability of the results. Preprocessing steps were critical to ensure data quality and consistency, involving …


Optimizing Phishing Site Detection: Analyzing the Impact of Feature Quantity in Random Forest Model

T Nabila, HH Nuha, SA Kristyan - 2024 International Conference on Data Science …, 2024

… The first dataset used is a dataset consisting of 5000 phishing sites and 5000 normal sites with 48 features using the Selenium Webdriver framework for extraction technique, This first dataset was collected and selected from Alexa and Common …


Optimizing Quality Estimation for Low-Resource Language Translations: Exploring the Role of Language Relatedness

A Sindhujan, D Kanojia, C Orasan - Proceedings of the Conference New Trends in …, 2024

Evaluation of machine translation (MT) is vital to determine the effectiveness of MT systems. This paper investigates quality estimation (QE) for machine translation (MT) for low-resource Indic languages. We analyse the influence of language relatedness …


Optimizing social determinant of health concept extraction

Generative Artificial Intelligence has recently garnered enormous attention for a varied number of reasons, one of them being its capacity to automate complex tasks. As the potential for integrating Generative Artificial Intelligence across technology …


ORCID

How are Asian and Black men and women stereotyped? Research from the gendered race and stereotype content perspectives has produced mixed empirical findings. Using BERT models pre-trained on English language books, news articles …


Organic Data-Driven Approach for Turkish Grammatical Error Correction and LLMs

A Ersoy, OT Yıldız - arXiv preprint arXiv:2405.15320, 2024

… 2020) which contains random paragraphs sampled from the CommonCrawl dataset3. Some studies put some effort into filling the gap and … 2020), a multilingual pre-trained encoder-decoder text-to-text transformer trained on a Common Crawl-based …


Outlier Detection in Serbian CommonCrawl Data

V Kalušev, D Ćulibrk - 2024 23rd International Symposium INFOTEH …, 2024

The surge in large language models (LLMs) has greatly advanced natural language processing. However, their development is often hampered by the limited availability and quality of training datasets, particularly for underrepresented languages. Our …


Outsmarting Artificial Intelligence in the Classroom—Incorporating Large Language Model-Based Chatbots into Teaching

J Wutzler - Issues in Accounting Education, 2024

Since the release of ChatGPT in November 2022, large language model-based chatbots have attracted much attention. Although businesses value their potential for efficiency gains, academics are concerned about their effects on learning and …


Overcoming the Labeled Training Data Bottleneck

R Starzl - The Cyber Defense Review, 2024

This article explores the potential of large language models (LLMs) to transform dataset creation and analysis in cybersecurity. The proposed method leverages LLMs to overcome the labeled data bottleneck by generating high-quality, task-specific …


Overcoming the Labeled Training Data Bottleneck: A Route to Specialized AI


Overdetermination of businesswomen in the British press: a corpus-based approach in critical discourse analysis

TNT Nguyen - ICTE Conference Proceedings, 2024

Within the strand of critical discourse analysis (CDA), research on media portrayal of individuals is prevalent, with the scope having been expanded beyond such subjects as refugees or asylum seekers, but also the breadth of research having …


Overview of Existing LLM Families

… data that are considered as redundant—namely the Common Crawl and Colossal Cleaned Common Crawl (C4). Researchers who created the … to Common Crawl when it came to training LLMs [30] and should be used instead of the whole …


Overview of the TREC 2023 NeuCLIR Track (Notebook Paper)

D Lawrie, S MacAvaney, J Mayfield, P McNamee…

… For more information about extracting the text for the documents from CommonCrawl News and how the collection can be obtained, see … of the track including HC4 — a CLIR collection built over three years of Common Crawl data in …


Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization

C Mavromatis, P Karypis, G Karypis - arXiv preprint arXiv:2404.11531, 2024

… 2019) is sa publicly available distribution of a Common Crawl snapshot. Following cBTM, we use 168B tokens of the no blocklist version (en.noblocklist) that is out of distribution to the OPT model’s pretraining corpus. S2ORC (Lo et al.…


PAL: Proxy-Guided Black-Box Attack on Large Language Models

C Sitawarin, N Mu, D Wagner, A Araujo - arXiv preprint arXiv:2402.09674, 2024

Large Language Models (LLMs) have surged in popularity in recent months, but they have demonstrated concerning capabilities to generate harmful content when manipulated. While techniques like safety fine-tuning aim to minimize harmful use …


Palöri at CheckThat! 2024 Shared Task 6: GloTa-Combining GloVe Embeddings with RoBERTa for Adversarial Attack

This paper describes the submission of attack methods and results for shared task 6 at CheckThat! Lab at CLEF 2024. We present two novel attack methods to test the robustness of credibility assessment (CA) classifiers across five tasks: fact-checking …


Parameter-Efficient Tuning of Large Convolutional Models

W Chen, Z Miao, Q Qiu - arXiv preprint arXiv:2403.00269, 2024

To address the high computational and parameter complexity associated with fine-tuning large pre-trained models, researchers have developed parameter-efficient methods, where only partial parameters are updated for downstream tasks. However, these …


Paraphrase Generation and Supervised Learning for Improved Automatic Short Answer Grading

L Ouahrani, D Bennouar - International Journal of Artificial Intelligence in …, 2024

We consider the reference-based approach for Automatic Short Answer Grading (ASAG) that involves scoring a textual constructed student answer comparing to a teacher-provided reference answer. The reference answer does not cover the variety of student …


Parsability revisited and reassessed

S Monakhov - Journal of Linguistics, 2023

This paper provides evidence that the inveterate way of assessing linguistic items’ degrees of analysability by calculating their derivation to base frequency ratios may obfuscate the difference between two meaning processing models: one based on …


Part-of-Speech Tagger for Bodo Language using Deep Learning approach

D Pathak, S Narzary, S Nandi, B Som - arXiv preprint arXiv:2401.03175, 2024

Language Processing systems such as Part-of-speech tagging, Named entity recognition, Machine translation, Speech recognition, and Language modeling (LM) are well-studied in high-resource languages. Nevertheless, research on these …


Pathological Liars: Algorithmic Knowing in the Rhetorical Ecosystem of Wallstreetbets

MH Yang, ZP Majdik - Rhetoric Society Quarterly, 2024

This essay demonstrates the value of using artificial intelligence (AI) technologies to address specific kinds of research questions in rhetoric. The essay builds on a study of a novel rhetorical object first observed by Yang on the Reddit subreddit r/wallstreetbets …


Pathological-Llama: an Explainable Medical Visual Question An

This thesis introduces Pathological-Llama, an explainable medical visual question answering system that integrates computer vision and natural language processing to accurately interpret medical images through a generative task approach. By …


Pedro L. Rodriguez Arthur Spirling Brandon M. Stewart Elisa M. Wirsching

… from Common Crawl and Wikipedia. A strength of the fastText model is that it uses subword … On inspection, we saw that Common Crawl includes many typos and rare terms (plus many … Beyond this potential for noise, Common Crawl is not …


Performance Analysis of Embedding Methods for Deep Learning-Based Turkish Sentiment Analysis Models

A Ba Alawi, F Bozkurt - Arabian Journal for Science and Engineering, 2024

… Then, we chose the model which is specifically for Turkish that was trained using data from Common Crawl and Wikipedia. It employs position-weighted Continuous Bag of Words (CBOW) architecture. The model has a size of 300 and includes 5-character …


Performance Comparison of Large Language Models, GPT and Gemini on Turkish News Classification Task

… This was achieved by training a Transformer-based masked language model on 100 languages using more than two terabytes of filtered CommonCrawl data. The balance between high and low-resource languages and the impact of language …


Performance Evaluation of Word Embedding Algorithms

S Kulshretha, L Lodha - Performance Evaluation, 2023

This study intends to explore the field of word embedding and thoroughly examine and contrast various word embedding algorithms. Words retain their semantic relationships and meaning when they are transformed into vectors using word …


Performance of ChatGPT on American Board of Surgery In-Training Examination Preparation Questions

CG Tran, J Chang, SK Sherman, JP De Andrade - Journal of Surgical Research, 2024

… The training dataset of ChatGPT 3.5, where the artificial intelligence (AI) “knowledge” is derived, is comprised of Common Crawl (a monthly updated web archive of data), books, journals, and Wikipedia and totals over 500 billion encoded “tokens.” Many …


Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models

Z Ankner, C Blakeney, K Sreenivasan, M Marion… - arXiv preprint arXiv …, 2024

… Secondly, while previous work only investigates pruning on datasets composed of just one domain (CommonCrawl1), we consider two … dataset, with 81.31% of its data being derived from the CommonCrawl. We find that successful pruning …


Personal Science Guide to AI

Imagine you have access to a zillion documents, preferably curated in some way reassures you about their quality and consistency. Wikipedia, for example, or maybe Reddit and other posts that have been sufficiently up-voted. Maybe you also have a …


Personality trait detection via transfer learning

B Alshouha, J Serrano-Guerrero, F Chiclana… - Comput Mater Continua …, 2024

Personality recognition plays a pivotal role when developing user-centric solutions such as recommender systems or decision support systems across various domains, including education, e-commerce, or human resources. Traditional machine …


Perspectives of Global and Hong Kong's Media on China's Belt and Road Initiative

LC Khoo, A Datta - arXiv preprint arXiv:2312.17013, 2023

This study delves into the media analysis of China's ambitious Belt and Road Initiative (BRI), which, in a polarized world, and furthermore, owing to the very polarizing nature of the initiative itself, has received both strong criticisms and …


PharmGPT: Domain-Specific Large Language Models for Bio-Pharmaceutical and Chemistry

L Chen, W Wang, Z Bai, P Xu, Y Fang, J Fang, W Wu… - arXiv preprint arXiv …, 2024

… reinforcing inherent biases present in the source materials (eg, CommonCrawl dumps) and those introduced through the filtering processes. Such … New tools were developed for extracting text from HTML in Common Crawl WARC files …


PHISHING DETECTION SYSTEM THROUGH MACHINE LEARNING BASED ON URL

PU Chandra, TV Teja, VN Reddy, S Agarwalla

… Legitimate site data is often gathered from Alexa's top-ranking website database or through common-crawl. Additionally, publicly available datasets like those from the UCI machine learning repository, containing over 11,000 records with 31 …


Phishing Detection Using Hybrid Algorithm Based on Clustering and Machine Learning

L Al-Shalabi, Y Hasan Jazyah - International Journal of Computing and Digital …, 2024

Phishing is a prevalent and evolving cyber threat that continues to exploit human vulnerability to deceive individuals and organizations into revealing sensitive information. Phishing attacks encompass a range of tactics, from deceptive emails …


Phishing URL Detection Using Machine Learning and Deep Learning

R Ferdaws, NE Majd - 2024 IEEE World AI IoT Congress (AIIoT), 2024

In the modern digital world, phishing attacks have become a persistent threat, jeopardizing the security and privacy of individuals and organizations, demanding prompt action to avoid excessive costs imposed to individuals and businesses. The …


Phishing Webpage Detection: Unveiling the Threat Landscape and Investigating Detection Techniques

A Kulkarni, V Balachandran, T Das - IEEE Communications Surveys & Tutorials, 2024

… Common Crawl [53] is a nonprofit organization that maintains and provides access to a large-scale web crawling dataset. This dataset, known as the Common Crawl corpus, contains a comprehensive collection of web pages and their …


Physics-informed Machine Learning with Uncertainty Quantification

Physics Informed Machine Learning (PIML) has emerged as the forefront of research in scientific machine learning with the key motivation of systematically coupling machine learning (ML) methods with prior domain knowledge often available in the …


PIE: A Personalized Information Embedded model for text-based depression detection

Y Wu, Z Liu, J Yuan, B Chen, H Cai, L Liu, Y Zhao… - Information Processing & …, 2024

… It undergoes pre-training on 2.5TB of filtered CommonCrawl data, encompassing 100 languages. By acquiring a comprehensive understanding of these 100 languages, the model can extract valuable features for downstream tasks. …


PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods

S Dadas, M Perełkiewicz, R Poświata - arXiv preprint arXiv:2402.13350, 2024

We present Polish Information Retrieval Benchmark (PIRB), a comprehensive evaluation framework encompassing 41 text information retrieval tasks for Polish. The benchmark incorporates existing datasets as well as 10 new, previously …


Pitfalls of Conversational LLMs on News Debiasing

IB Schlicht, D Altiok, M Taouk, L Flek - arXiv preprint arXiv:2404.06488, 2024

This paper addresses debiasing in news editing and evaluates the effectiveness of conversational Large Language Models in this task. We designed an evaluation checklist tailored to news editors' perspectives, obtained generated texts from three …


PLAID SHIRTTT for Large-Scale Streaming Dense Retrieval

D Lawrie, E Kayi, E Yang, J Mayfield, DW Oard - arXiv preprint arXiv:2405.00975, 2024

… With ten million documents from the news subset of CommonCrawl, NeuCLIR is presently the largest available test collection for MLIR; It is tiny though relative to ClueWeb, and thus has relatively small shards. The NeuCLIR MLIR task is to …


Please be polite to your peers: a multi-task model for assessing the tone and objectivity of critiques of peer review comments

PK Bharti, M Agarwal, A Ekbal - Scientometrics, 2024

The peer-review process plays a pivotal role in maintaining the quality and credibility of scientific publications. However, in recent times, there has been an increase in unhelpful and overly critical reviews, which can be detrimental to the …


Politehnica University of Bucharest, Splaiul Independentei Nr. 313, Cector 6, Bucharest, Romania radu. stefan@ live. de, radu. stefan3@ student. upt. ro

R Stefan, G Carutasu, M Mocan - The 17th International Conference Interdisciplinarity in …

This paper aims to investigate the ethical considerations surrounding the implementation and usage of large language models (LLMs). LLMs have shown tremendous advance in natural language processing and have the potential to …


Polysemy through the lens of psycholinguistic variables: a dataset and an evaluation of static and contextualized language models

A Bruera, F Zamani, M Poesio - Proceedings of the 13th Joint Conference on Lexical …, 2024

… This version was trained on a combination of Common Crawl and Wikipedia and has 300dimensional vectors. We extract word vectors for all nouns and verbs and create a phrase-specific representation for each noun as described for countpmi. …


Position: Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?

S Longpre, R Mahari, N Obeng-Marnu, W Brannon…

… Common Crawl. For pretraining text data, models routinely rely on the Common Crawl.For instance, sixty percent of GPT-3 training data is … Common Crawl is accurate with wide coverage, but not detailed. Hugging Face can be inaccurate with …


Position: Mission Critical–Satellite Data is a Distinct Modality in Machine Learning

E Rolf, K Klemmer, C Robinson, H Kerner - Forty-first International Conference on Machine …

Satellite data has the potential to inspire a seismic shift for machine learning---one in which we rethink existing practices designed for traditional data modalities. As machine learning for satellite data (SatML) gains traction for its real-world impact …


Position: Near to Mid-term Risks and Opportunities of Open-Source Generative AI

F Eiras, A Petrov, B Vidgen, CS de Witt, F Pizzati… - Forty-first International Conference …

In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about potential …


Position: Will we run out of data? Limits of LLM scaling based on human-generated data

P Villalobos, A Ho, J Sevilla, T Besiroglu, L Heim… - Forty-first International Conference …

… Assuming that Common Crawl is a representative sample of the indexed web,11 we can use it to estimate the average amount of plain text bytes per web page. This number has increased over time, from around 6100 bytes in 2013 to about 8200 …


Post-processing Techniques for Word Embedding

Word embedding has been a significant breakthrough in natural language processing (NLP). Although word representation has improved remarkably and resulted in better performance in downstream NLP applications, interpretability of …


Posthumanist AI

… In their now canonical paper on “Stochastic Parrots”, Bender, Gebru, McMillan-Major, and Mitchell note that "it is easy to imagine that very large datasets, such as Common Crawl “…petabytes of data collected over 8 years of web crawling” a …


Practical Unlearning for Large Language Models

C Gao, L Wang, C Weng, X Wang, Q Zhu - arXiv preprint arXiv:2407.10223, 2024

While LLMs have demonstrated impressive performance across various domains and tasks, their security issues have become increasingly severe. Machine unlearning (MU) has emerged as a promising solution to address these issues by …


Pre-Trained Acoustic-and-Textual Modeling for End-To-End Speech-To-Text Translation

W Zhang, H Zhang, C Liu, Z Ye, X Zhou, C Lin, L Dai - ICASSP 2024-2024 IEEE …, 2024

… The monolingual data are mainly from News Commentary, News crawl and Common Crawl. We apply language identification [23] to retain sentences predicted as desired language, remove sentences longer than 250 tokens and with a source/target …


Pre-trained language models in medicine: A survey

X Luo, Z Deng, B Yang, MY Luo - Artificial Intelligence in Medicine, 2024

With the rapid progress in Natural Language Processing (NLP), Pre-trained Language Models (PLM) such as BERT, BioBERT, and ChatGPT have shown great potential in various medical NLP tasks. This paper surveys the cutting-edge …


Pre-trained Large Language Models for Financial Sentiment Analysis

W Luo, D Gong - arXiv preprint arXiv:2401.05215, 2024

Financial sentiment analysis refers to classifying financial text contents into sentiment categories (eg positive, negative, and neutral). In this paper, we focus on the classification of financial news title, which is a challenging task due to a lack of …


Pre-training Small Base LMs with Fewer Tokens

S Sanyal, S Sanghavi, AG Dimakis - arXiv preprint arXiv:2404.08634, 2024

We study the effectiveness of a simple approach to develop a small base language model (LM) starting from an existing large base LM: first inherit a few transformer blocks from the larger LM, and then train this smaller model on a very small subset (0.1\%) …


Predicting fraud in MD&A sections using deep learning

S Velloor Sivasubramanian, D Skillicorn - Journal of Business Analytics, 2024

Conventional data analytic techniques have been successfully applied to detecting fraud in the Management’s Discussion and Analysis sections of company filings mandated by the SEC. Here, we investigate whether fraud detection can be …


Predicting semantic category of answers for question answering systems using transformers: a transfer learning approach

S CM, J Prakash, VS Alaparthi - Multimedia Tools and Applications, 2024

… The model uses a data set of web extracted text that has been created with the help of the Common Crawl and preprocessed to remove short, duplicate, and non-English content, resulting in a corpus called "Colossal Clean Crawled Corpus" (C4). An …


PREDICTING TARGETS WITH SEC 8-K FILINGS

There are large financial benefits in predicting acquisitions, but so far no adequate method has been found to that end. In this research textual data from 8-K Filings is used to improve the predictive performance of three classification models. 8-Ks are …


Prediction of Author's Profile basing on Fine-Tuning BERT model

B Bsir, N Khoufi, M Zrigui - Informatica, 2024

… Indeed, XLMRoBERTa is the multilingual variant of RoBERTa trained with a multilingual MLM on one hundred languages, with more than two terabytes of filtered Common Crawl data.XLM-RoBERTa showed its superiority over BERT by its …


Preemptive Answer" Attacks" on Chain-of-Thought Reasoning

R Xu, Z Qi, W Xu - arXiv preprint arXiv:2405.20902, 2024

Large language models (LLMs) showcase impressive reasoning capabilities when coupled with Chain-of-Thought (CoT) prompting. However, the robustness of this approach warrants further investigation. In this paper, we introduce a novel scenario …


Preprocessing Techniques for Clustering Arabic Text: Challenges and Future Directions.

T Almutairi, S Saifuddin, R Alotaibi, S Sarhan, S Nassif - International Journal of …, 2024

Arabic is a complex language for text analysis because of its orthographic features, rich synonyms, and semantic style. Thus, Arabic text must be prepared more carefully in the preprocessing stage for the analyzer to improve the quality of the …


Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions

M Miranda, ES Ruzzetti, A Santilli, FM Zanzotto… - arXiv preprint arXiv …, 2024

… In particular, in their experiments the language model generation is conditioned to some prompts that the attacker predefined: to maximize the probability of success of the attack, they choose prompts that are likely to be in the training data, like …


Pretraining and Updating Language-and Domain-specific Large Language Model: A Case Study in Japanese Business Domain

K Takahashi, T Omi, K Arima, T Ishigaki - arXiv preprint arXiv:2404.08262, 2024

… , mC4, and Common Crawl. These datasets contribute to general knowledge, which is essential for diverse natural language processing tasks. … the proportion of clean domain-specific data, we aim to mitigate the impact of noisy data sources such …


Pretraining Data and Tokenizer for Indic LLM

R Kumar, S Kakde, D Rajput, D Ibrahim, R Nahata… - arXiv preprint arXiv …, 2024

… Additionally, we perform deduplication on Common Crawl data to address the redundancy present in 70% of the crawled web pages. This study focuses on developing high-quality data, optimizing tokenization for our multilingual dataset for …


PRIVACY HARMS IN THE AI AGE

These problems run deeper than false accusations. Many AI models have trained on datasets that contain personal information, and every AI model is capable of regurgitating any data it has previously processed. In 2023, researchers …


Privacy in Federated Learning

The rise of Artificial Intelligence technology has raised concerns about the potential compromise of privacy due to the handling of personal data. Private AI prevents cybercrimes and falsehoods and protects human freedom and trust. While …


Probability-Aware Word-Confusion-Network-To-Text Alignment Approach for Intent Classification

E Villatoro-Tello, S Madikeri, B Sharma, D Khalil… - ICASSP 2024-2024 IEEE …, 2024

… The Language Model (LM) was trained with 34M utterances from publicly available English datasets including People’s Speech, Fisher, Switchboard, AMI, Wikitext103, and subsets of Common Crawl and Reddit datasets. The model was …


Probing and enhancing the reliance of Transformer models on poetic information

Transformer models have achieved remarkable success in the widest variety of domains, spanning not just a multitude of tasks within natural language processing, but also those in computer vision, speech, and reinforcement learning. The key to …


Probing Critical Learning Dynamics of PLMs for Hate Speech Detection

S Masud, MA Khan, V Goyal, MS Akhtar, T Chakraborty - arXiv preprint arXiv …, 2024

… Each variant is trained on a training corpus from Wikipedia, and Common-Crawl is curated and updated before the date associated with … that are pretrained on regular and latest Internet snapshots obtained via Common Crawl and Wikipedia …


Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics

F Lin, D Altshuler, JB Pierrehumbert - arXiv preprint arXiv:2404.03301, 2024

… 2018) trained on 600B Common Crawl data as a baseline. For the baseline, we add the representations of strongest and weakest adjectives on s as ⃗s, then also rank cos(⃗a, ⃗s) for each a to calculate MRR. Here, we only report the results for the …


PRODIS-a speech database and a phoneme-based language model for the study of predictability effects in Polish

Z Malisz, J Foremski, M Kul - arXiv preprint arXiv:2404.10112, 2024

We present a speech database and a phoneme-level language model of Polish. The database and model are designed for the analysis of prosodic and discourse factors and their impact on acoustic parameters in interaction with predictability effects. The …


Prometheus' Digital Fire: The Civic Responsibilities of Artificial Intelligence

[Prometheus], the noble son of Iapetus outwitted [Zeus] and stole the far-seen gleam of unwearying fire in a hollow fennel stalk. And Zeus who thunders on high was stung in spirit, and his dear heart was angered when he saw amongst men the far-seen …


Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars

Z Wu, X Lin, Z Dai, W Hu, Y Shu, SK Ng, P Jaillet… - arXiv preprint arXiv …, 2024

… [2], it is highly likely that GPT-3 has been trained on common benchmark datasets potentially contained in Common Crawl due to data contamination. Hence, further providing high-quality in-context exemplars could hardly improve the performance of …


Prompting Change: ChatGPT's Impact on Digital Humanities Pedagogy–A Case Study in Art History

Q Guo - International Journal of Humanities and Arts …, 2024

This article explores the transformative impact of ChatGPT, a Generative Pre-trained Transformers model, on humanities methodologies and its integration into a digital humanities (DH) course. The advent of ChatGPT enhances the capacity of people …


Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian

S Auriemma, M Miliani, M Madeddu, A Bondielli… - arXiv preprint arXiv …, 2024

… We may suppose that this is because UmBERTo was trained on Common-Crawl, which also contains legal and administrative texts in its Italian section. Very high results are obtained by UmBERTo for PER entities, reaching 0.83 in our zero-shot …


Prospect of large language models and natural language processing for lung cancer diagnosis: A systematic review

A Garg, S Gupta, S Vats, P Handa, N Goel - Expert Systems

Lung cancer, a leading cause of global mortality, demands a combat for its effective prevention, early diagnosis, and advanced treatment methods. Traditional diagnostic methods face limitations in accuracy and efficiency, necessitating …


Protein-Protein Interaction Prediction

Proteins are essential components of cellular processes, playing pivotal roles in various biological functions. Their interactions with other proteins are fundamental for maintaining cellular functionality, a phenomenon referred to as protein-protein …


PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models

H Tan, Z Guo, Z Shi, L Xu, Z Liu, X Li, Y Wang, L Shang… - arXiv preprint arXiv …, 2024

… Nevertheless, scaling the context window during the pretraining phase remains a daunting task as the computational demands surge quadratically with the length of the attention span, and a majority of texts within standard corpora, like Common …


Pruning Large Language Models with Semi-Structural Adaptive Sparse Training

W Huang, G Jian, Y Hu, J Zhu, J Chen - arXiv preprint arXiv:2407.20584, 2024

Transformer-based Large Language Models (LLMs) have demonstrated remarkable success across various challenging tasks. However, the deployment of LLMs is hindered by their substantial parameter count and memory consumption. Recently …


PSILENCE: A Pseudonymization Tool for International Law

LA Cabrera-Diego, A Gheewala - Proceedings of the Workshop on Computational …, 2024

Since the announcement of the GDPR, the pseudonymization of legal documents has become a high-priority task in many legal organizations. This means that for making public a document, it is necessary to redact the identity of certain entities …


PSQE: Personalized Semantic Query Expansion for user-centric query disambiguation

… Furthermore, they enable explainable intelligent systems, in that the results of querying the embedding for similar terms relate directly to the training set, rather than webscale collections such as Common Crawl. To our knowledge, our work is …


PSQS: Parallel Semantic Querying Service for Self-describing File Formats

C Niu, W Zhang, S Byna, Y Chen - 2023 IEEE International Conference on Big Data …, 2023

… Typically, theseword representations are learned from a training dataset suchas the Common Crawl web corpus (840 billion tokens, 2.2million words) or Wikipedia 2014 (6 billion tokens, 400,000words). However, this assumption does not always …


Public perception of the Chinese president's visit to Saudi Arabia and the China–Arab Summit: sentiment analysis of Arabic tweets

AAM Hassan - Social Network Analysis and Mining, 2024

In recent years, China and Saudi Arabia have had frequent exchanges in the political and economic fields, and public opinion evaluation is an essential aspect of evaluating the two countries’ current relations. This paper analyzes tweets in Arabic …


PUBLIC-PRIVATE PARTNERSHIPS IN HEALTH SECTOR INNOVATION: LESSONS FROM AROUND THE WORLD

CC Ebulue, OR Ebulue, CS Ekesiobi - International Medical Science Research …, 2024

Public-Private Partnerships (PPPs) have emerged as a crucial mechanism for fostering innovation in the health sector globally. This review encapsulates the lessons learned from diverse PPP models worldwide, highlighting their significance …


PyLoomer-The Automated Weaving of Python Text

A Sahani, RB Trivedi, T Singh, SR Goyal - 2024 2nd International Conference on …, 2024

The introduction of attention mechanisms has evolved NLP methodologies, giving rise to Large Language Models (LLMs) that excel in various sequence-to-sequence tasks like translation, question-answering, summarization, intent, and entity …


Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients

Z Zhang, A Jaiswal, L Yin, S Liu, J Zhao, Y Tian… - arXiv preprint arXiv …, 2024

… The C4 dataset is a massive collection of Common Crawl’s web crawl corpus, meticulously filtered and cleaned to ensure highquality language modeling and training. It is widely used for pre-training large language models due to its diverse …


Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora

S Ranathunga, N de Silva, M Velayuthan, A Fernando… - arXiv preprint arXiv …, 2024

… 2020) is a dataset created using 68 snapshots of CommonCrawl10. Document … of CommonCrawl. It contains around 4.5 billion parallel sentences across 576 language pairs. In building CCMatrix, it was assumed the aligned sentence could …


Quantifying Gender Bias in Arabic Pre-trained Language Models

W Alrajhi, H Al-Khalifa, AM Al-Salman - IEEE Access, 2024

The current renaissance in the development of Arabic Pre-trained Language models (APLMs) has yielded significant advancement across many fields. Nevertheless, no study has explored the dimensions of gender bias in these models. It is argued that …


Quantifying Geospatial in the Common Crawl Corpus

I Ilyankou, M Wang, J Haworth, S Cavazzi - arXiv e-prints, 2024

… Common Crawl corpus. However, the geospatial content within CC remains largely unexplored, impacting our understanding of LLMs' spatial reasoning. This paper investigates the prevalence of geospatial data in recent Common Crawl …


Quantifying semantic relatedness across base verbs and derivatives: English

Most authors on derivational semantics agree that the meaning of complex words builds on components contributed by the base, components contributed by the word formation process, as well as contextual knowledge components. There is far less …


Quantifying task-related gaze

K Walter, M Freeman, P Bex - Attention, Perception, & Psychophysics, 2024

… For this study, we used the pretrained word vector Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors), which can be obtained at https://nlp.stanford.edu/projects/glove/. To utilize GloVe with the LabelMe images, we used it in conjunction with LASS …


Quantifying the Importance of Data Alignment in Downstream Model Performance

K Chawla, A Sahai, M DePavia, S Sundar, B Miranda - 2024

Contrary to the conventional emphasis on dataset size, we explore the role of data alignment---an often overlooked aspect of data quality---in training capable Large Language Models (LLMs). To do so, we use the Task2Vec-based alignment …


Quantitative Analysis of the Relationship Between Optimal Learning Rate and Batch Size Scaling in Large Language Models

R Schneider, H Baumgartner, D Wohlgemuth

… Publicly available large-scale text datasets were utilized, including Common Crawl, Wikipedia, and BooksCorpus, which provided diverse and extensive language data. These datasets were chosen for their rich linguistic content and their …


Query Obfuscation for Information Retrieval through Differential Privacy

Protecting the privacy of a user querying an Information Retrieval (IR) system is of utmost importance. The problem is exacerbated when the IR system is not cooperative in satisfying the user’s privacy requirements. To address this …


Query of CC: Unearthing Large Scale Domain-Specific Knowledge from Public Corpora

Z Fei, Y Shao, L Li, Z Zeng, H Yan, X Qiu, D Lin - arXiv preprint arXiv:2401.14624, 2024

… become increasingly rich, including the Pile, RedPajama, and Common Crawl. To ensure the diversity of data sources for retrieval, we constructed our retrieval corpus based on Common CrawlCommon Crawl is an open-source web crawler project …


Query Performance Prediction using Relevance Judgments Generated by Large Language Models

C Meng, N Arabzadeh, A Askari, M Aliannejadi… - arXiv preprint arXiv …, 2024

Query performance prediction (QPP) aims to estimate the retrieval quality of a search system for a query without human relevance judgments. Previous QPP methods typically return a single scalar value and do not require the predicted …


Query Spelling Correction using Pre-trained Word Embeddings

… The GloVe word embeddings were trained on multiple dumps of Wikipedia, as well as on a Common Crawl corpus from 2014. Vector dimensions larger than 300 were empirically found to provide diminishing returns on semantic and syntactic …


QuRating: Selecting High-Quality Data for Training Language Models

A Wettig, A Gupta, S Malik, D Chen - arXiv preprint arXiv:2402.09739, 2024

… We visualize the distributions for the cluster in CommonCrawl and C4 in Figure 7 in the appendix. They are similarly encouraging, eg, the clusters associated with cells, protein, gene and energy, climate, species are rated highly on required …


R2GQA: Retriever-Reader-Generator Question Answering System to Support Students Understanding Legal Regulations in Higher Education

PTP Do, DND Cao, KQ Tran, K Van Nguyen - arXiv preprint arXiv:2409.02840, 2024

In this article, we propose the R2GQA system, a Retriever-Reader-Generator Question Answering system, consisting of three main components: Document Retriever, Machine Reader, and Answer Generator. The Retriever module employs …


RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors

L Dugan, A Hwang, F Trhlik, JM Ludan, A Zhu, H Xu… - arXiv preprint arXiv …, 2024

… The model was allegedly trained with a variety of data including the Common Crawl (fil- … Released on July 18th 2023, LLaMA 2 is the successor to the original LLaMA model which was trained on webpages from CommonCrawl, multilingual …


Ranking Generated Answers: On the Agreement of Retrieval Models with Humans on Consumer Health Questions

S Heineking, J Probst, D Steinbach, M Potthast… - arXiv preprint arXiv …, 2024

… We therefore obtained only the original Web documents from CommonCrawl, discarded those containing fewer than 50 characters in the HTML body, and extracted plain text using the Resiliparse library.We were able to restore 6,692 Web …


Rapidly Developing High-quality Instruction Data and Evaluation Benchmark for Large Language Models with Minimal Human Effort: A Case Study on Japanese

Y Sun, Z Wan, N Ueda, S Yahata, F Cheng, C Chu… - arXiv preprint arXiv …, 2024

The creation of instruction data and evaluation benchmarks for serving Large language models often involves enormous human annotation. This issue becomes particularly pronounced when rapidly developing such resources for a non-English …


Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning

J Hejna, C Bhateja, Y Jian, K Pertsch, D Sadigh - arXiv preprint arXiv:2408.14037, 2024

… Many breakthroughs in machine learning can be attributed to “Internet-scale” datasets, from the development of vision models like CLIP [1] to recent advancements in transformer-based language modeling powered by the Common …


Real-Time Intelligent Detection System Designs for Web Phishing Attacks

… First, they collected URLs using Common Crawl and extracted their domain names to build their vocabulary. Then, they combine this list of vocabulary with common English names. Next, they use the regular expression pattern [a-zA-Z]+|[0-9]+ …


Real-time phishing detection using deep learning methods by extensions

DM Linh, HD Hung, HM Chau, QS Vu, TN Tran - International Journal of Electrical …, 2024

… [2] used two algorithms, random forest (RF) and long short-term memory (LSTM), to evaluate the PhishTank and Common Crawl datasets, achieving accuracies of 93.5% and 98.7%, respectively. Similarly, Kumar et al. [3] evaluated using the …


Real-World Robot Applications of Foundation Models: A Review

K Kawaharazuka, T Matsushima, A Gambardella… - arXiv preprint arXiv …, 2024

… In terms of the data, the datasets for training LLMs are Internet-scale; for example, the pre-training of GPT-3 [47] uses CommonCrawl covering Internet data from years 2016 to 2019, which amounts to 45TB of compressed plain text and 410 billion …


Reasoning for fact verification using language models

In response to the proliferation of misinformation on social media platforms, this thesis introduces the Triple-R framework (Retriever, Ranker, Reasoner) to enhance fact-checking by leveraging the Web for evidence retrieval and generating …


Recent Advancements and Challenges of Turkic Central Asian Language Processing

Y Veitsman - arXiv preprint arXiv:2407.05006, 2024

Research in the NLP sphere of the Turkic counterparts of Central Asian languages, namely Kazakh, Uzbek, Kyrgyz, and Turkmen, comes with the typical challenges of low-resource languages, like data scarcity and a general lack of linguistic resources …


Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives

DH Hagos, R Battle, DB Rawat - arXiv preprint arXiv:2407.14962, 2024

The emergence of Generative Artificial Intelligence (AI) and Large Language Models (LLMs) has marked a new era of Natural Language Processing (NLP), introducing unprecedented capabilities that are revolutionizing various domains. This paper …


Recent Advances in Large Language Models for Healthcare

K Nassiri, MA Akhloufi - BioMedInformatics, 2024

Recent advances in the field of large language models (LLMs) underline their high potential for applications in a variety of sectors. Their use in healthcare, in particular, holds out promising prospects for improving medical practices. As we highlight in …


Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark

H Cao - arXiv preprint arXiv:2406.01607, 2024

… data: (post, comment) pairs from Reddit3, (question, upvoted answer) pairs from Stackexchange4, (entity name + section title, passage) pairs from English Wikipedia, (title, abstract) and citation pairs from Scientific papers [53], and (title, passage) pairs …


Recognizing Textual Inference in Mongolian Bar Exam Questions

G Khaltarkhuu, B Batjargal, A Maeda - Applied Sciences, 2024

This paper examines how to apply deep learning techniques to Mongolian bar exam questions. Several approaches that utilize eight different fine-tuned transformer models were demonstrated for recognizing textual inference in Mongolian bar exam …


Recording First-person Experiences to Build a New Type of Foundation Model

D Barcari, D Gamez, A Grig - arXiv preprint arXiv:2408.02680, 2024

… These models are trained on large amounts of text and image data scraped from the Internet - often accessed through the Common Crawl repository. Researchers and technology companies are starting to realize that this data source could become …


RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent

H Xu, W Zhang, Z Wang, F Xiao, R Zheng, Y Feng, Z Ba… - arXiv preprint arXiv …, 2024

Recently, advanced Large Language Models (LLMs) such as GPT-4 have been integrated into many real-world applications like Code Copilot. These applications have significantly expanded the attack surface of LLMs, exposing them to a variety of …


Referring expression segmentation: from conventional to generalized

In recent years, many remarkable achievements have been made in the field of deep machine learning in various data modalities, such as image processing and natural language comprehension. Based on the good performance of deep neural …


RegMix: Data Mixture as Regression for Language Model Pre-training

Q Liu, X Zheng, N Muennighoff, G Zeng, L Dou, T Pang… - arXiv preprint arXiv …, 2024

… is taken from the C4 dataset [47] and supposed to share a similar distribution as Pile-CC since they are all derived from the CommonCrawl corpus. Since CommonCrawl is a collection of diverse domains, we would expect the correlation …


Reliability Estimation of News Media Sources: Birds of a Feather Flock Together

S Burdisso, D Sánchez-Cortés, E Villatoro-Tello…

Evaluating the reliability of news sources is a routine task for journalists and organizations committed to acquiring and disseminating accurate information. Recent research has shown that predicting sources’ reliability represents an …


Reliability of large language models in managing odontogenic sinusitis clinical scenarios: a preliminary multidisciplinary evaluation

AM Saibene, F Allevi, C Calvo-Henriquez, A Maniaci… - European Archives of Oto …, 2024

Purpose This study aimed to evaluate the utility of large language model (LLM) artificial intelligence tools, Chat Generative Pre-Trained Transformer (ChatGPT) versions 3.5 and 4, in managing complex otolaryngological clinical scenarios …


ReLU $^ 2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs

Z Zhang, Y Song, G Yu, X Han, Y Lin, C Xiao, C Song… - arXiv preprint arXiv …, 2024

… For the corpus used for calculating the sparsity statistics, we a mixture of texts from Wikipedia, Pile, and Common Crawl. The total number of tokens is about 217. Since we need to store the activation values of all neurons, we only use a part of the …


Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

P Maini, S Seto, H Bai, D Grangier, Y Zhang, N Jaitly - arXiv preprint arXiv …, 2024

… There are different versions of CommonCrawl data and our selection of C4 for pretraining is driven by driven by its size and quality. … The dataset is also derived from the CommonCrawl, however has a more stringent filtering process. Our …


Representation Bias of Adolescents in AI: A Bilingual, Bicultural Study

R Wolfe, A Dangol, B Howe, A Hiniker - arXiv preprint arXiv:2408.01961, 2024

Popular and news media often portray teenagers with sensationalism, as both a risk to society and at risk from society. As AI begins to absorb some of the epistemic functions of traditional media, we study how teenagers in two countries speaking two …


Representation learning for computational sociopragmatics

Natural Language Processing (NLP) emerges as a critical solution for analyzing, manipulating, and understanding human language automatically and computationally, enabling the processing of vast amounts of language data swiftly …


Representation Learning for Generative Models with Applications to Healthcare, Astronautics, and Aviation

… Large Language Models (LLMs) like BERT are typically pre-trained on extensive open web datasets, such as the English Wikipedia [50], BookCorpus [180], and Common Crawl [126]. These datasets enable LLMs to develop a robust …


Representation Learning for Hierarchical Classification of Entity Titles

E Chistova - Proceedings of the 6th International Conference on …, 2023

We present a method for effective title encoding for hierarchical classification in a large taxonomy. The method enables taxonomy-aware encoding in pre-trained text encoders, such as fastText and BERT, which are additionally fine-tuned for the …


Representing Online Handwriting for Recognition in Large Vision-Language Models

A Fadeeva, P Schlattner, A Maksai, M Collier… - arXiv preprint arXiv …, 2024

The adoption of tablets with touchscreens and styluses is increasing, and a key feature is converting handwriting to text, enabling search, indexing, and AI assistance. Meanwhile, vision-language models (VLMs) are now the go-to solution …


Representing relational knowledge with language models

Relational knowledge is the ability to recognize the relationship between instances, and it has an important role in human understanding a concept or commonsense reasoning. We, humans, structure our knowledge by understanding individual …


Research and Application of Large Model-Based Intelligent Customer Service System

Y Xi - International Journal of Emerging Technologies and …, 2024

With the rapid development of artificial intelligence technology, intelligent customer service systems have been widely used. This paper addresses the limitations of traditional intelligent customer service systems, such as limited language …


Research on cross-lingual multi-label patent classification based on pre-trained model

Y Lu, L Chen, X Tong, Y Peng, H Zhu - Scientometrics, 2024

Patent classification is an important part of the patent examination and management process. Using efficient and accurate automatic patent classification can significantly improve patent retrieval performance. Current monolingual patent classification …


Research Scholar (JRF), Department of Philosophy, The University of Burdwan, Golapbug

In these days, technology advances very rapidly. Today, we are at the doorstep of constructing a super-intelligent machine capable of mimicking almost all human activities. Initially, we took recourse to science and technology to transcend our …


ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models

J Baek, SK Jauhar, S Cucerzan, SJ Hwang - arXiv preprint arXiv:2404.07738, 2024

Scientific Research, vital for improving human life, is hindered by its inherent complexity, slow pace, and the need for specialized experts. To enhance its productivity, we propose a ResearchAgent, a large language model-powered …


Rethinking Higher Education Teaching and Assessment In-Line with AI Innovations: A Systematic Review and Meta-Analysis

JN Lyanda, SO Owidi, AM Simiyu - African Journal of Empirical Research, 2024

With the rapid advancement of artificial intelligence (AI) technologies, higher education institutions are increasingly exploring innovative ways to rethink teaching and assessment practices. This research paper examines the implications of AI on …


Retrieval Augmented Generation via Context Compression Techniques for Large Language Models

Natural language processing has seen lots of improvements, yet optimizing large-scale models to efficiently handle vast amounts of contextual data remains a critical challenge. The novel approach presented integrates advanced context compression …


Retrieval-Based Grammatical Error Correction

… a cleaned version of the common crawl to produce nearly 200 million ungrammatical sentences with a realistic distribution of errors. … CWEB consists of randomly selected texts from the first 18 dumps of the Common Crawl, which include …


Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models

J Parmar, S Satheesh, M Patwary, M Shoeybi… - arXiv preprint arXiv …, 2024

… In our English corpus, the Web Crawl data is sourced from Common Crawl (CC) snapshots while the remaining categories are comprised of high-quality sets. For instance, the miscellaneous category consists of BigScience ROOTS (Lachaux et al.…


Revisión de modelos de lenguaje y aplicación a la automatización de tareas de oficina en empresas industriales

This project focuses on the evaluation and application of Large Language Models (LLMs) in natural language processing (NLP). The technical properties, advantages, and disadvantages of LLMs are analysed, highlighting their wide use in machine …


Revisiting Adversarial Training at Scale

Z Wang, X Li, H Zhu, C Xie - arXiv preprint arXiv:2401.04727, 2024

… DataComp-1B is a more recent dataset with about 1.3B samples filtered from a candidate pool of 12.8B image-text pairs from Common Crawl, which has been recorded to yield superior performance for contrastive training. To summarize, our …


Revisiting Joke Comprehension with Surprisal and Contextual Similarity: Implication from N400 and P600 Components

H Xu, M Nakanishi, S Coulson - Proceedings of the Annual Meeting of the Cognitive …, 2024

… Specifically, we used the version with a 2.2 million word vocabulary and 300-dimensional vectors trained on 840 billion tokens from the Common Crawl corpus. The contextual vector of all the words preceding the final word in each sentence was computed by …


Revisiting Pre-training of Embedding Layers in Transformer-based Neural Machine Translation

M Neishi, N Yoshinaga - Journal of Natural Language Processing, 2024

Recent trends in the pre-training and fine-tuning paradigm have made significant advances in several natural language processing tasks, including machine translation (MT), particularly for low-resource situations. However, it is reported that …


Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning

S Sarkar, L Lausen, V Cevher, S Zha, T Brox, G Karypis - arXiv preprint arXiv …, 2024

… It is sourced from the Common Crawl, a web archive that captures vast amounts of text data from the internet. We … the Common Crawl, covering 108 languages [46]. It is a subset of the Colossal Clean Crawled Corpus (C4), which itself is a cleaned …


Revolutionizing College English Reading with ChatGPT: Current Predicaments and Future Directions

W Li, Y Tang, S Wang - Transactions on Social Science, Education and …, 2024

… The training data mainly includes website data in Common Crawl, two internet-based books corpora, and Englishlanguage Wikipedia [11]. ChatGPT trains a large amount of Internet information without distinguishing the true from the false and unifying …


Revolutionizing Urdu Sentiment Analysis: Harnessing the Power of XLM-R and GPT-2

MR Ashraf, M Hussain, MA Jaffar, WY Ramay… - IEEE Access, 2024

… In our proposed approach XLM-R leverages much larger CommonCrawl web data across 100 languages and uses a larger 250k SentencePiece vocabulary to improve subword coverage of Urdu Text. Beyond masked language modeling, XLM-R …


Rho-1: Not All Tokens Are What You Need

Z Lin, Z Gou, Y Gong, X Liu, Y Shen, R Xu, C Lin… - arXiv preprint arXiv …, 2024

Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that "Not all tokens in a corpus are equally important for language model training". Our initial …


Risk Mitigation Strategies for the Open Foundation Model Value Chain

Policymakers worldwide are designing interventions to ensure AI development is safe and responsible. In the EU, the newly formed AI Office is drafting codes of practice for general purpose AI models, while in the US, state legislatures are …


Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Survey

MN Sakib, MA Islam, R Pathak, MM Arifin - arXiv preprint arXiv:2408.04643, 2024

Recent advancements in Large Language Models (LLMs), such as ChatGPT and LLaMA, have significantly transformed Natural Language Processing (NLP) with their outstanding abilities in text generation, summarization, and classification …


RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs

J Dang, A Ahmadian, K Marchisio, J Kreutzer, A Üstün… - arXiv preprint arXiv …, 2024

Preference optimization techniques have become a standard final stage for training state-of-art large language models (LLMs). However, despite widespread adoption, the vast majority of work to-date has focused on first-class citizen languages like …


Roberta-LightGBM: A hybrid model of deep fake detection with pre-trained and binary classification

V Rajkumar - INFOCOMP Journal of Computer Science, 2024

In May 2023, a fake image of an explosion near the Pentagon gained widespread social media traction. It dragged down US markets momentarily, perhaps marking the first time an artificial intelligence (AI)-generated image has affected the market …


RoBERTaNET: Enhanced RoBERTa Transformer Based Model for Cyberbullying Detection with GloVe Features

F Alrowais, AA Jamjoom, H Karamti, M Umer, S Alsubai… - IEEE Access, 2024

… 3) FastText FastText is a word illustration library, that encompasses 300dimensional 2 million common crawl words, resulting in 600 billion word vectors [31]. Particularly, hand-crafted ngrams are used as features alongside solo words. Its straightforward …


Robust Neural Machine Translation for Abugidas by Glyph Perturbation

H Kaing, C Ding, H Tanaka, M Utiyama - Proceedings of the 18th Conference of the …, 2024

Neural machine translation (NMT) systems are vulnerable when trained on limited data. This is a common scenario in low-resource tasks in the real world. To increase robustness, a solution is to intently add realistic noise in the training phase. Noise …


RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian

A Cosma, B Iordache, P Rosso - arXiv preprint arXiv:2402.13222, 2024

… Even if code is present in the dataset (from Common Crawl), it is described in English and comments and documentation are in English. The same argument can be made for mathematics and other scientific disciplines. A large, highly curated and …


RoMath: A Mathematical Reasoning Benchmark in Romanian

A Cosma, AM Bucur, E Radoi - arXiv preprint arXiv:2409.11074, 2024

… For pretraining, the general approach is to filter Common Crawl web pages and PDFs to obtain high quality math tokens. For instance, … 2024b) that would arise from dumps such as Common Crawl. Figure 1 showcases our approach. After …


ROME: Memorization Insights from Text, Probability and Hidden State in Large Language Models

B Li, Q Zhao, L Wen - arXiv preprint arXiv:2403.00510, 2024

Probing the memorization of large language models holds significant importance. Previous works have established metrics for quantifying memorization, explored various influencing factors, such as data duplication, model size, and prompt length …


RU22Fact: Optimizing Evidence for Multilingual Explainable Fact-Checking on Russia-Ukraine Conflict

Y Zeng, X Ding, Y Zhao, X Li, J Zhang, C Yao, T Liu… - arXiv preprint arXiv …, 2024

Fact-checking is the task of verifying the factuality of a given claim by examining the available evidence. High-quality evidence plays a vital role in enhancing fact-checking systems and facilitating the generation of explanations that are understandable to …


RuBia: A Russian Language Bias Detection Dataset

V Grigoreva, A Ivanova, I Alimova, E Artemova - arXiv preprint arXiv:2403.17553, 2024

Warning: this work contains upsetting or disturbing content. Large language models (LLMs) tend to learn the social and cultural biases present in the raw pre-training data. To test if an LLM's behavior is fair, functional datasets are employed, and due to their …


Sailor: Open Language Models for South-East Asia

L Dou, Q Liu, G Zeng, J Guo, J Zhou, W Lu, M Lin - arXiv preprint arXiv:2404.03608, 2024

We present Sailor, a family of open language models ranging from 0.5B to 7B parameters, tailored for South-East Asian (SEA) languages. These models are continually pre-trained from Qwen1.5, a great language model for multilingual use …


Sampling-based Pseudo-Likelihood for Membership Inference Attacks

M Kaneko, Y Ma, Y Wata, N Okazaki - arXiv preprint arXiv:2404.11262, 2024

… 2023): LLaMA-2 employs English CommonCrawl, C4, Github, Wikipedia, Books, ArXiv, and StackExchange as pre-training datasets. … or filtered to contain predominantly English text, but a small amount of non-English data is still present …


Scaffold-BPE: Enhancing Byte Pair Encoding with Simple and Effective Scaffold Token Removal

H Lian, Y Xiong, J Niu, S Mo, Z Su, Z Lin, P Liu, H Chen… - arXiv preprint arXiv …, 2024

… The Pile is composed of 22 diverse and high-quality datasets, the models trained on which significantly outperform both raw and filtered Common Crawl [10] models. The data distribution for our model training is identical to those described in the …


Scalable Distributed String Sorting

F Kurpicz, P Mehnert, P Sanders, M Schimek - arXiv preprint arXiv:2404.16517, 2024

… While different sampling and group assignment techniques may improve character balance, the CommonCrawl data set contains many duplicated strings (eg, standard legal disclaimers) which are always assigned to a single PE and cannot be …


Scalable Pre-training of Large Autoregressive Image Models

A El-Nouby, M Klein, S Zhai, MA Bautista, A Toshev… - arXiv preprint arXiv …, 2024

This paper introduces AIM, a collection of vision models pre-trained with an autoregressive objective. These models are inspired by their textual counterparts, ie, Large Language Models (LLMs), and exhibit similar scaling properties. Specifically …


Scalable Range Search over Temporal and Numerical Expressions

V Ohr, D Gupta - The 10th ACM SIGIR/The 14th International …

… a subset of Common Crawl amounting to … Common Crawl, published as C4 [30]. This document collection consists of approximately 13.81 million documents. The fourth and final document collection we use is the entire C4-En corpus derived from …


Scaling A Simple Approach to Zero-Shot Speech Recognition

J Zhao, V Pratap, M Auli - arXiv preprint arXiv:2407.17852, 2024

… Figure 2 Accuracy on FLEURS dev languages when increasing the amount of CommonCrawl text data to build lexicons and unigram LMs compared to using in-domain text data (FLEURS-topline; 3k utterances). … Data is drawn from CommonCrawl (CC; …


Scaling Laws for Data Filtering--Data Curation cannot be Compute Agnostic

S Goyal, P Maini, ZC Lipton, A Raghunathan, JZ Kolter - arXiv preprint arXiv …, 2024

… As we train for longer, the accuracy gains using the LAION-filtering subset that filters the common crawl to 10% of its initial size plateau. Surprisingly, even no-filtering of the common crawl is better than the popular LAION-filtering after seeing more …


Scaling neural machine translation to 200 languages

… To overcome this problem, we created training datasets through global bitext mining in publicly available web content (drawn from repositories such as CommonCrawl). The underlying idea of our bitext mining approach is first to learn a …


Scaling Retrieval-Based Language Models with a Trillion-Token Datastore

R Shao, J He, A Asai, W Shi, T Dettmers, S Min… - arXiv preprint arXiv …, 2024

Scaling laws with respect to the amount of training data and the number of parameters allow us to predict the cost-benefit trade-offs of pretraining language models (LMs) in different configurations. In this paper, we consider another …


ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws

R Li, Y Wei, M Zhang, N Yu, H Hu, H Peng - arXiv preprint arXiv:2408.08310, 2024

… We begin with five CommonCrawl … CommonCrawl data with no more than 25B tokens. Each dataset was used to train meta-models for 1 epoch. The results demonstrate that all experiments outperform the baseline of random selection. Training …


sCambiaMenti at ACTI: Ensemble model with majority voting for Automatic Conspiracy Detection

S Bianco, D Salusso - EVALITA Proceedings of the Eighth Evaluation …, 2024

In this paper we describe the methodology that we have implemented to solve the subtask A of the Automatic Conspiracy Detection (ACTI) task (EVALITA 2023). We have developed different classifiers and then used a majority voting approach to …


ScenicNL: Generating Probabilistic Scenario Programs from Natural Language

K Elmaaroufi, D Shanker, A Cismaru… - First Conference on Language …

… Most of these model’s are fine-tuned variants of a foundational model that was trained on a wider body of text such as Common Crawl. For instance Codex is ”a GPT language model fine-tuned on publicly available code from GitHub” and it’s …


SchemaPile: A Large Collection of Relational Database Schemas

T Döhmen, R Geacu, M Hulsebos, S Schelter - … of the ACM on Management of Data, 2024

… such as the aforementioned code dataset The Stack [25] or web crawls such as CommonCrawl [12] as alternative data sources. Furthermore, enterprises could adapt our data … Common Crawl – a free, open repository of web crawl data that …


Scoping: Towards Streamlined Entity Collections for Multi-Sourced Entity Resolution with Self-Supervised Agents

Linking multiple entities to a real-world object is a time-consuming and error-prone task. Entity Resolution (ER) includes techniques for vectorizing entities (signature), grouping similar entities into partitions (blocking), and matching entity pairs based …


Screening Articles for Systematic Reviews with ChatGPT

… ChatGPT is a Chatbot Generative Pre-trained Transformer that employs an auto-regressive language model pre-trained on large datasets with bil250 lions of tokens from CommonCrawl, Wikipedia, and other publicly available text sources. ChatGPT relies …


SD-WEAT: Towards Robustly Measuring Bias in Input Embeddings

M Gray, L Wu - International Conference on Human-Computer …, 2024

… 300d” from the Stanford NLP Group [9], which was trained on 840 billion tokens from Common Crawl data and generates 300-dimension vectors. Moreover, other input embedding methods, including those of BERT [10, 11], SciBERT [12, 13], and …


SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages

H Lovenia, R Mahendra, SM Akbar, LJV Miranda… - arXiv preprint arXiv …, 2024

Southeast Asia (SEA) is a region rich in linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, prevailing AI models suffer from a significant lack of representation of texts, images …


SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages

W Zhang, HP Chan, Y Zhao, M Aljunied, J Wang, C Liu… - arXiv preprint arXiv …, 2024

Large Language Models (LLMs) have shown remarkable abilities across various tasks, yet their development has predominantly centered on high-resource languages like English and Chinese, leaving low-resource languages underserved …


Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion

J Zhu, H Hunag, Z Lin, J Liang, Z Tang, K Almubarak…

This paper addresses the critical need for democratizing large language models (LLM) in the Arab world, a region that has seen slower progress in developing models comparable to state-of-the-art offerings like GPT-4 or ChatGPT 3.5, due to a …


Secret Collusion Among Generative AI Agents

SR Motwani, M Baranchuk, M Strohmeier, V Bolina… - arXiv preprint arXiv …, 2024

Recent capability increases in large language models (LLMs) open up applications in which teams of communicating generative AI agents solve joint tasks. This poses privacy and security challenges concerning the unauthorised sharing of information …


SECTION ALIGNMENT TO DETECT DOCUMENT LINKS EFFICIENTLY

When searching for information, users often use search engines like Google1 or Bing2. These search engines have access to a wealth of publicly available information. To make finding the information even simpler and for processing and …


Secured Weight Release for Large Language Models via Taylor Expansion

G Wang, YN Chuang, R Tang, S Zhong, J Yuan, H Jin…

Ensuring the security of released large language models (LLMs) poses a significant dilemma, as existing mechanisms either compromise ownership rights or raise data privacy concerns. To address this dilemma, we introduce TaylorMLP to protect the …


SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents

K Cheng, Q Sun, Y Chu, F Xu, Y Li, J Zhang, Z Wu - arXiv preprint arXiv:2401.10935, 2024

… We collected approximately 300k web pages from the latest Common Crawl repository 2 to serve as our training data for web UI. For each webpage, we collect two types of elements from the HTML code: (1) elements that display visible text …


Seeds of Stereotypes: A Large-Scale Textual Analysis of Race and Gender Associations with Diseases in Online Sources

L Hyldig Hansen, N Andersen, J Gallifant, LG McCoy… - arXiv e-prints, 2024

LH Hansen, N Andersen, J Gallifant, LG McCoy… - arXiv preprint arXiv …, 2024

… Methods We conducted a large-scale textual analysis using a dataset comprising diverse web sources, including Arxiv, Wikipedia, and Common Crawl. The study analyzed the context in which various diseases are discussed alongside markers of …


Selecting Large Language Model to Fine-tune via Rectified Scaling Law

H Lin, B Huang, H Ye, Q Chen, Z Wang, S Li, J Ma… - arXiv preprint arXiv …, 2024

The ever-growing ecosystem of LLMs has posed a challenge in selecting the most appropriate pre-trained model to fine-tune amidst a sea of options. Given constrained resources, fine-tuning all models and making selections afterward is …


Selective Metric Differential Privacy for Language Models

… We used a pre-trained GloVe model trained on Common Crawl data with 840B tokens with a 2.2M vocabulary size for the privatization mechanism. We chose to utilize this model because it has a large vocabulary and is case-sensitive. Since GPT-2 …


Self-Augmented In-Context Learning for Unsupervised Word Translation

Y Li, A Korhonen, I Vulić - arXiv preprint arXiv:2402.10024, 2024

… 2017): the version pretrained on Wikipedia7 is used for XLING and the version pretrained with Wikipedia plus Common Crawl8 is used for PanLex-BLI, as recommended by XLING and PanLex-BLI, respectively. The same WEs are used for …


SEMANT-Feature Group Selection Utilizing FastText-Based Semantic Word Grouping, Scoring, and Modeling Approach for Text Classification

D Voskergian, B Bakir-Gungor, M Yousef - International Conference on Database and …, 2024

Text classification presents a challenge due to its high-dimensional feature space. As such, devising an effective feature selection scheme is essential. In this study, we present SEMANT, a novel hybrid filter-wrapper feature selection method that utilizes …


SEMANTIC MATCHING FOR DATA CATALOGS

Modern organizations have a lot of data, and more is coming. This data is often spread across the entire organization, appearing in various forms, such as CSV files, databases, PPTX slides, emails, etc. To organize and manage the various forms of …


Semantic-Aware Methods for the Analysis of Bias and Underrepresentation in Language Resources

… Within this research I did not analyze Common Crawl5, even if it is mentioned in some LMs papers. This is because it is not a proper … CC-News is a snapshot of news texts derived from the CommonCrawl initiative9) that has been exploited for …


Semantic-guided spatio-temporal attention for few-shot action recognition

J Wang, B Liu - Applied Intelligence, 2024

… We utilize GloVe [49] to extract text embeddings for action labels, which is pre-trained on Common Crawl dataset of 840B tokens. For labels with multiple words, we average the embeddings of all their words. Finally, one action label is represented …


Semantically Enriched Cross-Lingual Sentence Embeddings for Crisis-related Social Media Texts

R Lamsal, MR Read, S Karunasekera - arXiv preprint arXiv:2403.16614, 2024

Tasks such as semantic search and clustering on crisis-related social media texts enhance our comprehension of crisis discourse, aiding decision-making and targeted interventions. Pre-trained language models have advanced performance in …


Semiotics of Machinic Co-Enunciation

1 In this paper, we propose a semiotic study on generative artificial intelligences, considering the work of Midjourney and DALL· E. 1 The aim is to frame their functioning from a semiotic point of view and to describe the operations that can be …


SeNSe: embedding alignment via semantic anchors selection

L Malandri, F Mercorio, M Mezzanzanica, F Pallucchini - International Journal of Data …, 2024

Word embeddings have proven extremely useful across many NLP applications in recent years. Several key linguistic tasks, such as machine translation and transfer learning, require comparing distributed representations of words belonging to …


SENTA: Sentence Simplification System for Slovene

A Žagar, M Klemen, M Robnik-Šikonja, I Kosem - Proceedings of the 2024 Joint …, 2024

… It uses sentence-level paraphrase data obtained by mining paraphrases from Common Crawl in combination with controllable generation mechanisms to adjust the length and lexical complexity of a simplified candidate. …


Sentence Embeddings for Massively Multilingual Speech and Text Processing

… CommonCrawl raw text data corresponds to partial snapshots of the internet totalling terabytes of text data extracted from web pages in many languages. The authors used the open-source FAISS library (Johnson et al. 2019) in order to …


Sentence Level Analysis Model for Phishing Detection Using KNN

Phishing emails have experienced a rapid surge in cyber threats globally, especially following the emergence of the COVID-19 pandemic. This form of attack has led to substantial financial losses for numerous organizations. Although various models …


Sentiment Analysis and Rating Prediction for App Reviews Using Transformer-based Models

In this study, we present the sentiment analysis of Spotify app reviews, the implementation of Natural Language Processing (NLP) methods, and the use of transformer-based models including BERT, DistilBERT, RoBERTa, and XLM-RoBERTa …


Sentiment Analysis of Solar Energy in US Cities: A 10-Year Analysis Using Transformer-Based Deep Learning

This study examines US public sentiment toward solar energy from 2013 to 2022, analyzing 8 million social media posts using RoBERTa, a transformer-based deep learning algorithm. Sentiment has generally been positive but has declined since …


Sentiment analysis on the issue of COVID-19 vaccination using LSTM and fasttext

WN Dewani, Y Azhar, N Hayatin - AIP Conference Proceedings, 2024

… The fasttext model is the result of a word vector previously trained from Common Crawl and Indonesian language Wikipedia data from Facebook. This model is trained using the Continuous Bag of Words (CBOW) method with position weight [13] …


Sequence-to-Sequence Architectures: Encoder-Decoders and Decoders

PM Nugues - … for Natural Language Processing: Programming with …, 2024

The transformer consisted originally of two parts: the encoder and the decoder. This chapter describes this complete structure and, to make it concrete, illustrates it with a machine-translation program in PyTorch. For sake of feasibility, we replace the word …


SETEM: Self-ensemble training with Pre-trained Language Models for Entity Matching

H Ding, C Dai, Y Wu, W Ma, H Zhou - Knowledge-Based Systems, 2024

… This benchmark is constructed by extracting schema.org data from the Common Crawl and using the following attributes: brand, title, currency and price. The WDC Products benchmark consists of overall nine training, nine validation and nine test …


Shapley Values-Powered Framework for Fair Reward Split in Content Produced by GenAI

A Glinsky, A Sokolsky - arXiv preprint arXiv:2403.09700, 2024

… DATACOMP introduces a new candidate pool of 12.8 billion image-text pairs sourced from Common Crawl, which is broader than the collection in LAION-5B. This extensive pool offers a more diverse base for dataset creation. Unlike LAION-5B …


ShieldGemma: Generative AI Content Moderation Based on Gemma

W Zeng, Y Liu, R Mullins, L Peran, J Fernandez… - arXiv preprint arXiv …, 2024

We present ShieldGemma, a comprehensive suite of LLM-based safety content moderation models built upon Gemma2. These models provide robust, state-of-the-art predictions of safety risks across key harm types (sexually explicit, dangerous …


Should we stay silent on violence? An ensemble approach to detect violent incidents in Spanish social media texts

D Sharma, V Gupta, VK Singh, D Pinto - Natural Language Processing

… For training, more than two terabytes of filtered CommonCrawl data was used. The implementation of XLM-RoBERTa includes a preprocessing step, as described earlier. After preprocessing the data, the tweets are tokenized using the XLM-RoBERTa …


SIGN LANGUAGE RECOGNITION OF WORDS AND SENTENCE PREDICTION USING LSTM AND NLP

R GAIKWAD, L ADMUTHE - Journal of Theoretical and Applied Information …, 2024

… The dataset used for the training of the T5 model is the C4 dataset called as Colossal Clean Crawled Corpus which is collected from Common Crawl, a publicly available web archive. It consists of 750 GB clean English text scraped from the …


Silencing the Risk, Not the Whistle: A Semi-automated Text Sanitization Tool for Mitigating the Risk of Whistleblower Re-Identification

D Staufer, F Pallas, B Berendt - arXiv preprint arXiv:2405.01097, 2024

Whistleblowing is essential for ensuring transparency and accountability in both public and private sectors. However, (potential) whistleblowers often fear or face retaliation, even when reporting anonymously. The specific content of their …


SimLESS: A Secure Deduplication System over Similar Data in Cloud Media Sharing

M Song, Z Hua, Y Zheng, T Xiang, X Jia - IEEE Transactions on Information Forensics …, 2024

With the growing popularity of cloud computing, sharing media data through the cloud has become a common practice. Due to high information redundancy, media data take up a significant amount of storage space. Moreover, similar media data …


SimLLM: Calculating Semantic Similarity in Code Summaries using a Large Language Model-Based Approach

X Jin, Z Lin - Proceedings of the ACM on Software Engineering, 2024

Code summaries are pivotal in software engineering, serving to improve code readability, maintainability, and collaboration. While recent advancements in Large Language Models (LLMs) have opened new avenues for automatic code …


Simple and Scalable Strategies to Continually Pre-train Large Language Models

A Ibrahim, B Thérien, K Gupta, ML Richter, Q Anthony… - arXiv preprint arXiv …, 2024

… We then generate a fixed token-length response for each of the models trained German Common Crawl. As a baseline, we also evaluate … of German-language outputs from the models trained on German Common Crawl when compared to the …


Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulation

J Barnett, K Kieslich, N Diakopoulos - arXiv preprint arXiv:2405.09679, 2024

… They utilize proprietary datasets as well as opensource web data such as the continuously growing Common Crawl, which contains more than 250 billion pages of over a decade’s worth of textual content. While these models primarily learn how …


Simulating Subjects: The Promise and Peril of AI Stand-ins for Social Agents and Interactions

Large Language Models (LLMs), through their exposure to massive collections of online text, learn to reproduce the perspectives and linguistic styles of diverse social and cultural groups. This capability suggests a powerful social scientific application–the …


Single Cells Are Biological Tokens: Towards Cell Language Models

The rapid advancement of single-cell technologies allows for simultaneous measurement of multiple molecular features within individual cells, providing unprecedented multimodal data through single-cell multi-omics and spatial omics …