Single Layer Single Gradient Unlearning

Z Cai, Y Tan, MS Asif - arXiv preprint arXiv:2407.11867, 2024

… Large-scale web scrapers, such as Common Crawl, archive billions of webpages monthly, providing massive data sources to train such revolutionary AI models. While these large unsupervised datasets significantly enhance performance under …


Single Parent Family: A Spectrum of Family Members from a Single Pre-Trained Foundation Model

H Hajimolahoseini, M Hassanpour, F Ataiefard, B Chen… - arXiv preprint arXiv …, 2024

This paper introduces a novel method of Progressive Low Rank Decomposition (PLRD) tailored for the compression of large language models. Our approach leverages a pre-trained model, which is then incrementally decompressed to smaller sizes using …


Sketch Grammar for Japanese

Sketch Grammar is a formalism used to extract frequent word collocations from text corpora. This thesis aims at analyzing the current sketch grammar for Japanese used in the Sketch Engine corpus manager and implementing a new version with …


Slight Corruption in Pre-training Data Makes Better Diffusion Models

H Chen, Y Han, D Misra, X Li, K Hu, D Zou… - arXiv preprint arXiv …, 2024

… For example, Stable Diffusion [26] was pre-trained on LAION-2B [17], which contains billion-scale image-text pairs collected from Common Crawl [27]. Despite the heavy filtering mechanisms used in collecting pre-training datasets [17, 28], they …


Small Language Models and Large Language Models in Oppositional thinking analysis: Capabilities and Biases and Challenges

Á Huertas-García, C Martí-González, J Muñoz… - Working Notes of CLEF, 2024

The proliferation of misinformation and conspiracy theories needs robust methods to differentiate legitimate critical discourse from harmful conspiratorial narratives. This study investigates discerning critical messages from conspiracy theories within …


Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs

J Tan, Z Dou, Y Zhu, P Guo, K Fang, JR Wen - arXiv preprint arXiv:2402.12052, 2024

The integration of large language models (LLMs) and search engines represents a significant evolution in knowledge acquisition methodologies. However, determining the knowledge that an LLM already possesses and the knowledge that requires the …


Smaller Can Be Better: Efficient Data Selection for Pre-training Models

G Fang, S Wang, M Wang, Y Yang, H Huang - Asia-Pacific Web (APWeb) and Web …, 2024

To reduce the resource consumption for pre-training models on large-scale corpora, existing research has primarily focused on refining network architectures and training methods. Nevertheless, the concomitant cost tends to be expensive. This …


Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors

N Mireshghallah, J Mattern, S Gao, R Shokri… - Proceedings of the 18th …, 2024

As large language models are becoming more embedded in different user-facing services, it is important to be able to distinguish between human-written and machine-generated text to verify the authenticity of news articles, product reviews, etc. Thus, in this paper …


Smart Automation Using LLM

P Ethape, R Kane, G Gadekar, S Chimane - International Research Journal of …, 2023

… Such large-scale models can ingest massive amounts of data, often from the internet, but also from sources such as the Common Crawl, which comprises more than 50 billion web pages, and Wikipedia, which has approximately 57 million pages. …


Smart Bilingual Focused Crawling of Parallel Documents

C García-Romero, M Esplà-Gomis, F Sánchez-Martínez - arXiv preprint arXiv …, 2024

Crawling parallel texts $\unicode{x2014}$texts that are mutual translations$\unicode{x2014}$ from the Internet is usually done following a brute-force approach: documents are massively downloaded in an unguided process, and only a fraction of them end up …


Smoky Quartz: A Transparent Bilingual Large Language Model

Q Huang, C Zhang, M Tao, J Lin, S Huang, Y Feng

Recent advancements in large-scale models have been remarkable; however, a significant challenge persists in their lack of transparency, particularly concerning the training data utilized. This obscurity raises concerns about potential data leaks …


SMW Cloud: A Corpus of Domain-Specific Knowledge Graphs from Semantic MediaWikis

Semantic wikis have become an increasingly popular means of collaboratively managing Knowledge Graphs. They are powered by platforms such as Semantic MediaWiki and Wikibase, both of which enable MediaWiki to store and publish …


Social Evolution of Published Text and The Emergence of Artificial Intelligence Through Large Language Models and The Problem of Toxicity and Bias

A Khan, P Saravanan, SK Venkatesan - arXiv preprint arXiv:2402.07166, 2024

We provide a birds eye view of the rapid developments in AI and Deep Learning that has led to the path-breaking emergence of AI in Large Language Models. The aim of this study is to place all these developments in a pragmatic broader historical social …


Social Media Sentiment Analysis on Multilingual Dataset

This thesis presents using unlabeled data and soft targets from a LLM to train a pre-trained model for sentiment analysis. Inspired by the study “What do LLMs Know about Financial Markets? A Case Study on Reddit Market Sentiment Analysis” by Deng et …


SocialQuotes: Learning Contextual Roles of Social Media Quotes on the Web

J Palowitch, H Alvari, M Kazemi, T Amin, F Radlinski - arXiv preprint arXiv …, 2024

… We build and release SocialQuotes1, a data set computed from the Common Crawl with over 32M social media quotes, accompanied by role labels for 8.3k quotes obtained from human annotators. 3. We validate our main hypothesis with …


Sociolinguistically Informed Interpretability: A Case Study on Hinglish Emotion Classification

K Tatariya, H Lent, J Bjerva, M de Lhoneux - arXiv preprint arXiv:2402.03137, 2024

Emotion classification is a challenging task in NLP due to the inherent idiosyncratic and subjective nature of linguistic expression, especially with code-mixed data. Pre-trained language models (PLMs) have achieved high performance for many tasks and …


SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training

N He, W Xiong, H Liu, Y Liao, L Ding, K Zhang, G Tang… - arXiv preprint arXiv …, 2024

… In summary, our experiments on the RedPajama CommonCrawl dataset substantiate that the soft deduplication method is capable of reducing perplexity and enhancing the accuracy of downstream tasks more efficiently compared to the …


SoK: SSO-MONITOR—The Current State and Future Research Directions in Single Sign-On Security Measurements

L Jannett, M Westers, T Wich, C Mainka, A Mayer…

Single Sign-On (SSO) with OAuth 2.0 and OpenID Connect 1.0 is essential for user authentication and authorization on the Internet. Billions of users rely on SSO services provided by Google, Facebook, and Apple. For large-scale measurements …


Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets

M Walsh, A Preus, M Antoniak - arXiv preprint arXiv:2406.18906, 2024

… About 30% of our poems are found in the Common Crawl data included in Dolma, with the C4 dataset close behind. Wikipedia and Semantic Scholar contain the fewest detected poems. Overall, if at least one line from a poem is detected, it is …


Sourceformer: Tool Integration into Transformer Large Language Models Through Source Code

LLMs have been shown to use tools well. By allowing specific tools to increase the capabilities that LLMs struggle with, these models can become much more useful. Previous works use handcrafted examples of simple tool use during self-training …


Spanning the Spectrum of Hatred Detection: A Persian Multi-Label Hate Speech Dataset with Annotator Rationales

Z Delbari, NS Moosavi, MT Pilehvar - Proceedings of the AAAI Conference on …, 2024

With the alarming rise of hate speech in online communities, the demand for effective NLP models to identify instances of offensive language has reached a critical point. However, the development of such models heavily relies on the …


Spatial Language Understanding: Deep Learning, Reasoning, and Evaluation

… Pretrained language models (PLM) are large neural network models that have been trained on vast amounts of text data such as the entire text of Wikipedia or the Common Crawl web corpus in a un/supervised way. These models use transformers …


Speaker independent recognition of low-resourced multilingual Arabic spoken words through hybrid fusion

S Mehra, V Ranga, R Agarwal, S Susan - Multimedia Tools and Applications, 2024

This article introduces a supervised strategy designed to enhance spoken word recognition within the constraints of a resource-limited multilingual dataset, specifically focusing on the Arabic language. Notably, existing methodologies often …


Specialising and Analysing Instruction-Tuned and Byte-Level Language Models for Organic Reaction Prediction

J Pang, I Vulić - arXiv preprint arXiv:2405.10625, 2024

Transformer-based encoder-decoder models have demonstrated impressive results in chemical reaction prediction tasks. However, these models typically rely on pretraining using tens of millions of unlabelled molecules, which can be time-consuming …


Specialized Language Models with Cheap Inference from Limited Domain Data

D Grangier, A Katharopoulos, P Ablin, A Hannun - arXiv preprint arXiv:2402.01093, 2024

Large language models have emerged as a versatile tool but are challenging to apply to tasks lacking large inference budgets and large in-domain training sets. This work formalizes these constraints and distinguishes four important variables …


Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language Models

A Kaushal, T Pandey, T Vaidhya, A Bhagat, I Rish - arXiv preprint arXiv:2407.12327, 2024

Post-training quantization is the leading method for addressing memory-related bottlenecks in LLM inference, but unfortunately, it suffers from significant performance degradation below 4-bit precision. An alternative approach involves …


SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models

D Liu, R Zhang, L Qiu, S Huang, W Lin, S Zhao, S Geng… - Forty-first International Conference …

P Gao, R Zhang, C Liu, L Qiu, S Huang, W Lin, S Zhao… - arXiv preprint arXiv …, 2024

… Specifically, we first collect large-scale PDF datasets from Common Crawl 1 and arXiv websites. Then, we utilize PyMuPDF 2 to get the rendering results of each page in the PDF file and also save all the text annotations along with their bounding …


Spike No More: Stabilizing the Pre-training of Large Language Models

S Takase, S Kiyono, S Kobayashi, J Suzuki - arXiv preprint arXiv:2312.16903, 2023

… 2020) that consists of clean English texts extracted from Common Crawl7 as our LLM pre-training corpus. We also used the separated part of C4 as our validation data. We used GPT-2 vocabulary (Radford et al.… https://commoncrawl.org/ …


SPML: A DSL for Defending Language Models Against Prompt Attacks

RK Sharma, V Gupta, D Grossman - arXiv preprint arXiv:2402.11755, 2024

… The pre-training data included trillions of tokens sourced from publicly available datasets, such as Common Crawl, Wikipedia, and public domain books from Project Gutenberg. We use the versions with 7 billion and 13 billion parameters, available …


Sports, crisis, and social media: a Twitter-based exploration of the Tokyo Olympics in the COVID-19 era

V Mehra, P Singh, S Bharany, RS Sawhney - Social Network Analysis and Mining, 2024

The global landscape underwent a transformative shift due to the unexpected arrival of the COVID-19 pandemic, disrupting various aspects of life, including the staging of sports events. Notably, the Tokyo Olympics, a renowned international sports …


Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text

A Hans, A Schwarzschild, V Cherepanova, H Kazemi… - arXiv preprint arXiv …, 2024

… When evaluating Binoculars on samples from languages that are not well represented in Common Crawl data (standard LLM pretraining data), we find that false-positives rates remain low, which is highly desirable from a harm reduction …


SRBerta—A Transformer Language Model for Serbian Cyrillic Legal Texts

M Bogdanović, J Kocić, L Stoimenov - Information, 2024

… CommonCrawl [5] is a collection of texts whose size is measured in petabytes. This collection was … , CommonCrawl contains texts of lower quality, so development teams often resorted to processing texts before training and extracting …


Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

W Du, T Luo, Z Qiu, Z Huang, Y Shen, R Cheng, Y Guo… - arXiv preprint arXiv …, 2024

LLMs are computationally expensive to pre-train due to their large scale. Model growth emerges as a promising approach by leveraging smaller models to accelerate the training of larger ones. However, the viability of these model growth …


Stage-1 Report: Everything everywhere all at once? Exploring the richness of visual experience beyond words

L Qianchen, M Yamada, Y Aizawa, N Tsuchiya

In consciousness research, understanding the nature and mechanism of visual experience has been one of the most debated topics. For humans, vision is arguably the primary sensory modality to understand the environment (Cole & Balcetis, 2021) …


StarCoder 2 and The Stack v2: The Next Generation

A Lozhkov, R Li, LB Allal, F Cassano, J Lamy-Poirier… - arXiv preprint arXiv …, 2024

The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on …


State of Title IX: A Knowledge Base for Title IX Documentation

Knowledge bases play a vital role in the modern world, offering a systematic and structured approach to integrate various entities, concepts, rules, and relationships associated with real-world information. In this study, we aggregated the official Title …


Statistical dataset evaluation: A case study on named entity recognition

C Wang, Q Dong, X Wang, Z Sui - Natural Language Processing

Datasets serve as crucial training resources and model performance trackers. However, existing datasets have exposed a plethora of problems, inducing biased models and unreliable evaluation results. In this paper, we propose a model-agnostic …


STFL: Utilizing a Semi-Supervised, Transfer-Learning, Federated-Learning Approach to Detect Phishing URL Attacks

I Sakazi, E Grolman, Y Elovici, A Shabtai - 2024 International Joint Conference on …, 2024

Phishing attacks are continually changing, so machine learning detection models must be continuously updated by collecting new URL data from users without compromising their privacy. Existing approaches for the detection of phishing URLs …


Strategies for Corpus Development for Low‐Resource Languages: Insights from Nepal

BK Bal, B Prasain, RR Ghimire, P Acharya - Automatic Speech Recognition and …, 2024

… 25 whereas the monolingual dataset were derived from Wikipedia and Common Crawl for languages under consideration. They also experimented with four variations of training settings: fully supervised, fully unsupervised, semisupervised …


Stratified Graph Indexing for E cient Search in Deep Descriptor Databases

Searching for unseen objects in extensive visual archives is challenging, demanding efficient indexing methods that can support meaningful similarity retrievals. This research paper presents the Stratified Graph (SG) approach for …


Strengthening the WiC: New Polysemy Dataset in Hindi and Lack of Cross Lingual Transfer

H Dubossarsky, F Dairkee - Proceedings of the 2024 Joint International Conference …, 2024

… TB of filtered CommonCrawl data as opposed to mBERT which was trained on Wikipedia data, and reported to outperform mBERT in a variety of cross-lingual benchmarks like XNLI, MLQA and NER. HindiBERT is a monolingual model trained …


Structured Packing in LLM Training Improves Long Context Utilization

K Staniszewski, S Tworkowski, S Jaszczur… - arXiv preprint arXiv …, 2023

Recent advances in long-context Large Language Models (LCLMs) have generated significant interest, especially in applications such as querying scientific research papers. However, their potential is often limited by inadequate context utilization. We …


Studies in conversational AI: multilingual capabilities, world knowledge, and evaluation strategies

This thesis studies the evolving landscape of conversational AI. The main research objective is to improve the conversational abilities of conversational agents, with a focus on integrating real-time knowledge and expanding multilingual capabil-ities …


Studying Peer Effects in Divergent Thinking: Theory and Method

In designing technology that supports user learning, an important first step is to understand how interactions among humans shape mutual learning. Much qualitative research in the realm of peer-assisted learning (PAL) has advanced the …


SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM

Q Wang, Y Yuan, X Yang, R Zhang, K Zhao, W Liu… - arXiv preprint arXiv …, 2024

While Large Language Models (LLMs) have achieved remarkable success in various fields, the efficiency of training and inference remains a major challenge. To address this issue, we propose SUBLLM, short for Subsampling-Upsampling-Bypass …


Supervised Learning

ER Vankov, K Gadhoumi - Statistical Methods in Epilepsy, 2024

… A corpus such as Common Crawl, which contains roughly 840 billion tokens of over two million unique words, provides such scale. There are other corpora used in these unsupervised processes, one such common candidate being a collection of all …


SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs

J Kim, J Nam, S Mo, J Park, SW Lee, M Seo, JW Ha… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have made significant advancements in various natural language processing tasks, including question answering (QA) tasks. While incorporating new information with the retrieval of relevant passages is a promising …


Suri: Multi-constraint Instruction Following for Long-form Text Generation

CM Pham, S Sun, M Iyyer - arXiv preprint arXiv:2406.19371, 2024

Existing research on instruction following largely focuses on tasks with simple instructions and short responses. In this work, we explore multi-constraint instruction following for generating long-form text. We create Suri, a dataset with 20K human-written …


Survey of different Large Language Model Ar-chitectures: Trends, Benchmarks, and Challenges

M SHAO, A BASIT, R KARRI, M SHAFIQUE - structure

Large Language Models (LLMs) represent a class of deep learning models adept at understanding natural language and generating coherent text in response to prompts or queries. These models significantly surpass conventional neural …


Synthetic continued pretraining

Z Yang, N Band, S Li, E Candès, T Hashimoto - arXiv preprint arXiv:2409.07431, 2024

Pretraining on large-scale, unstructured internet text has enabled language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient -- to learn a given fact, models must be trained on …


Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation

S Soni, A Kunchukuttan, T Chakraborty, MS Akhtar - arXiv preprint arXiv …, 2024

The widespread online communication in a modern multilingual world has provided opportunities to blend more than one language (aka code-mixed language) in a single utterance. This has resulted a formidable challenge for the computational …


Systematic Evaluation of Linear Transformations for Cross-lingual Semantic Spaces

In the field of natural language understanding (NLU), a fundamental element is the representation of word meanings, a process that is integral to a wide range of applications. These applications span numerous domains including machine …


Systematic exploration and in-depth analysis of ChatGPT architectures progression

D Banik, N Pati, A Sharma - Artificial Intelligence Review, 2024

The fast evolution of artificial intelligence frameworks has resulted in the creation of increasingly sophisticated large language models (LLM), ChatGPT being the most famous one. This study paper dives into this LLM with a case study of ChatGPT’s …


T5G2P: Text-to-Text Transfer Transformer based Grapheme-to-Phoneme Conversion

M Řezáčková, D Tihelka, J Matoušek - IEEE/ACM Transactions on Audio, Speech …, 2024

… For the Czech language, we collected the CommonCrawl corpus at the end of August 2020. It contains 47.9GB of clean text, 20.5 million … Similarly, for both Russian and German we collected CommonCrawl corpus in the season from August …


Tübingen-CL at SemEval-2024 Task 1: Ensemble Learning for Semantic Relatedness Estimation

L Zhang, Ç Çöltekin - Proceedings of the 18th International Workshop on …, 2024

The paper introduces our system for SemEval-2024 Task 1, which aims to predict the relatedness of sentence pairs. Operating under the hypothesis that semantic relatedness is a broader concept that extends beyond mere similarity of sentences …


Table Representation Learning

The increasing amount of data being collected, stored, and analyzed, induces a need for efficient, scalable, and robust methods to handle this data. Representation learning, ie, the practice of leveraging neural networks to obtain generic vector …


Tackling an Unbalanced Dataset for Classifying Indonesian E-Commerce Reviews Using Multi Word Embedding Model

R Adi, BR Irnawan, J Li - 2023 Eighth International Conference on Informatics …, 2023

… Our research used a pre-trained FastText that was trained on Indonesian Common Crawl and Wikipedia provided by [24]. 3) GloVe: GloVe is an unsupervised algorithm that can produce a vector representation of words created by [22]. Besides …


Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization

X Chen, Z Wang, D Sow, J Yang, T Chen, Y Liang… - arXiv preprint arXiv …, 2024

… subset of the Common Crawl’s web crawl corpus. More recently, LLaMA (Touvron et al.… datasets to encompass multiple domains such as Common Crawl, C4, and source codes from … contents are mostly unintelligible” in the Common Crawl. All …


Tamil Language Computing: the Present and the Future

K Sarveswaran - arXiv preprint arXiv:2407.08618, 2024

This paper delves into the text processing aspects of Language Computing, which enables computers to understand, interpret, and generate human language. Focusing on tasks such as speech recognition, machine translation, sentiment …


Task Oriented In-Domain Data Augmentation

X Liang, X Hu, S Zuo, Y Gong, Q Lou, Y Liu, SL Huang… - arXiv preprint arXiv …, 2024

Large Language Models (LLMs) have shown superior performance in various applications and fields. To achieve better performance on specialized domains such as law and advertisement, LLMs are often continue pre-trained on in-domain data …


Tasneef: A Fast and Effective Hybrid Representation Approach for Arabic Text Classification

M Louail, CKM Hamdi-Cherif, A Hamdi-Cherif - IEEE Access, 2024

The Arabic language role in actual global affairs entails sophisticated natural language processing techniques, especially in text classification. This paper presents Tasneef as a novel hybrid approach to tackle computational challenges by …


TEA+: A Novel Temporal Graph Random Walk Engine With Hybrid Storage Architecture

C Huan, Y Liu, H Zhang, S Song, S Pandey, S Chen… - ACM Transactions on Architecture …

Many real-world networks are characterized by being temporal and dynamic, wherein the temporal information signifies the changes in connections, such as the addition or removal of links between nodes. Employing random walks on these …


Team Aditya at EXIST 2024—Detecting Sexism In Multilingual Tweets Using Contrastive Learning Approach

A Shah, A Gokhale - Working Notes of CLEF, 2024

Due to the growing impact of social media, the necessity for automated mechanisms that can identify sexism and other forms of disrespectful and hateful conduct is rising, aiming to create a more inclusive and respectful digital space. However, it poses …


Team OpenWebSearch at CLEF 2024: LongEval

D Alexander, M Fröbe, G Hendriksen, F Schlatt… - 2024

We describe the OpenWebSearch group’s participation in the CLEF 2024 LongEval IR track. Our submitted runs explore how historical data from the past can be transferred into future retrieval systems. Therefore, we incorporate relevance …


TeClass: A Human-Annotated Relevance-based Headline Classification and Generation Dataset for Telugu

G Kanumolu, L Madasu, N Surange, M Shrivastava - arXiv preprint arXiv:2404.11349, 2024

… 2019) is a multilingual version of the RoBERTa model, and it was pre-trained on a vast 2.5TB CommonCrawl dataset, which included text from 100 languages. For our experiments, we utilized the xlm-roberta-base variant, boasting 270 million parameters. …


Tele-FLM Technical Report

X Li, Y Yao, X Jiang, X Fang, C Wang, X Liu, Z Wang… - arXiv preprint arXiv …, 2024

… CommonCrawl1 is often considered to be a repository containing diverse human experience … CommonCrawl are primarily concentrated in the English segment, with the Chinese content exhibiting relatively lower information density and quality …


Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications

A Maatouk, KC Ampudia, R Ying, L Tassiulas - arXiv preprint arXiv:2409.05314, 2024

… is the Common Crawl dataset, which comprises web archives from across the internet. To avoid issues with duplicates, non-English content, and potential profanity found in raw dumps, we utilized the refined web dataset [26]. This curated …


TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models

H Zou, Q Zhao, Y Tian, L Bariah, F Bader, T Lestable… - arXiv preprint arXiv …, 2024

Large Language Models (LLMs) have the potential to revolutionize the Sixth Generation (6G) communication networks. However, current mainstream LLMs generally lack the specialized knowledge in telecom domain. In this paper, for the …


Tell me a story: a framework for critically investigating AI language models

L Munn, L Henrickson - Learning, Media and Technology, 2024

Large language models are rapidly being rolled out into high-stakes fields like healthcare, law, and education. However, understanding of their design considerations, operational logics, and implicit biases remains limited. How might …


Temporal Blind Spots in Large Language Models

J Wallat, A Jatowt, A Anand - arXiv preprint arXiv:2401.12078, 2024

… A mixture of training datasets has been used: a filtered version of CommonCrawl, an expanded version of WebText [44], two not further specified internet-based book corpora, and the English Wikipedia. The most recent information in text-davinci-003’s …


Temporally-Varied News Headline Generation with GPT-3


TEXAS INTELLECTUAL PROPERTY LAW JOURNAL

J Nacht, A Neill, J Thomas, E Lee, KE Noonan… - 2024

… 4t Common Crawl contains information from over 250 billion web pages since 2007 using web crawlers … in Common Crawl was filtered before being incorporated into GPT-3 to maintain a higher level of quality for training purposes.Websites …


Text Emotion Recognition Using Fast Text Word Embedding in Bi-Directional Gated Recurrent Unit

C Akalya Devi, D Karthika Renuka, T Harisudhan…

Emotions are states of readiness in the mind that result from evaluations of one's own thinking or events. Although almost all of the important events in our lives are marked by emotions, the nature, causes, and effects of emotions are some of the …


Text Filtering Classifiers for Medium-Resource Languages

J Daðason, H Loftsson - Proceedings of the 2024 Joint International Conference …, 2024

… Online texts are often obtained from large datasets, such as those published by Common Crawl (CC).The GPT-3 model was pre-trained on 499B tokens, 410B of which were obtained from CC (Brown et al.… The Icelandic Common Crawl Corpus (IC3) …


Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges

J Becker, JP Wahle, B Gipp, T Ruas - arXiv preprint arXiv:2405.15604, 2024

Text generation has become more accessible than ever, and the increasing interest in these systems, especially those using large language models, has spurred an increasing number of related publications. We provide a systematic literature review …


TEXT SUMMARIZATION SYSTEM USING NATURAL LANGUAGE PROCESSING AND MACHINE LEARNING

Information overload, caused by the rapid growth of the Internet, has become a significant issue. The abundance of information available online necessitates the simplification of relevant content through summarization. Manual summarization of …


Text-to-text generative approach for enhanced complex word identification

P Śliwiak, SAA Shah - Neurocomputing, 2024

This paper presents a novel approach for solving the Complex Word Identification (CWI) task using the text-to-text generative model. The CWI task involves identifying complex words in text, which is a challenging Natural Language Processing task. To …


TextGram: Towards a Better Domain-Adaptive Pretraining

O Dhekane, S Hiwarkhedkar, R Joshi, A Ladkat - … , Perundurai, Erode, India, December 6–8 …

… The RealNews Dataset is a large corpus of news articles scraped from Common Crawl [16]. It consists of 5000 news domains that are indexed by Google news. IMDb Movie Reviews is a binary sentiment analysis dataset made up of 50000 reviews …


Textual Coverage of Eventive Entries in Lexical Semantic Resources

E Fucikova, CF Alcaina, J Hajic, Z Urešová - Proceedings of the 2024 Joint …, 2024

This short paper focuses on the coverage of eventive entries (verbs, predicates, etc.) of some well-known lexical semantic resources when applied to random running texts taken from the internet. While coverage gaps are often reported for manually …


Textual emotion detection–A systematic literature review

Textual emotion detection is a critical area of study with significant applications in business, education, and healthcare. Despite substantial theoretical advancements over the years, there is a notable gap in the practical implementation of these …


The advantages of context specific language models: the case of the Erasmian Language Model

J Gonçalves, N Jelicic, M Murgia, E Stamhuis - arXiv preprint arXiv:2408.06931, 2024

… The dominant trend in improving language models seems to be based on scaling up the number of parameters (eg GPT4 has approximately 1.7 trillion parameters) or the amount of training data fed into the model (eg the Common Crawl dataset …


The African Woman is Rhythmic and Soulful: Evaluation of Open-ended Generation for Implicit Biases

S Lim - arXiv preprint arXiv:2407.01270, 2024

This study investigates the subtle and often concealed biases present in Large Language Models (LLMs), which, despite passing explicit bias tests, can still exhibit implicit biases akin to those observed in humans who profess egalitarian beliefs yet …


The AI Companion in Education: Analyzing the Pedagogical Potential of ChatGPT in Computer Science and Engineering

Z He, T Nguyen, T Miari, M Aliasgari, S Rafatirad…

… ChatGPT3 employs a pre-trained large language model [29] trained over the Common Crawl dataset [30]. During its training process, it first preprocess data from the Common Crawl dataset, data quality is refined by eliminating irrelevant or noisy …


The AI Did My Job Well

Artificial intelligence (AI), an idea first proposed back in 1955 (McCarthy, Minsky, & Rochester) seemingly has burst into public consciousness nearly 70 years later. Like the chained prisoners emerging from Plato’s allegorical cave, many of us find …


The Aranea Corpora Family: Ten+ Years of Processing Web-Crawled Data

V Benko - International Conference on Text, Speech, and …, 2024

Aranea is a project to create a family of web-crawled corpora for languages taught at Slovak Universities. Since 2013, more than two dozen languages have been added to the project: they are often represented with Gigaword+ size corpora and …


The BEA 2024 Shared Task on the Multilingual Lexical Simplification Pipeline

M Shardlow, F Alva-Manchego, RT Batista-Navarro… - … on Innovative Use of NLP for …, 2024

We report the findings of the 2024 Multilingual Lexical Simplification Pipeline shared task. We released a new dataset comprising 5,927 instances of lexical complexity prediction and lexical simplification on common contexts across 10 languages, split …


The Case For Data Centre Hyperloops

G López-Paradís, IM Hair, S Kannan, R Rabbat… - 2024 ACM/IEEE 51st …, 2024

G López-Paradıs, IM Hair, S Kannan, R Rabbat…

Data movement is a hot-button topic today, with workloads like machine learning (ML) training, graph processing, and data analytics consuming datasets as large as 30PB. Such a dataset would take almost a week to transfer at 400gbps while consuming …


The Case for Developing a Foundation Model for Planning-like Tasks from Scratch

B Srivastava, V Pallagani - arXiv preprint arXiv:2404.04540, 2024

Foundation Models (FMs) have revolutionized many areas of computing, including Automated Planning and Scheduling (APS). For example, a recent study found them useful for planning problems: plan generation, language translation, model …


The Case for Nurturing AI Literacy in Law Schools

S Migliorini, JI Moreira - Asian Journal of Legal Education, 2024

The debate surrounding the permissibility of generative artificial intelligence (AI) tools in legal education has garnered widespread attention. However, this discourse has largely oscillated between the advantages and disadvantages of generative AI …


The correlation between nativelike selection and prototypicality: a multilingual onomasiological case study using semantic embedding

H Zhang - arXiv preprint arXiv:2405.13529, 2024

In native speakers' lexical choices, a concept can be more readily expressed by one expression over another grammatical one, a phenomenon known as nativelike selection (NLS). In previous research, arbitrary chunks such as collocations have …


The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models

A Birhane, S Dehdashtian, VU Prabhu, V Boddeti - arXiv preprint arXiv:2405.04623, 2024

… This is particularly important in the context of multimodal datasets whose main source is the World Wide Web, condensed and packaged as the Common Crawl dump, which is known to exhibit numerous drawbacks. In this paper, we evaluate the …


The Dark Side of Language Models: Exploring the Potential of LLMs in Multimedia Disinformation Generation and Dissemination

D Barman, Z Guo, O Conlan - Machine Learning with Applications, 2024

Disinformation - the deliberate spread of false or misleading information poses a significant threat to our society by undermining trust, exacerbating polarization, and manipulating public opinion. With the rapid advancement of artificial intelligence and …


The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational study

DM Levine, R Tuwani, B Kompa, A Varma… - The Lancet Digital Health, 2024

… GPT-3 is one of the largest AI models ever created, containing over 175 billion model parameters, and was trained on approximately 570 gigabytes of data from the common crawl dataset, which contains nearly all unstructured text available on the …


The effect of generative artificial intelligence on managerial behavior across functions

GAI is becoming readily available and easily accessible for those who are willing to take advantage of it. As a result, the world is witnessing a transformation regarding how managers perform their daily tasks. This study aims to understand the …


THE ENHANCEMENT OF OPEN DATA USABILITY BY DEFINING A CATEGORIZATION METHOD BASED ON THE OPEN DATA PORTAL METADATA

Due to numerous data transparency and open government initiatives, a large volume of data was published on open data portals. To make it more accessible and visible, these portals have introduced data filtering by category, tags, format …


The ethical situation of DALL-E 2

E Hogea, J Rocafortf - arXiv preprint arXiv:2405.19176, 2024

A hot topic of Artificial Intelligence right now is image generation from prompts. DALL-E 2 is one of the biggest names in this domain, as it allows people to create images from simple text inputs, to even more complicated ones. The company that made this …


The Ethics and Challenges of Legal Personhood for AI

Hon. Katherine B. Forrest (Fmr.) abstract. AI’s increasing cognitive abilities will raise challenges for judges.“Legal personhood” is a flexible and political concept that has evolved throughout American history. In determining whether to expand that concept …


The Evolution of Multimodal Model Architectures

SN Wadekar, A Chaurasia, A Chadha, E Culurciello - arXiv preprint arXiv:2405.17927, 2024

This work uniquely identifies and characterizes four prevalent multimodal model architectural patterns in the contemporary multimodal landscape. Systematically categorizing models by architecture type facilitates monitoring of developments in …


The Fill-Mask Association Test (FMAT): Measuring Propositions in Natural Language

HWS Bao - Journal of Personality and Social Psychology, 2024

L Lin, B Wang, X Wang, ZX Wang, A Wiśniowski…

Recent advances in large language models are enabling the computational intelligent analysis of psychology in natural language. Here, the Fill-Mask Association Test (FMAT) is introduced as a novel and integrative method leveraging …


The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

G Penedo, H Kydlíček, A Lozhkov, M Mitchell, C Raffel… - arXiv preprint arXiv …, 2024

… In this work, we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets. To advance the understanding of how best to curate high-quality …


The Future of Chatbots: Survey on Generative Pretrained Models, Ernie, and Gemini

A Raut, J Katti - 2024 International Conference on Innovations and …, 2024

The evolution of chatbots from rule-based, retrieval-based, contextual systems, to hybrid and generative AIpowered models. It aimsto solve complex natural language processing and task-specific challenges. The finding explores innovative self-attention …


The Future of Data Science in Materials Science

P Chundi, V Bommanapally, V Gadhamshetty - Machine Learning in 2D Materials …, 2023

… GPT-3 is an autoregressive model built with 96 layers [31], increased embedding vector dimension of 12888, trained with 175B parameters on five datasets including Common crawl, WebText2, Books1, Books2, and Wikipedia. Though GPT-3 has …


The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare

… Namely, datasets used for training LLMs such as Common Crawl contain petabytes of data [260]. Therefore, it is not possible to have complete human oversight over the data cleaning process. Hence, this is why heuristic methods are …


The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)

S Zeng, J Zhang, P He, Y Xing, Y Liu, H Xu, J Ren… - arXiv preprint arXiv …, 2024

… 2021), we randomly select chunks from the Common Crawl dataset to serve as the {information} component. … 2021), we randomly select chunks from the Common Crawl dataset to serve as the {information} component. Due to the random …


The High Stakes of AI Ethics: Evaluating OpenAI's Potential Shift to “NSFW” Content and Other Concerns

… These address concerns surrounding the dubious curation practices used to generate these datasets, the sordid quality of alt-text data available on the world wide web, the problematic content of the CommonCrawl dataset often used as a …


The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis

M Zhang, V Gautam, M Wang, JO Alabi, X Shen… - arXiv preprint arXiv …, 2024

… The order of languages follows their data ratio in the CommonCrawl corpus10 from high-resource to low-resource. We observe large variations in model performance across different languages. For instance, there exists a large …


The Impact of Geometric Complexity on Neural Collapse in Transfer Learning

M Munn, B Dherin, J Gonzalvo - arXiv preprint arXiv:2405.15706, 2024

Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical …


The Impact of Infrastructure: Considerations of Generative AI in the Classroom

ES Kehlenbach - Journal of Political Science Education, 2024

How are we to grapple with the increasing intrusion of technology in the classroom? What considerations need to be addressed before we begin bringing advanced technological systems like ChatGPT into our classroom spaces? A close attention to …


The impact of language family on D2T generation in under-resourced languages

The paper delves deep into the challenges and methods of generating text from structured data(RDF triples) for under-resourced languages based on the WebNLG challenge. The main question of this paper is to assess how important language …


The Impact of Online Censorship on LLMs

M Ahmed, J Knockel - Free and Open Communications on the Internet, 2024

… the Common Crawl contains censored content, there may be downstream effects on the models. To investigate whether the Common Crawl … Of which, we found pages indexed from 325 of the censored websites in the Common Crawl dataset that …


The Impact of OpenAI on the Work Environment located in Finland

In the modern era, technology has undergone a rapid transformation that touches every aspect of our lives. At the forefront of this transformation stands artificial intelligence (AI), a field rapidly evolving through groundbreaking innovations, and …


The life cycle of large language models in education: A framework for understanding sources of bias

J Lee, Y Hicke, R Yu, C Brooks, RF Kizilcec - British Journal of Educational …, 2024

Large language models (LLMs) are increasingly adopted in educational contexts to provide personalized support to students and teachers. The unprecedented capacity of LLM‐based applications to understand and generate natural language can …


The model student: GPT-4 performance on graduate biomedical science exams

D Stribling, Y Xia, MK Amer, KS Graim, CJ Mulligan… - Scientific Reports, 2024

… As these study materials may have been incorporated into the GPT-4 training data such as the Common Crawl 43 , standardized examinations may not be an accurate assessment of domain-specific model knowledgebase and capability …


The OECD-UNSD Multinational Enterprise Information Platform

… (3) Common Crawl is an open source initiative to provide a monthly snapshot of the internet through web scraping. As part of the project, they provide a quarterly network graph of hyperlinks between domains, which can be used to determine …


The Open Web Index

G Hendriksen, M Dinzinger, SM Farzana, NA Fathima…

… To operationalize our crawling pipeline, we introduce a re-crawl mechanism specifically for Common Crawl dumps. Unlike Common Crawl, we crawl on a daily basis and save the results as WARC (Web ARChive) files.This approach allows us …


The Open Web Index: Crawling and Indexing the Web for Public Use

G Hendriksen, M Dinzinger, SM Farzana, NA Fathima… - European Conference on …, 2024

… To operationalize our crawling pipeline, we introduce a re-crawl mechanism specifically for Common Crawl dumps. Unlike Common Crawl, we crawl on a daily basis and save the results as WARC (Web ARChive) files. Footnote 11 This …


The Power of Absence: Thinking with Archival Theory in Algorithmic Design

J Sherman, R Morrison, L Klein, DK Rosner - arXiv preprint arXiv:2405.05420, 2024

… , for example–but also of datasets like the Colossal Clean Common Crawl corpus, or C4, which explicitly documents its decision to include … [33] have shown how the quality filters imposed on the original Common Crawl dataset exhibit a preference …


The Practical Epistemologies of Design and Artificial Intelligence

W Billingsley - Science & Education, 2024

This article explores the epistemological trade-offs that practical and technology design fields make by exploring past philosophical discussions of design, practitioner research, and pragmatism. It argues that as technologists apply Artificial …


The principal components of meaning, revisited

C Westbury, M Yang, K Anderson - Psychonomic Bulletin & Review, 2024

… the Common Crawl corpus (https://commoncrawl.org/), a web-derived corpus. As its name suggests, the model has vectors of length 300. … ://nlp.stanford.edu/projects/glove/), a matrix with vectors of length 300 that was trained on 840 billion tokens from …


The research landscape on generative artificial intelligence: a bibliometric analysis of transformer-based models

Purpose The aim of this study is to offer valuable insights to businesses and facilitate better understanding on transformer-based models (TBMs), which are among the widely employed generative artificial intelligence (GAI) models, garnering …


The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources

S Longpre, S Biderman, A Albalak, H Schoelkopf… - arXiv preprint arXiv …, 2024

… These corpora often merge and deduplicate multiple years of common crawl scrapes, or supplement additional sources including biomedical text (PubMed), legal text (Freelaw, … 2023) provides a 3 trillion token, 2023 processed split of …


The Revealed Preferences of Pre-authorized Licenses and Their Ethical Implications for Generative Models

VM Suriyakumar, P Menell, D Hadfield-Menell…

This work examines the revealed preferences of creators as reflected in open and quasi-open licensing regimes based on the most commonly used licenses by copyright holders of images in the Creative Commons and copyright holders of code …


The Right Prompts for the Job: Repair Code-Review Defects with Large Language Model

Z Zhao, Z Xu, J Zhu, P Di, Y Yao, X Ma - arXiv preprint arXiv:2312.17485, 2023

Automatic program repair (APR) techniques have the potential to reduce manual efforts in uncovering and repairing program defects during the code review (CR) process. However, the limited accuracy and considerable time costs associated with …


The Right to Scrap Data on the Internet: From the US Case hiQLabs, Inc. v. LinkedIn Corp. to the ChatGPT Scraping Cases: Differences Between US and EU Law [pre …

JTS y López - Global Privacy Law Review, 2024

… a doctor when she was undergoing treatment for a rare genetic condition ended up in the Common Crawl’s archive and LAION’s dataset, and therefore in the GhatGPT datasets (because the Common Crawl is one of the ChatGPT training …


The Role and Prospects of Using ChatGPT Artificial Intelligence-Based Chatbots in Healthcare Improvement

NA Mumtaz - … OF INTERNATIONAL SEMINAR AND WORKSHOP ON …

In the age of digital transformation, healthcare is undergoing a revolution through information technology and artificial intelligence (AI). This article explores the role of AI-based chatbots, like ChatGPT, in healthcare enhancement. Utilizing a literature …


The Science of Data Filtering: Data Curation cannot be Compute Agnostic

S Goyal, P Maini, ZC Lipton, A Raghunathan, JZ Kolter - ICLR 2024 Workshop on …

… In fact, we show that even a model trained on *unfiltered common crawl* obtains higher accuracy than that trained on the LAION dataset post 40 or more repetitions. While past research in neural scaling laws has considered web data to be …


The science of implicit race bias: Evidence from the Implicit Association Test

KN Morehouse, MR Banaji - Daedalus, 2024

Beginning in the mid-1980s, scientific psychology underwent a revolution–the implicit revolution–that led to the development of methods to capture implicit bias: attitudes, stereotypes, and identities that operate without full conscious awareness …


The Self in Action: Narrating Agentic Moments

S Zilberstein, E Ayala-Hurtado, M Sanchez, D Robey - RSF: The Russell Sage …, 2024

This article develops a cultural and contextual approach to studying agency that attends to variation in how people narrate their experiences. Drawing on the large-scale, nationally representative American Voices Project data, the article uses …


The Shape of Word Embeddings: Recognizing Language Phylogenies through Topological Data Analysis

O Draganov, S Skiena - arXiv preprint arXiv:2404.00500, 2024

Word embeddings represent language vocabularies as clouds of $d$-dimensional points. We investigate how information is conveyed by the general shape of these clouds, outside of representing the semantic meaning of each token. Specifically, we …


The Silicone Ceiling: Auditing GPT's Race and Gender Biases in Hiring

L Armstrong, A Liu, S MacNeil, D Metaxa - arXiv preprint arXiv:2405.04412, 2024

Large language models (LLMs) are increasingly being introduced in workplace settings, with the goals of improving efficiency and fairness. However, concerns have arisen regarding these models' potential to reflect or exacerbate social biases …


The Sociolinguistic Foundations of Language Modeling

J Grieve, S Bartl, M Fuoli, J Grafmiller, W Huang… - arXiv preprint arXiv …, 2024

In this paper, we introduce a sociolinguistic perspective on language modeling. We claim that large language models are inherently models of varieties of language, and we consider how this insight can inform the development and deployment of …


The Tokenization Problem: Understanding Generative AI's Computational Language Bias

M Asprovska, N Hunter - Ubiquity Proceedings, 2024

… This paper used the Common Crawl corpus as a stand-in due to its wide linguistic representation, but the actual distribution within the pre-training dataset may differ. The discrepancies between these datasets might significantly influence tokenization …


The Trade-off between Performance, Efficiency, and Fairness in Adapter Modules for Text Classification

MD Bui, K von der Wense - arXiv preprint arXiv:2405.02010, 2024

… 2019) is derived from 393,423 online biographies in English from the Common Crawl corpus, each including the subject’s occupation and gender. The dataset contains 28 occupations, assuming a binary gender classification. Gender identification …


The use of machine learning and deep learning models in detecting depression on social media: A systematic literature review

WA Gadzama, D Gabi, MS Argungu, HU Suru - Personalized Medicine in Psychiatry, 2024

Depression is regarded as one of the world's primary concerns. Recent researchers use artificial intelligence techniques like machine learning and deep learning to identify depressive symptoms automatically. This literature review focuses on using …


The Web Data Commons Schema. org Table Corpora

R Peeters, A Brinkmann, C Bizer - Companion Proceedings of the ACM on Web …, 2024

… This paper presented the WDC Schema.org Table Corpora that we extracted from the Common Crawl. To the best of our knowledge, the corpora are the largest public table corpora that contain tables from many different sources which all share a …


The Web Is Missing an Essential Part of Infrastructure: An Open Web Index A proposal for building an index of the Web that separates the infrastructure part of the …

The web as it currently exists would not be possible without search engines. They are an integral part of the Web and can also be seen as a part of the Web’s infrastructure. Google alone now serves over two trillion search queries per year. 11 …


The work of art in the age of artificial intelligibility

The emergence of complex deep-learning models capable of producing novel images on a practically innumerable number of subjects and in an equally wide variety of artistic styles is beginning to highlight serious inadequacies in the ethical …


Themes and sentiments in conversations about food waste on Twitter: Proposal of a framework using neural topic modeling

MW Barbosa, A Gomes - Food Quality and Preference, 2024

… In this database, Twitter data was extracted from CommonCrawl. This method resulted in a collection spanning 80,000 distinct users. Historical snapshots from CommonCrawl, which spanned two years, were also considered to augment the …


Theoretical and Methodological Framework for Studying Texts Produced by Large Language Models

J Milička - arXiv preprint arXiv:2408.16740, 2024

This paper addresses the conceptual, methodological and technical challenges in studying large language models (LLMs) and the texts they produce from a quantitative linguistics perspective. It builds on a theoretical framework that …


Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications

SB Tete - arXiv preprint arXiv:2406.11007, 2024

The advent of Large Language Models (LLMs) has revolutionized various applications by providing advanced natural language processing capabilities. However, this innovation introduces new cybersecurity challenges. This paper …


Time for a Pause: Without Effective Public Oversight, AI in Schools Will Do More Harm Than Good.

B Williamson, A Molnar, F Boninger - 2024

Public education is a public and private good essential to democratic civic life. The public must, therefore, be able to provide meaningful direction over schools through transparent democratic governance structures. And yet artificial intelligence (AI) 1 …


Time Machine GPT

F Drinkall, E Rahimikia, JB Pierrehumbert, S Zohren - arXiv preprint arXiv:2404.18543, 2024

… To scale to even larger models, processing the annual Common Crawl datasets is a necessary step, though the dataset has proved problematic due to its scale and lack of … Aside from this, cleaning Common Crawl would also demand significant …


Timely Quality Problem Resolution in Peer-Production Systems: The Impact of Bots, Policy Citations, and Contributor Experience

V Mindel, A Aaltonen, A Rai, L Mathiassen, W Jabr - Information Systems Research, 2024

Although online peer-production systems have proven to be effective in producing high-quality content, their open call for participation makes them susceptible to ongoing quality problems. A key concern is that the problems should be addressed …


To Code, or Not To Code? Exploring Impact of Code in Pre-training

V Aryabumi, Y Su, R Ma, A Morisot, I Zhang, A Locatelli… - arXiv preprint arXiv …, 2024

Including code in the pre-training data mixture, even for models not specifically designed for code, has become a common practice in LLMs pre-training. While there has been anecdotal consensus among practitioners that code data plays a vital role …


To store or not: Online cost optimization for running big data jobs on the cloud

X Fu, L Pan, S Liu - Future Generation Computer Systems, 2024

As businesses increasingly rely on cloud-based big data analytics services to drive insights, reducing the cost of storing and analyzing large volumes of data in the cloud has become a major concern. During the execution of big data analysis jobs …


TOFU: A Task of Fictitious Unlearning for LLMs

P Maini, Z Feng, A Schwarzschild, ZC Lipton, JZ Kolter - arXiv preprint arXiv …, 2024

… LLMs trained on Common Crawl data1 may be able to correctly answer factual questions about this person and they may wish to have their data removed from an LLM. In fact, regulations around the Right to be Forgotten that focus on this situation …


TOMGPT: Reliable Text-Only Training Approach for Cost-Effective Multi-modal Large Language Model

Y Chen, Q Wang, S Wu, Y Gao, T Xu, Y Hu - ACM Transactions on Knowledge …, 2024

… For instance, c4 [37] dataset, a colossal and cleaned version of the web crawl corpus from Common Crawl, serves as a suitable option. Since the CLIP text encoder has limitations in processing long text, we subdivided the text in the c4 …


Tonirodriguez at CheckThat! 2024: Is it Possible to Use Zero-Shot Cross-Lingual Methods for Subjectivity Detection in Low-Resources Languages?

A Rodríguez, E Golobardes, J Suau - 2024

… on 2.5TB of filtered CommonCrawl data containing 100 languages. Following the work of XLM and RoBERTa, the XLM-RoBERTa or XLM-R model takes multilingual pretraining one step further by massively upscaling the training data [19]. Using the …


Tooling for big data extraction

R Beilich - HPI Future SOC Lab–Proceedings 2020, 2024

… , provided by Common Crawl, what may causes libraries to crumble and others to rise. … The Common Crawl Foundation created “an open repository of web crawl data that can be … [5] For the Common Crawl dataset this results in three types of …


Topic Modelling Swahili Using LDA and Contextualized Embeddings

DW Kahenya, W Mwangi, R Rimiru - 2024 Intelligent Methods, Systems, and …, 2024

… The XLMRoBERTa model, developed by [19] in their study on Unsupervised Cross-lingual Representation Learning at Scale, was trained on 2.5TB of filtered CommonCrawl data containing 100 languages. Flair framework [20] was used to …


Toward Better Education Quality through Students' Sentiment Analysis Using AutoML

C Simionescu, D Marcu, MS Măciucă - … . Broad Research in Artificial Intelligence and …, 2024

Sentiment analysis from students' interactions with learning environments is a topic of interest for researchers in the field of education because it can make important contributions to improving the quality of instructional processes through …


Toward Inference-optimal Mixture-of-Expert Large Language Models

L Yun, Y Zhuang, Y Fu, EP Xing, H Zhang - arXiv preprint arXiv:2404.02852, 2024

Mixture-of-Expert (MoE) based large language models (LLMs), such as the recent Mixtral and DeepSeek-MoE, have shown great promise in scaling model size without suffering from the quadratic growth of training cost of dense transformers …


Toward Robust Arabic AI-Generated Text Detection: Tackling Diacritics Challenges

H Alshammari, K Elleithy - Information, 2024

Current AI detection systems often struggle to distinguish between Arabic human-written text (HWT) and AI-generated text (AIGT) due to the small marks present above and below the Arabic text called diacritics. This study introduces robust Arabic text …


Toward Understanding Training Data Attacks In Artificial Intelligence And Machine Learning Models

… With all of these changes, the most accessible source of training data continues to be open source data, such as the Common Crawl datasets(Tom B. Brown, 2020) and others that have been made publicly available. Only extremely large technology …


Towards a Framework for Openness in Foundation Models: Proceedings from the Columbia Convening on Openness in Artificial Intelligence

A Basdevant, C François, V Storchan, K Bankston… - arXiv preprint arXiv …, 2024

Over the past year, there has been a robust debate about the benefits and risks of open sourcing foundation models. However, this discussion has often taken place at a high level of generality or with a narrow focus on specific technical attributes. In …


Towards a Scalable Geoparsing Approach for the Web

The ongoing surge in web data generation and storage, coupled with embedded geographic information, holds immense potential for enhancing search applications across diverse domains. However, extracting geographic information for further …


Towards a small language model powered chain‐of‐reasoning for open‐domain question answering

J Roh, M Kim, K Bae - ETRI Journal, 2024

… Unlike the ONUS approach [10], which extracts simple sub-question examples from a Common Crawl, our method emphasizes the autonomous generation of a dataset without human intervention facilitated by the use of LLMs. Consequently, we …


Towards a sociology of recurrent events. Constellations of cultural change around Eurovision in 18 countries (1981–2021)

L Carbone, J Mijs, T van Dooremalen, S Daenekindt - Poetics, 2024

… First, we used CMD with a corpus that was pre-trained on Common Crawl data using fastText English embeddings. Common Crawl is a vast and heterogeneous source of information that crawls web archives and freely provides their data to the …


Towards a Unified Terminology: A study on Siemens Energy Technical Information Systems

… The dataset includes the Common Crawl archive, an extensive collection of web-extracted text [5], as well as the WebText dataset, books … GPT-3’s training includes a staggering 400 billion tokens from Common Crawl, 19 billion from WebText, 12 …


Towards adaptive support for self‐regulated learning of causal relations: Evaluating four Dutch word vector models

HJ Pijeira‐Díaz, S Braumann, J van de Pol, T van Gog… - British Journal of …, 2024

Advances in computational language models increasingly enable adaptive support for self‐regulated learning (SRL) in digital learning environments (DLEs; eg, via automated feedback). However, the accuracy of those models is a common concern …


Towards Automated Solution Recipe Generation for Industrial Asset Management with LLM

N Zhou, D Patel, S Lin, F O'Donncha - arXiv preprint arXiv:2407.18992, 2024

This study introduces a novel approach to Industrial Asset Management (IAM) by incorporating Conditional-Based Management (CBM) principles with the latest advancements in Large Language Models (LLMs). Our research introduces an …


Towards automatic question generation using pre-trained model in academic field for Bahasa Indonesia

D Suhartono, MRN Majiid, R Fredyan - Education and Information Technologies, 2024

… This model is a potent tool for transforming our text data into numerical representations because it was previously trained on the sizable Common Crawl Dataset in Indonesian. These embeddings are essential for later modeling and …


Towards Better Statistical Understanding of Watermarking LLMs

Z Cai, S Liu, H Wang, H Zhong, X Li - arXiv preprint arXiv:2403.13027, 2024

In this paper, we study the problem of watermarking large language models (LLMs). We consider the trade-off between model distortion and detection ability and formulate it as a constrained optimization problem based on the green-red algorithm …


Towards Building Multilingual Language Model for Medicine

P Qiu, C Wu, X Zhang, W Lin, H Wang, Y Zhang… - arXiv preprint arXiv …, 2024

In this paper, we aim to develop an open-source, multilingual language model for medicine, that the benefits a wider, linguistically diverse audience from different regions. In general, we present the contribution from the following aspects: first, for …


Towards Chapterisation of Podcasts Detection of Host and Structuring Questions in Radio Transcripts

This Master thesis investigates the application of Bidirectional Encoder Representations from Transformers (BERT) on podcast to identify the host and detect structuring questions within each episode. This research is conducted on an …


Towards detection and explanation of factual inconsistencies

Factual inconsistencies in text, which include a range of errors from minor inaccuracies to substantial distortions, present a significant challenge in the realm of information dissemination. These inconsistencies, whether unintentional or the …


Towards Disfluency Annotated Corpora for Indian Languages

C Kochar, V Mujadia, P Mishra, DM Sharma

In the natural course of spoken language, individuals often engage in thinking and self-correction during speech production. These instances of interruption or correction are commonly referred to as disfluencies. When preparing data for …


Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models

J Wang, A Jatowt, Y Cai - arXiv preprint arXiv:2406.01863, 2024

In the evolving field of Natural Language Processing, understanding the temporal context of text is increasingly crucial. This study investigates methods to incorporate temporal information during pre-training, aiming to achieve effective time-aware …


Towards Estimating Personal Values in Song Lyrics

AM Demetriou, J Kim, S Manolios, C Liem - arXiv preprint arXiv:2408.12694, 2024

Most music widely consumed in Western Countries contains song lyrics, with US samples reporting almost all of their song libraries contain lyrics. In parallel, social science theory suggests that personal values - the abstract goals that guide our …


Towards Green AI. A methodological survey of the scientific literature

E Barbierato, A Gatti - IEEE Access, 2024

The pervasive deployment of Deep Learning models has recently prompted apprehensions regarding their ecological footprint, owing to the exorbitant levels of energy consumption necessitated by the training and inference processes. The term …


Towards interactive information seeking: conversational question answering

… To further advance the field, additional conversational search benchmarks have been introduced, such as ORQuAC [93], which extends CAsT and QuAC with more conversations constructed from Natural Questions and additional passages from …


Towards Large Language Models for Everyone: Instruction Following, Knowledge Retrieval and Multilingualism

Large language models (LLMs) have significantly advanced the field of Natural Language Processing and demonstrated the potential to fuel a variety of AI applications. Nonetheless, building them in a way that maximally benefits the very …


Towards Machine Translation Based on Monolingual Texts

… The CommonCrawl1 project carries out periodic web crawls and publishes the crawled data in an open … the CommonCrawl corpus. For example, the open source OSCAR2 project compiled a large multilingual corpus by language …


Towards More Factual Large Language Models: Parametric and Non-parametric Approaches

Large language models (LLMs) are increasingly important in assisting people to access information, ranging from simple factoid questions such as “where is the world’s largest ice sheet located” to complex questions that require accessing real-time …


Towards Privacy Preserving LLMs Training

B Buesser - Large Language Models in Cybersecurity: Threats …, 2024

Privacy-preserving training of machine learning models aims to avoid or minimize (mitigate) the exact or similar reproduction (leakage) of information contained in the training data. This chapter introduces pre-processing methods (filtering and de-duplication) …


Towards quantifying politicization in foreign aid project reports

S Wang, G Eggers, ART Georgiadis, TA Đo, L Gontard… - Proceedings of the Second …, 2024

We aim to develop a metric of politicization by investigating whether this concept can be operationalized computationally using document embeddings. We are interested in measuring the extent to which foreign aid is politicized. Textual reports of foreign …


Towards Region-aware Bias Evaluation Metrics

A Borah, A Garimella, R Mihalcea - arXiv preprint arXiv:2406.16152, 2024

… To measure biases in different data domains and regions, we extract region-aware topics using the GeoWAC dataset which spans Common Crawl separated by regions, and create a WEAT-style evaluation setup using these topics. Data. We consider two …


Towards reliability and interactive debugging for large language models

Large language models (LLMs) have permeated our everyday lives and are used in critical decision-making scenarios that can affect millions of people. Despite their impressive progress, model deficiencies may result in exacerbating harmful biases …


Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question Answering in the Era of Large Language Models

A Srivastava, A Memon - arXiv preprint arXiv:2406.13232, 2024

Open Domain Question Answering (ODQA) within natural language processing involves building systems that answer factual questions using large-scale knowledge corpora. Recent advances stem from the confluence of several factors …


Towards Systematic Monolingual NLP Surveys: GenA of Greek NLP

J Bakagianni, K Pouli, M Gavriilidou, J Pavlopoulos - arXiv preprint arXiv:2407.09861, 2024

Natural Language Processing (NLP) research has traditionally been predominantly focused on English, driven by the availability of resources, the size of the research community, and market demands. Recently, there has been a noticeable shift …


Towards the automatic creation of NER systems for new domains

E Matos, M Rodrigues, A Teixeira - Proceedings of the 16th International Conference …, 2024

… ner-roberta (ROBERTA): xlm-roberta-base-trimmed-pt-600002 model based in xlm-roberta-base, pretrained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Crosslingual …


Towards Video Understanding through Language in Real-life Settings

Videos have become an integral part of our daily lives, with a rapidly growing number on YouTube, Netflix, and TikTok serving as testimony to their widespread popularity. Behind the simplicity of their interfaces and user experiences, the …


Toxicity Detection for Free

Z Hu, J Piet, G Zhao, J Jiao, D Wagner - arXiv preprint arXiv:2405.18822, 2024

… An analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


Training and evaluation of vector models for Galician

M Garcia - Language Resources and Evaluation, 2024

This paper presents a large and systematic assessment of distributional models for Galician. To this end, we have first trained and evaluated static word embeddings (eg, word2vec, GloVe), and then compared their performance with that of current …


Training BERT Models to Carry over a Coding System Developed on One Corpus to Another

D Galambos, P Zsamboki - Proceedings of the 2024 Joint International Conference …, 2024

This paper describes how we train BERT models to carry over a coding system developed on the paragraphs of a Hungarian literary journal to another. The aim of the coding system is to track trends in the perception of literary translation around …


Training Data for the Price of a Sandwich1

… , this report sheds light on the implications of Common Crawl’s popularity by exploring the values that guide Common Crawl itself in-depth. … ) whether Common Crawl’s data had been used and b) what specific version of Common Crawl was …


Training Task Experts through Retrieval Based Distillation

J Ge, X Jia, V Viswanathan, H Luo, G Neubig - arXiv preprint arXiv:2407.05463, 2024

One of the most reliable ways to create deployable models for specialized tasks is to obtain an adequate amount of high-quality task-specific data. However, for specialized tasks, often such datasets do not exist. Existing methods address this by …


Transfer Learning in the Era of Large Language Models

Natural language processing has made great strides in recent years, largely driven by developments in large-scale neural networks trained on large quantities of textual data. This thesis is a collection of work studying transfer learning—the transfer of …


Transformer-Based Named Entity Recognition Model-Tamil Language Check for updates

K Dhayalan, N Sultanova, J Mustafina, P Daud - Data Science and Emerging Technologies …

This work presents different transformer-based approaches to the named entity recognition problem in the Tamil language. The WikiANN-ta dataset is used for training, validating and testing the model. Person, organization and location tags are …


Transformer-Driven Models for Language, Vision, and Multimodality

M Luo, T Gokhale, N Varshney, Y Yang, C Baral - Advances in Multimodal Information …, 2024

In this chapter, we will learn about the modeling and learning techniques that drive multimodal applications. We will focus specifically on the recent advances in transformer-based modeling for natural language understanding, and image …


Transformers and large language models in healthcare: A review

S Nerella, S Bandyopadhyay, J Zhang, M Contreras… - Artificial Intelligence in …, 2024

With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning …


Transformers and tradition: using Generative AI and Deep Learning for financial markets prediction

Artificial intelligence has revolutionized numerous industries, and financial markets are no exception. With the ability to process vast amounts of data quickly and accurately, AI algorithms have been increasingly used in finance to predict stock …


Transformers at HSD-2Lang 2024: Hate Speech Detection in Arabic and Turkish Tweets Using BERT Based Architectures

K Singhal, J Bedi - Proceedings of the 7th Workshop on Challenges and …, 2024

Over the past years, researchers across the globe have made significant efforts to develop systems capable of identifying the presence of hate speech in different languages. This paper describes the team Transformers’ submission to the subtasks …


Transformers for Detection of Distressed Cardiac Patients with an ICD Based on Danish Text Messages

JDW Andersen, ML Jensen, UK Wiil, S Skovbakke… - 2023 IEEE International …, 2023

Cardiac patients with implantable cardioverter defibrillator devices frequently exhibit signs of anxiety and depression (termed "Distressed"). Early detection of these patients is vital for evaluation, intervention, and prevention against relapse …


Transformers on Dynamical Systems-An Exploration of In-context Learning

Large Language Models (LLMs) have shown to be highly effective at performing in-context learning, where, given a prompt, the model can learn from the prompt and complete the sequence without needing to perform additional gradient steps or fine-tuning. In …


Transformers@ DravidianLangTech-EACL2024: Sentiment Analysis of Code-Mixed Tamil Using RoBERTa

K Singhal, J Bedi - Proceedings of the Fourth Workshop on Speech, Vision …, 2024

In recent years, there has been a persistent focus on developing systems that can automatically identify the hate speech content circulating on diverse social media platforms. This paper describes the team Transformers’ submission to the Caste/Immigration …


Transformers@ LT-EDI-EACL2024: Caste and Migration Hate Speech Detection in Tamil Using Ensembling on Transformers

K Singhal, J Bedi - Proceedings of the Fourth Workshop on Language …, 2024

In recent years, there has been a persistent focus on developing systems that can automatically identify the hate speech content circulating on diverse social media platforms. This paper describes the team “Transformers” submission to the Caste …


Transforming Conversational AI

We were motivated to write this book by the launch in November 2022 of ChatGPT and by the ensuing excitement and disruption across the world of Conversational AI. The book is written for a broad audience who are already working, starting to work …


Transforming Dutch: Debiasing Dutch Corefence Resolution Systems for Non-binary Pronouns

Gender-neutral pronouns are increasingly being introduced across Western languages, and are continuously more frequently being adopted by non-binary individuals. Recent evaluations have however demonstrated that English language …


TransGPT: Multi-modal Generative Pre-trained Transformer for Transportation

P Wang, X Wei, F Hu, W Han - arXiv preprint arXiv:2402.07233, 2024

Natural language processing (NLP) is a key component of intelligent transportation systems (ITS), but it faces many challenges in the transportation domain, such as domain-specific knowledge and data, and multi-modal inputs and outputs. This …


Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation

E Yang, D Lawrie, J Mayfield, DW Oard, S Miller - arXiv preprint arXiv:2401.04810, 2024

Prior work on English monolingual retrieval has shown that a cross-encoder trained using a large number of relevance judgments for query-document pairs can be used as a teacher to train more efficient, but similarly effective, dual-encoder student …


Translating and predicting document structure for medical domain scientific abstracts

SA Rauf, F Yvon - Computer Speech & Language, 2024

Abstract Machine Translation (MT) technologies have improved in many ways and generate usable outputs for a growing number of domains and language pairs. Yet, most sentence based MT systems struggle with contextual dependencies …


Translating QA is Enough: A Key to Unlocking In-context Cross-lingual Performance

S Kim, D Ki, Y Kim, J Lee - ICML 2024 Workshop on In-Context Learning

Multilingual large language models (MLLMs) have demonstrated significant cross-lingual capabilities through in-context learning. Existing approaches typically construct monolingual few-shot examples, either in the source or target language. However …


Translating scientific abstracts in the bio-medical domain with structure-aware models

SA Rauf, F Yvon - Computer Speech and Language, 2024

Machine Translation (MT) technologies have improved in many ways and generate usable outputs for a growing number of domains and language pairs. Yet, most sentence based MT systems struggle with contextual dependencies, processing …


Translation from Historical to Contemporary Japanese Using Japanese T5

H Usui, K Komiya - Proceedings of the Joint 3rd International Conference …, 2023

This paper presents machine translation from historical Japanese to contemporary Japanese using a Text-to-Text Transfer Transformer (T5). The result of the previous study that used neural machine translation (NMT), Long Short Term Memory (LSTM) …


TransLSTM: A hybrid LSTM-Transformer model for fine-grained suggestion mining

S Riaz, A Saghir, MJ Khan, H Khan, HS Khan, MJ Khan - Natural Language …, 2024

… Each model, excluding BERT, which employs BERT-BASE-Uncased, undergoes testing using simple text vectorization, GloVe, and Common Crawl. Notably, the GloVe word embedding surpasses common text vectorization and Common Crawl in …


TraQuLA: Transparent Question Answering Over RDF Through Linguistic Analysis

E Zimina, K Järvelin, J Peltonen, A Ranta… - International Conference on …, 2024

Answering complex questions over knowledge graphs has gained popularity recently. Systems based on large language models seem to achieve top performance. However, these models may generate content that looks reasonable …


Treading water: new data on the impact of AI ethics information sessions in classics and ancient language pedagogy

EAS Ross, J Baines - Journal of Classics Teaching, 2024

Over 2023, many universities and policy organisations in the higher education (HE) sector are working to create guiding principles and guidelines for the use of generative artificial intelligence (AI) in HE Teaching and Learning (T&L). Despite …


TriLM vs FloatLM: Ternary LLMs are more Performant than Quantized FP16 LLMs

A Kaushal, T Vaidhya, T Pandey, A Bhagat, I Rish - … 2024 Workshop on Foundation Models in …

Ternary LLMs offer significantly better performance for their size (measured in bits) than the models trained and deployed in FP16/BF16. Given the widespread usage of quantization before deployment and advancements in Post Training Quantization …


Trustworthy and Responsible AI for Human-Centric Autonomous Decision-Making Systems

F Dehghani, M Dibaji, F Anzum, L Dey, A Basdemir… - arXiv preprint arXiv …, 2024

Artificial Intelligence (AI) has paved the way for revolutionary decision-making processes, which if harnessed appropriately, can contribute to advancements in various sectors, from healthcare to economics. However, its black box nature …


Unboxing Occupational Bias: Grounded Debiasing LLMs with US Labor Data

A Gorti, M Gaur, A Chadha - arXiv preprint arXiv:2408.11247, 2024

Large Language Models (LLMs) are prone to inheriting and amplifying societal biases embedded within their training data, potentially reinforcing harmful stereotypes related to gender, occupation, and other sensitive categories. This issue …


Understanding College Students' Satisfaction With ChatGPT: An Exploratory And Predictive Machine Learning Approach Using Feature Engineering

K Pabreja, N Pabreja - MIER Journal of Educational Studies Trends and …, 2024

Artificial Intelligence (AI) technologies are continually improving and becoming more pervasive in many facets of our lives. ChatGPT is one such cutting-edge artificial intelligence application, and it has received a lot of worldwide media attention …


Understanding Emergent Abilities of Language Models from the Loss Perspective

Z Du, A Zeng, Y Dong, J Tang - arXiv preprint arXiv:2403.15796, 2024

Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) …


Understanding Large Language Models

Today, finding someone who hasn’t heard of ChatGPT, the AI chatbot that took the world by storm, is hard. ChatGPT—and its competitors such as Google Bard, Microsoft Bing Chat, etc.—are part of a broader area in AI known as large language …


Understanding latent affective bias in large pre-trained neural language models

A Kadan, P Deepak, S Bhadra, MP Gangan, VL Lajish - Natural Language …, 2024

Groundbreaking inventions and highly significant performance improvements in deep learning based Natural Language Processing are witnessed through the development of transformer based large Pre-trained Language Models (PLMs). The …


Understanding LLMs: A Comprehensive Overview from Training to Inference

Y Liu, H He, T Han, X Zhang, M Liu, J Tian, Y Zhang… - arXiv preprint arXiv …, 2024

… from the CommonCrawl. However, due to the presence of a substantial amount of low-quality data in web archives, preprocessing is essential when working with CommonCrawl data. Currently, four commonly used filtered datasets based on …


Understanding New Machine Learning Architectures: Practical Generative Artificial Intelligence for Anesthesiologists

Recent advances in neural networks have given rise to generative artificial intelligence, systems able to produce fluent responses to natural questions or attractive and even photorealistic images from text prompts. These systems were …


Understanding Privacy Risks of Embeddings Induced by Large Language Models

Z Zhu, N Shao, D Lian, C Wu, Z Liu, Y Yang, E Chen - arXiv preprint arXiv …, 2024

… CC-News (CommonCrawl News dataset) is a dataset containing news articles from international news websites. It contains 708241 English-language news articles published between January 2017 and December 2019. Pile-PubMed is a …


Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT

M Ali, S Panda, Q Shen, M Wick, A Kobren - arXiv preprint arXiv:2407.21058, 2024

… We experiment with English Wikipedia and the CC-100 English subset of Common Crawl. For each model size and type of training data, we compute biases in the representations upstream and performance disparities downstream …


UNIVERSAL IMITATION GAMES

In 1950, Alan Turing proposed a framework called an imitation game in which the participants are to be classified Human or Machine solely from natural language interactions. Using mathematics largely developed since Turing–category theory–we …


University Students' Attitudes and Perceptions on Learning English with ChatGPT

M Atakuziev, AM Gultom - Formosa Journal of Sustainable Research, 2024

This study examines 43 master's students' perceptions of ChatGPT in English language learning at Universitas Negeri Yogyakarta. Using a descriptive quantitative methodology, data was gathered via questionnaires and interviews …


Unlock the Power of Frozen LLMs in Knowledge Graph Completion

B Xue, Y Xu, Y Song, Y Pang, Y Ren, J Ding, L Fu… - arXiv preprint arXiv …, 2024

Classical knowledge graph completion (KGC) methods rely solely on structural information, struggling with the inherent sparsity of knowledge graphs (KGs). Large Language Models (LLMs) learn extensive knowledge from large corpora with …


Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset

H Laurençon, L Tronchon, V Sanh - arXiv preprint arXiv:2403.09029, 2024

… available online (and often captured in web crawls like Common Crawl) presents a tempting resource for generating pairs of screenshots and corresponding HTML codes by simply rendering the HTML and capturing the output. However, this …


UNLOCKING THE POWER OF LANGUAGE: IMPROVING MULTILINGUAL CAPABILITIES IN GENERATIVE AI FOR GLOBAL ACCESSIBILITY

Generative AI has emerged as a transformative technology with the potential to revolutionize global communication and accessibility. However, the current state of language support in generative AI applications presents both opportunities and …


Unmasking Memorization: Assessing Dutch Language Memorization in mT5 Models

This study investigates the memorization of Dutch language content in mT5 models, a multilingual variant of the T5 Transformer-based models. A fill-mask evaluation technique is used to assess memorization and how it varies across different model …


Unraveling the Mystery of Scaling Laws: Part I

H Su, Z Tian, X Shen, X Cai - arXiv preprint arXiv:2403.06563, 2024

Scaling law principles indicate a power-law correlation between loss and variables such as model size, dataset size, and computational resources utilized during training. These principles play a vital role in optimizing various aspects of model pre-training …


Unseen but influential associates: Properties of words' associates influence lexical and semantic processing

EJ Muraki, PM Pexman - Psychonomic Bulletin & Review, 2024

In many models of lexical and semantic processing, it is assumed that single word processing is a function of the characteristics of the words presented and the distributional properties of the words’ networks. Recent research suggests that …


Unsupervised approach for misinformation detection in Russia-Ukraine war news

N Khairova, A Galassi, FL Scudo, B Ivasiuk, I Redozub - 2024

The Russian-Ukrainian war has attracted considerable global attention; however, fake news often obstructs the formation of public opinion and disseminates false information. To address this issue, we have curated the RUWA dataset, comprising …


Unsupervised LLM Adaptation for Question Answering

K Saito, K Sohn, CY Lee, Y Ushiku - arXiv preprint arXiv:2402.12170, 2024

… Commoncrawl 2. The topics cover politics, sports, business, etc. To reduce the number of tokens, we ask ChatGPT to generate summaries. … We employ NewsFetch7 to crawl the news from Commoncrawl. The published date of the article …


Unsupervised Outlier Detection for Language-Independent Text Quality Filtering

J Daðason, H Loftsson - Proceedings of the 3rd Annual Meeting of the Special …, 2024

… Common Crawl (CC) is an organization that maintains a massive repository of data crawled … corpora, all of which are derived from Common Crawl. It consists of 6.3 trillion tokens in 167 … crawled sources, such as Common Crawl. The total size …


Untangling the Unrestricted Web: Automatic Identification of Multilingual Registers

E Henriksson, A Myntti, A Eskelinen… - arXiv preprint arXiv …, 2024

This article explores deep learning models for the automatic identification of registers - text varieties such as news reports and discussion forums - in web-based datasets across 16 languages. Web register (or genre) identification would provide a …


Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models

J Tian, D Zheng, Y Cheng, R Wang, C Zhang, D Zhang - arXiv preprint arXiv …, 2024

… (2024), who emphasize the influence of data quality and diversity in training models, our curated dataset incorporates sources such as Common Crawl, books, Wikipedia, code, and academic papers. Additionally, our dataset is multilingual, with …


Unveiling Gender Bias in Large Language Models: Using Teacher's Evaluation in Higher Education As an Example

Y Huang - arXiv preprint arXiv:2409.09652, 2024

This paper investigates gender bias in Large Language Model (LLM)-generated teacher evaluations in higher education setting, focusing on evaluations produced by GPT-4 across six academic subjects. By applying a comprehensive analytical …


UNVEILING THE COGNITIVE CAPACITY OF CHATGPT: ASSESSING ITS HUMAN-LIKE REASONING ABILITIES

… OpenAI utilizes a dataset termed Common Crawl, which is a collection of publicly available websites. This dataset includes billions of web pages and is one of the largest text datasets available. Meanwhile, this dataset is just a starting line. …


USDC: A Dataset of $\underline {U} $ ser $\underline {S} $ tance and $\underline {D} $ ogmatism in Long $\underline {C} $ onversations

M Marreddy, SR Oota, VC Chinni, M Gupta, L Flek - arXiv preprint arXiv:2406.16833, 2024

Identifying user's opinions and stances in long conversation threads on various topics can be extremely critical for enhanced personalization, market research, political campaigns, customer service, conflict resolution, targeted advertising, and …


Use of Large Language Models to Aid Analysis of Textual Data

LR Bentley, X Xia, JM Sitt, SC Fankhauser…

… The use of the data generated by the Common Crawl discussed earlier in this paper mitigates this limitation, drawing from the vast and growing body of content generated through the internet. The quality of the input data also plays a vital role in …


User Story Classification with Machine Learning and LLMs

P Chuor, A Ittoo, S Heng - … Conference on Knowledge Science, Engineering and …, 2024

We address the problem of classifying Capability, Task, Hard-goal, and Soft-goal in user stories. Such a classification is essential for generating Rationale Tree. Several articles have attempted to classify different aspects of user stories in the past …


Using Large Language Models to Probe Cognitive Constructs, Augment Data, and Design Instructional Materials

F Kieser, P Wulff - … in Educational Sciences: Approaches, Applications and …, 2024

… It was shown that transformer-based large language models (LLMs could be pretrained on large language data sets such as the Common Crawl of the Internet, the Books Corpus, or Wikipedia, and then further fine-tuned in specific tasks [9, 10] …


Using Large Language Models to Understand Leadership Perception and Expectation

Y Zhang, X Wang, Z Zhang, X Wang, X Ma, Y Wu… - 2024 IEEE International …, 2024

Using multimodal approaches for measuring propositions in natural language (the Fill-Mask Association Test, FMAT) and ChatGPT 4.0, based on the Five-Factor Model of personality traits, we examined how leaders are perceived and expected in …


Using Large Language Models to Understand Telecom Standards

A Karapantelakis, M Shakur, A Nikou, F Moradi… - arXiv preprint arXiv …, 2024

… They have billions of parameters and are trained on corpora such as Common Crawl [5]. For foundation models to adapt to application domains, such as telecommunications, prompt engineering and fine-tuning approaches are used …


Using LLMs for the Extraction and Normalization of Product Attribute Values

A Brinkmann, N Baumann, C Bizer - … Conference on Advances in Databases and …, 2024

Product offers on e-commerce websites often consist of a product title and a textual product description. In order to enable features such as faceted product search or to generate product comparison tables, it is necessary to extract structured attribute-value …


Using LLMs for the Extraction

ABDN Baumann, C Bizerb - Advances in Databases and Information Systems: 28th …


Using rhetorical strategies to design prompts: a human-in-the-loop approach to make AI useful

N Ranade, M Saravia, A Johri - AI & SOCIETY, 2024

… It consumed the text content of the popular CommonCrawl dataset of web pages from 2016 to 2019. Although OpenAI curated it to eliminate redundancy and improve quality, it is nominally 45-terabytes worth of compressed text data. …


Utilizing Large Language Models for Non-trained Binary Sentiment Classification

Prompt Return a VALID JSON object with key “sentiment”. Return sentiment must be “positive” or “negative”. Do not say anything other than JSON object result. There is no “neutral”,“mixed” answer. Answer must start with ‘1’. Classify the sentiment of the …


Utilizing Large Language Models in Ophthalmology: The Current Landscape and Challenges

P Chotcomwongse, P Ruamviboonsuk, A Grzybowski - Ophthalmology and Therapy, 2024

A large language model (LLM) is an artificial intelligence (AI) model that uses natural language processing (NLP) to understand, interpret, and generate human-like language responses from unstructured text input. Its real-time response capabilities …


UzRoberta: A pre-trained language model for Uzbek

R Davronov, F Adilova - AIP Conference Proceedings, 2024

… Common Crawl datasets, was generated distributed word representations for 157 different languages [8]. Author’s Uzbek Wikipedia model has 110K words, compared to -830K words in the Common Crawl … It is pre-trained on 2.5TB of filtered …


UZROBERTA: AN UZBEK LANGUAGE PRE-TRAINED MODEL

F Adilova, R Davronov, R Safarov - Universum: технические науки, 2023

… Distributed word representations for 157 distinct languages were created using the Wikipedia and Common Crawl databases [8]. The author's Uzbek Wikipedia model has 110K words, but the Common Crawl model only has 830K words. They …


Validating and Exploring Large Geographic Corpora

J Dunn - arXiv preprint arXiv:2403.08198, 2024

… Beginning with a 427 billion word corpus derived from the Common Crawl, three methods are used to improve the quality of sub-corpora representing specific language-country pairs … This paper starts with an existing web corpus derived …


VERB: Visualizing and Interpreting Bias Mitigation Techniques Geometrically for Word Representations

YAN ZHENG, CCM YEH, J WANG, WEI ZHANG - 2024

Complicated and massive datasets are becoming more and more commonly represented as vectorized embeddings. They are part of a de facto model for words in word vector embeddings such as Word2Vec [51] and GloVe [54], and are …


Verifying Claims About Metaphors with Large-Scale Automatic Metaphor Identification

K Aono, R Sasano, K Takeda - arXiv preprint arXiv:2404.01029, 2024

… Third, in this study, the examples were collected from CommonCrawl, which is a corpus of texts on the Web, and thus contains mostly written language. It is unknown whether similar results can be obtained when analyzing examples collected from a …


VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation

Y Song, Y Kim, M Iyyer - arXiv preprint arXiv:2406.19276, 2024

Existing metrics for evaluating the factuality of long-form text, such as FACTSCORE (Min et al., 2023) and SAFE (Wei et al., 2024), decompose an input text into "atomic claims" and verify each against a knowledge base like Wikipedia. These metrics are not …


Viewpoints Detection in Political Speeches

Politics has significant impacts on our daily lives. It influences laws, policies, and resources allocation. The Internet and social media confront readers with an overwhelming amount of political texts. Readers struggle to analyze the information …


ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model

LT Nguyen - arXiv preprint arXiv:2405.14141, 2024

… Leveraging a Transformer architecture and trained on a massive dataset of 2.5TB filtered CommonCrawl text across 100 languages, it employs a masked language modeling (MLM) objective to learn crosslingual representations. This approach …


ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models

TH Nguyen, AC Le, VC Nguyen - arXiv preprint arXiv:2404.11086, 2024

The rapid advancement of large language models (LLMs) necessitates the development of new benchmarks to accurately assess their capabilities. To address this need for Vietnamese, this work aims to introduce ViLLM-Eval, the …


VISION2UI: A Real-World Dataset with Layout for Code Generation from UI Designs

Y Gui, Z Li, Y Wan, Y Shi, H Zhang, Y Su, S Dong… - arXiv preprint arXiv …, 2024

… we opt to create our dataset on top of the Common Crawl3 dataset. The Common Crawl dataset is a vast collection of Web page data from … This partial dataset from the original Common Crawl dataset includes 3.35 billion pages, amounting to a total …


Visual Artists, Technological Shock, and Generative AI

The impact of generative AI (GenAI) programs on visual art is comparable to earlier historical moments of technological shock, when literary and visual artists grappled with unprecedented reproductive tools such as the printing press, photography, and …


Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey

J Huang, J Zhang, K Jiang, H Qiu, S Lu - arXiv preprint arXiv:2312.16602, 2023

Traditional computer vision generally solves each single task independently by a dedicated model with the task instruction implicitly designed in the model architecture, arising two limitations: (1) it leads to task-specific models, which require …


VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding

PNT Do, SQ Tran, PG Hoang, K Van Nguyen… - arXiv preprint arXiv …, 2024

… 250002 multilingual CommonCrawl XLM-Robertalarge … This model was trained on a Transformers-based masked language task using two terabytes of CommonCrawl data across more than a hundred languages. The model has two …


W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering

J Nian, Z Peng, Q Wang, Y Fang - arXiv preprint arXiv:2408.08444, 2024

… The corpus may comprise chuncked Wikipedia articles or from online sources such as Reddit and Common Crawl, while the question-answer pairs could be sourced from OpenQA platforms such as Quora and StackOverflow. The simple first-stage …


WaCadie: Towards an Acadian French Corpus

J Robichaud, P Cook - Proceedings of the 2024 Joint International Conference …, 2024

… We treat each post as one document, similar to one website’s HTML from the Common Crawl corpus. We refer to this corpus as r/acadie. … We used CommonCrawl’s extensive database and scraped New Brunswick (.nb.ca) websites …


WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset

J Qiu, H Lv, Z Jin, R Wang, W Ning, J Yu, CB Zhang… - arXiv preprint arXiv …, 2024

… Moreover, Common Crawl contains a large amount of unsafe content, such as toxic and pornographic … Common Crawl data, which includes data extraction, heuristic rule filtering, fuzzy deduplication, content safety filtering, and data quality …