Building a Large Japanese Web Corpus for Large Language Models

N Okazaki, K Hattori, H Shota, H Iida, M Ohi, K Fujii… - arXiv preprint arXiv …, 2024

… The corpus developed in this study is made from 21 snapshots of Common Crawl (from CC-MAIN-2020-40 to CC-MAIN-2023-23), and the size of the corpus after cleaning is 173,350,375 pages and 312,093,428,689 characters. To confirm the …


Building a Specialized ChatGpt-Like System

… be examined, which was sampled from a mix primarily made up of the Common Crawl dataset [11]. This dataset is large enough to train the … This means that Common Crawl and Books2 datasets are sampled less than once during training …


Building and AI Chatbot using LLM

In our digital world, where communication is key, the emergence of AI-powered chatbots has revolutionized the way we interact with technology. This project delves into the development of an AI chatbot utilizing Large Language Models (LLM), a …


Building and better understanding vision-language models: insights and future directions

H Laurençon, A Marafioti, V Sanh, L Tronchon - arXiv preprint arXiv:2408.12637, 2024

The field of vision-language models (VLMs), which take images and texts as inputs and output texts, is rapidly evolving and has yet to reach consensus on several key aspects of the development pipeline, including data, architecture, and training …


Building Better Datasets: Seven Recommendations for Responsible Design from Dataset Creators

W Orr, K Crawford - arXiv preprint arXiv:2409.00252, 2024

The increasing demand for high-quality datasets in machine learning has raised concerns about the ethical and responsible creation of these datasets. Dataset creators play a crucial role in developing responsible practices, yet their …


Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers

X Wei, S Moalla, R Pascanu, C Gulcehre - arXiv preprint arXiv:2406.16450, 2024

… Dataset We use the RefinedWeb dataset [44], a carefully curated subset of CommonCrawl, processed for optimal filtering and deduplication. It provides 600B tokens available for public use. As it is a substantial dataset, we shuffle, extract, and …


Building Question-Answer Data Using Web Register Identification

A Eskelinen, A Myntti, E Henriksson, S Pyysalo… - Proceedings of the 2024 …, 2024

… (2015), is a corpus of Finnish collected between 2015 and 2016 from Common Crawl and by crawling .fi domains. As all of these resources are Common Crawl based, in the post-processing stage we make sure that duplicate QA pairs are …


By Senior Judge Stephanie Domitrovich and Judge Herbert B. Dixon Jr.

CJJ Federal Judiciary - Judges Journal, 2024

… For example, the datasets used to train ChatGPT 3 included text from the following sources: (1) Common Crawl, containing petabytes of data collected from the internet since 2008; (2) WebText2, text from webpages with highly ranked Reddit …


Bye-bye Bias: What to Consider When Training Generative AI Models on Subjective Marketing Metrics

… However, these models are trained on vast data sets scraped from the Internet, such as Common Crawl or LAION, that contain little information on the types of perceptual measures marketing is often interested in, such as perceived brand …


Cache Me If You Can: The Case For Retrieval Augmentation in Federated Learning

A Muhamed, P Thaker, MT Diab, V Smith - Privacy Regulation and Protection in Machine …

We propose retrieval augmentation (RA) as an enhancement to federated learning (FL) that can improve privacy protection and ensure regulatory compliance. FL, primarily designed for data privacy preservation, faces challenges with conventional …


CAM 2.0: End-to-End Open Domain Comparative Question Answering System

A Shallouf, H Herasimchyk, M Salnikov, RG Veliz…

… After extracting objects and aspects, we look for their matches in the Common Crawl text. It is also important to mention, that for the final … (lemmatized) corpus containing 14.3 billion English sentences from the Common Crawl. To retrieve …


Can ChatGPT learn Chinese or Swahili? Considering how large language models might act differently if trained in different languages.

… Thien Nguyen, a professor of computer science at the University of Oregon, looked at the CommonCrawl corpus, a selection of text scraped … Then, because only a small percentage of the CommonCrawl corpus is in Korean, the company …


Can ChatGPT Plan Your Retirement?: Generative AI andFinancial Advice

We identify some of the most pressing issues facing the adoption of large language models (LLMs) in practical settings, and propose a research agenda to reach the next technological inflection point in generative AI. We focus on three challenges …


Can cross-domain term extraction benefit from cross-lingual transfer and nested term labeling?

HTH Tran, M Martinc, A Repar, N Ljubešić, A Doucet… - Machine Learning, 2024

Automatic term extraction (ATE) is a natural language processing task that eases the effort of manually identifying terms from domain-specific corpora by providing a list of candidate terms. In this paper, we treat ATE as a sequence-labeling task and …


Can Large Language Models Accelerate Digital Transformation by Generating Expert-Like Systems Engineering Artifacts? Insights from an Empirical Exploration

M Husain, P Wach, TG Topcu - The Proceedings of the 2024 Conference on Systems …

The old saga of complex system development continues as the vast majority of government and industry programs continue to result in cost and schedule overruns [1–7]. While numerous researchers and government reports attribute this trend to the ever-increasing …


Can large language models understand molecules?

S Sadeghi, A Bui, A Forooghi, J Lu, A Ngom - BMC bioinformatics, 2024

Purpose Large Language Models (LLMs) like Generative Pre-trained Transformer (GPT) from OpenAI and LLaMA (Large Language Model Meta AI) from Meta AI are increasingly recognized for their potential in the field of cheminformatics, particularly …


Can Models Learn Skill Composition from Examples?

H Zhao, S Kaur, D Yu, A Goyal, S Arora - ICML 2024 Workshop on LLMs and Cognition

As large language models (LLMs) become increasingly capable, their ability to exhibit *compositional generalization* of skills has garnered significant attention. Yu et al. 2023 recently introduced Skill-Mix evaluation, where models are tasked with …


Cascaded transformer-based networks for wikipedia large-scale image-caption matching

N Messina, DA Coccomini, A Esuli, F Falchi - Multimedia Tools and Applications, 2024

… It uses an XLM-RoBERTa masked language model pre-trained on the CommonCrawl dataset and fine-tuned on the image URL-caption match classification task. The CLS token in output from XLM-RoBERTa is attached to a feed-forward …


Causality-driven multivariate stock movement forecasting

A Díaz Berenguer, Y Da, MN Bossa, MC Oveneke… - PloS one, 2024

Our study aims to investigate the interdependence between international stock markets and sentiments from financial news in stock forecasting. We adopt the Temporal Fusion Transformers (TFT) to incorporate intra and inter-market …


CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl

I Ilyankou, J Haworth, S Cavazzi - arXiv preprint arXiv:2405.11039, 2024

The Common Crawl (CC) corpus is the largest open web crawl dataset containing 9.5+ petabytes of data captured since 2008. The dataset is instrumental in training large language models, and as such it has been studied for (un)desirable content …


CCSUM: A large-scale and high-quality dataset for abstractive news summarization

… Accordingly, among 35 million CommonCrawl News articles, we identify pairs of articles about the same news story and use one article’s first sentence as the summary for the other article. To ensure high quality, we apply strict filters whose …


CERM: Context-aware Literature-based Discovery via Sentiment Analysis

JC Young, U Akujuobi - arXiv preprint arXiv:2402.01724, 2024

Driven by the abundance of biomedical publications, we introduce a sentiment analysis task to understand food-health relationship. Prior attempts to incorporate health into recipe recommendation and analysis systems have primarily focused on …


Challenge design roadmap

HJE Balderas, I Guyon, A Howard, W Reade, S Treguer - AI Competitions and …, 2023

Challenges can be seen as a type of game that motivates participants to solve serious tasks. As a result, competition organizers must develop effective game rules. However, these rules have multiple objectives beyond making the game enjoyable …


Challenges and Contributions in Table-to-text Generation: A Survey

… The pre-training of T5 involves an unsupervised “span masking” objective applied to Common Crawl data, where spans of text are masked and the model learns to predict them, thus gaining a deep understanding of language patterns and …


Changes and challenges of legal education in the era of generative artificial intelligence: Chinese experience

W Wang, Z Xu, Z Xu - Journal of Infrastructure, Policy and Development, 2024

Using generative artificial intelligence systems in the classroom for law case analysis teaching can enhance the efficiency and accuracy of knowledge delivery. They can create interactive learning environments that are appropriate, immersive …


Chatbots in Airport Customer Service—Exploring Use Cases and Technology Acceptance

I Auer, S Schlögl, G Glowka - Future Internet, 2024

… Those pre-trained transformer systems like BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) have been trained on vast language corpora such as Wikipedia and Common Crawl …


ChatGPT Alternative Solutions: Large Language Models Survey

H Alipour, N Pendar, K Roy - arXiv preprint arXiv:2403.14469, 2024

In recent times, the grandeur of Large Language Models (LLMs) has not only shone in the realm of natural language processing but has also cast its brilliance across a vast array of applications. This remarkable display of LLM capabilities has ignited a …


ChatGPT and GPT-4: utilities in the legal sector

Artificial intelligence systems such as ChatGPT, the OpenAI chatbot, based on the language model family GPT (generative pre-trained transformers), as well as other solutions built on this technology and fine-tuned for specific tasks, have generated …


ChatGPT and Language Translation: A Small Case Study Evaluating English–Mandarin Translation

C Woodrum - International Conference on Human-Computer …, 2024

… has been informed speculation that it relies heavily on common crawl [13], a compendium of publicly available … common crawl. The \(15^{th}\) most common language, Indonesian, makes up less than 1% of the data set [14, 15]. It is important …


ChatGPT as the Marketplace of Ideas: Should Truth-Seeking Be the Goal of AI Content Governance?

J Zhang - arXiv preprint arXiv:2405.18636, 2024

As one of the most enduring metaphors within legal discourse, the marketplace of ideas has wielded considerable influence over the jurisprudential landscape for decades. A century after the inception of this theory, ChatGPT emerged as a …


ChatGPT in Cyber Onslaught and Fortification: Past, Present, and Future

M Gunda, V Manda, P Naradasu, S Mekala… - 2024 IEEE 9th International …, 2024

… GPT-1 was unveiled in 2018 and trained on a pair of data sets: over 11,000 books from Book Corpus and the enormous Common Crawl … With 1.5 billion features and a much more extensive and diversified training dataset comprising Common Crawl …


ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis

W Ahmed, M Saturno, R Rajjoub, AH Duey, B Zaidat… - European Spine Journal, 2024

… Furthermore, the largest dataset used to train both ChatGPT 3.5 and 4.0 is CommonCrawl, a platform that does not include PubMed and consists predominantly of open access articles, some of which may be of less rigorous if not …


ChatGPT vs Gemini vs LLaMA on Multilingual Sentiment Analysis

A Buscemi, D Proverbio - arXiv preprint arXiv:2402.01715, 2024

… It relied on a dataset referred to as the Common Crawl [30], a publicly accessible repository that comprises billions of web pages, making it one of the most extensive text databases available. It is important to note that the choice of dataset significantly …


ChatGPT's applications in marketing: a topic modeling approach

W Tafesse, A Wien - Marketing Intelligence & Planning, 2024

Purpose ChatGPT is a versatile technology with practical use cases spanning many professional disciplines including marketing. Being a recent innovation, however, there is a lack of academic insight into its tangible applications in the marketing …


ChatGPT, the perfect virtual teaching assistant? Ideological bias in learner-chatbot interactions

M Van Poucke - Computers and Composition, 2024

This paper examines ChatGPT's use of evaluative language and engagement strategies while addressing information-seeking queries. It assesses the chatbot's role as a virtual teaching assistant (VTA) across various educational settings. By …


ChatGPT-3.5 can create and justify short Afrikaans poems: Implications for practice

CJ van Staden - Suid-Afrikaanse Tydskrif vir Natuurwetenskap en …, 2024

ChatGPT-3.5 can create poems but the implications thereof has not yet been sufficiently investigated. The purpose of this explorative case study was to determine to which extent ChatGPT-3.5 can create and justify Afrikaans poems. Because …


ChatQA: Building GPT-4 Level Conversational QA Models

Z Liu, W Ping, R Roy, P Xu, M Shoeybi, B Catanzaro - arXiv preprint arXiv …, 2024

… We collect 7k documents (average ∼1k words per document) from common crawl, which cover a wide range of domains. Each document is used for generation of a single conversational QA sample, which leads to a total of 7k multi-turn QA dialogues …


Chatting Over Course Material: The Role of Retrieval Augmented Generation Systems in Enhancing Academic Chatbots.

Large Language Models (LLMs) have the potential to enhance learning among students. These tools can be used in chatbot systems allowing students to ask questions about course material, in particular when plugged with the so-called …


Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data

L Castro-Gonzalez, YL Chung, HR Kirk, J Francis… - arXiv preprint arXiv …, 2024

The field of machine learning has recently made significant progress in reducing the requirements for labelled training data when building new models. These `cheaper' learning techniques hold significant potential for the social sciences, where …


Check for Extending the Comparative Argumentative Machine: Multilingualism and Stance Detection

I Nikishina¹, A Bondarenko, S Zaczek¹, OL Haag… - Robust Argumentation Machines …

The comparative argumentative machine CAM can retrieve arguments that answer comparative questions questions that ask which of several to-be-compared options should be favored in some scenario. In this paper, we describe how we equipped …


Check for updates A Study of Transductive Graph-Based Regression

ADB Valejo - Intelligent Systems Design and Applications: Deep …

Regression methods play an important role in many realworld applications such as econometric, pattern recognition, and prediction of protein chains to cite a few tasks. In the semi-supervised transductive learning problem, some studies present graph-based …


Check for updates Digilog: Enhancing Website Embedding on Local Governments-A Comparative Analysis

J Gerber, B Kreiner, J Saxer, A Weiler - Foundations of Intelligent Systems: 27th International …

The ability to understand and process websites, known as website embedding, is crucial across various domains. It lays the foundation for machine understanding of websites. Specifically, website embedding proves invaluable when monitoring local …


CheckIT!: A Corpus of Expert Fact-checked Claims for Italian Jacopo Gili', Lucia Passaro² and Tommaso Caselli³ ¹Department of Computer Science, University of Turin …

LP Jacopo Gili, T Caselli - Proceedings of the 9th Italian Conference on …, 2024

This paper introduces CheckIT!, a resource of expert fact-checked claims, filling a gap for the development of fact-checking pipelines in Italian. We further investigate the use of three state-of-the-art generative text models to create variations of claims …


Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model

X Du, Z Yu, S Gao, D Pan, Y Cheng, Z Ma, R Yuan… - arXiv preprint arXiv …, 2024

In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional …


CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems

S Rosenthal, A Sil, R Florian, S Roukos - arXiv preprint arXiv:2404.02103, 2024

… However, they use sentencelevel matching (by encoding sentences for semantic similarity comparisons) to retrieve up to top 7 documents from Common Crawl while avoiding exact matches as the abstractive dataset. In the extractive version, the …


CLASSLA-web: Comparable Web Corpora of South Slavic Languages Enriched with Linguistic and Genre Annotation

N Ljubešić, T Kuzman - arXiv preprint arXiv:2403.12721, 2024

This paper presents a collection of highly comparable web corpora of Slovenian, Croatian, Bosnian, Montenegrin, Serbian, Macedonian, and Bulgarian, covering thereby the whole spectrum of official languages in the South Slavic language space …


Cleaner Pretraining Corpus Curation with Neural Web Scraping

Z Xu, Z Liu, Y Yan, Z Liu, C Xiong, G Yu - arXiv preprint arXiv:2402.14652, 2024

… The web-crawled datasets, such as Common Crawl, have been widely used for pretraining, facilitating the development of language … htmlparser serves as the text pre-extraction tool for CommonCrawl6 WET file (containing pre-extracted text), while …


CLEARNESS: Coreference Resolution for Generating and Ranking Arguments Extracted from Debate Portals for Queries

… Refining Arg-CTRL with data from Common Crawl leads to a higher quality of generated arguments compared to using user discussions from Reddit comments. In the mentioned studies, new arguments are generated through the use of knowledge …


CLIP and the City: Addressing the Artificial Encoding of Cities in Multimodal Foundation Deep Learning Models

In this project, we propose and explore a computational pipeline to examine urban cultural landscapes through the lens of artificial intelligence, and for questioning modes of embedding culture in machine learning models. By employing machine …


CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination

K Yang, T Gu, X An, H Jiang, X Dai, Z Feng, W Cai… - arXiv preprint arXiv …, 2024

Contrastive Language-Image Pre-training (CLIP) has achieved excellent performance over a wide range of tasks. However, the effectiveness of CLIP heavily relies on a substantial corpus of pre-training data, resulting in notable consumption …


Co-constructing AI Authoring Using Ethical Theories

B Jones - 88th Annual International Conference

We must show ethical humility, historically contextualizing ethical standards, as we implement AI tools in courses. The pace of change requires agile approaches to defining standards for the ethical use of AI. We invited students to co-construct a …


CoastTerm: a Corpus for Multidisciplinary Term Extraction in Coastal Scientific Literature

J Delaunay, HTH Tran, CE González-Gallardo… - arXiv preprint arXiv …, 2024

… – Multilingual pre-trained model: We opt for XLMR [9], a transformer-based model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. This multilingual version of RoBERTa, achieves benchmark performance in ATE for rich-resourced …


Community OSCAR: A Community Effort for Multilingual Web Data

M Brack, M Ostendorff, PO Suarez, JJ Saiz, IL Castilla…

The development of large language models (LLMs) relies heavily on extensive, highquality datasets. Publicly available datasets focus predominantly on English, leaving other language communities behind. To address this issue, we introduce …


Comparative Analysis of Cognitive Agreement between Human Analysts and Generative AI in Construction Safety Risk Assessment

U Ray, C Arteaga, JW Park - ISARC. Proceedings of the International Symposium on …, 2024

The construction industry struggles with safety risk assessment complexities due to evolving work environments, diverse labor forces, time constraints, regulatory intricacies, and inconsistent practices. While previous studies have highlighted the …


Comparative Analysis of Generative AI Tools in Enhancing Educational Engagement

F Borović, K Aleksić-Maslać, P Vranešić - 2024 47th MIPRO ICT and Electronics …, 2024

Technological development is continuously reshaping the traditional educational setting, presenting various possibilities for educational refinement. It is pivotal to motivate active engagement among all participants, and generative AI tools emerge …


Comparative evaluation of machine learning algorithms for phishing site detection

NF Almujahid, MA Haq, M Alshehri - PeerJ Computer Science, 2024

The advent of Internet technologies has resulted in the proliferation of electronic trading and the use of the Internet for electronic transactions, leading to a rise in unauthorized access to sensitive user information and the depletion of resources for …


Comparative Performance of Advanced NLP Models and LLMs in Multilingual Geo-Entity Detection

K Kopanov - Proceedings of the Cognitive Models and Artificial …, 2024

The integration of advanced Natural Language Processing (NLP) methodologies and Large Language Models (LLMs) has significantly enhanced the extraction and analysis of geospatial data from multilingual texts, impacting sectors such as …


Comparative Study on Synthetic and Natural Error Analysis with BART & MarianMT

R Rohit, SA Gandheesh, GS Sannala, PB Pati - 2024 IEEE 9th International …, 2024

Text is essential for communication, information sharing, knowledge acquisition, and analysis. It shapes public opinion, supports education, and drives online content, making it crucial in various domains. While there are various language models …


Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization

H Bansal, A Suvarna, G Bhatt, N Peng, KW Chang… - arXiv preprint arXiv …, 2024

A common technique for aligning large language models (LLMs) relies on acquiring human preferences by comparing multiple generations conditioned on a fixed context. This only leverages the pairwise comparisons when the generations are …


Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models

Z Zhao, N Aletras - arXiv preprint arXiv:2403.12809, 2024

In many real natural language processing application scenarios, practitioners not only aim to maximize predictive performance but also seek faithful explanations for the model predictions. Rationales and importance distribution given by feature …


Comparing Fine-Tuning, Zero and Few-Shot Strategies with Large Language Models in Hate Speech Detection in English

R Pan, JA García-Díaz, R Valencia-García

… This is a multilingual model based on RoBERTa, pre-trained with a 2.5 TB filtered CommonCrawl dataset containing 100 languages, using an MLM goal, where it randomly masks 15% of the words in the input, runs the entire masked sentence …


Comparing Foundations: Insights into the Construction of Financial Causal Knowledge Graphs with and without Ontology

Z Xu, R Ichise - 人工知能学会全国大会論文集 第 38 回 (2024), 2024

Simplicity in information navigation, interpretation, and reasoning has positioned mono-relation knowledge graphs (KGs) as a focus of attention, particularly in targeted scenarios. However, their deficiency in hierarchical interactions between …


Comparing generative and retrieval-based chatbots in answering patient questions regarding age-related macular degeneration and diabetic retinopathy

KX Cheong, C Zhang, TE Tan, BJ Fenner, WM Wong… - British Journal of …, 2024

… ChatGPT was trained on multiple extremely large datasets, of which Common Crawl (open repository of web crawl data) constituted the bulk of the datasets.30 31 Google Bard was trained on Infiniset (data comprising public forum dialogue …


Comparison of Common Crawl News & GDELT

A El Ouadi, D Beskow - 2024 IEEE International Systems Conference (SysCon), 2024

… Common Crawl data has proven valuable for research and training artificial intelligence, noting that 82% of raw tokens used to train GPT-3 come from the Common Crawl dataset. CC-News, a derived corpus of news data from Common …


Comparison of Machine Learning Algorithms and Large Language Models for Product Categorization

A İhsanoğlu, M Zaval, OT Yıldız - 2024 32nd Signal Processing and Communications …, 2024

This study explores the efficacy of traditional machine learning algorithms and Large Language Models (LLMs) in automating product categorization for online e-commerce platforms. By comparing these methodologies, we assess their performance in …


Compass: Large Multilingual Language Model for South-east Asia

S Maria - arXiv preprint arXiv:2404.09220, 2024

… We categorized these sources into seven types: CommonCrawl, C4, Wikipedia, WebText, Academic, Books, and Code. We upsampled and … We expect that the introduction of biases may originate from CommonCrawl, as this substantial dataset …


Competition concerns with foundation models: a new feast for big tech?

S Mitra - European Competition Journal, 2024

… Some well-known FMs such as GPT-3 and LLaMA have been trained on data from public sources (such as Common Crawl). However, there are concerns that publicly available data would eventually have been completely saturated for training …


Compilation of a Synthetic Judeo-French Corpus

I Nikolova-Stoupak, G Lejeune, E Schaeffer-Lacroix - Proceedings of the 8th Joint …, 2024

This is a short paper describing the process of derivation of synthetic Judeo-French text. Judeo-French is one of a number of rare languages used in speaking and writing by Jewish communities as confined to a particular temporal and …


Complex Word Identification for Italian Language: a dictionary-based approach

L Occhipinti - Sixth International Conference, 2024

… The model used in our study was trained on Wikipedia and Common Crawl datasets6. These embeddings provided vector representations for each word in our dataset, primarily comprising isolated items, allowing us to incorporate contextual …


Compositional Text-to-Image Generation with Dense Blob Representations

W Nie, S Liu, M Mardani, C Liu, B Eckart, A Vahdat - arXiv preprint arXiv:2405.08246, 2024

Existing text-to-image models struggle to follow complex text prompts, raising the need for extra grounding inputs for better controllability. In this work, we propose to decompose a scene into visual primitives - denoted as dense blob representations …


Comprehensive analysis of natural language processing

RK Yadav, A Madaan - Global Journal of Engineering and Technology …, 2024

… Huge datasets like the Penn Treebank and the Common Crawl drove the development of more sophisticated NLP models. Transfer learning techniques like BERT enabled finetuning of pre-trained models for specific tasks. TensorFlow and …


Comprehensive review and comparative analysis of transformer models in sentiment analysis

H Bashiri, H Naderi - Knowledge and Information Systems, 2024

Sentiment analysis has become an important task in natural language processing because it is used in many different areas. This paper gives a detailed review of sentiment analysis, including its definition, challenges, and uses. Different …


COMPREHENSIVE STUDY OF CLINICAL ENTITY EXTRACTION AND CLASSIFICATION USING LARGE LANGUAGE MODELS

M Faedi, P Torroni, DA Galassi, DG Grundler…

This project aims to evaluate various techniques for extraction and categorization of clinical terms in unstructured documents. The purpose of this study is to assist automated systems operating in the biomedical domain by highlighting relevant …


Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models

A Saxena, AK Bishwas, AA Mishra, R Armstrong - arXiv preprint arXiv:2407.15904, 2024

… For pruning language models, we used XLM-RoBERTA [30] trained on 2.5Tb of filtered CommonCrawl data containing 100 languages, BERTsmall [31] trained on disaster tweets dataset [32] and GPT-2 small [33] finetuned on E2E NLG [34]. …


Compressing Lengthy Context With UltraGist

P Zhang, Z Liu, S Xiao, N Shao, Q Ye, Z Dou - arXiv preprint arXiv:2405.16635, 2024

Compressing lengthy context is a critical but technically challenging problem. In this paper, we propose a new method called UltraGist, which is distinguished for its high-quality compression of lengthy context due to the innovative design of the compression and …


Compression Represents Intelligence Linearly

Y Huang, J Zhang, Z Shan, J He - arXiv preprint arXiv:2404.09937, 2024

… Concretely, for assessing knowledge and commonsense, we have compiled texts from the latest Common Crawl dataset. To evaluate coding ability, we have sourced data from GitHub repositories mainly on the Python language since the downstream …


Computational Collective Intelligence: 16th International Conference, ICCCI 2024, Leipzig, Germany, September 9–11, 2024, Proceedings, Part II

This volume contains the first part of the proceedings of the 16th International Conference on Computational Collective Intelligence (ICCCI 2024), held in Leipzig, Germany from 9–11 September 2024. The conference was organized in a hybrid …


Computational criminology: at-scale quantitative analysis of the evolution of cybercrime forums

Cybercrime forums and marketplaces are used by members to share hacking techniques, general community-building discussions, and trade hacking tools. While there is a large corpus of literature studying these platforms, from a cross-forum …


Computational Measures of Language Variation in Textual Utterances

The use of language as a tool for self-expression and social communication is marked by extensive variation in the way that language is used by different people and communities. The computational modeling of this variation studies the social …


Computerized diagnostic decision support systems–a comparative performance study of Isabel Pro vs. ChatGPT4

… ChatGPT4 is trained on Common Crawl, a publicly available dataset and one of the most extensive text datasets. While not trained explicitly for medical diagnosis, ChatGPT4 accesses medical textbooks, medical websites, medical papers, and …


Comquest: an Adaptive Crawler for User Comments on the Web

This thesis introduces Comquest, an adaptive framework designed for the large-scale collection and integration of user comments from the Web. User comments are featured on many websites and there is growing interest in mining and studying user …


Comquest: Large Scale User Comment Crawling and Integration

Z Chen, L He, A Mukherjee, E Dragut - 2024

… To train the model, we collect an extensive dataset from the Common Crawl Project. We filter the webpages that show apparent code signals (eg, loading the commenting system libraries) of the supported commenting systems. …


Confounders in Instance Variation for the Analysis of Data Contamination

B Mehrbakhsha, D Garigliottid, F Martínez-Plumeda…

… 2020) the same approach has been taken and all data points from the evaluation sets that had a 13-gram collision in the pre-training Common Crawl (C4) dataset were removed to tackle contamination. As contamination can involve minor …


Consent in Crisis: The Rapid Decline of the AI Data Commons

S Longpre, R Mahari, A Lee, C Lund, H Oderinwale… - arXiv preprint arXiv …, 2024

… In our study, we focus on three popular, open-source, and permissively licensed data sources which are derived from Common Crawl, the largest publicly available crawl of the web, which has collected and stored hundreds of billions of web pages …


Considering the Role of Fairness in Copyright Fair Use

Moral intuitions regarding when it is "fair" to engage in unauthorized copying or derivation from a copyrighted work may often take into account factors that the fair use statute, 17 USC § 107, does not enumerate among fair use considerations. This …


Conspiracy Theory Detection using Transformers with Multi-task and Multilingual Approaches

The COVID-19 pandemic sparked a new age of conspiracy theories in society. This has become an issue, especially since these theories are mixed in with reasonable arguments that criticize the measures taken by governments and their effects. A way …


Construction of Text Summarization Corpus in Economics Domain and Baseline Models

S Jumpathong, A Takhom, P Boonkwan… - 2024

… It was pre-trained with a data collection of 101 languages from Common Crawl Due to time and resource constraints, we chose mT5-small… It was pre-trained with a data collection of 101 languages from Common Crawl. Due to time and resource …


Context-aware embeddings for robust multiclass fraudulent URL detection in online social platforms

S Afzal, M Asim, MO Beg, T Baker, AI Awad, N Shamim - Computers and Electrical …, 2024

… Their dataset comprised 2 million URLs from Common Crawl and PhishTank, and they applied three-fold cross-validation with 90% training data. Feature-ranking techniques were used to identify the most important among 14 features. They …


Context-aware Transliteration of Romanized South Asian Languages

C Kirov, C Johny, A Katanova, A Gutkin, B Roark - Computational Linguistics, 2024

While most transliteration research is focused on single tokens such as named entities – eg, transliteration of “અમદાવાદ” from the Gujarati script to the Latin script “Ahmedabad” – the informal romanization prevalent in South Asia and elsewhere …


Contextual Chart Generation for Cyber Deception

DD Nguyen, D Liebowitz, S Nepal, SS Kanhere… - arXiv preprint arXiv …, 2024

… (1) HPN-T5 is based on Text-to-Text Transfer Transformer (T5) [42], a highly scalable sequence-to-sequence Transformer pre-trained on a large corpus harvested from Common Crawl. T5 introduced a novel pre-training objective that …


Contextual Word Embedding for Biomedical Knowledge Extraction: a Rapid Review and Case Study

D Vithanage, P Yu, L Wang, C Deng - Journal of Healthcare Informatics Research, 2024

Recent advancements in natural language processing (NLP), particularly contextual word embedding models, have improved knowledge extraction from biomedical and healthcare texts. However, limited comprehensive research compares these models …


Contextually Enriched Meta-Learning Ensemble Model for Urdu Sentiment Analysis Symmetry 2023, 15, 645

K Ahmed, MI Nadeem, D Li, Z Zheng, N Al-Kahtani… - 2023

… On the other hand, this model makes use of pre-trained word vectors that were trained on “common crawl” and “Wikipedia” through the use of the fastText model [82]. The CBOW algorithm with position-weighting is used to train this word vector. After …


Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

K Fujii, T Nakamura, M Loem, H Iida, M Ohi, K Hattori… - arXiv preprint arXiv …, 2024

Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an …


Controllable Sentence Simplification in Dutch

T Seidl, V Vandeghinste - Computational Linguistics in the Netherlands Journal, 2024

Text simplification aims to reduce complexity in vocabulary and syntax, enhancing the readability and comprehension of text. This paper presents a supervised sentence simplification approach for Dutch using a pre-trained large language …


Controllable Sentence Simplification in Swedish Using Control Prefixes and Mined Paraphrases

J Monsen, A Jönsson - Proceedings of the 2024 Joint International Conference …, 2024

… 2020), an opensource repository with tools to download and clean Common Crawl snapshots. Each file in the CC-100 corpus contains … The Common Crawl snapshots contain a large portion of low-quality data, including offensive and …


Copilot for Microsoft 365: A Comprehensive End-user Training Plan for Organizations

This thesis presents a comprehensive end user training plan for Copilot for Microsoft 365, a generative AI tool that integrates with Microsoft’s productivity suite of applications. The research is conducted within the context of Sulava Ltd., a Finnish …


Copyleft for Alleviating AIGC Copyright Dilemma: What-if Analysis, Public Perception and Implications

X Guo, Y Li, Y Peng, X Wei - arXiv preprint arXiv:2402.12216, 2024

As AIGC has impacted our society profoundly in the past years, ethical issues have received tremendous attention. The most urgent one is the AIGC copyright dilemma, which can immensely stifle the development of AIGC and greatly cost the entire …


CoRePooL—Corpus for Resource‐Poor Languages: Badaga Speech Corpus

HB Barathi Ganesh, G Jyothish Lal, R Jairam… - … Speech Recognition and …, 2024

This chapter presents a corpus named CoRePooL that stands for Corpus for Resource‐Poor Languages. As voice‐specific human‐machine interaction applications are accelerated by deep learning algorithms, the lack of resources …


Corpus Considerations for Annotator Modeling and Scaling

OO Sarumi, B Neuendorf, J Plepi, L Flek, J Schlötterer… - arXiv preprint arXiv …, 2024

Recent trends in natural language processing research and annotation tasks affirm a paradigm shift from the traditional reliance on a single ground truth to a focus on individual perspectives, particularly in subjective tasks. In scenarios where …


CorpusNÓS: A massive Galician corpus for training large language models

I de-Dios-Flores, SP Suárez, CC Pérez, DB Outeiriño… - Proceedings of the 16th …, 2024

CorpusNÓS is a massive Galician corpus made up of 2.1 B words primarily devised for training large language models. The corpus sources are varied and represent a relatively wide range of genres. CorpusNÓS is, to the best of our knowledge, the …


Correcting Auditory Spelling Mistakes in Jordanian Dialect Using Machine Learning Techniques

M Smadi, G Abandah - 2024 15th International Conference on Information and …, 2024

This paper explores the application of machine learning techniques, specifically leveraging an efficient transformer model known as Byt5, to rectify Arabic auditory spelling mistakes. The model is trained using datasets with synthetic errors …


Cost-Effective Big Data Orchestration Using Dagster: A Multi-Platform Approach

H Picatto, G Heiler, P Klimek - arXiv preprint arXiv:2408.11635, 2024

… The Common Crawl dataset is a particular promising avenue for this kind of research, as it also contains historic data which allows one to … The temporal partitioning aligns with the Common Crawl1 dataset used, facilitating efficient data …


Cost-Effective Event Mining on the Web via Event Source Page Discovery and Data API Construction

YH Lin, CH Chang, HM Chuang, XS Lin, T Yeh… - IEEE Access, 2024

… • Compared to the less than 1% of event pages found in the Common Crawl and Clueweb datasets when crawling the entire web, a significantly higher proportion of events, 13.3%, were identified when specifically targeting the websites of potential …


Cost-efficient prompt engineering for unsupervised entity resolution in the product matching domain

N Nananukul, K Sisaengsuwanchai, M Kejriwal - Discover Artificial Intelligence, 2024

… Web Data Commons (WDC) [52]: The full version of this dataset contains 26 million products and descriptions from different e-commerce websites scraped from the Web as part of the Web Data Commons (Common Crawl). The products are …


Counter (media) Intelligence and Visioning: An Interview with Adam Harvey

… Among the largest and most popular datasets used in AI research projects are Common Crawl (for NLP), and ImageNet and COCO (for computer vision), which are all derived from user-generated content. In my research project Exposing.ai, I’ve …


Crafting clarity: Leveraging large language models to decode consumer reviews

SV Praveen, P Gajjar, RK Ray, A Dutt - Journal of Retailing and Consumer Services, 2024

Large Language Models (LLMs) have emerged as powerful tools for understanding consumer perceptions and extracting insights from unstructured textual data. This study investigates the effectiveness of LLMs in comprehending consumer opinions …


Creating Ad Campaigns Using Generative AI Check for updates

A Bulut, B Arslan - Applications of Generative AI

Search campaigns consist of ad groups. An ad group contains a related set of keywords and ads. During an online campaign, search advertisers experiment with different marketing messages such as subtle vs. strong being used in ad copies, with …


Creating Ad Campaigns Using Generative AI

A Bulut, B Arslan - Applications of Generative AI, 2024

… In particular, T5 and PEGASUS models were trained on 750 GB of English-language text from the public Common Crawl web scrape while the BART model was trained on the CNN/Daily Mail dataset, which contains roughly 300K unique news articles …


Creating Parallel Corpora for Ukrainian: a German-Ukrainian Parallel Corpus (ParaRook|| DE-UK)

M Shvedova, A Lukashevskyi - Proceedings of the Third Ukrainian Natural Language …, 2024

Parallel corpora are currently a popular and vibrantly developing category of linguistic resources, used both in literature and translation studies, as well as in the field of NLP. For Ukrainian, though, there are still not enough significant parallel …


CroissantLLM: A Truly Bilingual French-English Language Model

M Faysse, P Fernandes, N Guerreiro, A Loison, D Alves… - arXiv preprint arXiv …, 2024

We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local …


Cross-cultural Inspiration Detection and Analysis in Real and LLM-generated Social Media Data

O Ignat, GG Lakshmy, R Mihalcea - arXiv preprint arXiv:2404.12933, 2024

Inspiration is linked to various positive outcomes, such as increased creativity, productivity, and happiness. Although inspiration has great potential, there has been limited effort toward identifying content that is inspiring, as opposed to just engaging …


Cross-Language Harmonization of Linguistic Resources

The presented work consists of two parts. In the first part I summarize the main directions of my research since the defense of my PhD thesis in 2005. I start with cross-language transfer of parsing models to languages that have little or no …


Cross-Lingual Cross-Modal Retrieval with Noise-Robust Fine-Tuning

R Cai, J Dong, T Liang, Y Liang, Y Wang, X Yang… - IEEE Transactions on …, 2024

… from snapshots of the CommonCrawl public dataset2. Experiments in this paper concern parallel data between English and other four languages, namely Chinese (ZH), French (FR), German (DE) and Czech (CZ). Evaluation metrics. … https://commoncrawl.org/ …


Cross-lingual Editing in Multilingual Language Models

H Beniwal, M Singh - arXiv preprint arXiv:2401.10521, 2024

The training of large language models (LLMs) necessitates substantial data and computational resources, and updating outdated LLMs entails significant efforts and resources. While numerous model editing techniques (METs) have emerged to …


Cross-lingual Transfer and Multilingual Learning for Detecting Harmful Behaviour in African Under-Resourced Language Dialogue

TO Ajayi, M Arcan, P Buitelaar - Proceedings of the 25th Annual Meeting of the …, 2024

Most harmful dialogue detection models are developed for high-resourced languages. Consequently, users who speak under-resourced languages cannot fully benefit from these models in terms of usage, development, detection and …


Cross-lingual transfer for the annotation of the SynSemClass ontology

This work compares two approaches to automatic preannotation of semantic class to verbs in a sentence for the purpose of adding a new language to the SynSemClass ontology. Both approaches rely on a multilingual deep learning classification model …


Cross-lingual transfer of multilingual models on low resource African Languages

H Thangaraj, A Chenat, JS Walia, V Marivate - arXiv preprint arXiv:2409.10965, 2024

… 2019) introduce XLM-R, a large-scale multilingual language model trained on 100 languages using two terabytes of CommonCrawl data. XLM-R offers better performance than models such as mBERT, particularly in low-resource languages such as …


Current strategies to address data scarcity in artificial intelligence-based drug discovery: A comprehensive review

A Gangwal, A Ansari, I Ahmad, AK Azad… - Computers in Biology and …, 2024

Artificial intelligence (AI) has played a vital role in computer-aided drug design (CADD). This development has been further accelerated with the increasing use of machine learning (ML), mainly deep learning (DL), and computing hardware and software …


Cyber Risks of Machine Translation Critical Errors: Arabic Mental Health Tweets as a Case Study

H Saadany, A Tantawy, C Orasan - arXiv preprint arXiv:2405.11668, 2024

With the advent of Neural Machine Translation (NMT) systems, the MT output has reached unprecedented accuracy levels which resulted in the ubiquity of MT tools on almost all online platforms with multilingual content. However, NMT systems, like …


Cyberbullying Detection Using PCA Extracted GLOVE Features and RoBERTaNet Transformer Learning Model

M Umer, EA Alabdulqader, AA Alarfaj, L Cascone… - IEEE Transactions on …, 2024

Online platforms are nurturing social interactions, yet regrettably, they have also led to the proliferation of antisocial behaviors such as cyberbullying, trolling, and hate speech on a global scale. The identification of hate speech and aggression has …


D3. 2: Report on Citizen discourses and attitudes towards controversies

P Troullinou, I Bonavita, FU TRI, FT ERI, S Stoyanov…

… Social media listening analysed data from the open web repository, CommonCrawl, for the period 2013-2021. The analysis indicated which … Social listening The findings of the analysis conducted on the CommonCrawl database will …


DANSK: Domain Generalization of Danish Named Entity Recognition

K Enevoldsen, ET Jessen, R Baglini - Northern European Journal of Language …, 2024

Named entity recognition is an important application within Danish NLP, essential within both industry and research. However, Danish NER is inhibited by a lack coverage across domains and entity types. As a consequence, no current models …


Data Analysis of Twitter's Nasdaq100 Sentiments and Topics as Indicators for News Articles Retrieval: Fine-Tuning RoBERTa and RAG

This study explores the combination of sentiment analysis with vader-lexicon and semantic analysis with latent dirichlet allocation to identify real-life events, particularly in the context of Twitter datasets. While sentiment analysis alone may not …


Data Analyst Competencies: A Theory-Driven Investigation of Industry Requirements in the Field of Data Analytics

CA Collier, AL Powell - Journal of Information Systems Education, 2024

… First, the job postings utilized in Dong and Triche’s study were not comprehensive but rather only those captured by the Common Crawl tool to facilitate the longitudinal nature of their study. Second, the wildcard searches used by Dong and Triche …


Data Augmentation and Large Language Model for Legal Case Retrieval and Entailment

MQ Bui, DT Do, NK Le, DH Nguyen, KVH Nguyen… - The Review of Socionetwork …, 2024

The Competition on Legal Information Extraction and Entailment (COLIEE) is a well-known international competition organized each year with the goal of applying machine learning algorithms and techniques in the analysis and understanding of legal …


Data augmentation for language generation inspired by machine translation

The field of natural language processing has witnessed a surge in the adoption of deep learning, which faces notable hurdles when the training data is scarce. This thesis aims to study automatic data augmentation for language generation tasks …


Data Augmentation with Semantic Enrichment for Deep Learning Invoice Text Classification

WW Chi, TY Tang, NM Salleh, M Mukred, H AlSalman… - IEEE Access, 2024

Natural language processing (NLP) is a research field that provides huge potential to automate accounting tasks dealing with text data. This research studies the application of NLP in automatically categorizing invoices based on the invoice text …


Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?

S Longpre, R Mahari, N Obeng-Marnu, W Brannon… - arXiv preprint arXiv …, 2024

… Common Crawl is accurate with wide coverage, but not detailed. Hugging Face can be inaccurate with varying levels of detail, but it has extensive coverage. The Data Provenance Initiative is highly accurate and detailed but is currently limited in scope. …


Data Collection Pipeline for Low-Resource Languages: A Case Study on Constructing a Tetun Text Corpus

G de Jesus, SS Nunes - Proceedings of the 2024 Joint International Conference …, 2024

This paper proposes Labadain Crawler, a data collection pipeline tailored to automate and optimize the process of constructing textual corpora from the web, with a specific target to low-resource languages. The system is built on top of Nutch, an …


Data Contamination Report from the 2024 CONDA Shared Task

O Sainz, I García-Ferrero, A Jacovi, JA Campos… - arXiv preprint arXiv …, 2024

… The reported corpora are mainly based on CommonCrawl snapshots, GitHub, or a mix of sources. For CommonCrawlbased corpora, there are 35 events reported for C4 (Raffel et al.… Ortiz Su’arez et al., 2019) and 6 for CommonCrawl (Rana…


Data Engineering for Scaling Language Models to 128K Context

Y Fu, R Panda, X Niu, X Yue, H Hajishirzi, Y Kim… - arXiv preprint arXiv …, 2024

We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in particular \textit{the ability to utilize information at arbitrary input …


Data Mixing Made Efficient: A Bivariate Scaling Law for Language Model Pretraining

C Ge, Z Ma, D Chen, Y Li, B Ding - arXiv preprint arXiv:2405.14908, 2024

Large language models exhibit exceptional generalization capabilities, primarily attributed to the utilization of diversely sourced data. However, conventional practices in integrating this diverse data heavily rely on heuristic schemes, lacking …


Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions

J Hayase, A Liu, Y Choi, S Oh, NA Smith - ICML 2024 Workshop on Foundation Models in …

… 2022), which is based on the Nov/Dec 2022 dump from Common Crawl. We consider the 112 languages with at least 1MB of data. … , containing English Wikipedia dumps from Jun-Aug 2022, Web, Common Crawl data that was de-duplicated …


Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?

J Hayase, A Liu, Y Choi, S Oh, NA Smith - arXiv preprint arXiv:2407.16607, 2024

… on the Nov/Dec 2022 dump from Common Crawl. We consider the 112 languages with at least 1 MB of data. … The training data is reportedly sourced from Common Crawl … November/December 2022 dump of Common Crawl. We only …


Data Selection for Generalization in Unimodal & Multimodal Models

In this thesis, I present research on improving datasets for deep learning models with automateddata transformation methods. The immediate goal of this work is to maximize the in-domainperformance, out-of-domain generalization, and robustness …


Data, Data Everywhere: A Guide for Pretraining Dataset Construction

J Parmar, S Prabhumoye, J Jennings, B Liu… - arXiv preprint arXiv …, 2024

… We close this gap by conducting a large-scale analysis on over 90 Common Crawl web snapshots for the attributes of domain, quality, toxicity, and type of speech. We then show how such data attributes can aid in pretraining set construction to …


Data-Centric Methods for Decentralizing Large Language Models

… from large web crawl services like Common Crawl. Web crawl is perceived to be wholly representative of activity on the Internet [Brügger… Indeed, web dumps like Common Crawl offer the promise of more diverse text than what is available in …


Data-to-Text Generation with Neural Language Models

Data-to-text generation systems need to produce texts with high levels of semantic accuracy. Rule-based systems can guarantee this aspect, but their fluency and adaptability to new domains remain limited. Meanwhile, neural language models …


DataComp Challenge

… billion sample image-text dataset filtered from a comparable pool of samples to Common Crawl. For the creation of the LAION-2B dataset, first language filtering is applied to the original Common Crawl pool so that only samples with English texts …


DataComp-LM: In search of the next generation of training sets for language models

J Li, A Fang, G Smyrnis, M Ivgi, M Jordan, S Gadre… - arXiv preprint arXiv …, 2024

… As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark …


DataSculpt: Crafting Data Landscapes for LLM Post-Training through Multi-objective Partitioning

K Lu, Z Liang, X Nie, D Pan, S Zhang, K Zhao, W Chen… - arXiv preprint arXiv …, 2024

… For instance, less than 5% of documents within CommonCrawl exceed 2k tokens in length [8], which hinders LLMs’ exposure to complex narratives. Despite these challenges, significant strides have been made toward enhancing LLMs’ capabilities …


Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum

H Pouransari, CL Li, JHR Chang, PKA Vasu, C Koc… - arXiv preprint arXiv …, 2024

… Although the benefits of ICLM with large-scale Common Crawl data (used in our experiments) are marginal in regular evaluation, it significantly boosts long-context evaluation metrics from 27.5 to 28.7 by constructing long data sequences through …


Dataset Growth

Z Qin, Z Xu, Y Zhou, Z Zheng, Z Cheng, H Tang… - arXiv preprint arXiv …, 2024

Deep learning benefits from the growing abundance of available data. Meanwhile, efficiently dealing with the growing data scale has become a challenge. Data publicly available are from different sources with various qualities, and it is …


Datasets for Large Language Models: A Comprehensive Survey

Y Liu, J Cao, C Liu, K Ding, L Jin - arXiv preprint arXiv:2402.18041, 2024

… The first method involves building upon Common Crawl1. Common Crawl is a massive, unstructured, multilingual web corpus that provides public access to web archives by regularly crawling and storing webpage data from the Internet. However …


Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models

H Park, S Lee, G Gim, Y Kim, D Kim, C Park - arXiv preprint arXiv:2403.19340, 2024

To address the challenges associated with data processing at scale, we propose Dataverse, a unified open-source Extract-Transform-Load (ETL) pipeline for large language models (LLMs) with a user-friendly design at its core. Easy addition of …


Dated Data: Tracing Knowledge Cutoffs in Large Language Models

J Cheng, M Marone, O Weller, D Lawrie, D Khashabi… - arXiv preprint arXiv …, 2024

… Our analysis reveals two reasons for these inconsistencies: (1) temporal biases of CommonCrawl data due to non-trivial amounts of old data in new dumps and (2) complications in LLM deduplication schemes involving semantic duplicates and …


DDK: Distilling Domain Knowledge for Efficient Large Language Models

J Liu, C Zhang, J Guo, Y Zhang, H Que, K Deng, Z Bai… - arXiv preprint arXiv …, 2024

Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands. Knowledge Distillation (KD) has emerged as an effective strategy to improve the …


DE-COP: Detecting Copyrighted Content in Language Models Training Data

AV Duarte, X Zhao, AL Oliveira, L Li - arXiv preprint arXiv:2402.09910, 2024

How can we detect if copyrighted content was used in the training process of a language model, considering that the training data is typically undisclosed? We are motivated by the premise that a language model is likely to identify verbatim …


Dear GPT-3: Collaborative Writing with Neural Networks

J Becker - Artificial Intelligence-Intelligent Art?: Human-Machine …, 2024

GPT-3 and I engaged in this dialogue about four months ago. I wanted to develop a collaborative writing project, not only to collectively realise a novel, but also to negotiate poetological questions which might arise during the process. These …


DeBERTa-BiLSTM: A multi-label classification model of arabic medical questions using pre-trained models and deep learning

BS Al-Smadi - Computers in Biology and Medicine, 2024

… Using data from CommonCrawl in 100 different languages, the authors in [38] introduce XLM-RoBERTa, an advanced masked multilingual language model. XLM-RoBERTa outperformed previous multilingual models such as XLM and mBERT. [39] …


Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Models

R Abbasi, MH Rohban, MS Baghshah - arXiv preprint arXiv:2407.05897, 2024

… In our experiments, we evaluate CLIP models trained on a diverse selection of datasets, including OpenAI’s private dataset, LAION, YFCC15m, CC12m, DataComp, DFN-5B, WebLI, and CommonCrawl. These models leverage a variety of backbone …


Decoding the Diversity: A Review of the Indic AI Research Landscape

S KJ, V Jain, S Bhaduri, T Roy, A Chadha - arXiv preprint arXiv:2406.09559, 2024

This review paper provides a comprehensive overview of large language model (LLM) research directions within Indic languages. Indic languages are those spoken in the Indian subcontinent, including India, Pakistan, Bangladesh, Sri Lanka, Nepal, and …


Deconstructing AI literacy for librarians

In this presentation, Samantha shares ways we can think about AI literacy-how it is similar and dissimilar from traditional information literacy, what we need to know about the mechanics of AI tools, and how SMU Libraries support the different users …


Deconstructing In-Context Learning: Understanding Prompts via Corruption

N Shivagunde, V Lialin, S Muckatira, A Rumshisky - arXiv preprint arXiv:2404.02054, 2024

The ability of large language models (LLMs) to "learn in context" based on the provided prompt has led to an explosive growth in their use, culminating in the proliferation of AI assistants such as ChatGPT, Claude, and Bard. These AI …


Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languages

Multilingual neural machine translation systems learn to map sentences of different languages into a common representation space. Intuitively, with a growing number of seen languages the encoder sentence representation grows more flexible and …


Deep learning aided clinical decision support

Medical professionals create vast amounts of clinical texts during patient care. Often, these documents describe medical cases from anamnesis to the final clinical outcome. Automated understanding and selection of relevant medical records pose …


Deep Learning Based Multi-document Summarization

In this era of rapidly advancing technology, the exponential increase of data availability makes analyzing and understanding text files a tedious, labor-intensive, and time-consuming task. Multi-document summarization (MDS) is an effective tool …


Deep Learning based Visually Rich Document Content Understanding: A Survey

Y Ding, J Lee, SC Han - arXiv preprint arXiv:2408.01287, 2024

Visually Rich Documents (VRDs) are essential in academia, finance, medical fields, and marketing due to their multimodal information content. Traditional methods for extracting information from VRDs depend on expert knowledge and manual labor …


Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlook

X Zou, Y Yan, X Hao, Y Hu, H Wen, E Liu, J Zhang, Y Li… - arXiv preprint arXiv …, 2024

As cities continue to burgeon, Urban Computing emerges as a pivotal discipline for sustainable development by harnessing the power of cross-domain data fusion from diverse sources (eg, geographical, traffic, social media, and environmental data) …


Deep Learning for Economists

M Dell - arXiv preprint arXiv:2407.15339, 2024

Deep learning provides powerful methods to impute structured information from large-scale, unstructured text and image datasets. For example, economists might wish to detect the presence of economic activity in satellite images, or to measure the topics or …


Deep Learning Impacts in the Field of Artificial Intelligence

R Gulwani, M Aggarwal - Deep Learning Concepts in Operations Research, 2024

This book chapter explores the profound impacts of deep learning on the field of artificial intelligence (AI). Deep learning has revolutionized various domains within AI, enabling significant advancements in computer vision, natural language processing …


Deep Learning Model for Tamil Part-of-Speech Tagging

H Visuwalingam, R Sakuntharaj, J Alawatugoda… - The Computer Journal, 2024

… These pre-trained word embeddings are trained on Common Crawl and Wikipedia. This model was trained using CBOW with position-weights, in dimension 300, with character n-grams of length 5, a window of size 5 and 10 negatives. …


DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

X Bi, D Chen, G Chen, S Chen, D Dai, C Deng, H Ding… - arXiv preprint arXiv …, 2024

… Our analysis revealed that deduplicating the entire Common Crawl corpus results in higher removal of duplicate instances compared to deduplicating within a single dump. Table 1 illustrates that deduplicating across 91 dumps eliminates four times …


DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

Q Zhu, D Guo, Z Shao, D Yang, P Wang, R Xu, Y Wu… - arXiv preprint arXiv …, 2024

… For the math corpus, we collect 221B math-related tokens sourced from CommonCrawl using the same pipeline, which approximately doubles the size of the 120B DeepSeekMath corpus (Shao et al., 2024), while for the natural language …


DeepSeek-VL: Towards Real-World Vision-Language Understanding

H Lu, W Liu, B Zhang, B Wang, K Dong, B Liu, J Sun… - arXiv preprint arXiv …, 2024

We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions: We strive to ensure our data is diverse …


DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Z Shao, P Wang, Q Zhu, R Xu, J Song, M Zhang, YK Li… - arXiv preprint arXiv …, 2024

… the DeepSeekMath Corpus from Common Crawl. As depicted in Figure 2, … Common Crawl, we employ URL-based deduplication and near-deduplication techniques, resulting in 40B HTML web pages. We then recall mathematical web …


Defining Boundaries: The Impact of Domain Specification on Cross-Language and Cross-Domain Transfer in Machine Translation

L Shahnazaryan, M Beloucif - arXiv preprint arXiv:2408.11926, 2024

… 2021) and mined from the Common Crawl corpus.While all languages used in our experiments, except for Italian, are considered bridge languages in the M2M-100 model, the lack of information on the exact data sizes mined for each language limits …


Deliverable 3.1: WP3 1st Interim technical report

1.1. Background This document is the first Interim Technical Report (deliverable 3.1) prepared within the WP3 New usecases for the ESSnet Trusted Smart Statistics–Web Intelligence Network project (ESSnet TSS-WIN). The report covers the period from …


Deriving the Meaning of Out-of-Vocabulary Words

… It consists of the articles that were collected using CommonCrawl[34]. The articles originated from five Czech news portals: … It is trained on more than 2TB of data from the filtered CommonCrawl[34] containing 100 languages. They report that the …


Design of Artificial Intelligence Companion Chatbot

With the development of cities and the prevalence of networks, interpersonal relationships have become increasingly distant. When people crave communication, they hope to find someone to confide in. With the rapid advancement of deep …


Design, Implementation and Evaluation of a Chatbot for Accounting Firm: A Fine-Tuning Approach With Two Novel Dataset

Artificial intelligence, particularly in the field of Chatbots, is fundamentally reshaping learning, communication, and work paradigms. This phenomenon has sparked growing interest among businesses, viewing Chatbots as a means to streamline …


Designing a course for pre-service science teachers using ChatGPT: what ChatGPT brings to the table

HZ Okulu, N Muslu - Interactive Learning Environments, 2024

ChatGPT holds significant potential for enhancing learning through integration into education as an advanced chatbot. With the goal of harnessing this potential, our research focused on exploring the utilization of ChatGPT in designing a course plan …


Designing an Intelligent System to Map Global Connections

E Bellamy, K Farrell, A Hopping, J Pinter, M Saju… - 2024

… Extended Abstract: This study develops a knowledge graph from the Common Crawl News Dataset to provide situational awareness and … We developed a data pipeline to extract semantic content from the Common Crawl News feed and filter it …


Designing and developing a dedicated Natural Language Processing Framework for Healthcare Information Technology Management and Assessment

The escalating complexity of the hospital environment, propelled by technological advancements, necessitates a comprehensive exploration of the integration and management of diverse tools and technologies in healthcare settings. In this context …


Designing Heterogeneous LLM Agents for Financial Sentiment Analysis

F Xing - arXiv preprint arXiv:2401.05799, 2024

… GPT-3.5 was trained mainly on the Common Crawl corpus [2], which archives the web. BLOOMZ was trained on an even larger Open-science Open-collaboration Text Sources corpus [19], which is mainly crowd-sourced scientific datasets. The five …


Detecting Dementia from Transcribed Speech in Slovak using Pre-trained BERT Models

J Staš, D Hládek, A Kopnický - 2024 34th International Conference Radioelektronika …, 2024

… TB of texts written in 100 languages from the Common Crawl dataset [14]; • RemBERT model6: a Rebalanced multilingual BERT model pre-trained on a large unlabeled text created by the combination of mC4 Common Crawl and Wikipedia …


Detecting Depression and Anxiety on Social Media Using Selective Masking

Mental health problems are one of the major problems in the world. It is estimated that once in their life, at least one mental health condition will affect one in four people. Depression is the most common condition, with 5\% of adults suffering from …


Detecting Hidden Meaning in Stock Images

OBJECTIVE − Automated extraction of hidden meaning in stock images− Is it possible to distinguish what is shown from what is meant?− Examine the divergence of text and image− Analyze the usage of stock images and textual descriptions on the web …


Detecting Machine-Generated News Using Fine-Tuned Transformers

… The dataset for this research was a collection of pre-generated GPT-2 news articles, along with real news articles gathered from the internet using Common Crawl. This data combined with the latest popular LLMs such as Llama, Gemma …


Detection of adversarial phishing attack using machine learning techniques

KM Sudar, M Rohan, K Vignesh - Sādhanā, 2024

The frequency of cyberattacks, particularly phishing attacks, is increasing exponentially every day. Many users fall victim to clicking on malicious URLs, leading to the exploitation of their information. The traditional methodologies …


Detection of bullying with MachineLearning: Using Supervised Machine Learning and LLMs to classify bullying in text

In recent years, there has been a increase in the issue of bullying, particularly in academic settings [1]. This degree project examines the use of supervised machine learning techniques to identify bullying in text data from school surveys provided by …


Detection of Hate Speech and Offensive Language CodeMix Text in Dravidian Languages using Cost-Sensitive Learning Approach

K Sreelakshmi, B Premjith, BR Chakravarthi… - IEEE Access, 2024

… It was trained using more than two terabytes of filtered CommonCrawl data. It has significantly improved performance on various cross-lingual transfer tasks and outperformed the mBERT model. XLM-R has 24 layers and 16 attention heads …


Detection of phishing addresses and pages with a data set balancing approach by generative adversarial network (GAN) and convolutional neural network (CNN) …

S Jafari, N Aghaee‐Maybodi - Concurrency and Computation: Practice and …

Phishing attacks have a remarkable ability to steal user information by using simple techniques. Phishing attacks steal valuable information, such as user names and passwords. The loss caused by phishing attacks is significant, and every year …


Detection of Sarcasm in Urdu Tweets using Deep Learning and Transformer based Hybrid Approaches

ME Hassan, M Hussain, I Maab, U Habib, MA Khan… - IEEE Access, 2024

Sarcasm has a significant role in human communication especially on social media platforms where users express their sentiments through humor, satire, and criticism. The identification of sarcasm is crucial in comprehending the sentiment and the …


Detection of Twitter Spam Using GLoVe Vocabulary Features, Bidirectional LSTM and Convolution Neural Network

P Manasa, A Malik, I Batra - SN Computer Science, 2024

… The training of the GloVe model involves a substantial text corpus, like Wikipedia or Common Crawl, which facilitates the learning of connections between words and phrases. For every word in the corpus, the model creates a vector, wherein each …


Developing and Deploying AI Applications on NVIDIA Jetson Orin NX: A Comprehensive Guide

The NVIDIA Jetson Orin NX is a cutting-edge platform that brings powerful AI capabilities to the edge, enabling a wide range of applications from real-time computer vision to advanced robotics. This comprehensive guide provides detailed …


Developing ChatGPT for Biology and Medicine: A Complete Review of Biomedical Question Answering

Q Li, L Li, Y Li - arXiv preprint arXiv:2401.07510, 2024

ChatGPT explores a strategic blueprint of question answering (QA) in delivering medical diagnosis, treatment recommendations, and other healthcare support. This is achieved through the increasing incorporation of medical domain data via natural …


Development and Evaluation of a German Language Model for the Financial Domain

N Kozaeva, S Hamotskyi, C Hänig - Proceedings of the Joint Workshop of the 7th …, 2024

… The German colossal, cleaned Common Crawl corpus15 was employed, comprising texts of varying lengths. Further the results of LM performance for mixed datasets (financial corpus mixed with common language sentences and financial …


Development and Evaluation of Pre-trained Language Models for Historical Danish and Norwegian Literary Texts

A Al-Laith, A Conroy, J Bjerring-Hansen… - Proceedings of the 2024 …, 2024

We develop and evaluate the first pre-trained language models specifically tailored for historical Danish and Norwegian texts. Three models are trained on a corpus of 19th-century Danish and Norwegian literature: two directly on the corpus with no …


Development of a Geographical Question-Answering System in the Kazakh Language

A Mukanova, A Barlybayev, A Nazyrova, L Kussepova… - IEEE Access, 2024

… Among the pre-trained models, the KazBERT model, trained on cleansed CommonCrawl data in the Kazakh language, achieved the most favorable outcomes. For the test dataset, the KazBERT model demonstrated an Exact Match (EM) …


Development of an Adaptive Multi-Domain Artificial Intelligence System Built using Machine Learning and Expert Systems Technologies

J Straub - arXiv preprint arXiv:2406.11272, 2024

Producing an artificial general intelligence (AGI) has been an elusive goal in artificial intelligence (AI) research for some time. An AGI would have the capability, like a human, to be exposed to a new problem domain, learn about it and then use …


Development of an Interface for Multi-Dimensional Clinical Language Analysis of Dementia and Assessment of its Market Viability

… This model, from Microsoft, is initialized from the transformer model XLM-RoBERTa-Large, which is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages [24]. CommonCrawl data consists of a comprehensive corpus of web …


Development of Natural Language Processing Tools and Resources for Assamese Text

It is my immense pleasure to convey my heartfelt gratitude to my supervisors, Prof. Sukumar Nandi and Prof. Priyankoo Sarmah, for their unwavering support and guidance during my research work. I am grateful to both of my supervisors for …


Diagonalization & Forcing FLEX: From Cantor to Cohen and Beyond

Background: Maybe very high energy cosmic radiation started a synaptic butterfly cascade 2... who knows? For one reason or another, I decided to embark on a journey of exploring and discussing the possible role of Artificial Intelligence (AI) on …


Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning

XY Liu, R Zhu, D Zha, J Gao, S Zhong, M Qiu - arXiv preprint arXiv:2312.17493, 2023

The surge in interest and application of large language models (LLMs) has sparked a drive to fine-tune these models to suit specific applications, such as finance and medical science. However, concerns regarding data privacy have emerged …


Diffused Seeing: The Epistemological Challenge

This article examines the transformation of the relationship between seeing and understanding in humans and machines by the technologies of machine learning known as ‘generative AI’. Taking Stable Diffusion as the main case study, while also …


Digilog: Enhancing Website Embedding on Local Governments-A Comparative Analysis

J Gerber, B Kreiner, J Saxer, A Weiler - International Symposium on Methodologies …, 2024

The ability to understand and process websites, known as website embedding, is crucial across various domains. It lays the foundation for machine understanding of websites. Specifically, website embedding proves invaluable when monitoring local …


Digital Protection and Innovative Development Path of Red Culture Resources Based on Distributed Machine Learning Supported by Intelligent Information

M Huang, X Zeng - Journal of Combinatorial Mathematics and …

As a product of the revolutionary war years, red culture, with its strong vitality, strong cohesion and extraordinary charm, with its incomparable positive energy, resists vulgar and flattering culture, promotes people to rebuild their faith, purify their minds …


Digital Threats

S Samtani, E Raff, H Anderson, E Domschot… - 2024

… First, we used the common crawl GloVe embedding, created by Stanford [28]. This embedding was trained over 42 billion tokens, with a 300-dimensional vector space, and was used as the embedding layer of the BiLSTM model. Results are …


DiPaCo: Distributed Path Composition

A Douillard, Q Feng, AA Rusu, A Kuncoro, Y Donchev… - arXiv preprint arXiv …, 2024

Progress in machine learning (ML) has been fueled by scaling neural network models. This scaling has been enabled by ever more heroic feats of engineering, necessary for accommodating ML approaches that require high bandwidth …


Disambiguating Homographs and Homophones Simultaneously: A Regrouping Method for Japanese

Y Sato - Proceedings of the 2024 Joint International Conference …, 2024

We present a method that re-groups surface forms into clusters representing synonyms, and help disambiguate homographs as well as homophone. The method is applied post-hoc to trained contextual word embeddings. It is beneficial to …


Disambiguating natural language via aligning meaningful descriptions

Artificial Intelligence (AI) technologies are increasingly pervading aspects of our lives. Because people use natural language to communicate with each other, computers should also use natural language to communicate with us. One of the principal …


Display options

Chemical patents are a commonly used channel for disclosing novel compounds and reactions, and hence represent important resources for chemical and pharmaceutical research. Key chemical data in patents is often presented in tables …


Dissecting Whiteness: consistencies and differences in the stereotypes of lower-and upper-class White US Americans

T Morgenroth, CT Begeny, TA Kirby, B Paaßen, Y Zeng - Self and Identity, 2024

Economic inequality is increasing in the United States, making categorization and stereotyping based on social class more likely. Yet, social class stereotypes have received relatively little attention. Focusing on spontaneously generated stereotypes …


Distance Comparison Operators for Approximate Nearest Neighbor Search: Exploration and Benchmark

Z Wang, H Xiong, Z He, P Wang - arXiv preprint arXiv:2403.13491, 2024

Approximate nearest neighbor search (ANNS) on high-dimensional vectors has become a fundamental and essential component in various machine learning tasks. Prior research has shown that the distance comparison operation is the bottleneck of …


Distortions in Judged Spatial Relations in Large Language Models

N Fulman, A Memduhoğlu, A Zipf - The Professional Geographer, 2024

We present a benchmark for assessing the capability of large language models (LLMs) to discern intercardinal directions between geographic locations and apply it to three prominent LLMs: GPT-3.5, GPT-4, and Llama-2. This benchmark specifically …


Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages

R van Noord, T Kuzman, P Rupnik, N Ljubešić… - arXiv preprint arXiv …, 2024

… rest of corpora compared in this paper, MaCoCu corpora are not obtained by processing Common Crawl data. Instead, a strategy consisting of crawling relevant internet top-level domains directly for the targeted languages is followed (eg, .al for …


Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks

The concept of localization in LLMs is often mentioned in prior work; however, methods for localization have never been systematically and directly evaluated. We propose two complementary benchmarks that evaluate the ability of localization …


Do Multilingual Large Language Models Mitigate Stereotype Bias?

S Nie, M Fromm, C Welch, R Görge, A Karimi, J Plepi… - arXiv preprint arXiv …, 2024

While preliminary findings indicate that multilingual LLMs exhibit reduced bias compared to monolingual ones, a comprehensive understanding of the effect of multilingual training on bias mitigation, is lacking. This study addresses this gap by …


Do not shut up and do dribble: social media and TV consumption

M Pazzona, N Spagnolo - Journal of Population Economics, 2024

… Footnote 11 The NLP model utilized in this study, XLM-RoBERTa, undergoes pre-training on 2.5TB of filtered CommonCrawl data encompassing 100 languages. For tweet classification, we employed the general-purpose Python library TweetNLP (Camacho-Collados …


Document-Level Machine Translation with Large-Scale Public Parallel Corpora

Despite the fact that document-level machine translation has inherent advantages over sentence-level machine translation due to additional information available to a model from document context, most translation systems continue to operate at a …


Documenting Geographically and Contextually Diverse Language Data Sources

A McMillan-Major, F De Toni, Z Alyafeai, S Biderman… - Northern European Journal …, 2024

… Typically, this data is collected from online sources, ranging from highly edited and structured text such as Wikipedia to the myriad text and audiovisual components of web pages, eg, collected by the Common Crawl Founda… Data from Common …


Does Context Help Mitigate Gender Bias in Neural Machine Translation?

H Gete, T Etchegoyhen - arXiv preprint arXiv:2406.12364, 2024

Neural Machine Translation models tend to perpetuate gender bias present in their training data distribution. Context-aware models have been previously suggested as a means to mitigate this type of bias. In this work, we examine this claim by …


Does Lack of Knowledge and Hardship of Information Access Signify Powerful AI? A Large Language Model Perspective

IA Zahid, SS Joudar - Applied Data Science and Analysis, 2023

Large Language Models (LLMs) are evolving and expanding enormously. With the consistent improvement of LLMs, more complex and sophisticated tasks will be tackled. Handling various tasks and fulfilling different queries will be more precise …


Does sentiment help in asset pricing? A novel approach using large language models and market-based labels

F Audrino, J Schüttler, F Sigrist - 2024

We present a novel approach to sentiment analysis in financial markets by using a state-ofthe-art large language model, a market data-driven labeling approach, and a large dataset consisting of diverse financial text sources including earnings call …


Does the Language Matter? Curriculum Learning over Neo-Latin Languages

G Pucci, L Ranaldi - Proceedings of the 2024 Joint International Conference …, 2024

Curriculum Learning (CL) is emerging as a relevant technique to reduce the cost of pre-training Large Language Models. The idea, tested for the English language, is to train LLMs by organizing training examples from the simplest to the most complex …


Does your data spark joy? Performance gains from domain upsampling at the end of training

C Blakeney, M Paul, BW Larsen, S Owen, J Frankle - arXiv e-prints, 2024

Pretraining datasets for large language models (LLMs) have grown to trillions of tokens composed of large amounts of CommonCrawl (CC) web scrape along with smaller, domain-specific datasets. It is expensive to understand the impact of these …


Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

L Soldaini, R Kinney, A Bhagia, D Schwenk, D Atkinson… - arXiv preprint arXiv …, 2024

… In total, we find 0.02% of documents in the 25 Common Crawl snapshots match this filter. … Head, Middle, and Tail parts of our Common Crawl data. The correlation is computed for 24M, … The correlation among the documents flagged for removal …


Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical Field

T Kerner - arXiv preprint arXiv:2407.14076, 2024

… CommonCrawl, or collecting multiple datasets that are at least slightly related to our domain. Large-scale webscraping is a timeand resource-intensive task, so it is recommended to consider using data from CommonCrawl … [3] Assuming we chose …


DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classification

AE Mahdaouy, S Lamsiyah, MJ Idrissi, H Alami… - arXiv preprint arXiv …, 2024

… This is a cleaned version of the Common Crawl’s web corpus, curated by the Allen Institute for Artificial Intelligence [38], containing … This dataset is compiled from CommonCrawl, using strict filtering and extensive deduplication [39], and …


Downstream bias mitigation is all you need

A Baksi, R Singh, T Joshi - arXiv preprint arXiv:2408.00612, 2024

The advent of transformer-based architectures and large language models (LLMs) have significantly advanced the performance of natural language processing (NLP) models. Since these LLMs are trained on huge corpuses of data from the web and …


DPHANet: Discriminative Parallel and Hierarchical Attention Network for Natural Language Video Localization

R Chen, J Tan, Z Yang, X Yang, Q Dai, Y Cheng, L Lin - IEEE Transactions on …, 2024

Natural Language Video Localization (NLVL) has recently attracted much attention because of its practical significance. However, the existing methods still face the following challenges: 1) When the models learn intra-modal semantic association …


Dr. Nahid Ebrahimi Majd

The issue of web phishing attacks has been more prevalent in recent years, and phishing is one of the riskiest online crimes that can have disastrous consequences. The aim of phishing is to collect confidential information by tricking a target and …


DrawL: Understanding the Effects of Non-Mainstream Dialects in Prompted Image Generation

JN Williams, M FitzMorris, O Aka, S Laszlo - arXiv preprint arXiv:2405.05382, 2024

… Stable Diffusion is trained in part using the LAION-5B dataset [50], which consists of text-image pairs from Common CrawlCommon Crawl includes archived data from a large variety of sources, including sites such as Reddit and a variety of blogs …


DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domain

Y Labrak, A Bazoge, OE Khettari, M Rouvier, N Grabar… - arXiv preprint arXiv …, 2024

The biomedical domain has sparked a significant interest in the field of Natural Language Processing (NLP), which has seen substantial advancements with pre-trained language models (PLMs). However, comparing these models has proven …


Dredge Word, Social Media, and Webgraph Networks for Unreliable Website Classification and Identification

EM Williams, P Carragher, KM Carley - arXiv preprint arXiv:2406.11423, 2024

In an attempt to mimic the complex paths through which unreliable content spreads between search engines and social media, we explore the impact of incorporating both webgraph and large-scale social media contexts into website credibility …


DsDm: Model-Aware Dataset Selection with Datamodels

L Engstrom, A Feldmann, A Madry - arXiv preprint arXiv:2401.12926, 2024

When selecting data for training large-scale models, standard practice is to filter for examples that match human notions of data quality. Such filtering yields qualitatively clean datapoints that intuitively should improve model behavior. However, in …


Dual Modalities of Text: Visual and Textual Generative Pre-training

Y Chai, Q Liu, J Xiao, S Wang, Y Sun, H Wu - arXiv preprint arXiv:2404.10710, 2024

… Meanwhile, The C4 dataset represents a substantial refinement of the Common Crawl corpus. This dataset, derived from the extensive Common Crawl web scrape, undergoes rigorous cleaning and preprocessing to ensure the quality and relevance …


Duth at semeval 2024 task 8: Comparing classic machine learning algorithms and llm based methods for multigenerator, multidomain and multilingual machine …

T Kyriakou, I Maslaris, A Arampatzis - Proceedings of the 18th International Workshop …, 2024

Text-generative models evolve rapidly nowadays. Although, they are very useful tools for a lot of people, they have also raised concerns for different reasons. This paper presents our work for SemEval2024 Task-8 on 2 out of the 3 subtasks. This …


Dynamic decoding and dual synthetic data for automatic correction of grammar in low-resource scenario

A Musyafa, Y Gao, A Solyman, S Khan, W Cai… - PeerJ Computer Science, 2024

… The data was collected from January–December 2018 using Commoncrawl snapshots in the CC-Net repository and organized into a single text document with a total data size of 36 GB. We selected the CC100 Indonesian corpus due to its open …


Dynamic Few-shot Learning for Computational Social Science

R Malla, TG Coan, V Srinivasan, C Boussalis

Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) and have recently demonstrated remarkable potential in social science research through their capacity to efficiently perform a wide range of text-as-data tasks …


Easy Problems That LLMs Get Wrong

S Williams, J Huckle - arXiv preprint arXiv:2405.19616, 2024

We introduce a comprehensive Linguistic Benchmark designed to evaluate the limitations of Large Language Models (LLMs) in domains such as logical reasoning, spatial intelligence, and linguistic understanding, among others. Through a series of …


Echoes of culture: Relationships of implicit and explicit attitudes with contemporary English, historical English, and 53 non-English languages

TES Charlesworth, K Morehouse, V Rouduri… - Social Psychological and …, 2024

Attitudes are intertwined with culture and language. But to what extent? Emerging perspectives in attitude research suggest that cultural representations in language are more related to implicitly measured (vs. explicitly measured) attitudes, and that …


Edge-LLMs: Edge-Device Large Language Model Competition

S Liu, K Han, A Fernandez-Lopez, A Jaiswal…

… The C4 dataset is a colossal, cleaned version of Common Crawl’s web crawl corpus, which is mainly intended to pre-train language models and word representations. Language models like MPT-7B and T5 are pre-trained with the C4 …


Effektivisering av fakturaklassificering enligt UNSPSC-standarden: en maskininlärningslösning

Procurement requirements and procurement analysis are approaches used by the Swedish National Agency for Public Procurement to ensure and maintain a sustainable societal development. The aim is to safeguard tax funds and ensure that …


Efficiency Comparison of Dataset Generated by LLMs using Machine Learning Algorithms

P Pawade, M Kulkarni, S Naik, A Raut, KS Wagh - 2024 International Conference on …, 2024

… • Bing Chat was trained on web text data, including Common Crawl, Wikipedia, news articles, social media posts, and other publicly available text, via a technique called denoising autoencoder (DAE). In DAE, the model is given a noisy version of a …


Efficient learning in spiking neural networks

A Rast, MA Aoun, EG Elia, N Crook - Neurocomputing, 2024

… Training data required is similarly massive: the CommonCrawl dataset is 242 TB in size, and hyperparameter tuning remains an expensive, search-based process that can require ∼ 1000 − 10 , 000 + sweeps. After evaluating a number of possible …


Efficient Model-Relational Data Management: Challenges and Opportunities

V Sanca, A Ailamaki - IEEE Transactions on Knowledge and Data …, 2024

As modern data pipelines continue to collect, produce, and store various data formats, extracting and combining value from traditional and context-rich sources becomes unsuitable for RDBMS. To tap into the dark data, domain experts analyze …


Efficient query clustering and information retrieval using Sequenced User Search Pattern Query Optimization

S Surya, P Sumitra - Multimedia Tools and Applications, 2024

The rapid growth of the internet has ushered in an era of unprecedented information availability. However, this expansion has also given rise to the formidable challenge of efficiently extracting relevant data from the vast expanse of web pages …


Efficient Training and Inference: Techniques for Large Language Models Using Llama

… The training dataset consisted of a mixture of the Common Crawl, Wikipedia, and BooksCorpus datasets, encompassing a diverse range of linguistic structures and vocabularies. For evaluation, the GLUE benchmark suite was utilized, providing a …


Efficient URL and URI Compression

F Savins, K Saric, GS Ramachandran, R Jurdak - 2024 33rd International Conference …, 2024

… The dataset explored was taken from the Common Crawl project and consists of 5 million URIs in 5 sets. The URIs were found to be 99.4% URLs with the rest being other URI formats such as “Javascript:” and “Tel:”. These URIs were selected …


Eir: Thai Medical Large Language Models

Y Thiprak, R Ngodngamthaweesuk… - arXiv preprint arXiv …, 2024

We present Eir Thai Medical LLM, a large language model with 8 billion parameters, specifically designed to enhance the accuracy of handling medical tasks in the Thai language. This model focuses on providing clear and easy-to-understand answers …


Eliciting Big Five Personality Traits in Large Language Models: A Textual Analysis with Classifier-Driven Approach

A Hilliard, C Munoz, Z Wu, AS Koshiyama - arXiv preprint arXiv:2402.08341, 2024

… GPT-3 followed, with 175 billion parameters and training on datasets like Common Crawl [15]. GPT-3.5 Turbo was introduced to enhance real-time performance, and the latest, GPT-4, boasts 1.76 trillion parameters and capabilities …


EMBA: Entity Matching using Multi-Task Learning of BERT with Attention-over-Attention

… The WDC Product Data Corpus for Large-scale Product Matching [34], was built by extracting product offers from Common Crawl. The WDC datasets serve as a popular benchmark and have been used for evaluation in DITTO, JointBERT, and …


Embedding-based Query Spelling Correction

For many retrieval systems, correcting spelling errors in the queries that searchers submit is an essential step of query understanding. Inspired by a blog post on spelling correction from 2018, we implement and analyze a simple embedding-based …


Emergent Abilities in Reduced-Scale Generative Language Models

S Muckatira, V Deshpande, V Lialin, A Rumshisky - arXiv preprint arXiv:2404.02204, 2024

Large language models can solve new tasks without task-specific fine-tuning. This ability, also known as in-context learning (ICL), is considered an emergent ability and is primarily seen in large language models with billions of parameters. This …


Emerging AI Tools for Education and Research: Perspective and Policies for IISc

The Director, IISc, constituted a committee on August 21, 2023 with the following mandate:(a) Explore the challenges and benefits of emerging AI tools in the context of academic teaching and learning,(b) provide guidance and recommendations to …


Emotion Analysis in Narrative Economics

The purpose of this quantitative cross-sectional design using a structural equation model was to determine if emotions mediate the viral spread of narratives with potentially far-reaching economic and political consequences. This dissertation …


Emotional significance in Cross-Cultural Semantic Crossmodal Correspondences

J Alvarado - Proceedings of the Annual Meeting of the Cognitive …, 2024

Crossmodal correspondences are associations between perceptual features from different senses that aid in crossmodal binding. The semantic coding of these correspondences is expected to capture and mediate the emergence of perceptual …


Employing Siamese MaLSTM Model and ELMO Word Embedding for Quora Duplicate Questions Detection

A Altamimi, M Umer, D Hanif, S Alsubai, TH Kim… - IEEE Access, 2024

… 3) FastText Subword FastText Subword comprises a collection of 2 million word vectors trained on the Common Crawl dataset, which consists of a massive 600 billion tokens. In contrast to traditional word embeddings, sub-word embeddings …


Empowering hate speech detection: leveraging linguistic richness and deep learning

IGBJ Abasan, EB Setiawan - Bulletin of Electrical Engineering and Informatics, 2024

… What they don’t realize is the use of pre-trained models, namely CommonCrawl and Wiki, in extraction using FastText or GloVe. The pre-trained model used is not specific for hate speech detection and, of course, consists of many languages, and …


Empowering Humans with Machines: From Explanations to Teaching

… This dataset contains hundreds of thousands of online biographies from the Common Crawl corpus. The task is to predict a person’s profession given a biography. The original dataset consists of 29 professions, and we narrow it down to …


Empowering knowledge through AI: open scholarship proactively supporting well trained generative AI

… The bulk of the content consists of the common crawl dataset, a free-to-use dataset consisting of over 250 billion webpages from the last 17 … Both the common crawl dataset and the WebText2 dataset rely heavily on well-linked and well-described …


Enabling action crossmodality for a pretrained large language model

A Caesar, O Özdemir, C Weber, S Wermter - Natural Language Processing Journal, 2024

Natural language processing and vision tasks have seen large improvements recently through the rise of Transformer architectures. The high performing large language models (LLMs) benefit from large textual datasets that are numerously …


Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment

A Agarwalla, A Gupta, A Marques, S Pandit, M Goin… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have revolutionized Natural Language Processing (NLP), but their size creates computational bottlenecks. We introduce a novel approach to create accurate, sparse foundational versions of performant LLMs that achieve full …


Enabling self-identification in intelligent agent: insights from computational psychoanalysis

L Li, C Li - arXiv preprint arXiv:2403.07664, 2024

Building upon prior framework of computational Lacanian psychoanalysis with the theory of active inference, this paper aims to further explore the concept of self-identification and its potential applications. Beginning with two classic paradigms in psychology …


End to End Urdu Abstractive Text Summarization with Dataset and Improvement in Evaluation Metric

Urdu, being a common language in South Asia, has not received significant attention in terms of language processing compared to more advanced languages. In the field of Natural Language Processing (NLP), the task of text summarization …


Energy and AI

J Göpfert, JM Weinand, P Kuckertz, D Stolten

… Because models can be trained on this objective in a self-supervised setting, large unlabeled corpora (such as Wikipedia, book corpora, and Common Crawl data) can be used for training. The computational advantages of the transformer …


English-language abstract text summarization using the T5 model

RD Darshan, I Surya, G Malarselvi - AIP Conference Proceedings, 2024

… The Common Crawl public web scrape is the source for the 750GB of English-language text that makes up the C4 dataset. The news summary dataset that we've used in this instance has 4515 samples and includes the following information: Headlines, Short …


EnhancedBERT: A Feature-rich Ensemble Model for Arabic Word Sense Disambiguation with Statistical Analysis and Optimized Data Collection

S Kaddoura, R Nassar - Journal of King Saud University-Computer and …, 2024

Accurate assignment of meaning to a word based on its context, known as Word Sense Disambiguation (WSD), remains challenging across languages. Extensive research aims to develop automated methods for determining word senses in …