Inference-Based No-Learning Approach on Pre-Trained BERT Model Retrieval

HL Pham, R Mibayashi, T Yamamoto, MP Kato… - 2024 IEEE International …, 2024

In recent years, the practice of leveraging pre-trained machine learning models for specific tasks has gained traction. Instead of training models from the ground up, it is now common to fine-tune existing pre-trained models. However, users have the …


Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks

N Yax, PY Oudeyer, S Palminteri - arXiv preprint arXiv:2404.04671, 2024

This paper introduces PhyloLM, a method applying phylogenetic algorithms to Large Language Models to explore their finetuning relationships, and predict their performance characteristics. By leveraging the phylogenetic distance metric, we …


Information Retrieval Chatbot on Military Policies and Standards

C Gunasekara, A Sharafeldin, M Triff, Z Kabir…

In the Canadian Armed Forces (CAF), navigating through extensive policies and standards can be a challenging task. To address the need for streamlined access to these vital documents, this paper explores the usage of artificial intelligence (AI) and …


Information Retrieval with Dense and Sparse Representations

… However, they use 1TB of the training data from Common Crawl dumps while our model only use 115MB of the Wikipedia data for pretraining. We put their scores in Table 2.2 for reference. In SimCSE, the authors propose to use MLM as an auxiliary …


Initial exploration into sarcasm and irony through machine translation

ZL Chia, M Ptaszynski, M Karpinska, J Eronen, F Masui - Natural Language …, 2024

In this paper, we investigate sarcasm and irony as seen through a novel perspective of machine translation. We employ various techniques for translation, comparing both manually and automatically translated datasets of irony and sarcasm. We first …


InjectBench: An Indirect Prompt Injection Benchmarking Framework

… Many datasets such as CommonCrawl cover a much wider distribution of the types of website online compared to the few datasets we have selected. However, we have chosen to use the following datasets to better understand and emphasize …


Innovative Use of Self-Attention-Based Ensemble Deep Learning for Suicide Risk Detection in Social Media Posts

HS Choi, J Yang - Applied Sciences, 2024

… Specifically, we employed the extensive corpus variant, known as “Common Crawl (840 B tokens, 2.2 M vocab, cased, 300 d vectors)”, which encompasses an impressive 840 billion tokens, a vocabulary of 2.2 million cased terms, and provides …


Inside the Black Box: Detecting Data Leakage in Pre-trained Language Encoders

Y Xin, Z Li, N Yu, D Chen, M Fritz, M Backes, Y Zhang - arXiv preprint arXiv …, 2024

Despite being prevalent in the general field of Natural Language Processing (NLP), pre-trained language models inherently carry privacy and copyright concerns due to their nature of training on large-scale web-scraped data. In this paper, we pioneer a …


Inspecting and Measuring Fairness of unlabeled Image Datasets

R Görge, M Mock, M Akila - 2024 IEEE 40th International Conference on Data …, 2024

Bias in training data can lead to algorithmic unfairness in machine learning tasks. Therefore, a general requirement for trustworthy AI is that data should be representative and free of bias. There are several approaches to measure fairness …


Instruction-tuned Language Models are Better Knowledge Learners

Z Jiang, Z Sun, W Shi, P Rodriguez, C Zhou, G Neubig… - arXiv preprint arXiv …, 2024

… However, its scope is limited to Wikipedia, which restricts the trained models’ adaptability to other sources like web pages from Common Crawl or scientific documents from arXiv. We focus on eliciting factual knowledge with instruction-tuning …


Integrating Generative AI Literacy into the Information Retrieval Course at a university in Canada: towards critical evaluation of online search results

… • Trained on information (billions of words) on the open web prior to 2021 • Among other open sources, dataset comes from Common Crawl (crawls the web) and Wikipedia • Length of answers limited - between 500 and 700 words, “leaving …


Intelligent Pharmacy

The modern language generation model ChatGPT, created by Open Artificial Intelligence (AI), is recognised for its capacity to comprehend context and produce pertinent content. This model is built on the transformer architecture, which enables …


Intelligent Phishing Website Detection Model Powered by Deep Learning Techniques

U Chetachi, O Henry, OA Agbugba - Asian Journal of Research in Computer …, 2024

… Approximately one million authentic and fraudulent URLs were employed in the dataset gathered from PhishTank and Common Crawl. Over 10,000 images and one million URLs were used in training for the CNN classifier and LSTM to create the …


Intercity relationships between 293 Chinese cities quantified based on toponym co-occurrence

W Tongjing, Z Yin, Z Bao, E Meijers - Cybergeo: European Journal of Geography, 2024

… 14The primary dataset of this study was obtained from the Common Crawl, a web archive that has periodically crawled the Internet since … We used the entire Common Crawl text archive from April 2019 for processing and conducting …


International Journal of Cognitive Computing in Engineering

… MC4 dataset contains 101 language variants drawn from the Common Crawl web scrape. MT5 did not deviates much from the original T5 model and attempted to extend the capacities of the T5 to multilingual settings. T5 is a pre-trained language …


Internet-scale Topic Modeling using Large Language Models

… This proportion is estimated from the Common Crawl dataset’s statistics [28]. In short, the model should have an excellent understanding of English, an adequate understanding of languages such as German and Spanish, and be satisfactory at …


InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

H Ying, S Zhang, L Li, Z Zhou, Y Shao, Z Fei, Y Ma… - arXiv preprint arXiv …, 2024

The math abilities of large language models can represent their abstract reasoning ability. In this paper, we introduce and open-source our math reasoning LLMs InternLM-Math which is continue pre-trained from InternLM2. We unify chain-of-thought …


InternLM2 Technical Report

Z Cai, M Cao, H Chen, K Chen, K Chen, X Chen… - arXiv preprint arXiv …, 2024

… Our web page data mainly comes from Common Crawl1. Firstly, we need to decompress the original Warc format files and use Trafilatura (… The entire filtering process eliminates a large proportion of Web page data (Common Crawl) and Patent …


Interpretable Tensor Fusion

S Varshneya, A Ledent, P Liznerski, A Balinskyy… - arXiv preprint arXiv …, 2024

Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce …


Interpreting Themes from Educational Stories

Y Zhang, FA González, T Solorio - arXiv preprint arXiv:2404.05250, 2024

Reading comprehension continues to be a crucial research focus in the NLP community. Recent advances in Machine Reading Comprehension (MRC) have mostly centered on literal comprehension, referring to the surface-level …


Interrupted time series analysis of clickbait on worldwide news websites, 2016-2023

A McCutcheon, C Brogly - arXiv preprint arXiv:2408.06376, 2024

… This research provides an analysis of clickbait at the scale of the web on primarily English-language news sites from the Common Crawl. In this analysis, 3 of 5 selected newsgenerating events produced significant terms that suggest major …


Intersectional Male-Centric and White-Centric Biases in Collective Concepts

AH Bailey, A Williams, A Poddar, A Cimpian - 2024

… Here we used NLP tools (specifically, static word embeddings trained on the Common Crawl corpus) to investigate collective understanding of three fundamental concepts: PERSON, WOMAN, and MAN.These three concepts organize much of …


Introducing the Djinni Recruitment Dataset: A Corpus of Anonymized CVs and Job Postings

N Drushchak, M Romanyshyn - Proceedings of the Third Ukrainian Natural …, 2024

This paper introduces the Djinni Recruitment Dataset, a large-scale open-source corpus of candidate profiles and job descriptions. With over 150,000 jobs and 230,000 candidates, the dataset includes samples in English and Ukrainian, thereby …


Invasion of ChatGPT and LLMs in Review and Writing Spaces: A Revolution or Radical

A Hadap, V Khatri - Scientific Publishing Ecosystem

Though AI flourished before 1970s but in the last five years, the invasion of artificial intelligence (AI) tools, such as ChatGPT and Large Language Model (LLMs), has received much attention in scientific research writing. These AI tools assist the …


Investigating Complex Answer Attribution Approaches with Large Language Models

This master’s thesis explores the attribution of answers in complex question-answering scenarios utilizing large language models (LLMs). The research aims to assess and enhance the traceability of answers back to their source documents, a critical aspect …


Investigating Conversational Agents to Support Secondary School Computer Science Exploratory Search

Online educational resources (eg, curricula, tutorials, documentation, Q&A sites) increasingly serve as key sources for secondary school students learning Computer Science Principles (CSP). A big obstacle to using these resources is finding …


Investigating Gender Bias in Turkish Language Models

O Caglidil, M Ostendorff, G Rehm - arXiv preprint arXiv:2404.11726, 2024

… an analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


Investigating the Classification of radicalisation phases of Dutch social media messages upon physical threats

The increase of social media platforms such as Twitter and Telegram has transformed communication, enhanced free speech, but also facilitated the spread of harmful content. This rise in online threats against Dutch politicians highlights the …


Investigating the pre-training bias in low-resource abstractive summarization

D Chernyshev, B Dobrov - IEEE Access, 2024

Recent advances in low-resource abstractive summarization were largely made through the adoption of specialized pre-training, pseudo-summarization, that integrates the content selection knowledge through various centrality-based …


IPA Transcription of Bengali Texts

K Fatema, FD Haider, NF Turpa, T Azmal, S Ahmed… - arXiv preprint arXiv …, 2024

The International Phonetic Alphabet (IPA) serves to systematize phonemes in language, enabling precise textual representation of pronunciation. In Bengali phonology and phonetics, ongoing scholarly deliberations persist concerning the …


IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models

DI Adelani, J Ojo, IA Azime, JY Zhuang, JO Alabi, X He… - arXiv preprint arXiv …, 2024

Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (eg African languages) are often …


Is AI changing learning and assessment as we know it? Evidence from a ChatGPT experiment and a conceptual framework

O Kolade, A Owoseni, A Egbetokun - Heliyon, 2024

ChatGPT, a state-of-the-art chatbot built upon Open AI's generative pre-trained transformer, has generated a major public interest and caused quite a stir in the higher education sector, where reactions have ranged from excitement to …


Is ChatGPT taking over the language classroom? How language ideologies of large language models impact teaching and learning

M Lau - Working papers in Applied Linguistics and Linguistics …, 2024

… Thus, using datasets such as the Common Crawl can lead to the over-representation of English and the perspectives of English-speaking content … This means that LLMs that use Common Crawl for training, such as ChatGPT, may perform with …


Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search Engines

J Bevendorff, M Wiegmann, M Potthast, B Stein

Many users of web search engines have been complaining in recent years about the supposedly decreasing quality of search results. This is often attributed to an increasing amount of search-engineoptimized but low-quality content. Evidence for …


Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models

C Liu, W Zhang, Y Zhao, AT Luu, L Bing - arXiv preprint arXiv:2403.10258, 2024

… We categorize languages larger than 1% frequency in Common Crawl2 as high-resource languages (ie, de, ru, fr, zh, es, ja, it and vi), and the rest as low-resource languages. We exclude English since we want to evaluate the efficient prompting strategy for …


It is Time to Develop an Auditing Framework to Promote Value Aware Chatbots

Y Wang, L Singh - arXiv preprint arXiv:2409.01539, 2024

… According to OpenAI, 60% of the training data come from Common Crawl, a large data set consisting of web pages, extracted metadata and text since 2008. Another 22% of data are from WebText2, containing all Reddit posts until December 2017 …


It's About Time: Incorporating Temporality in Retrieval Augmented Language Models

A Gade, J Jetcheva - arXiv preprint arXiv:2401.13222, 2024

… Additionally both models are fine-tuned jointly as a Fusion-in-Decoder [3] on common-crawl data using Wikipedia as a document index. The authors showed that Atlas performs well out-of-box and can be adapted to different tasks such as question-answering …


IUST at ClimateActivism 2024: Towards Optimal Stance Detection: A Systematic Study of Architectural Choices and Data Cleaning Techniques

G Mahmoudi, S Eetemadi - Proceedings of the 7th Workshop on Challenges and …, 2024

This work presents a systematic search of various model architecture configurations and data cleaning methods. The study evaluates the impact of data cleaning methods on the obtained results. Additionally, we demonstrate that a combination of …


Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent

Q Gallouédec, E Beeching, C Romac, E Dellandréa - arXiv preprint arXiv:2402.09844, 2024

The search for a general model that can operate seamlessly across multiple domains remains a key goal in machine learning research. The prevailing methodology in Reinforcement Learning (RL) typically limits models to a single task …


Japanese Pointer Network based Entity Linker for Wikidata

Y Sawamura, T Morita, S Egami, T Ugai, K Fukuda - 2023

… Pre-trained fastText is a model trained on Common Crawl and Wikipedia and provides pre-computed vectors for a word. The length of the POS Tags was changed from 36 to 18. The other five embeddings, namely Entity Embedding, Text Match …


JetMoE: Reaching Llama2 Performance with 0.1 M Dollars

Y Shen, Z Guo, T Cai, Z Qin - arXiv preprint arXiv:2404.07413, 2024

Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence. This report introduces JetMoE-8B …


Jina CLIP: Your CLIP Model Is Also Your Text Retriever

A Koukounas, G Mastrapas, M Günther, B Wang… - arXiv preprint arXiv …, 2024

H Xiao, G Mastrapas, B Wang - Multi-modal Foundation Model meets Embodied AI …

Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks …


JiuZhang3. 0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models

K Zhou, B Zhang, J Wang, Z Chen, WX Zhao, J Sha… - arXiv preprint arXiv …, 2024

Mathematical reasoning is an important capability of large language models~(LLMs) for real-world applications. To enhance this capability, existing work either collects large-scale math-related texts for pre-training, or relies on stronger LLMs (\eg GPT-4) …


JLBert: Japanese Light BERT for Cross-Domain Short Text Classification

C Kayal, S Chattopadhyay, A Gupta, S Abrol, A Gugol - Proceedings of the 2024 Joint …, 2024

… Second, there are limited models that have been trained on diverse datasets beyond Wikipedia and Common Crawl, limiting experimentation with other forms of text data. Moreover, with the English language being the focus of research interest …


Journal of Business Academy

This study examines in detail the potential of artificial intelligence (AI)-assisted ChatGPT in the field of accounting education and the opportunities and challenges that this utilisation may bring. ChatGPT can benefit by reducing error rates and …


Juru: Legal Brazilian Large Language Model from Reputable Sources

RM Junior, R Pires, R Romero, R Nogueira - arXiv preprint arXiv:2403.18140, 2024

The high computational cost associated with pretraining large language models limits their research. Two strategies have emerged to address this issue: domain specialization and pretraining with high-quality data. To explore these strategies, we …


Just Rewrite It Again: A Post-Processing Method for Enhanced Semantic Similarity and Privacy Preservation of Differentially Private Rewritten Text

S Meisenbacher, F Matthes - arXiv preprint arXiv:2405.19831, 2024

The study of Differential Privacy (DP) in Natural Language Processing often views the task of text privatization as a $\textit{rewriting}$ task, in which sensitive input texts are rewritten to hide explicit or implicit private information. In order to evaluate the …


KazQAD: Kazakh Open-Domain Question Answering Dataset

R Yeshpanov, P Efimov, L Boytsov, A Shalkarbayuli… - arXiv preprint arXiv …, 2024

… Kaz-RoBERTa6 is a monolingual model trained on a collection of Kazakh texts from Common Crawl, the Leipzig Corpora Collection, the … XLM-R was pre-trained on a larger multilingual corpus derived from Common Crawl, in which Kazakh is …


KazSAnDRA: Kazakh Sentiment Analysis Dataset of Reviews and Attitudes

R Yeshpanov, HA Varol - arXiv preprint arXiv:2403.19335, 2024

This paper presents KazSAnDRA, a dataset developed for Kazakh sentiment analysis that is the first and largest publicly available dataset of its kind. KazSAnDRA comprises an extensive collection of 180,064 reviews obtained from various sources …


KDPII: A New Korean Dialogic Dataset for the Deidentification of Personally Identifiable Information

L Fei, Y Kang, S Park, Y Jang, J Lee, H Kim - IEEE Access, 2024

… Using this framework, they examined the "Common Crawl's web crawl corpus," a frequently utilized resource for training LMs, to determine whether it contains high-risk personal information. They also compared various detection methods, including …


Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI

N Pangakis, S Wolken - arXiv preprint arXiv:2409.09467, 2024

… Because LLMs are trained on huge volumes of text data, using digital resources such as the Common Crawl, they are able to “memorize” publicly available information during the training process. This issue is compounded due to the fact …


KEPA-CRF: Knowledge expansion prototypical amortized conditional random field for few-shot event detection

R Wu, L Yu, S Tian, J Long, T Zhou, B Wang - Journal of Intelligent & Fuzzy Systems

Event Detection (ED) has long struggled with the ambiguous definition of event categories, making it challenging to accurately classify events. Previous endeavors aimed to tackle this problem by constructing prototypes for specific event categories …


Kernel Least Squares Transformations for Cross-Lingual Semantic Spaces

A Mištera, T Brychcín - International Conference on Text, Speech, and …, 2024

The rapid development in the field of natural language processing (NLP) and the increasing complexity of linguistic tasks demand the use of efficient and effective methods. Cross-lingual linear transformations between semantic spaces play a …


Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks

N Chirkova, V Nikoulina - arXiv preprint arXiv:2402.12279, 2024

… For Intermediate tuning we finetune models for 100k steps on the CommonCrawl data uniformly sampled across all target languages and English, with the batch size of 5k tokens and the LR chosen to optimize fluency of model generations, inspected …


Knowledge Induced Transformer Network for Causality Prediction

T Dasgupta, M Sinha, A Naskar - Companion Proceedings of the ACM on Web …, 2024

… The CausalBank dataset3 contains 314 million pairs of cause-effect statements scraped from the Common Crawl corpus using causal lexical patterns. Baselines: Most of the earlier work on causality detection is based on identifying cause, effect …


Knowledge Sources

M Jiang, BY Lin, S Wang, Y Xu, W Yu, C Zhu - Knowledge-augmented Methods for …, 2024

… To be more specific, the dataset includes Pile-CommonCrawl (227.12G, web crawling data), PubMed Central (90.27G, biomedical articles), Books (100.96G, a mix of fiction and nonfiction books), OpenWebText (262.77G, content from Reddit) …


Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models

R Vemulapalli, H Pouransari, F Faghri, S Mehta… - Forty-first International Conference …

… 2023) which contains 1.28B images filtered from Common Crawl (web-crawled data). We take 10 randomly augmented crops from each image and create a gallery of 12.8B images. See Appendix B.2 for additional details about this gallery set and …


Knowledge-augmented Methods for Natural Language Processing

Ever since language was invented and used, humans have leveraged this powerful tool to pass down experience over generations. These kinds of knowledge summarize precious findings and inspiring ideas, which constitute the human …


Knowledge-Enhanced Neural Networks for Machine Reading Comprehension

Machine Reading Comprehension is a language understanding task where a system is expected to read a given passage of text and typically answer questions about it. When humans assess the task of reading comprehension, in addition to the …


KOMPYUTER LINGVISTIKASIDA TA'MINOT MASALASI VA SINONIM BIRLIKLARNING LINGVISTIK TA'MINOTI

N SOTVOLDIYEVA - News of UzMU journal, 2024

… Lingvistik ma'lumotlar konsorsiumi (LDC) va Common Crawl[8] loyihasi kabi ochiq hamkorlik tashabbuslari lingvistik resurslarning mavjudligiga sezilarli hissa qo'shdi. Bu hamkorlik resurslar almashinuvini targ‘ib qiladi, ortiqchalikni kamaytiradi va …


Krishiq-BERT: A Few-Shot Setting BERT Model to Answer Agricultural-Related Questions in the Kannada Language

P Ajawan, V Desai, S Kale, S Patil - Journal of The Institution of Engineers (India) …, 2024

… corpora for 11 languages from two language families obtained primarily by sourcing articles from news crawls [3], and OSCAR (Open Super-large Crawled ALMAnaCH coRpus), a huge multilingual corpus procured by language …


KVP10k: A Comprehensive Dataset for Key-Value Pair Extraction in Business Documents

O Naparstek, R Pony, I Shapira, FA Dahood, O Azulai… - arXiv preprint arXiv …, 2024

… : extensive web data from Common Crawl and a collection of images from publicfiles.fcc.gov. From Common Crawl, we employed a systematic … A schematic describing the data acquisition process for the common crawl data is shown in Fig.2 …


L3i++ at SemEval-2024 Task 8: Can Fine-tuned Large Language Model Detect Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text?

HTH Tran, TN Nguyen, A Doucet, S Pollak - … of the 18th International Workshop on …, 2024

… Meanwhile, XLM-R is a multilingual version of RoBERTa that was pretrained on 2.5TB of filtered CommonCrawl data containing 100 languages. These models are also suggested as the baseline methods from SemEval2024 Task 8 organizers. …


Labadain-30k+: A Monolingual Tetun Document-Level Audited Dataset

G de Jesus, S Nunes - Proceedings of the 3rd Annual Meeting of the Special …, 2024

This paper introduces Labadain-30k+, a monolingual dataset comprising 33.6 k documents in Tetun, a low-resource language spoken in Timor-Leste. The dataset was acquired through web crawling and augmented with Wikipedia documents …


Label-dependent semantic associations for existing concepts

In a series of studies, we examined whether a label assigned to a cue concept (political event or its actor) modulates(a) its semantic associations (semantic effects) and (b) valence of the associations and attitude judgments (valence effects). We introduced …


Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Level

Z Feng, R Chen, Y Zhang, Z Meng, Z Liu - arXiv preprint arXiv:2406.15741, 2024

General-purpose Large Language Models (LLMs) like GPT-4 have achieved remarkable advancements in machine translation (MT) by leveraging extensive web content. On the other hand, translation-specific LLMs are built by pre-training on …


Language as a Lens: A Hybrid Text Summarization and Sentiment Analysis Approach for Multiclass Stock Return Prediction

F Balaneji - Intelligent Systems Conference, 2024

This research explores the application of text summarization and sentiment analysis techniques in the multiclass classification of hourly stock price returns, studying six companies from the Dow Jones Index between 2017 and the first quarter of 2020 …


Language as a Lens: A Hybrid Text Summarization and Sentiment Analysis Approach for Multiclass Stock Return

F Balaneji - Intelligent Systems and Applications: Proceedings of …


Language Model Frame Filling for Low Resource Languages

Acceptability judgment tasks are key tools used in the understanding of the grammar of languages by linguists. A typical way to construct these tasks is to use semantic frames. We provide evidence that the fill-mask objective for language models (LMs) …


Language Models Fine-Tuning for Automatic Format Reconstruction of SEC Financial Filings

G Lombardo, G Trimigno, M Pellegrino, S Cagnoni - IEEE Access, 2024

The analysis of financial reports is a crucial task for investors and regulators, especially the mandatory annual reports (10-K) required by the SEC (Securities and Exchange Commission) that provide crucial information about a public company in …


Language Models for Online Depression Detection: A Review and Benchmark Analysis on Remote Interviews

R Qin, R Cook, K Yang, A Abbasi, D Dobolyi, S Seyedi… - ACM Transactions on …, 2024

… The project primarily uses data from the English CommonCrawl (67%) but also includes other public data sources including Wikipedia, GitHub, and ArXiv. LLaMA’s ability to outperform GPT-3 on the RACE Reading Comprehension benchmark …


Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining

N Ljubešić, V Suchomel, P Rupnik, T Kuzman… - arXiv preprint arXiv …, 2024

The world of language models is going through turbulent times, better and ever larger models are coming out at an unprecedented speed. However, we argue that, especially for the scientific community, encoder models of up to 1 billion parameters …


Language Models Pre-training

U Kamath, K Keenan, G Somers, S Sorenson - Large Language Models: A Deep Dive …, 2024

… The Common Crawl data is the most notable publicly available web scrape. … They demonstrated that these sources improved downstream performance over models trained on less diverse corpora such as Common Crawl. Taking this idea …


Language models scale reliably with over-training and on downstream tasks

SY Gadre, G Smyrnis, V Shankar, S Gururangan… - arXiv preprint arXiv …, 2024

Scaling laws are useful guides for developing language models, but there are still gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal …


Large Language Model-Based Chatbots in Higher Education

D Yigci, M Eryilmaz, AK Yetisen, S Tasoglu, A Ozcan - Advanced Intelligent Systems, 2024

Large language models (LLMs) are artificial intelligence (AI) platforms capable of analyzing and mimicking natural language processing. Leveraging deep learning, LLM capabilities have been advanced significantly, giving rise to generative …


Large Language Models (LLMs) on Tabular Data: Predic-tion, Generation, and Understanding-A Survey

X Fang, W Xu, FA Tan, J Zhang, Z Hu, Y Qi… - arXiv preprint arXiv …, 2024

Recent breakthroughs in large language modeling have facilitated rigorous exploration of their application in diverse tasks related to tabular data modeling, such as prediction, tabular data synthesis, question answering, and table …


Large Language Models and Rule-Based Approaches in Domain-Specific Communication

D Halvoník, J Kapusta - IEEE Access, 2024

Currently, we are once again experiencing a frenzy related to artificial intelligence. Generative Pre-trained Transformers (GPT) models are highly effective at various natural language processing tasks. Different varieties of GPT models are widely …


Large Language Models and Sentiment Analysis in Financial Markets: A Review, Datasets and Case Study

C Liu, A Arulappan, R Naha, A Mahanti… - IEEE Access, 2024

This paper comprehensively examines Large Language Models (LLMs) in sentiment analysis, specifically focusing on financial markets and exploring the correlation between news sentiment and Bitcoin prices. We systematically categorize various …


Large Language Models and the Disappearing Private Sphere

A Cappello, M Dada, V Grigoreva, R Khan, C Stinson… - 2024

… The higher quality datasets are sampled more often during training, but the Reddit approved contents of CommonCrawl still … of CommonCrawl contains fan fiction, video game chats, conspiracy theories, pornography, junk advertising, and …


Large Language Models aren't all that you need

KV Holla, C Kumar, A Singh - arXiv preprint arXiv:2401.00698, 2024

This paper describes the architecture and systems built towards solving the SemEval 2023 Task 2: MultiCoNER II (Multilingual Complex Named Entity Recognition) [1]. We evaluate two approaches (a) a traditional Conditional Random …


LARGE LANGUAGE MODELS AS AN AID FOR SOFTWARE DEVELOPMENT USING GRAPHICAL PROGRAMMING LANGUAGE.

… In pre-training, the model is trained with the Common Crawl dataset, internet books, and Python code, summing up to 400B tokens. After the model is pre-trained, the phase preference model pre-training comes, and after that, the preference model …


Large Language Models As Genetic Counselors: AI-DRIVEN TOOLS FOR HEALTH MANAGEMENT

This thesis, in collaboration with Arkus-AI, explores the development of an AI-driven genetic counselor designed to enhance genetic counseling services at Eiira. By utilizing advanced large language models, this project aims to improve the quality …


Large Language Models As Genetic Counselors: Personalized AI Driven Tool for Health Management


Large language models as oracles for instantiating ontologies with domain-specific knowledge

G Ciatto, A Agiollo, M Magnini, A Omicini - arXiv preprint arXiv:2404.04108, 2024

… Models following the RoBERTa [30] approach were trained on (i) the same datasets as BERT, plus (ii) a sample of the CommonCrawl4 News dataset (a dump of news … 4https://commoncrawl.org 5https://mistral.ai/news/mixtral-of-experts …


Large Language Models for Biomedical Text Simplification: Promising But Not There Yet

Z Li, S Belkadi, N Micheletti, L Han, MSG Nenadic

… the common crawl corpus and filtered to keep only natural text and de-duplication processing. They extracted 750GB of clean English data to feed into the model for multi-task pretraining. Different masking strategies are integrated into the T5 model …


Large language models for biomedicine: foundations, opportunities, challenges, and best practices

SS Sahoo, JM Plasek, H Xu, Ö Uzuner, T Cohen… - Journal of the American …, 2024

… In the pre-training phase, generic training data, such as the Common Crawl dataset or social media posts, are used for unsupervised learning.With further training on prompt-response pairs (such as the Alpaca dataset developed at Stanford23) …


Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review

J Su, C Jiang, X Jin, Y Qiao, T Xiao, H Ma, R Wei… - arXiv preprint arXiv …, 2024

… The training data for GPT-3 includes the Common Crawl dataset for lower quality, as well as the WebText2 dataset for higher quality, as well as the Books1, Books2, and Wikipedia datasets for higher quality [148]. GPT-3 assigns different weights to …


Large Language Models for Human-Machine Collaborative Particle Accelerator Tuning through Natural Language

J Kaiser, A Eichler, A Lauscher - arXiv preprint arXiv:2405.08888, 2024

Autonomous tuning of particle accelerators is an active and challenging field of research with the goal of enabling novel accelerator technologies cutting-edge high-impact applications, such as physics discovery, cancer research and material sciences. A …


Large Language Models for identification of medical data in unstructured records

… Sources for the training data include web pages extracted by CommonCrawl, opensource GitHub repositories, Wikipedia in twenty languages, Project Gutenberg books in the public domain, LaTeX source code of scientific articles in ArXiv and …


Large Language Models for Relevance Judgement in Table Retrieval

… The tables come from the English subset of the Web Data Commons (WDC) table corpus 2015, which contains 50.8M HTML tables from the July 2015 Common Crawl The creators of the WTR collection further filtered the collection by identifying entities …


Large Language Models for Reliable Information Extraction

This project addresses the ongoing challenge of achieving reliable Information Extraction (IE), particularly in domains which require near-perfect precision. ELICIT (Butcher et al., 2023) introduced a novel approach that combines the processing speed of …


Large Language Models for Software Engineering: A Systematic Mapping Study

MK Görmez, M Yılmaz, PM Clarke - European Conference on Software Process …, 2024

In this research, we aim to conduct a systematic mapping study on Large Language Models (LLMs) for Software Engineering (SE). The significantly enhanced capabilities of LLMs have led to their use in many fields, including the important …


LARGE LANGUAGE MODELS FOR TEXT CLASSIFICATION: FROM ZERO-SHOT LEARNING TO INSTRUCTION-TUNING

Advances in large language models (LLMs) have transformed the field of natural language processing and have enormous potential for social scientific analysis. We explore the application of LLMs to supervised text classification. As a case study, we …


Large language models in healthcare: from a systematic review on medical examinations to a comparative analysis on fundamentals of robotic surgery online test

A Moglia, K Georgiou, P Cerveri, L Mainardi… - Artificial Intelligence Review, 2024

Large language models (LLMs) have the intrinsic potential to acquire medical knowledge. Several studies assessing LLMs on medical examinations have been published. However, there is no reported evidence on tests related to robot-assisted …


Large Language Models on Tabular Data--A Survey

Recent breakthroughs in large language modeling have facilitated rigorous exploration of their application in diverse tasks related to tabular data modeling, such as prediction, tabular data synthesis, question answering, and table …


Large Language Models Understand Layouts

W Li, M Duan, D An, Y Shao - arXiv preprint arXiv:2407.05750, 2024

… We find that GLM, Llama, and GPT-3 all use datasets such as CommonCrawl, Wikipedia, and Books (Pile includes CommonCrawl, Wikipedia, and Books) in their pretraining. CommonCrawl is a large-scale, unstructured, multilingual web dataset …


Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation

M Wysocka, O Wysocki, M Delmas, V Mutel, A Freitas - Journal of Biomedical …, 2024

Objective: The paper introduces a framework for the evaluation of the encoding of factual scientific knowledge, designed to streamline the manual evaluation process typically conducted by domain experts. Inferring over and extracting information from …


Large Language Models: A Case Study on Corporate Values and Transformative Technologies

Design (VSD) a technique that provides theories and methods to systematically “account for human values… throughout the design process”(Value Sensitive Design Lab, nd). This paper employs VSD as a theoretical lens to dissect the dual challenges of …


Large Language Models: A Deep Dive

U Kamath, K Keenan, G Somers, S Sorenson

In the panorama of technological evolution, Large Language Models (LLMs) have emerged as a cornerstone, transforming our interaction with information, reshaping industries, and redefining the boundaries of artificial intelligence. As we stand on the …


Large Language Models: A Survey

S Minaee, T Mikolov, N Nikzad, M Chenaghlu… - arXiv preprint arXiv …, 2024

… Common Crawlbased dataset consisting of texts in 101 languages. … CommonCrawl. They also released an extract of 600 billion tokens from our REFINEDWEB dataset, and 1.3/7.5B parameters language models trained on it. 27 …


Large Language Models: An Introduction

U Kamath, K Keenan, G Somers, S Sorenson - Large Language Models: A Deep Dive …, 2024

This chapter begins with a discussion of the historical context and progression of natural language processing. Beginning with the origins of human linguistic capabilities, this chapter explains the gradual transition to computational language …


Large Malaysian Language Model Based on Mistral for Enhanced Local Language Understanding

H Zolkepli, A Razak, K Adha, A Nazhan - arXiv preprint arXiv:2401.13565, 2024

… An analysis of the widely utilized Common Crawl dataset reveals a mere 0.0742% contribution from the Malay language based on CCMAIN-… We also replicated the same technique for Malaysia Hansard and MS CommonCrawl samples. All synthetic …


Large Scale Transfer Learning for Tabular Data via Language Modeling

J Gardner, JC Perdomo, L Schmidt - arXiv preprint arXiv:2406.12031, 2024

… Given that T4 consists of 3.1Mtables sourced from public data sources (Common Crawl, Github) and that our evaluations are also comprised of public benchmarks, we investigate the extent and possible impact of data contamination – that is …


Large Vocabulary Size Improves Large Language Models

S Takase, R Ri, S Kiyono, T Kato - arXiv preprint arXiv:2406.16508, 2024

This paper empirically investigates the relationship between subword vocabulary size and the performance of large language models (LLMs) to provide insights on how to define the vocabulary size. Experimental results show that larger vocabulary …


Large-language models facilitate discovery of the molecular signatures regulating sleep and activity

D Peng, L Zheng, D Liu, C Han, X Wang, Y Yang… - Nature Communications, 2024

Sleep, locomotor and social activities are essential animal behaviors, but their reciprocal relationships and underlying mechanisms remain poorly understood. Here, we elicit information from a cutting-edge large-language model (LLM), generative pre-trained …


Large-Language-Models (LLM)-Based AI Chatbots: Architecture, In-Depth Analysis and Their Performance Evaluation

V Kumar, P Srivastava, A Dwivedi, I Budhiraja… - International Conference on …, 2023

… LLaMA models span from 7 billion to 65 billion parameters and were trained on trillions of tokens derived from publicly accessible datasets, including Common Crawl and Wikipedia. The primary objective of LLaMA models is to serve as …


Large-scale machine unlearning in transformer-based pre-trained language models: Evaluation and adaptation of existing machine unlearning frameworks KGA and …

In the past few years, Machine Learning (ML) models such as GPT-4, utilized by the prominent chatbot ChatGPT, have faced data breaches, raising concerns about data privacy and awareness about the "right to be forgotten." Consequently, the concept …


Latxa: An Open Language Model and Evaluation Suite for Basque

J Etxaniz, O Sainz, N Perez, I Aldabe, G Rigau, E Agirre… - arXiv preprint arXiv …, 2024

We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. Addressing the …


Law and the Political Economy of AI Production

The governance of Artificial Intelligence (AI) is at a historical juncture. Legislative acts, global treaties, export controls, and technical standards are now dominating the discourse over what used to be a predominantly market-driven space. Amidst all this …


Law, Technology and Humans

… Around 60% of GPT-3 training data comes from a filtered version of Common Crawl—that is, a non-profit organisation that archives and provides open-access petabytes of data collected since 2011. [30] The remaining 40% comes from diverse …


LAWSUIT: a LArge expert-Written SUmmarization dataset of ITalian constitutional court verdicts

L Ragazzi, G Moro, S Guidi, G Frisoni - Artificial Intelligence and Law, 2024

Large-scale public datasets are vital for driving the progress of abstractive summarization, especially in law, where documents have highly specialized jargon. However, the available resources are English-centered, limiting research …


Learning entity and relation representation for low-resource medical language processing

Recent advancements in natural language processing have led to growing interest in critical domains such as legal, finance, and healthcare. In particular, medical language processing has emerged as a focus of its own. As the number of …


Learning Fine-Grained Grounded Citations for Attributed Large Language Models

L Huang, X Feng, W Ma, Y Gu, W Zhong, X Feng, W Yu… - arXiv preprint arXiv …, 2024

… 2021), a pre-processed and cleaned version of the Common Crawl corpus, serving as a proxy web search index. In particular, for a given user query sampled from the AQuAMuSe dataset, we initially retrieve the top 100 relevant documents …


Learning from Noisy Data for Cross Lingual Text Generation in Low-Resource Languages

With Large Language Models (LLMs) and Language models in general becoming a more significant part of our daily content consumption, it is paramount to ensure that languages with fewer resources do not get excluded. As most language models are …


Learning to Generate Answers with Citations via Factual Consistency Models

R Aly, Z Tang, S Tan, G Karypis - arXiv preprint arXiv:2406.13124, 2024

Large Language Models (LLMs) frequently hallucinate, impeding their reliability in mission-critical situations. One approach to address this issue is to provide citations to relevant sources alongside generated content, enhancing the verifiability of …


Learning with Noisy Foundation Models

H Chen, J Wang, Z Wang, R Tao, H Wei, X Xie… - arXiv preprint arXiv …, 2024

… GPT2 is pre-trained on WebText [29], a scraped web dataset from Common Crawl that contains low-quality raw texts. We also leverage OpenAI’s API service “text-ada-002”. We cannot use larger and more recent language models such as LLaMA [30], since …


Leveraging AI to Deliver Culturally Responsive Mental Health Care at Scale

A Cerezo, D Cooper, V Palat, A Jolley, S Peregrine

… One of the most commonly used datasets to train LLMs is The Common Crawl dataset, a large collection of web pages scraped from the internet since 2014. These pages contain the gamut from high-quality news, science, and literature to the …


Leveraging distant supervision and deep learning for twitter sentiment and emotion classification

M Kastrati, Z Kastrati, A Shariq Imran, M Biba - Journal of Intelligent Information …, 2024

Nowadays, various applications across industries, healthcare, and security have begun adopting automatic sentiment analysis and emotion detection in short texts, such as posts from social media. Twitter stands out as one of the most popular …


Leveraging Large Language Models to Measure Gender Bias in Gendered Languages

E Derner, SS de la Fuente, Y Gutiérrez, P Moreda… - arXiv preprint arXiv …, 2024

Gender bias in text corpora used in various natural language processing (NLP) contexts, such as for training large language models (LLMs), can lead to the perpetuation and amplification of societal inequalities. This is particularly …


Leveraging LLMs for Collaborative Ontology Engineering in Parkinson Disease Monitoring and Alerting

G Bouchouras, D Doumanas, A Soularidis, K Kotis…

This paper explores the integration of Large Language Models (LLMs) in the engineering of a Parkinson’s Disease (PD) monitoring and alerting ontology through four key methodologies: One Shot (OS) prompt techniques, Chain of Thought (CoT) …


Leveraging Open Large Language Models for Multilingual Policy Topic Classification: The Babel Machine Approach

M Sebők, Á Máté, O Ring, V Kovács, R Lehoczki - Social Science Computer Review, 2024

The article presents an open-source and freely available natural language processing system for comparative policy studies. The CAP Babel Machine allows for the automated classification of input files based on the 21 major policy topics of …


Leveraging Parameter Efficient Training Methods for Low Resource Text Classification: A case study in Marathi

P Deshmukh, N Kulkarni, S Kulkarni, K Manghani… - 2024 IEEE 9th International …, 2024

With the surge in digital content in low-resource languages, there is an escalating demand for advanced Natural Language Processing (NLP) techniques tailored to these languages. BERT (Bidirectional Encoder Representations from Transformers) …


Leveraging Universal Speech Representations for Detecting and Assessing the Severity of Mild Cognitive Impairment Across Languages

A Favaro, T Cao, N Dehak, L Moro-Velazquez

This study examines the suitability of language-agnostic features for automatically detecting Mild Cognitive Impairment (MCI) and predicting Mini-Mental State Examination (MMSE) scores in a multilingual framework. We explored two methods …


Leveraging Zero-Shot Prompting for Efficient Language Model Distillation

L Vöge, V Gurgul, S Lessmann - arXiv preprint arXiv:2403.15886, 2024

… Specifically, we use the T5 v1.1 models which were only pretrained on the C4 Common Crawl dataset for general language understanding without additional task-specific finetuning. The use of text-to-text models allows for straightforward processing, as …


LexEval: A Scalable LLM Evaluation Framework

As Large Language Models (LLMs) become more powerful and their parameter counts increase, it is crucial for them to exhibit consistent recall capabilities for terms that appear less frequently in their pre-training data. The importance of accurate …


Lib2Life–Digital Library Services Empowered with Advanced Natural Language Processing Techniques

M Nitu, M Dascalu, MD Dascalu, LM Neagu…

Educational institutions are struggling to keep up with the accelerated technological advancements; hence, sustainable and supportive tools have become essential to reshape traditional models into intelligent learning systems. This paper introduces …


LiBERTa: Advancing Ukrainian Language Modeling through Pre-training from Scratch

M Haltiuk, A Smywiński-Pohl - Proceedings of the Third Ukrainian Natural Language …, 2024

… CC-100, a multilingual text corpus sourced from Wikipedia and CommonCrawl, was processed following the CCNet1 methodology. The Ukrainian segment of CC-100 encompasses 6.5 billion tokens, equivalent to 84 GiB of data2. This corpus primarily …


Limitations of Language Models in The Oil & Gas Upstream Operations

A Alsultan, FA Razak - International Petroleum Technology Conference, 2024

… Although the process is simple in principle, it is done on very large training corpra which can be as big as dataset provided by the Common Crawl project (428.81 TB as of the date of writing this paper). Thus, training these models using these large …


Limits to Predicting Online Speech Using Large Language Models

M Remeli, M Hardt, RC Williamson - arXiv preprint arXiv:2407.12850, 2024

We study the predictability of online speech on social media, and whether predictability improves with information outside a user's own posts. Recent work suggests that the predictive information contained in posts written by a user's peers …


LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages

AM Bean, S Hellsten, H Mayne, J Magomere, EA Chi… - arXiv preprint arXiv …, 2024

… ], while the second and fourth rows are sorted by Common Crawl Share [16]. The upper two … of pages tagged with a given language in the Common Crawl dataset [16]. Figure 12 show the … LLMs are trained on, using the Common Crawl share and …


Linguistic Profiling of Deepfakes: An Open Database for Next-Generation Deepfake Detection

Y Wang, Z Huang, Z Ma, X Hong - arXiv preprint arXiv:2401.02335, 2024

The emergence of text-to-image generative models has revolutionized the field of deepfakes, enabling the creation of realistic and convincing visual content directly from textual descriptions. However, this advancement presents considerably greater …


Linguistic Structure from a Bottleneck on Sequential Information Processing

R Futrell, M Hahn - arXiv preprint arXiv:2405.12109, 2024

… Table 3: English expressions for the given meanings, along with frequencies from the English Common Crawl web corpus [64]. Example … n-gram counts and language models from the common crawl. In Language Resources and Evaluation …


Linguistic_Hygenist at PAN 2024 TextDetox: HybridDetox-A Combination of Supervised and Unsupervised Methods for Effective Multilingual Text Detoxification

S Gangopadhyay, MT Khan, H Jabeen - 2024

… To handle these, we used the XLM-RoBERTa large model [25], which is pretrained in a self-supervised manner on 2.5TB of filtered CommonCrawl data spanning 100 languages, including all languages featured in the competition. The model employs a …


LITS: An Optimized Learned Index for Strings (An Extended Version)

Y Yang, S Chen - arXiv preprint arXiv:2407.11556, 2024

Index is an important component in database systems. Learned indexes have been shown to outperform traditional tree-based index structures for fixed-sized integer or floating point keys. However, the application of the learned solution to variable-length …


LITS: An Optimized Learned Index for Strings

Y Yang, S Chen - Proceedings of the VLDB Endowment, 2024


Llama (LLM)

… Data was mostly from CommonCrawl and C4. Llama-2-70B saw 2T tokens but Llama-3-70B saw 15T tokens. Llama-3-8B and Llama-3-70B took 1.3M and 6.4M GPU hours for pretraining. …


LLaMA Beyond English: An Empirical Study on Language Capability Transfer

J Zhao, Z Zhang, Q Zhang, T Gui, X Huang - arXiv preprint arXiv:2401.01055, 2024

In recent times, substantial advancements have been witnessed in large language models (LLMs), exemplified by ChatGPT, showcasing remarkable proficiency across a range of complex tasks. However, many mainstream LLMs (eg LLaMA) are …


LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

T Zhu, X Qu, D Dong, J Ruan, J Tong, C He, Y Cheng - arXiv preprint arXiv …, 2024

… As to the Dev-to-Train differences in Figure 9b, we find HellaSwag and ARC-c share the most similar expert preferences with CommonCrawl and C4, and GSM-8K is similar to arXiv. This may provide some insights for continual pre-training to further …


LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages

Y Lu, W Zhu, L Li, Y Qiao, F Yuan - arXiv preprint arXiv:2407.05975, 2024

Large Language Models~(LLMs) demonstrate remarkable translation capabilities in high-resource language tasks, yet their performance in low-resource languages is hindered by insufficient multilingual data during pre-training. To address this, we …


LLM-Assistance for Quality Control of LLM Output

K Sandkuhl - Perspectives in Business Informatics Research: 23rd …

… GPT-3, the model behind ChatGPT, operates with 175 billion parameters and utilizes datasets that include nearly a trillion words sourced from various corpora, such as the Common Crawl, WebText2, internet-based books, and the English …


LLM-Based Edge Intelligence: A Comprehensive Survey on Architectures, Applications, Security and Trustworthiness

O Friha, MA Ferrag, B Kantarci, B Cakmak, A Ozgun… - IEEE Open Journal of the …, 2024

The integration of Large Language Models (LLMs) and Edge Intelligence (EI) introduces a groundbreaking paradigm for intelligent edge devices. With their capacity for human-like language processing and generation, LLMs empower edge …


LLM-Datasets: An Open Framework for Pretraining Datasets of Large Language Models

M Ostendorff, PO Suarez, LF Lage, G Rehm

… All of the numbers are given by Common Crawl dump. … 2 show the number of documents by both date of the CommonCrawl dump and language, the percentage of documents removed at each filtering step, and the final document and token …


LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs

A Aizawa, E Aramaki, B Chen, F Cheng, H Deguchi… - arXiv preprint arXiv …, 2024

… The statistics of the Japanese corpus from Common Crawl dumps are presented in Figure 2. We used the publicly available Common Crawl dumps from 2013 to the middle of 2023. We merged the Common Crawl dumps from 2013 to 2016 because …


LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement

N Lee, T Wattanawong, S Kim, K Mangalam, S Shen… - arXiv preprint arXiv …, 2024

… We assume that we are given an LLM model M (eg, GPT-3.5 or LLaMA2-7B) that is pre-trained on some source dataset (eg, Common Crawl). The goal is to adapt M (hereon called the student model) to a new target domain by using a small seed dataset D …


LLM4Schema. org: Generating Schema. org Markups with Large Language Models

MH Dang, THT Pham, P Molli, H Skaf-Molli, A Gaignard

The integration of Schema. org markup in web pages has resulted in the creation of billions of RDF triples, yet approximately 75% of all web pages still lack these essential markups. Large Language Models (LLMs) offer a potential solution by …


LLMs achieve adult human performance on higher-order theory of mind tasks

W Street, JO Siy, G Keeling, A Baranes, B Barnett… - arXiv preprint arXiv …, 2024

This paper examines the extent to which large language models (LLMs) have developed higher-order theory of mind (ToM); the human ability to reason about multiple mental and emotional states in a recursive manner (eg I think that you …


LLMs for Low Resource Languages in Multilingual, Multimodal and Dialectal Settings

F Alam, SA Chowdhury, S Boughorbel, M Hasanain - … of the 18th Conference of the …, 2024

The recent breakthroughs in Artificial Intelligence (AI) can be attributed to the remarkable performance of Large Language Models (LLMs) across a spectrum of research areas (eg, machine translation, question-answering, automatic speech …


LLMs Still Can't Avoid Instanceof: An Investigation Into GPT-3.5, GPT-4 and Bard's Capacity to Handle Object-Oriented Programming Assignments

BP Cipriano, P Alves - arXiv preprint arXiv:2403.06254, 2024

… are high-level descriptions of the training datasets (eg a filtered version of Common Crawl 14, English Wikipedia, among others) [3], it was not possible to perform this analysis for those models. 13https:github.com/camilaabrantes/CursoJava …


LMaaS: Exploring Pricing Strategy of Large Model as a Service for Communication

P Wu, Q Liu, Y Dong, F Wang - arXiv preprint arXiv:2401.02675, 2024

The next generation of communication is envisioned to be intelligent communication, that can replace traditional symbolic communication, where highly condensed semantic information considering both source and channel will be extracted and …


LMCK: pre-trained language models enhanced with contextual knowledge for Vietnamese natural language inference

NLT Nguyen, KTK Phan, TV Huynh, KV Nguyen - Multimedia Tools and Applications, 2024

… trained with a cross-lingual masked language modeling objective on data in 100 languages, including Vietnamese from Common Crawl. … a combination of span masking and sentence shuffling objectives on a subset of 25 languages, including …


LogogramNLP: Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLP

D Chen, F Shi, A Agarwal, J Myerston… - arXiv preprint arXiv …, 2024

… 2023), a pixel-based language model pre-trained on the Common Crawl dataset with a masked image modeling objective, to encode the … Among the aforementioned models, T5, ByT5, and PIXEL are pre-trained on large-scale text …


Long-input summarization using Large Language Models

… the colossal and cleaned version of common crawl CNN … T5 was pre-trained on the C4 (The Colossal and Cleaned version of Common Crawl) dataset. C4 contains text from 350M web pages; the dataset is 750 GB. A maximum sequence length of …


Long-Term Ad Memorability: Understanding & Generating Memorable Ads

SI Harini, S Singh, Y Kumar, A Bhattacharyya, V Baths…

Marketers spend billions of dollars on advertisements, but to what end? At purchase time, if customers cannot recognize the brand for which they saw an ad, the money spent on the ad is essentially wasted. Despite its importance in marketing, until now …


LongWanjuan: Towards Systematic Measurement for Long Text Quality

K Lv, X Liu, Q Guo, H Yan, C He, X Qiu, D Lin - arXiv preprint arXiv:2402.13583, 2024

… In the English data, the CommonCrawl domain predominates, accounting for over 50% of the data. Apart from a significant amount of aggregated texts in the CommonCrawl domain, the majority of data in other domains consists of holistic texts …


Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining

Q Zhang, T Du, H Huang, Y Wang, Y Wang - Forty-first International Conference on Machine …

In recent years, the rise of generative self-supervised learning (SSL) paradigms has exhibited impressive performance across visual, language, and multi-modal domains. While the varied designs of generative SSL objectives lead to distinct properties in …


Look Hear: Gaze Prediction for Speech-directed Human Attention

S Mondal, S Ahn, Z Yang, N Balasubramanian… - arXiv preprint arXiv …, 2024

For computer systems to effectively interact with humans using spoken language, they need to understand how the words being generated affect the users' moment-by-moment attention. Our study focuses on the incremental prediction of attention as a person is …


LoRA-Enhanced Language Alignments for Robust Code-Mixed Text Representation

X Ran, Y Zhang, J Wang, D Xu, X Zhang - 2024 International Joint Conference on …, 2024

… • XLM-RoBERTa-Large [3] is a multilingual version of RoBERTa, which is pre-trained on a filtered Common Crawl dataset containing 100 languages. • XMFGM [31] applies adversarial training to construct some adversarial samples via adding …


LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Models

H Elesedy, PM Esperança, SV Oprea, M Ozay - arXiv preprint arXiv:2407.02987, 2024

Guardrails have emerged as an alternative to safety alignment for content moderation of large language models (LLMs). Existing model-based guardrails have not been designed for resource-constrained computational portable devices, such …


Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training

Z Zhong, M Xia, D Chen, M Lewis - arXiv preprint arXiv:2405.03133, 2024

… We suspect that segment-level routing may be particularly effective at handling out-of-domain evaluation data (we assume that there is only a very small part of Python code in Commoncrawl). Our analysis in Section 5.4 shows that segment-level routing …


Low-Complexity and Secure Clustering-Based Similarity Detection for Private Files

Detection of the similarity between files is a requirement for many practical applications, such as copyright protection, file management, plagiarism detection, and detecting duplicate submissions of scientific articles to multiple journals or …


M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets

G Thakkar, S Hakimov, M Tadić - arXiv preprint arXiv:2404.01753, 2024

… 2020b) is a large multilingual language model trained on 2.5 TB of filtered Common Crawl data containing 100 languages. The model was trained with the Masked Language Modelling (MLM) objective, with 15% of the input words masked …


Machine Learning and the Analysis of Culture

The focus of this chapter is on how machine learning (ML) impacts the analysis of culture in sociology. It shows how ML has greatly advanced the analysis of culture, with new tools enabling a massive and fine-grained extraction of information from …


Machine Translation that Peeks at the Reference

Abstract Machine translation with lexical constraints is a popular research topic, especially for terminology translation. Existing approaches for lexical control in MT are usually complex and not easily applicable to all existing MT toolkits. We propose …


Machine-Learning Phishing Detection Model Used in the E-Banking Environment

M Manala, J Jansen van Vuuren - IFIP International Conference on Human Choice …, 2024

… Rashid, Mahmood [20] proposed a machine-learning approach that uses Alexa and the Common Crawl archive dataset with 5000 URLs. … Geyik, Erensoy [21] used the PhishTank Alexa Common-Crawl datasets. The performance attained …


Machines Do See Color: A Guideline to Classify Different Forms of Racist Discourse in Large Corpora

DD Gordillo, J Timoneda, SV Vera - arXiv preprint arXiv:2401.09333, 2024

Current methods to identify and classify racist language in text rely on small-n qualitative approaches or large-n approaches focusing exclusively on overt forms of racist discourse. This article provides a step-by-step generalizable guideline to …


MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions

K Zhang, Y Luan, H Hu, K Lee, S Qiao, W Chen, Y Su… - arXiv preprint arXiv …, 2024

… We collect all images with the same URL from Common Crawl2 as a group of images from the same web page for potential pairing. Due to the inevitable noisy images introduced by simple grouping, we remove duplicated, low-resolution, and …


MAiDE-up: Multilingual Deception Detection of GPT-generated Hotel Reviews

O Ignat, X Xu, R Mihalcea - arXiv preprint arXiv:2404.12938, 2024

Deceptive reviews are becoming increasingly common, especially given the increase in performance and the prevalence of LLMs. While work to date has addressed the development of models to differentiate between truthful and …


MaiNLP at SemEval-2024 Task 1: Analyzing Source Language Selection in Cross-Lingual Textual Relatedness

S Zhou, H Shan, B Plank, R Litschko - arXiv preprint arXiv:2404.02570, 2024

This paper presents our system developed for the SemEval-2024 Task 1: Semantic Textual Relatedness (STR), on Track C: Cross-lingual. The task aims to detect semantic relatedness of two sentences in a given target language without access to …


MambaByte: Token-free Selective State Space Model

J Wang, T Gangavarapu, JN Yan, AM Rush - arXiv preprint arXiv:2401.13660, 2024

Token-free language models learn directly from raw bytes and remove the bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences, and standard autoregressive Transformers scale poorly in such settings …


MAmmoTH2: Scaling Instructions from the Web

X Yue, T Zheng, G Zhang, W Chen - arXiv preprint arXiv:2405.03548, 2024

… We argue that the pre-training corpus (eg, Common Crawl) already contains a vast amount of high-quality instruction data for LLM reasoning. For example, the web corpus contains a large amount of educational materials in the form of instruction-following …


MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series

G Zhang, S Qu, J Liu, C Zhang, C Lin, CL Yu, D Pan… - arXiv preprint arXiv …, 2024

… Inspired by [89], which adopts an iterative pipeline to facilitate the acquisition of large-scale, high-quality data from Common Crawl, we propose to select high-quality data for mathematics, scientific exam synthetic data, and wiki-based content in our Matrix. …


Mapping the media landscape: predicting factual reporting and political bias through web interactions

D Sánchez-Cortés, S Burdisso, E Villatoro-Tello… - International Conference of …, 2024

… In this work we used the graph G built in [8] consisting of 17K news sources obtained after processing 100M news articles from Common Crawl News. Following [3, 8] we report 5-fold cross-validation evaluation results on our MBFC datasets …


Mapping the Positive Self-Bias Embedded in Human Languages

… Here, we employed the fastText algorithm 62, pre-trained on the Common Crawl and Wikipedia corpora, for the analyses across Studies 1a-c. We opted for this model due to its leading performance, widespread acceptance, and multilingual …


MapPool-Bubbling up an extremely large corpus of maps for AI

R Schnürer - 2024 ICA Workshop on AI, Geovisualization, and …, 2024

MapPool is a dataset of 75 million potential maps and textual captions. It has been derived from CommonPool, a dataset consisting of 12 billion text-image pairs from the Internet. The images have been encoded by a vision transformer and classified …


Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking

Y Fung, R Zhao, J Doo, C Sun, H Ji - arXiv preprint arXiv:2402.09369, 2024

Pretrained large language models have revolutionized many applications but still face challenges related to cultural bias and a lack of cultural commonsense knowledge crucial for guiding cross-culture communication and interactions …


MASTERARBEIT/MASTER'S THESIS

Today, a huge amount of unstructured data from various sources is available. Natural language processing (NLP) techniques using such data are applied to several tasks such as named entity recognition (NER). This study is designed to …


MASTERARBEIT| MASTER'S THESIS

The exponential growth of the social media user base has led to an alarming rise in the use of offensive language and profanity, necessitating effective detection mechanisms. This thesis explores the challenges and advancements in automatic …


Mastering Transformers: The Journey from BERT to Large Language Models and Stable Diffusion

Explore transformer-based language models from BERT to GPT, delving into NLP and computer vision tasks, while tackling challenges effectively Key Features Understand the complexity of deep learning architecture and transformers …


Materials science in the era of large language models: a perspective

G Lei, R Docherty, SJ Cooper - arXiv preprint arXiv:2403.06949, 2024

Large Language Models (LLMs) have garnered considerable interest due to their impressive natural language capabilities, which in conjunction with various emergent properties make them versatile tools in workflows ranging from complex …


Matthew Coscia and

… URL: https://github.com/commoncrawl/ia-web-commons. [9] Image Details - Container Registry - Google Cloud Platform. 2022. …


MDSAA

In the field of digital marketing, efficient content gap analysis is crucial for developing effective SEO strategies. Traditional approaches to this task are time-consuming, representing a challenge for Organic Performance teams, who must process large …


Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse

M Song, SH Sim, R Bhardwaj, HL Chieu, N Majumder… - arXiv preprint arXiv …, 2024

… To do this, we query Wikipedia and Common Crawl to retrieve the 100 most relevant documents. We filter seed question for which the retriever fails to retrieve relevant documents. Furthermore, we identify 5 documents that are equally effective …


Measuring and Improving the Energy Efficiency of Large Language Models Inference

MF Argerich, M Patiño-Martínez - IEEE Access, 2024

… 4) RedPajama [42]: another family of models based on Pythia with 7B and 3B parameters, fine-tuned on a dataset with the same name as the model, with over 100B text documents coming from 84 CommonCrawl snapshots. Similar to Dolly …


Measuring complex psychological and sociological constructs in large-scale text

A Herderich, J Lasser, S Aroyehun, D Garcia…

In recent years, there has been an increasing exchange between social and computational sciences. Methods from natural language processing enable social scientists to systematically process large amounts of text. Rich psycho-sociological …


Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain

I García-Ferrero, R Agerri, AA Salazar, E Cabrio… - arXiv preprint arXiv …, 2024

… mT5 was trained using mC4, a 1 Trillion token Common Crawl-based dataset covering 101 languages. The pre-training is based on a masked language modeling “spancorruption” objective, where consecutive spans of input tokens are replaced …


Meeting the challenge: A benchmark corpus for automated Urdu meeting summarization

B Sadia, F Adeeba, S Shams, K Javed - Information Processing & Management, 2024

Meeting summarization has become crucial as the world is gradually shifting towards remote work. Nowadays, automation of meeting summary generation is really needed in order to minimize the time and effort. The surge in online meetings …


MELTing point: Mobile Evaluation of Language Transformers

S Laskaridis, K Kateveas, L Minto, H Haddadi - arXiv preprint arXiv:2403.12844, 2024

Transformers have revolutionized the machine learning landscape, gradually making their way into everyday tasks and equipping our computers with ``sparks of intelligence''. However, their runtime requirements have prevented them from being …


Memorization and Privacy Risks in Domain-Specific Large Language Models

X Yang, Z Wen, W Qu, Z Chen, Z Xiang, B Chen, H Yao - ICLR 2024 Workshop on Reliable …

… Common Crawl3 is a nonprofit organization that provides a large and open web crawl data repository for public use. It collects web pages from the internet every month and stores them on Amazon Web Services. Common Crawl’s … C4 cleanses …


Mentions of prejudice in news media–an international comparison

D Rozado - Journal of Computational Social Science, 2024

Previous research has identified a post-2010 sharp increase of terms used to denounce prejudice (ie racism, sexism, homophobia, Islamophobia, anti-Semitism, etc.) in US and UK news media content. Here, we extend previous analysis to an …


MeSH2Matrix: Combining MeSH keywords and machine learning for biomedical relation classification based on PubMed

H Turki, BFP Dossou, CC Emezue, A Toluwase…

Biomedical relation classification has been significantly improved by the application of advanced machine learning techniques on the raw texts of scholarly publications. Despite this improvement, the reliance on large chunks of raw text makes these …


Methods Futures Report–Sea, Sky, and Land: Engaging in Solidarity in Endangered Ecologies–4S Conference Nov 2023

This is a report of the 4S 2023 conference. The 4S (Society for the Social Studies of Science) conference 2023 was a large, disciplinarily diverse gathering with over 400 sessions spread over four days. Navigating a particular route through the conference …


mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval

X Zhang, Y Zhang, D Long, W Xie, Z Dai, J Tang, H Lin… - arXiv preprint arXiv …, 2024

We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained …


Milestones in Bengali Sentiment Analysis leveraging Transformer-models: Fundamentals, Challenges and Future Directions

S Sengupta, S Ghosh, P Mitra, TI Tamiti - arXiv preprint arXiv:2401.07847, 2024

Sentiment Analysis (SA) refers to the task of associating a view polarity (usually, positive, negative, or neutral; or even fine-grained such as slightly angry, sad, etc.) to a given text, essentially breaking it down to a supervised (since we have the view …


MindLLM: Lightweight large language model pre-training, evaluation and domain application

Y Yang, H Sun, J Li, R Liu, Y Li, Y Liu, Y Gao, H Huang - AI Open, 2024

Large Language Models (LLMs) have demonstrated remarkable performance across various natural language tasks, marking significant strides towards general artificial intelligence. While general artificial intelligence is leveraged by developing …


MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

S Hu, Y Tu, X Han, C He, G Cui, X Long, Z Zheng… - arXiv preprint arXiv …, 2024

The burgeoning interest in developing Large Language Models (LLMs) with up to trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This …


MINNESOTA JOURNAL OF LAW, SCIENCE & TECHNOLOGY

Z Niesel, J Villasenor, J Perl, DE Ho, A Vallebueno

In administrative law, there is perhaps no more misunderstood phrase than the" arbitrary and capricious" standard of judicial review. American administrative law's cornerstone is the judicial review of agency decision-making-a review which has …


MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens

A Awadalla, L Xue, O Lo, M Shu, H Lee, EK Guha… - arXiv preprint arXiv …, 2024

… We follow OBELICS’s method for extracting interleaved multimodal documents from CommonCrawl WARC files by parsing each WARC entry’s DOM tree. While OBELICS only processed documents from February 2020 to February 2023 …


Misinformation Resilient Search Rankings with Webgraph-based Interventions

P Carragher, EM Williams, KM Carley - arXiv preprint arXiv:2404.08869, 2024

… To this end, we investigate the distribution of changes in CommonCrawl PageRank as a result of our interventions for all domains in the webgraph, not just news domains. We then take the domains whose PR score changes significantly ( 50%) …


Mission Critical--Satellite Data is a Distinct Modality in Machine Learning

E Rolf, K Klemmer, C Robinson, H Kerner - arXiv preprint arXiv:2402.01444, 2024

Satellite data has the potential to inspire a seismic shift for machine learning -- one in which we rethink existing practices designed for traditional data modalities. As machine learning for satellite data (SatML) gains traction for its real-world impact …


Mitigating Catastrophic Forgetting in Language Transfer via Model Merging

A Alexandrov, V Raychev, MN Müller, C Zhang… - arXiv preprint arXiv …, 2024

As open-weight large language models (LLMs) achieve ever more impressive performances across a wide range of tasks in English, practitioners aim to adapt these models to different languages. However, such language adaptation is often …


Mitigating the Impact of Adversarial Fine-tuning on Large Language Models through Perturbation Attenuation

Z Jejesi, A McBride, F Kovacic, L O'Connor - 2024

The increasing complexity and scale of machine learning models have rendered them vulnerable to adversarial manipulation, particularly during fine-tuning processes where subtle perturbations can lead to significant degradation in model …


Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data

S Zeng, J Zhang, P He, J Ren, T Zheng, H Lu, H Xu… - arXiv preprint arXiv …, 2024

… 2021), we randomly select segments from the Common Crawl dataset to function as the information component. However, the randomness of the input may affect the command component. To mitigate this issue, we limit the maximum length of the …


MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures

J Ni, F Xue, X Yue, Y Deng, M Shah, K Jain, G Neubig… - arXiv preprint arXiv …, 2024

… In the detection phase, we train open-source LLMs on self-collected data to detect queries in Common Crawl splits. During filtering, we utilize GPT-4 Turbo to exclude non-query sentences. In classification, we categorize the filtered queries by input …


ML-EAT: A Multilevel Embedding Association Test for Interpretable and Transparent Social Science

R Wolfe, A Hiniker, B Howe - arXiv preprint arXiv:2408.01966, 2024

This research introduces the Multilevel Embedding Association Test (ML-EAT), a method designed for interpretable and transparent measurement of intrinsic bias in language technologies. The ML-EAT addresses issues of ambiguity and difficulty in …


MMRA: A Benchmark for Multi-granularity Multi-image Relational Association

S Wu, K Zhu, Y Bai, Y Liang, Y Li, H Wu, J Liu, R Liu… - arXiv preprint arXiv …, 2024

Given the remarkable success that large visual language models (LVLMs) have achieved in image perception tasks, the endeavor to make LVMLs perceive the world like humans is drawing increasing attention. Current multi-modal benchmarks …


Mobile Generative AI: Opportunities and Challenges

Y Zhang, J Zhang, S Yue, W Lu, J Ren, X Shen - IEEE Wireless Communications, 2024

Recently, generative artificial intelligence (GenAI) has gained significant interest on a global scale, particularly with the explosion of some killer GenAl applications, like ChatGPT. However, due to the excessively large sizes of generative models, most …


MobileVLM: A Fast, Reproducible and Strong Vision Language Assistant for Mobile Devices

X Chu, L Qiao, X Lin, S Xu, Y Yang, Y Hu, F Wei… - arXiv preprint arXiv …, 2023

We present MobileVLM, a competent multimodal vision language model (MMVLM) targeted to run on mobile devices. It is an amalgamation of a myriad of architectural designs and techniques that are mobile-oriented, which comprises a set of language …


Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation

Ensuring the verifiability of model answers is a fundamental challenge for retrieval-augmented generation (RAG) in the question answering (QA) domain. Recently, self-citation prompting was proposed to make large language models (LLMs) generate citations …


Modeling Caption Diversity in Contrastive Vision-Language Pretraining

S Lavoie, P Kirichenko, M Ibrahim, M Assran… - arXiv preprint arXiv …, 2024

There are a thousand ways to caption an image. Contrastive Language Pretraining (CLIP) on the other hand, works by mapping an image and its caption to a single vector -- limiting how well CLIP-like models can represent the diverse ways to describe an …


Modeling the Sacred: Considerations when Using Considerations when Using Religious Texts in Natural Language Processing

B Hutchinson - arXiv preprint arXiv:2404.14740, 2024

This position paper concerns the use of religious texts in Natural Language Processing (NLP), which is of special interest to the Ethics of NLP. Religious texts are expressions of culturally important values, and machine learned models have a …


Modelling Multimodal Integration in Human Concept Processing with Vision-and-Language Models

A Bavaresco, MH Kloots, S Pezzelle, R Fernández - arXiv preprint arXiv:2407.17914, 2024

Representations from deep neural networks (DNNs) have proven remarkably predictive of neural activity involved in both visual and linguistic processing. Despite these successes, most studies to date concern unimodal DNNs, encoding either …


Modelling Visual Semantics via Image Captioning to extract Enhanced Multi-Level Cross-Modal Semantic Incongruity Representation with Attention for Multimodal …

S Aggarwal, A Pandey, DK Vishwakarma - arXiv preprint arXiv:2408.02595, 2024

… With over 2 terabytes of cleaned CommonCrawl data [45], the masked language model [44] has already been pre-trained on over a hundred languages, including Hindi. We chose this particular variant of [46] for our investigation since the majority …


Mono-lingual text reuse detection for the Urdu language at lexical level

A Noreen, I Muneer, RMA Nawab - Engineering Applications of Artificial Intelligence, 2024

Text reuse is the process of creating new texts from pre-existing ones. In recent years, Urdu Text Reuse Detection (U-TRD) has garnered the attention of researchers due to the ready availability of digital text all over the internet, which can be copied or …


mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus

M Futeral, A Zebaze, PO Suarez, J Abadji, R Lacroix… - arXiv preprint arXiv …, 2024

… This motivated us to collect and release the first large-scale multilingual and multimodal document dataset derived from Common Crawl.Our dataset, multimodal OSCAR (mOSCAR), follows the OSCAR initiative [Ortiz Suárez et al., 2019, Abadji et …


MoVA: Adapting Mixture of Vision Experts to Multimodal Context

Z Zong, B Ma, D Shen, G Song, H Shao, D Jiang, H Li… - arXiv preprint arXiv …, 2024

As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in …


mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models

P Lin, C Hu, Z Zhang, AFT Martins, H Schütze - Findings of the Association for …, 2024

Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining. It remains an open question whether it is feasible to employ mPLMs to …


mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding

A Hu, H Xu, J Ye, M Yan, L Zhang, B Zhang, C Li… - arXiv preprint arXiv …, 2024

Structure information is critical for understanding the semantics of text-rich images, such as documents, tables, and charts. Existing Multimodal Large Language Models (MLLMs) for Visual Document Understanding are equipped with text recognition …


MQM-Chat: Multidimensional Quality Metrics for Chat Translation

Y Li, J Suzuki, M Morishita, K Abe, K Inui - arXiv preprint arXiv:2408.16390, 2024

The complexities of chats pose significant challenges for machine translation models. Recognizing the need for a precise evaluation metric to address the issues of chat translation, this study introduces Multidimensional Quality Metrics for Chat …


MS MARCO Web Search: a Large-scale Information-rich Web Dataset with Millions of Real Click Labels

Q Chen, X Geng, C Rosset, C Buractaon, J Lu, T Shen… - Companion Proceedings of …, 2024

Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals. In this paper, we introduce MS MARCO Web Search, the first large-scale information-rich web dataset, featuring millions of real clicked …


Multi Class Depression Detection Through Tweets using Artificial Intelligence

MO Nusrat, W Shahzad, SA Jamal - arXiv preprint arXiv:2404.13104, 2024

… Typically, GloVe embeddings were pretrained on extensive corpora like Wikipedia or Common Crawl and fine-tuned for specific tasks. 300 LSTM units were used with a dropout rate of 0.4. For depression classification, the dense layer was …


Multi-class hate speech detection in the Norwegian language using FAST-RNN and multilingual fine-tuned transformers

E Hashmi, SY Yayilgan - Complex & Intelligent Systems, 2024

… FastText, a word representation tool developed by Facebook’s research division, offers both unsupervised and supervised modes and features a comprehensive database of 2 million words from Common Crawl, each represented by 300 …


Multi-Head Mixture-of-Experts

X Wu, S Huang, W Wang, F Wei - arXiv preprint arXiv:2404.15045, 2024

Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activation, where only a small subset of experts are activated for …


Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation

J Choi, J Yun, K Jin, YB Kim - arXiv preprint arXiv:2404.09682, 2024

… an analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


Multi-Reference Benchmarks for Russian Grammatical Error Correction

FP Gomez, A Rozovskaya - Proceedings of the 18th Conference of the European …, 2024

This paper presents multi-reference benchmarks for the Grammatical Error Correction (GEC) of Russian, based on two existing single-reference datasets, for a total of 7,444 learner sentences from a variety of first language backgrounds. Each …


Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation

M Guan, Y Wang, G Ma, J Liu, M Sun - arXiv preprint arXiv:2405.05672, 2024

Sign language serves as a non-vocal means of communication, transmitting information and significance through gestures, facial expressions, and bodily movements. The majority of current approaches for sign language recognition (SLR) …


Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings

I Mohr, M Krimmel, S Sturua, MK Akram, A Koukounas… - arXiv preprint arXiv …, 2024

… Seeking further diversity and volume, we utilize Common Crawl data to create two types of pairs: one from web page titles and their contents, and another by mining questionanswer pairs from FAQ and support-related pages. Additionally, we pair …


Multidimensional Text Feature Analysis: Unveiling the Veil of Automatically Generated Text

… Given the widespread use of transformer-based pre-trained models in various downstream tasks and the availability of multilingual datasets in official task data, XLM-RoBERTa[19] is a multilingual version of Roberta that has been pre-trained on …


Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixture

J Song, Y Huang, Z Zhou, L Ma - arXiv preprint arXiv:2407.07342, 2024

… We consider most state-of-the-art LLMs to have trained on these 55 source languages, as these languages are enclosed in the CommonCrawl corpus. All multilingual translations are conducted using Google Translate API3. All codes for …


Multilingual Detection of Cyberbullying in Mixed Urdu, Roman Urdu, and English Social Media Conversations

… The pre-training data is carefully selected from Common Crawl, a vast collection of web pages, to ensure that it’s both high-quality and diverse. This diverse training data allows m-BERT to learn language-agnostic representations, making it easier to …


Multilingual Diversity Improves Vision-Language Representations

T Nguyen, M Wallingford, S Santy, WC Ma, S Oh… - arXiv preprint arXiv …, 2024

… Data We experiment with the medium pool of the DataComp benchmark [14], which consists of 128M image-text pairs randomly sampled from Common Crawl dumps between 2014 and 2022, and deduplicated. Unlike other heavily filtered …


Multilingual E5 Text Embeddings: A Technical Report

L Wang, N Yang, X Huang, L Yang, R Majumder, F Wei - arXiv preprint arXiv …, 2024

This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding models of different sizes (small / base / large) are provided, offering a …


Multilingual Fake News Detection: A Study on Various Models and Training

R Chalehchaleh, R Farahbakhsh, N Crespi - … Systems and Applications: Proceedings of the …

… XLM-RoBERTa model is pre-trained on 2.5 TB of filtered CommonCrawl data containing 100 languages it was introduced in [8]. LASER (Language-Agnostic SEntence Representations)[3] is trained to capture the semantic meaning of sentences …


Multilingual Hate Speech Detection: Comparison of Transfer Learning Methods to Classify German, Italian, and Spanish Posts

J Fillies, MP Hoffmann, A Paschke - 2023 IEEE International Conference on Big Data …, 2023

… This datasetincludes Wikipedia dumps for all supported languages anda substantial 2.5 terabytes of Common Crawl Data collectedfrom the Internet. The ”RoBERTa” component in the model’sname indicates the adherence to the training procedure …


Multilingual Large Language Models and Curse of Multilinguality

D Gurgurov, T Bäumel, T Anikina - arXiv preprint arXiv:2406.10602, 2024

… Common Crawl The Common Crawl corpus is a vast collection of web data accumulated over more than 10 years through web crawling. It includes raw web page data, metadata extracts, and text extracts. This noncurated dataset includes …


Multilingual Model Fine-tuning for Sentiment Analysis

ML Elrefai, MI Khalil, HM Abbas - … on Intelligent Computing and Information Systems …, 2023

… A sizable multilingual language model was created using 2.5 TB of filtered CommonCrawl data. One of the outstanding XLMRoBERTa in low-resource languages, Low-resource languages are languages with limited available data for …


Multilingual Models in Neural Machine Translation

… XGLM [19] uses the decoder-only model architecture similar to that of GPT-3 [6], and was pre-trained on CC100-XL that covers 68 Common Crawl (CC) snapshots and 134 languages. Experiment results show that XGLM (7.5B) outperforms GPT-3 (6.7B) …


Multilingual Speech and Language Analysis for the Assessment of Mild Cognitive Impairment: Outcomes from the Taukadial Challenge

PA Perez-Toro, T Arias-Vergara, P Klumpp, T Weise…

… Our analysis utilized the BERTBase and RoBERTa-Base models, trained on Wikicorpus data spanning 102 languages and CommonCrawl data filtered from 100 languages [16], respectively. For both models, the final layer, comprising 768 units …


MultiLS: A Multi-task Lexical Simplification Framework

K North, T Ranasinghe, M Shardlow, M Zampieri - arXiv preprint arXiv:2402.14972, 2024

Lexical Simplification (LS) automatically replaces difficult to read words for easier alternatives while preserving a sentence's original meaning. LS is a precursor to Text Simplification with the aim of improving text accessibility to various target …


Multimodal Large Language Models for Bioimage Analysis

S Zhang, G Dai, T Huang, J Chen - arXiv preprint arXiv:2407.19778, 2024

Rapid advancements in imaging techniques and analytical methods over the past decade have revolutionized our ability to comprehensively probe the biological world at multiple scales, pinpointing the type, quantity, location, and even temporal …


Multimodal Learning for Visual Question Answering using World Knowledge

Navigating the frontier of the Visual Turing Test, this research delves into multimodal learning to bridge the gap between visual perception and linguistic interpretation, a foundational challenge in artificial intelligence. It scrutinizes the integration of visual …


Multimodal predictive model for analyzing news data

The aim of this thesis is to create a predictive tool utilising a multimodal machine-learning model for the combined textual and visual data from a dataset of news articles to predict the political orientation of news articles. A thorough exploration of multimodal …


Multiscale cascaded domain-based approach for Arabic fake reviews detection in e-commerce platforms

N Qandos, G Hamad, M Alharbi, S Alturki, W Alharbi… - Journal of King Saud …, 2024

Fake reviews in e-commerce can lead to customer deception and financial losses. Despite the importance of fake reviews detection, studies for Arabic language are scarce due to the lack of comprehensive datasets. This study addresses this gap by …


Multivariate Log-based Anomaly Detection for Distributed Database

L Zhang, T Jia, M Jia, Y Li, Y Yang, Z Wu - arXiv preprint arXiv:2406.07976, 2024

… Following this, MultiLog utilizes pre-trained word vectors based on the Common Crawl Corpus with the FastText algorithm[45], which is adept at capturing the inherent relationships among words in natural language. This means each word in a …


Musical Word Embedding for Music Tagging and Retrieval

SH Doh, J Lee, D Jeong, J Nam - arXiv preprint arXiv:2404.13569, 2024

… We used two publicly available embeddings trained on Common Crawl, a large-scale general word corpus, with the GloVe and skipgram … It demonstrates that word embeddings trained with a general word corpus, such as Common Crawl and Wiki …


Myanmar XNLI: Building a Dataset and Exploring Low-resource Approaches to Natural Language Inference with Myanmar

Despite dramatic recent progress in NLP, it is still a major challenge to apply Large Language Models (LLM) to low-resource languages. This is made visible in benchmarks such as Cross-Lingual Natural Language Inference (XNLI), a key task …


Named Entity Recognition in Italian Lung Cancer Clinical Reports using Transformers

D Paolo, A Bria, C Greco, M Russano, S Ramella… - 2023 IEEE International …, 2023

The widespread adoption of electronic health records (EHRs) offers a valuable opportunity to support clinical research by containing crucial patient information, including diagnoses, symptoms, medications, lab tests, and more. Despite the …


NATALITY AT RISK? RAISING DOUBTS ON THE EDUCATIONAL IMPORTANCE OF ChatGPT

Digital tools are part of our daily life. Thus, they also enter educational contexts. With reference to Hannah Arendt’s writings this paper explores if, and to what extent, digital tools can be considered to be helpful and desirable in education. For this …


Native Design Bias: Studying the Impact of English Nativeness on Language Model Performance

M Reusens, P Borchert, J De Weerdt, B Baesens - arXiv preprint arXiv:2406.17385, 2024

Large Language Models (LLMs) excel at providing information acquired during pretraining on large-scale corpora and following instructions through user prompts. This study investigates whether the quality of LLM responses varies depending on …


Natural Language Processing Advancements: Breaking Barriers in Human-Computer Interaction

JGC Ramírez - Journal of Artificial Intelligence General science (JAIGS …, 2024

Natural Language Processing (NLP) advancements have revolutionized human-computer interaction, breaking barriers and opening new frontiers in technology. NLP techniques enable machines to understand, interpret, and generate human …


Natural Language Processing Approaches for Monitoring Health Activities

Monitoring health activities is vital for individual and public health. It identifies trends, detects risks early, and tailors interventions. At the individual level, it informs customized treatments. At the population level, monitoring broad health patterns …


Natural Language Processing for Dialects of a Language: A Survey

A Joshi, R Dabre, D Kanojia, Z Li, H Zhan, G Haffari… - arXiv preprint arXiv …, 2024

State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language …


Natural Language Processing Journal

MN Hoque, U Salma, MJ Uddin, MM Ahamad, S Aktar

In the sentiment analysis (SA) task, we can obtain a positive or negative-typed comment or feedback from an online user or a customer about any object, such as a movie, drama, food, and others. This user’s sentiment may positively impact various …


Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey

DVT Le, L Bigo, M Keller, D Herremans - arXiv preprint arXiv:2402.17467, 2024

Several adaptations of Transformers models have been developed in various domains since its breakthrough in Natural Language Processing (NLP). This trend has spread into the field of Music Information Retrieval (MIR), including studies …


Natural Language Processing Tools for Romanian–Going Beyond a Low-Resource Language

… Oscar (Open Super-large Crawled Aggregated coRpus) [64] is a multilingual dataset obtained via filtering the common crawl corpus. It encompasses 151 languages. The Romanian sub-corpus contains over 4,5 million documents …


Natural Language Processing

CH Focke, RM Sylvester - Statistical Methods in Epilepsy, 2024

In this chapter, we introduce modern natural language processing (NLP) libraries, techniques and their applications. This chapter focuses on deep learning methods and less on computational linguistics that require nuanced knowledge of linguistics …


Navigating Challenges and Technical Debt in Large Language Models Deployment

A Menshawy, Z Nawaz, M Fahmy - Proceedings of the 4th Workshop on Machine …, 2024

… These models, trained on an internet scale data including common crawl, books, and articles, have acquired an extensive understanding of language patterns, nuances, and contexts. This profound comprehension allows LLMs to perform a …


Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging

S Woerner, CF Baumgartner - arXiv preprint arXiv:2408.08058, 2024

… LAION-2B contains 2 billion image-text pairs extracted from common crawl [3] and is the English language subset of the larger LAION-5B [12] … https://commoncrawl.org. [4] Alexey Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image …


Navigating Multimodal Complexity: Advances in Model Design, Dataset Creation, and Evaluation Techniques

Ibn Sina, a philosopher of 11th-century Persia, wrote of a `Floating Man'. This man is floating through a void, without the use of his sight or touch or any of the senses which make us human. Yet as he has a human brain this man, according to Ibn Sina …


Navigating the ethical landscape behind ChatGPT

L Peng, B Zhao - Big Data & Society, 2024

… While ChatGPT and similar text-generating AIs claim to be trained on mostly public data such as the Common Crawl archives, rendering … content in public datasets used for AI training, such as the Common Crawl, which is beyond the …


Navigating the Legal Landscape: Technical Implementation of Copyright Reservations for Text and Data Mining in the Era of AI Language Models

L Löbling, C Handschigl, K Hofmann, J Schwedhelm - JIPITEC–Journal of Intellectual …, 2023

… However, limitations arose from the potential omission of specific pages of a website in individual monthly crawls by Common Crawl. Through manual investigation, a section containing the terms of use could be identified for 40 …


Ncl-uor at semeval-2024 task 8: Fine-tuning large language models for multigenerator, multidomain, and multilingual machine-generated text detection

F Xiong, T Markchom, Z Zheng, S Jung, V Ojha… - Proceedings of the 18th …, 2024

SemEval-2024 Task 8 introduces the challenge of identifying machine-generated texts from diverse Large Language Models (LLMs) in various languages and domains. The task comprises three subtasks: binary classification in monolingual …