WCC-EC 2.0: Enhancing Neural Machine Translation with a 1.6 M+ Web-Crawled English-Chinese Parallel Corpus

J Zhang, K Su, Y Tian, T Matsumoto - Electronics, 2024

This research introduces WCC-EC 2.0 (Web-Crawled Corpus—English and Chinese), a comprehensive parallel corpus designed for enhancing Neural Machine Translation (NMT), featuring over 1.6 million English-Chinese sentence pairs …


We have no idea what we are walking into: AI and ethical considerations

KB Forrest - Annals of the New York Academy of Sciences, 2024

… called “Common Crawl” provide data, “C4” is a curated (de-duplicated and some nonsensical junk eliminated) set of the Common Crawl data, then there is a data set well named “The Pile”, and others that are compilations of books, music …


Weakly supervised learning for an effective focused web crawler

PRJ Dhanith, K Saeed, G Rohith, SP Raja - Engineering Applications of Artificial …, 2024

Focused crawler traverses the Web to only collect pages that are relevant to a particular topic, and is increasingly considered as a way to get around the scalability issues with current general-purpose search engines. But the data diversity in the …


WEATHER-5K: A Large-scale Global Station Weather Dataset Towards Comprehensive Time-series Forecasting Benchmark

T Han, S Guo, Z Chen, W Xu, L Bai - arXiv preprint arXiv:2406.14399, 2024

Global Station Weather Forecasting (GSWF) is crucial for various sectors, including aviation, agriculture, energy, and disaster preparedness. Recent advancements in deep learning have significantly improved the accuracy of weather predictions by …


Web Crawl Refusals: Insights From Common Crawl

… Abstract: This study investigates server-side blocks encountered by Common Crawl, a major … that approximately 1.68% of websites in a Common Crawl snapshot exhibit some form of explicit … of these blocks and the effectiveness of …


Web Data Commons-RDFa, Microdata, Embedded JSON-LD, and Microformats Data Sets-October 2023

The Web Data Commons RDFa, Microdata and Microformats data sets has been extracted from the September/October 2023 release of the Common Crawl. In summary, we found structured data within 1.7 billion HTML pages out of the 3.4 …


Web Data Commons-Schema. org Table Corpus 2023

The WDC schema.org Table Corpus consists of ~5 million relational tables which were generated by extracting schema.org data from the Common Crawl and grouping the data into seperate tables for each class/host combination, eg all …


Web scraping and Generative Models training in the Directive 790/19

… The companies that created GM never disclosed the exact corpus of training data they employed to train their models; however, it has become clear sources that many copyrighted materials were employed as part of larger datasets such as the …


WebCiteS: Attributed Query-Focused Summarization on Chinese Web Search Results with Citations

H Deng, C Wang, X Li, D Yuan, J Zhan, T Zhou, J Ma… - arXiv preprint arXiv …, 2024

Enhancing the attribution in large language models (LLMs) is a crucial task. One feasible approach is to enable LLMs to cite external sources that support their generations. However, existing datasets and evaluation methods in this domain still …


WebGraph: The Next Generation (Is in Rust)

T Fontana, S Vigna, S Zacchiroli - ACM Web Conference 2024, 2024

We report the results of a yearlong effort to port the WebGraph framework from Java to Rust. For two decades WebGraph has been instrumental in the analysis and distribution of large graphs for the research community of TheWebConf, but the …


What are Large Language Models Made of?

C Stinson - Leading the Way: Envisioning the Future of Higher …, 2024

… CommonCrawl is the lowest quality but largest of these datasets, consisting of text scraped from all over the web. Their data from the years 2016 to 2019 were used to train GPT-3. OpenAI’s quality control measure for filtering CommonCrawl was to …


What do BERT word embeddings learn about the French language?

E Goliakova, D Langlois - Sixth International Conference, 2024

Pre-trained word embeddings (for example, BERT-like) have been successfully used in a variety of downstream tasks. However, do all embeddings, obtained from the models of the same architecture, encode information in the same way? Does the …


What's in a Name? Experimental Evidence of Gender Bias in Recommendation Letters Generated by ChatGPT

DM Kaplan, R Palitsky, SJ Arconada Alvarez… - Journal of Medical Internet …, 2024

… In the case of GPT-3.5 (the current version at the time of this study), the model contains 175 billion parameters and was trained with commonly available text training sets that represent a broad swath of data on the internet (Common Crawl) …


When Context Leads but Parametric Memory Follows in Large Language Models

Y Tao, A Hiatt, E Haake, AJ Jetter, A Agrawal - arXiv preprint arXiv:2409.08435, 2024

Large language models (LLMs) have demonstrated remarkable progress in leveraging diverse knowledge sources. This study investigates how nine widely used LLMs allocate knowledge between local context and global parameters when …


When Young Scholars Cooperate with LLMs in Academic Tasks: The Influence of Individual Differences and Task Complexities

J Wang, C Huang, S Yan, W Xie, D He - International Journal of Human–Computer …, 2024

As a novel AI-powered conversational system, large language models (LLMs) have the potential to be used in various applications. Recent advances in LLMs like ChatGPT have made LLM-based academic tools possible. However, most of the …


Where does In-context Translation Happen in Large Language Models

S Sia, D Mueller, K Duh - arXiv preprint arXiv:2403.04510, 2024

Self-supervised large language models have demonstrated the ability to perform Machine Translation (MT) via in-context learning, but little is known about where the model performs the task with respect to prompt instructions and demonstration …


Which Word Embeddings for Modeling Web Search Queries? Application to the Study of Search Strategies

In order to represent the global strategies deployed by a user during an information retrieval session on the Web, we compare different pretrained vector models capable of representing the queries submitted to a search engine. More precisely …


Who's Afraid of ChatGPT?: Large Language Models and Conspiracies

… GPT-3 added Common Crawl to the training datasets, 570GB of text obtained after filtering from 45TB of plaintext. The filtering was done by comparing new documents to documents in WebText, with the latter acting as a proxy for high-quality …


Willkommens-Merkel, Chaos-Johnson, and Tore-Klose: Modeling the Evaluative Meaning of German Personal Name Compounds

We present a comprehensive computational study of the under-investigated phenomenon of personal name compounds (PNCs) in German such as Willkommens-Merkel (‘Welcome-Merkel’). Prevalent in news, social media, and …


Worldwide Federated Training of Language Models

A Iacob, L Sani, B Marino, P Aleksandrov, ND Lane - arXiv preprint arXiv:2405.14446, 2024

… To simulate geographically distributed systems, we use a subset of Multilingual Colossal Common Crawl (mC4)[81], covering high and low-… For IID experiments, we use the standard Cleaned Colossal Common Crawl (C4) English dataset …


WRITTEN IN PERSIAN USING A TRANSFORMER-BASED MODEL

T Firoozi, MJ Gierl - The Routledge International Handbook of Automated …, 2024

… This model is created using contents from Common Crawl and Wikipedia. It is trained using CBOW with position weights, in dimension 300, with character n-grams of length 5, and a window of size 5 and 10. The model gives 300-dimensional vector …


X-Phishing-Writer: A Framework for Cross-Lingual Phishing Email Generation

SW Guo, YC Fan - ACM Transactions on Asian and Low-Resource …, 2024

… The mBART model employs various language corpora from the Common Crawl dataset and undergoes multilingual pretraining based on the BART architecture. The training data is a concatenation of data from � diferent languages, denoted as � = �1,�2...�� …


XLM for Autonomous Driving Systems: A Comprehensive Review

S Fourati, W Jaafar, N Baccar, S Alfattani - arXiv preprint arXiv:2409.10484, 2024

Large Language Models (LLMs) have showcased remarkable proficiency in various information-processing tasks. These tasks span from extracting data and summarizing literature to generating content, predictive modeling, decision-making …


XLMR4MD: New Vietnamese dataset and framework for detecting the consistency of description and permission in Android applications using large language models

QN Nguyen, NT Cam, K Van Nguyen - Computers & Security, 2024

… The most significant update that XLMRoBERTa provides compared to the original is that the amount of training data has increased significantly, the cleaned CommonCrawl data it was trained on takes up a considerable memory block of up to …


XLORE 3: A Large-scale Multilingual Knowledge Graph from Heterogeneous Wiki Knowledge Resources

K Zeng, H Jin, X Lv, F Zhu, L Hou, Y Zhang, F Pang… - ACM Transactions on …, 2024

In recent years, Knowledge Graph (KG) has attracted significant attention from academia and industry, resulting in the development of numerous technologies for KG construction, completion, and application. XLORE is one of the largest …


Yi: Open Foundation Models by 01. AI

A Young, B Chen, C Li, C Huang, G Zhang, G Zhang… - arXiv preprint arXiv …, 2024

… We start with web documents from Common Crawl, use the CCNet pipeline [79] for language identification and perplexity scoring. Then … Notably, the Chinese content extracted from Common Crawl present unique challenges, particularly with a …


Yuan 2.0-M32: Mixture of Experts with Attention Router

S Wu, J Luo, X Chen, L Li, X Zhao, T Yu, C Wang… - arXiv preprint arXiv …, 2024

Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a mixture-of-experts architecture with 32 experts of which 2 experts are active. A new router network, Attention Router, is proposed and adopted for a more efficient selection of experts …


YuLan: An Open-source Large Language Model

Y Zhu, K Zhou, K Mao, W Chen, Y Sun, Z Chen, Q Cao… - arXiv preprint arXiv …, 2024

… In addition to these sources, we process raw data from Common Crawl (CC) dumps, particularly focusing on events that occurred between January 2021 and February 2023. To manage the practical challenges of HTML content extraction and …


Zero and Few Short Learning Using Large Language Models for De-Identification of Medical Records

YS Yashwanth, R Shettar - IEEE Access, 2024

… Among the word embeddings, CommonCrawl, a general English word embedding, demonstrated superior performance compared to the other four. The highest F-measure for CommonCrawl was 96.46, with corresponding precision and recall scores of …


Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Data

N Popp, JH Metzen, M Hein - arXiv preprint arXiv:2404.16637, 2024

… The first one involves relying on large-scale data such as common crawl datasets [7,38]. While this approach is feasible for large foundation models, it poses challenges for smaller models as these lack the same level of generalization capabilities due to …


Zero-shot learning for multilingual discourse relation classification

E Metheniti, P Muller, C Braud, M Hernández-Casas - 5th Workshop on …, 2024

Classifying discourse relations is a hard task: discourse-annotated data is scarce, especially for languages other than English, and there exist different theoretical frameworks that affect textual spans to be linked and the label set used. Thus, work …


Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon

F Koto, T Beck, Z Talat, I Gurevych, T Baldwin - arXiv preprint arXiv:2402.02113, 2024

Improving multilingual language models capabilities in low-resource languages is generally difficult due to the scarcity of large-scale data in those languages. In this paper, we relax the reliance on texts in low-resource languages by using …


Zero-Shot vs. Few-Shot Multi-Speaker TTS Using Pre-trained Czech SpeechT5 Model

J Lehečka, Z Hanzlíček, J Matoušek, D Tihelka - arXiv preprint arXiv:2407.17167, 2024

… Text Data As a source of unlabeled text data, we used the Common Crawl project4 which is a huge public web archive consisting of petabytes of crawled web pages. We used the language information provided in the index files to select Czech …


ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using ChatGPT

A Kabir, S Wang, Y Tian, M Asaduzzaman, W Zhang - arXiv preprint arXiv …, 2024

Technical question and answering (Q&A) sites such as Stack Overflow have become an important source for software developers to seek knowledge. However, code snippets on Q&A sites are usually uncompilable and semantically incomplete for …


Zyda: A 1.3 T Dataset for Open Language Modeling

Y Tokpanov, B Millidge, P Glorioso, J Pilault, A Ibrahim… - arXiv preprint arXiv …, 2024

… Additionally, unlike these works which directly filter from Common Crawl, we aim to instead collect and collate existing open-source datasets under one banner, and to uniformly filter and deduplicate them against one another so as to create a dataset …