Spaces:

znation
/

xet-repo-dedupe

Running

znation HF Staff commited on Jan 7

Commit

eb4214d

1 Parent(s): 990a2a3

clarifications

Files changed (2) hide show

index.html CHANGED Viewed

@@ -18,7 +18,8 @@
 	<body>
 		<div class="card">
 			<h1>Visualizing Repo-level Dedupe</h1>
-      <p>This visualization demonstrates the amount of <a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/blog/from-files-to-chunks">chunk-level dedupe</a> within a repo or across a selection of repos. (For now, demonstrates a hardcoded selection.)</p>
 		</div>
     <div id="vis"></div>
     <script>

 	<body>
 		<div class="card">
 			<h1>Visualizing Repo-level Dedupe</h1>
+      <p>This visualization demonstrates the amount of <a target="_blank" rel="noopener noreferrer" href="https://huggingface.co/blog/from-files-to-chunks">chunk-level dedupe</a> across all public repos.</p>
+      <p>"Dedupe factor" is defined as the number of re-uses of a given "xorb". A "xorb" is a collection of content-defined chunks, typically around 1,000 chunks comprising up to 64 MB of total data.</p>
 		</div>
     <div id="vis"></div>
     <script>

xorbs.json CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:efd8947a02dabec8b1b0228fb97d12ba923d4c33335e36598096fad3cde7f96f
 size 629739

 version https://git-lfs.github.com/spec/v1
+oid sha256:feaca61dcd2c355fa10b3a69d844299b798587574d985693b31234ed550b1a66
 size 629739