Upload all models and assets for dz (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: dz
-language_name: DZ
+language_name: Dzongkha
 language_family: tibetoburman_tibetic
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-tibetoburman_tibetic
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 7.097
+    value: 5.510
   - name: best_isotropy
     type: isotropy
-    value: 0.7372
+    value: 0.6999
   - name: vocabulary_size
     type: vocab
-    value: 4041
-generated: 2025-12-30
+    value: 0
+generated: 2026-01-04
 ---
 
-# DZ - Wikilangs Models
+# Dzongkha - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **DZ** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Dzongkha** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -44,12 +54,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +70,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,58 +80,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 4.839x | 4.82 | 0.1103% | 866,377 |
-| **16k** | 5.558x | 5.53 | 0.1267% | 754,348 |
-| **32k** | 6.305x | 6.28 | 0.1438% | 664,948 |
-| **64k** | 7.097x 🏆 | 7.07 | 0.1618% | 590,703 |
+| **8k** | 4.484x | 4.49 | 0.0965% | 813,691 |
+| **16k** | 4.768x | 4.77 | 0.1026% | 765,197 |
+| **32k** | 5.092x | 5.09 | 0.1096% | 716,539 |
+| **64k** | 5.510x 🏆 | 5.51 | 0.1185% | 662,175 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `ཡུ་ནའི་ཊེཊ་ཨ་རབ་ཨེ་མི་རེཊསི 
-
- དུ་བ་ཡེ
-
-Category:རྒྱལ་ཁབ
-Category:ཨེ་ཤི་ཡ`
+**Sample 1:** `རྒྱལ་ཁབ ཇཱ་པཱན། 日本 ཇ་པན་གྱི་རྒྱལ་ཁབ་འདི་ཤར་ཨེ་ཤི་ཡ་ལུ་ཆགས་ཏི་ཡོད་མི་མཚོ་གླིང་གྱི...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ཡུ་ ནའི་ ཊེ ཊ་ ཨ་ར བ་ ཨེ་ མི་ རེ ཊ ... (+12 more)` | 22 |
-| 16k | `▁ཡུ་ནའི་ཊེཊ་ ཨ་རབ་ ཨེ་ མི་ རེ ཊ སི ▁ད ུ་ བ་ ... (+7 more)` | 17 |
-| 32k | `▁ཡུ་ནའི་ཊེཊ་ ཨ་རབ་ ཨེ་ མི་རེ ཊ སི ▁དུ་ བ་ ཡེ ▁category ... (+5 more)` | 15 |
-| 64k | `▁ཡུ་ནའི་ཊེཊ་ ཨ་རབ་ཨེ་མི་རེ ཊ སི ▁དུ་བ་ཡེ ▁category : རྒྱལ་ཁབ ▁category : ... (+1 more)` | 11 |
+| 8k | `▁རྒྱལ་ཁབ ▁ཇ ཱ་ པ ཱན། ▁ 日本 ▁ཇ་པ ན་ གྱི་རྒྱལ་ཁབ་ ... (+31 more)` | 41 |
+| 16k | `▁རྒྱལ་ཁབ ▁ཇཱ་པཱན། ▁ 日本 ▁ཇ་པན་ གྱི་རྒྱལ་ཁབ་ འདི་ ཤར་ཨེ་ཤི་ཡ་ ལུ་ཆག���་ ཏི་ ... (+23 more)` | 33 |
+| 32k | `▁རྒྱལ་ཁབ ▁ཇཱ་པཱན། ▁ 日本 ▁ཇ་པན་ གྱི་རྒྱལ་ཁབ་ འདི་ཤར་ཨེ་ཤི་ཡ་ ལུ་ཆགས་ཏི་ ཡོད་མི་ མཚོ་གླིང་གྱི་ ... (+12 more)` | 22 |
+| 64k | `▁རྒྱལ་ཁབ ▁ཇཱ་པཱན། ▁ 日本 ▁ཇ་པན་ གྱི་རྒྱལ་ཁབ་ འདི་ཤར་ཨེ་ཤི་ཡ་ ལུ་ཆགས་ཏི་ ཡོད་མི་ མཚོ་གླིང་གྱི་ ... (+12 more)` | 22 |
 
-**Sample 2:** `གཟའནེཔ་ཊུན ༺རྟགས༔ 20px|♆༻
-Category:གནམ་རིག`
+**Sample 2:** `སེམས་ཅན བྱི་ལི ཁྱི ཉ སྟག​ བྱམོ དོམ ལུག རྟ བྱི་ཙི པར་རིས་བར་འཁྱམས། ཁུངས་གཏུག། ཕྱི...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁གཟ འ ན ེ པ་ ཊ ུན ▁ ༺ རྟ ... (+13 more)` | 23 |
-| 16k | `▁གཟ འ ནེ པ་ ཊ ུན ▁ ༺ རྟ གས༔ ... (+9 more)` | 19 |
-| 32k | `▁གཟའ ནེ པ་ཊ ུན ▁ ༺ རྟགས༔ ▁ 2 0 ... (+6 more)` | 16 |
-| 64k | `▁གཟའ ནེ པ་ཊ ུན ▁ ༺ རྟགས༔ ▁ 2 0 ... (+6 more)` | 16 |
+| 8k | `▁སེམས་ ཅན ▁བྱི་ ལི ▁ཁྱ ི ▁ཉ ▁སྟ ག ▁བྱ ... (+15 more)` | 25 |
+| 16k | `▁སེམས་ཅན ▁བྱི་ལི ▁ཁྱ ི ▁ཉ ▁སྟ ག ▁བྱ མོ ▁ད ... (+13 more)` | 23 |
+| 32k | `▁སེམས་ཅན ▁བྱི་ལི ▁ཁྱི ▁ཉ ▁སྟག ▁བྱམོ ▁དོམ ▁ལུག ▁རྟ ▁བྱི་ཙི ... (+5 more)` | 15 |
+| 64k | `▁སེམས་ཅན ▁བྱི་ལི ▁ཁྱི ▁ཉ ▁སྟག ▁བྱམོ ▁དོམ ▁ལུག ▁རྟ ▁བྱི་ཙི ... (+5 more)` | 15 |
 
-**Sample 3:** `གཟའསྤེན་པ ༺རྟགས༔ 20px|♄༻
-Category:གནམ་རིག`
+**Sample 3:** `ཞི་ཆོག་གི་སྐབས་ལུ་འཕུ་ནི་གི་ཆོས་ཆས། རྒྱ་མཚོ་ནང་གི་སེམས་ཅན་ཅིག་གི་ཕྱི་ཤུབས། དུང་ད...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁གཟ འ སྤ ེན་ པ ▁ ༺ རྟ གས ༔ ... (+11 more)` | 21 |
-| 16k | `▁གཟ འ སྤ ེན་ པ ▁ ༺ རྟ གས༔ ▁ ... (+8 more)` | 18 |
-| 32k | `▁གཟའ སྤ ེན་ པ ▁ ༺ རྟགས༔ ▁ 2 0 ... (+6 more)` | 16 |
-| 64k | `▁གཟའསྤེན་ པ ▁ ༺ རྟགས༔ ▁ 2 0 px | ... (+4 more)` | 14 |
+| 8k | `▁ཞི་ ཆོག་ གི་ སྐབས་ལུ་ འཕ ུ་ ནི་གི་ ཆོས་ ཆས། ▁རྒྱ་མཚོ་ ... (+15 more)` | 25 |
+| 16k | `▁ཞི་ ཆོག་ གི་སྐབས་ལུ་ འཕ ུ་ ནི་གི་ ཆོས་ ཆས། ▁རྒྱ་མཚོ་ ནང་གི་ ... (+12 more)` | 22 |
+| 32k | `▁ཞི་ཆོག་ གི་སྐབས་ལུ་ འཕུ་ནི་གི་ ཆོས་ཆས། ▁རྒྱ་མཚོ་ ནང་གི་སེམས་ཅན་ ཅིག་གི་ཕྱི་ཤུབས། ▁དུང་དཀར་གྱི་ མིང་གཞན་ ▁སྐྱེ་བ་ལྔ་པ་ ... (+1 more)` | 11 |
+| 64k | `▁ཞི་ཆོག་ གི་སྐབས་ལུ་ འཕུ་ནི་གི་ ཆོས་ཆས། ▁རྒྱ་མཚོ་ ནང་གི་སེམས་ཅན་ ཅིག་གི་ཕྱི་ཤུབས། ▁དུང་དཀར་གྱི་ མིང་གཞན་ ▁སྐྱེ་བ་ལྔ་པ་ ... (+1 more)` | 11 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 7.097x compression
-- **Lowest UNK Rate:** 8k with 0.1103% unknown tokens
+- **Best Compression:** 64k achieves 5.510x compression
+- **Lowest UNK Rate:** 8k with 0.0965% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -128,57 +139,111 @@ Category:གནམ་རིག`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 405 🏆 | 8.66 | 5,365 | 59.5% | 96.2% |
-| **2-gram** | 305 🏆 | 8.25 | 3,288 | 65.5% | 97.8% |
-| **3-gram** | 2,415 | 11.24 | 27,806 | 28.8% | 71.5% |
-| **3-gram** | 1,793 | 10.81 | 19,053 | 28.1% | 78.4% |
-| **4-gram** | 11,256 | 13.46 | 95,263 | 14.3% | 42.2% |
-| **4-gram** | 7,832 | 12.94 | 68,338 | 14.3% | 45.4% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 11,790 | 13.53 | 28,884 | 11.2% | 35.3% |
+| **2-gram** | Subword | 488 🏆 | 8.93 | 5,527 | 57.6% | 90.8% |
+| **3-gram** | Word | 34,131 | 15.06 | 59,067 | 5.7% | 18.6% |
+| **3-gram** | Subword | 3,461 | 11.76 | 28,498 | 24.5% | 62.8% |
+| **4-gram** | Word | 80,153 | 16.29 | 114,752 | 2.9% | 10.7% |
+| **4-gram** | Subword | 15,479 | 13.92 | 106,273 | 12.4% | 37.5% |
+| **5-gram** | Word | 77,316 | 16.24 | 96,422 | 2.3% | 8.9% |
+| **5-gram** | Subword | 44,243 | 15.43 | 194,726 | 7.1% | 23.4% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ཡོདཔ ཨིན` | 3,325 |
+| 2 | `རྒྱལ ཁབ` | 2,719 |
+| 3 | `སྤྱི ལོ` | 1,933 |
+| 4 | `ཨིན མས` | 1,872 |
+| 5 | `ནང ལུ` | 1,628 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `རིན པོ ཆེ` | 778 |
+| 2 | `ཡོདཔ ཨིན མས` | 778 |
+| 3 | `རྒྱལ ཁབ ནང` | 732 |
+| 4 | `སྤྱི ལོ ལུ` | 688 |
+| 5 | `འབྲུག རྒྱལ ཁབ` | 623 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `རྒྱལ ཁབ ནང ལུ` | 309 |
+| 2 | `འབྲུག རྒྱལ ཁབ ནང` | 288 |
+| 3 | `དཔལ ལྡན འབྲུག པའི` | 272 |
+| 4 | `གུ རུ རིན པོ` | 250 |
+| 5 | `སྡེ སྲིད ཁྲི རབས` | 223 |
+
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `གུ རུ རིན པོ ཆེ` | 184 |
+| 2 | `གནམ ལོ མེད སྤྱི ལོ` | 162 |
+| 3 | `ཞབས དྲུང རིན པོ ཆེ` | 150 |
+| 4 | `རྒྱལ ཡོངས དགའ སྐྱིད དཔལ` | 127 |
+| 5 | `ཡོངས དགའ སྐྱིད དཔལ འཛོམས` | 125 |
+
+**2-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ས ་` | 123,525 |
+| 2 | `ང ་` | 91,851 |
+| 3 | `ན ་` | 70,834 |
+| 4 | `་ _` | 62,281 |
+| 5 | `་ བ` | 59,589 |
+
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ི ་` | 117,504 |
-| 2 | `་ ས` | 79,905 |
-| 3 | `ན ་` | 68,413 |
-| 4 | `་ ར` | 62,786 |
-| 5 | `་ ལ` | 54,596 |
+| 1 | `ག ས ་` | 25,075 |
+| 2 | `ད ང ་` | 18,381 |
+| 3 | `་ ད ང` | 17,725 |
+| 4 | `། _ །` | 15,647 |
+| 5 | `་ པ ་` | 15,536 |
 
-**3-grams:**
+**4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ྱ ི ་` | 22,148 |
-| 2 | `་ དང ་` | 20,477 |
-| 3 | `་ པ ་` | 18,904 |
-| 4 | `་ ལ ུ` | 18,737 |
-| 5 | `ི ན ་` | 18,711 |
+| 1 | `་ ད ང ་` | 17,384 |
+| 2 | `་ པ འི ་` | 13,232 |
+| 3 | `་ ལ ས ་` | 12,579 |
+| 4 | `་ འ དི ་` | 8,184 |
+| 5 | `་ ན ང ་` | 6,539 |
 
-**4-grams:**
+**5-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `་ ར ྒ ྱ` | 16,554 |
-| 2 | `་ པའ ི ་` | 15,863 |
-| 3 | `་ ལ ུ ་` | 15,197 |
-| 4 | `་ མ ི ་` | 11,786 |
-| 5 | `་ ག ི ་` | 11,404 |
+| 1 | `་ ཡོ ད པ ་` | 5,559 |
+| 2 | `་ ལ ས ་ _` | 4,930 |
+| 3 | `་ ད ང ་ _` | 4,145 |
+| 4 | `་ འ བ ད ་` | 3,971 |
+| 5 | `ས ་ པ འི ་` | 3,925 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 305
+- **Best Perplexity:** 2-gram (subword) with 488
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~45% of corpus
+- **Coverage:** Top-1000 patterns cover ~23% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -186,55 +251,86 @@ Category:གནམ་རིག`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.4630 | 1.378 | 3.64 | 8,836 | 53.7% |
-| **1** | 1.1778 | 2.262 | 9.63 | 727 | 0.0% |
-| **2** | 0.3272 | 1.255 | 2.83 | 32,129 | 67.3% |
-| **2** | 0.9588 | 1.944 | 5.50 | 7,001 | 4.1% |
-| **3** | 0.2937 🏆 | 1.226 | 2.25 | 90,896 | 70.6% |
-| **3** | 0.6936 🏆 | 1.617 | 3.16 | 38,506 | 30.6% |
-| **4** | 0.3266 | 1.254 | 2.05 | 204,174 | 67.3% |
-| **4** | 0.5052 | 1.419 | 2.36 | 121,621 | 49.5% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 1.1820 | 2.269 | 14.12 | 12,061 | 0.0% |
+| **1** | Subword | 0.8884 | 1.851 | 7.57 | 1,607 | 11.2% |
+| **2** | Word | 0.5611 | 1.475 | 2.65 | 170,162 | 43.9% |
+| **2** | Subword | 0.6433 | 1.562 | 5.02 | 12,152 | 35.7% |
+| **3** | Word | 0.2267 | 1.170 | 1.41 | 449,950 | 77.3% |
+| **3** | Subword | 0.5247 | 1.439 | 3.26 | 61,009 | 47.5% |
+| **4** | Word | 0.0989 🏆 | 1.071 | 1.15 | 633,460 | 90.1% |
+| **4** | Subword | 0.3500 | 1.275 | 2.11 | 199,035 | 65.0% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `དང སྲས ཚུ གིས ས ཐག མ སླེབས ཚེ འདི ལེགས སོ བཅོམ ཡིད གསུམ གྱི`
+2. `པ ཨིན པས དེ མ མི ཡུལ བྱིན ཅན ཁྱོད འདི གི ལཱ འབད ནི ཀ`
+3. `ལུ བདག སྐྱོང ལེགས སོ དཀརཔོ ཅིག གཅིག པུར ལུ ༡༡ ག གིས པདྨ རིགས མ`
+
+**Context Size 2:**
+
+1. `ཡོདཔ ཨིན པས རྒྱབ རྟེན ༡ དྲག ཤོས ཀྱི གསོལ ར ༤ གློག འཕྲིན གྱི ཁྱབ བདག`
+2. `རྒྱལ ཁབ ཀྱི སྐུ རིམ དང པོ ནས བློ གྲོས བཟང མོ གིས ཨ ལུ འདི ཆ`
+3. `སྤྱི ལོ སྤྱི ཟླ ༤ པ ༡༡ པ ལས འཛིན ཟེར བཙུགས མི མཐོ ཚད ཀི ལོ`
 
-### Generated Text Samples
+**Context Size 3:**
+
+1. `རིན པོ ཆེ སངས རྒྱས ཀུན གྱི སྐུ འཆང བ སངས རྒྱས ཀུན གྱི གསུང ཡང ཡིན རྡོ`
+2. `ཡོདཔ ཨིན མས ཨོ རྒྱན ཆོས གླིང ལྷ ཁང འདི དུས རབས ༨ པའི ནང གུ རུ རིན`
+3. `རྒྱལ ཁབ ནང ལུ ཡང དམངས གཙོའི རིང ལུགས ཀྱི རྒྱལ པོའི བརྟན བཞུགས གི རྩ ཚིག གསར`
+
+**Context Size 4:**
+
+1. `རྒྱལ ཁབ ནང ལུ དཔལ འབྱོར གྱི སྡེ ཚན ཅིག ཡང གཞི གཙུགས འབད དེ འདུག དེ ཡང སྔོན`
+2. `འབྲུག རྒྱལ ཁབ ནང ཡོད པའི རྒྱལ ཁབ ཅིག ཨིན དེ ཡང གྷི རེཊ བིརི ཊེན ཟེར མི འདི`
+3. `དཔལ ལྡན འབྲུག པའི གདུང བརྒྱུད ཅིག ཞུ ནིའི དོན ལུ ཚེས ཉེར དགུ ལུ བླ མ གུ རུ`
 
-Below are text samples generated from each Markov chain model:
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `་ མ ་ གཡ ོ ད ་ པ ོ ག ་ དང ་ བའ ི ང`
-2. `ི ་ ཚ ུ ག ི ན ་ འཕ ེ ༌ འད ྲ ྭ གས ་`
-3. `ོ ། ད ་ པ ་ ད ་ མ ་ ག ་ ས ་ བཙག ་`
+1. `་_ཚར་ཏེ་གིས་ཡོདཔོན་འ`
+2. `_རྫོང་སྟེངས་_ཞེང་བ་རུའི`
+3. `སལཔ་ལུ་ག་བཏུབཟོཔ་ཡིག`
 
 **Context Size 2:**
 
-1. `ི ་ གནས ་ ཀ ྱ ི ་ ལ ུ ་ ར ྒ ྱ ལ ་ མང`
-2. `་ ས ྤ ྱ ོ བས ་ ལ ུ ་ འཐད ་ པ ་ དང ་ གཅ`
-3. `ན ་ ཌ ེ མ ་ ག ི ་ ལ ེ འ ི ་ གསར ་ ར`
+1. `ས་ལུང་ཞིན་པ་འབྲུག་འབྲུག`
+2. `ང་ཁྲུང་ཁབ་སྦྲུལ་ཙ་ཝཊ་ཛ`
+3. `ན་ནང་བླ་མཆོད་ཆོས་དཔ་`
 
 **Context Size 3:**
 
-1. `ྱ ི ་ ཡ ུ དཔ ་ ཐ ེ ངས ་ ཏ ེ ། ། ཡར ་ ཡར`
-2. `་ དང ་ ། ། མས ༌ ཀ ྱ ི ས ་ ག ླ ི ང ་ ག`
-3. `་ པ ་ ཟ ེ ར ་ ག ྱ ི ་ མཐ ུ ད ་ འད ི ་`
+1. `གས་རིག་པའི་ནུས་པ་སྦེ་ཐོན`
+2. `དང་རའི་ཨཔ་ཟླཝ་ག་རང་འ`
+3. `་དང་ཕྱི་མས།_།ཉི་ཟླ་_༢༩`
 
 **Context Size 4:**
 
-1. `་ ར ྒ ྱ ལ ་ ཁབ ་ ནང ་ ལ ུ ་ ཐད ་ ཀར ་ ག ླ`
-2. `་ པའ ི ་ གས ེ ར ་ ག ྱ ི ་ ས ྒ ྲ ུ བ ་ ཆ`
-3. `་ ལ ུ ་ མཁས ་ པ ་ འགའ ་ ཡང ་ མ ི ་ ས ྡ ེ ་`
+1. `་དང་གཅིག་ནང་_ཡན་ལག་ཁ`
+2. `་པའི་བླ་མ་ཐུབ།_།དགེ་བ་སྟོ`
+3. `་ལས་_འབྱུང་ཁུངས།_།དགའ་`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-3 with 70.6% predictability
+- **Best Predictability:** Context-4 (word) with 90.1% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (121,621 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (199,035 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -250,64 +346,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 4,041 |
-| Total Tokens | 1,481,034 |
-| Mean Frequency | 366.50 |
-| Median Frequency | 4 |
-| Frequency Std Dev | 4331.41 |
+| Vocabulary Size | 6,761 |
+| Total Tokens | 898,876 |
+| Mean Frequency | 132.95 |
+| Median Frequency | 6 |
+| Frequency Std Dev | 709.47 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ས | 138,469 |
-| 2 | ག | 99,582 |
-| 3 | ར | 94,106 |
-| 4 | ན | 91,997 |
-| 5 | ལ | 90,195 |
-| 6 | ད | 76,047 |
-| 7 | མ | 55,299 |
-| 8 | ང | 48,577 |
-| 9 | བ | 46,244 |
-| 10 | གས | 35,001 |
+| 1 | དང | 18,802 |
+| 2 | པ | 17,903 |
+| 3 | ལུ | 15,384 |
+| 4 | པའི | 14,560 |
+| 5 | ལས | 14,391 |
+| 6 | མི | 11,348 |
+| 7 | དེ | 11,091 |
+| 8 | མ | 10,372 |
+| 9 | གི | 10,307 |
+| 10 | འདི | 9,382 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | yongla | 2 |
-| 2 | pelbar | 2 |
-| 3 | dargeychhoeling | 2 |
-| 4 | fortress | 2 |
-| 5 | gods | 2 |
-| 6 | shaba | 2 |
+| 1 | printer | 2 |
+| 2 | fortress | 2 |
+| 3 | gods | 2 |
+| 4 | wordpress | 2 |
+| 5 | phurdo | 2 |
+| 6 | gonpa | 2 |
 | 7 | assam | 2 |
 | 8 | pelgen | 2 |
-| 9 | bjoka | 2 |
-| 10 | ༡༨༨༩ | 2 |
+| 9 | anecdotes | 2 |
+| 10 | kheng | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.7500 |
-| R² (Goodness of Fit) | 0.979006 |
+| Zipf Coefficient | 1.8277 |
+| R² (Goodness of Fit) | 0.959592 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 91.1% |
-| Top 1,000 | 99.3% |
-| Top 5,000 | 0.0% |
+| Top 100 | 49.0% |
+| Top 1,000 | 92.3% |
+| Top 5,000 | 99.6% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9790 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 91.1% of corpus
-- **Long Tail:** -5,959 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9596 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 49.0% of corpus
+- **Long Tail:** -3,239 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -320,24 +416,79 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 1,914 | 32 | 4.382 | 1.097 | 0.7372 🏆 |
-| **mono_64d** | 1,914 | 64 | 4.602 | 1.013 | 0.4928 |
-| **mono_128d** | 1,914 | 128 | 4.665 | 1.008 | 0.1299 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.6999 🏆 | 0.3567 | N/A | N/A |
+| **mono_64d** | 64 | 0.4345 | 0.3403 | N/A | N/A |
+| **mono_128d** | 128 | 0.1109 | 0.3305 | N/A | N/A |
+| **aligned_32d** | 32 | 0.6999 | 0.3594 | 0.0547 | 0.2644 |
+| **aligned_64d** | 64 | 0.4345 | 0.3388 | 0.1307 | 0.4103 |
+| **aligned_128d** | 128 | 0.1109 | 0.3270 | 0.2340 | 0.4742 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.7372 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 1,914 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.6999 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3421. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 23.4% R@1 in cross-lingual retrieval.
+- **Recommendation:** 128d aligned for best cross-lingual performance
+
+---
+## 6.  Morphological Analysis (Experimental)
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **-0.621** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+*No productive affixes detected.*
+
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+*No significant bound stems detected.*
+
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+*No significant affix co-occurrences detected.*
+
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+*Insufficient data for recursive segmentation.*
+
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language Dzongkha shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
 
 ---
-## 6. Summary & Recommendations
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -345,11 +496,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (7.10x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (305) |
-| Markov | **Context-4** | Highest predictability (70.6%) |
+| Tokenizer | **64k BPE** | Best compression (5.51x) |
+| N-gram | **2-gram** | Lowest perplexity (488) |
+| Markov | **Context-4** | Highest predictability (90.1%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -539,7 +691,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -555,7 +708,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-30 08:46:14*
+*Report Date: 2026-01-04 03:00:40*

models/embeddings/aligned/dz_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1dcfe07db51e338f29b11cc3948400bc263bdaa9ee20bff094b937df3a88eafe
+size 1025712069

models/embeddings/aligned/dz_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "dz", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/dz_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae9d51cd6d1f20f33ad43e0ebc040e9c60e5efef1660c66c48e1753e7f29e7d1
+size 65664

models/embeddings/aligned/dz_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "dz",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 329,
+  "vocab_size": 1602
+}

models/embeddings/aligned/dz_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3933e9d9dffccc98f99b10e7f95f6656ef45b6fa711dc8726dc64e374a14aa6
+size 256481733

models/embeddings/aligned/dz_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "dz", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/dz_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17308e828737ebd9ac6a3c706dae164e4bcc335a5173fecdc73408ccedf7da6c
+size 4224

models/embeddings/aligned/dz_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "dz",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 329,
+  "vocab_size": 1602
+}

models/embeddings/aligned/dz_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34e82a789619b535b455c2b7f657a7f8bda7d16f597d1daedd1096e57ba64065
+size 512891845

models/embeddings/aligned/dz_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "dz", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/dz_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55de110e901818649692cfc6a7e97343d7cd34e63a6fa97e21046355a71de3e0
+size 16512

models/embeddings/aligned/dz_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "dz",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 329,
+  "vocab_size": 1602
+}

models/embeddings/monolingual/dz_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d722d0680f671d8ae327caa3ee8603c53cb43740cfd380d68f04ef1fdc0564e9
-size 1026040985
+oid sha256:1dcfe07db51e338f29b11cc3948400bc263bdaa9ee20bff094b937df3a88eafe
+size 1025712069

models/embeddings/monolingual/dz_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 1914
+  "vocab_size": 1602
 }

models/embeddings/monolingual/dz_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:de6fde9cee2ca33805b66192484f75a384fc767a4e41c9e3e1349e5df4efbbe0
-size 256571033
+oid sha256:e3933e9d9dffccc98f99b10e7f95f6656ef45b6fa711dc8726dc64e374a14aa6
+size 256481733

models/embeddings/monolingual/dz_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 1914
+  "vocab_size": 1602
 }

models/embeddings/monolingual/dz_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3145c5cc9d22a1770f9210ca05d9737c7c503b50ffe23ea80a6eb9ceb9258651
-size 513061017
+oid sha256:34e82a789619b535b455c2b7f657a7f8bda7d16f597d1daedd1096e57ba64065
+size 512891845

models/embeddings/monolingual/dz_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 1914
+  "vocab_size": 1602
 }

models/subword_markov/dz_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b4bb0e27f1c210e8aab111267e19d1d5e1d18721ec5e52e6b277feae0adde466
-size 60805
+oid sha256:3087bea20e8eb6b2ec3a5f0aa9584c7311e41fe5a4c01f823fa4834212620662
+size 111860

models/subword_markov/dz_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "dz",
-  "unique_contexts": 727,
-  "total_transitions": 4381675
+  "unique_contexts": 1607,
+  "total_transitions": 2899198
 }

models/subword_markov/dz_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3747acef076c0fe34becbcf80d7e30260918fc21406b3e37de57d81f61ce0562
-size 310779
+oid sha256:6292dd07a274165c86703180f2425201d1d9674e4ba2c33f61d9d5938c3467e1
+size 489356

models/subword_markov/dz_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "dz",
-  "unique_contexts": 7001,
-  "total_transitions": 4380562
+  "unique_contexts": 12152,
+  "total_transitions": 2898160
 }

models/subword_markov/dz_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95233f33b057a7c0b853eb01e55d5fea5a9a3405b48f24242e21c2ba56259a16
-size 1040905
+oid sha256:388847f6c74b750ae03e7f5efd4aaeacf73b7bd43240dd42b5f2aa721dbdc066
+size 1647571

models/subword_markov/dz_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "dz",
-  "unique_contexts": 38506,
-  "total_transitions": 4379449
+  "unique_contexts": 61009,
+  "total_transitions": 2897122
 }

models/subword_markov/dz_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2dca69cb48bb57a909fbca7cb5406a84c25f6c5f445ab67e8d62f592fc6ac5a2
-size 2665013
+oid sha256:fed702bf6148ce9dc1a7ea84a35755fc292d0a1697d7d3752899d2b7ed2b2c5b
+size 4311752

models/subword_markov/dz_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "dz",
-  "unique_contexts": 121621,
-  "total_transitions": 4378336
+  "unique_contexts": 199035,
+  "total_transitions": 2896084
 }

models/subword_ngram/dz_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:09d5275b8f276ba3da20aca7a059f240b755a5848b95578cd0e57576e8d2b173
-size 43022
+oid sha256:98106b25d70e32c0869643449157893becaaa5cd2d9c93a574d49972755b142f
+size 77884

models/subword_ngram/dz_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "dz",
-  "unique_ngrams": 3288,
-  "total_ngrams": 4381675
+  "unique_ngrams": 5527,
+  "total_ngrams": 2899198
 }

models/subword_ngram/dz_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:67ecae4be4d51a96f235bd1b6cdecd6fa9034a1a947167dcebb6838e6424000f
-size 244220
+oid sha256:19e9ca58532c279df9875e0fc70fc0ca01a0dc4a6eeebb0d90cc9611de440cca
+size 417712

models/subword_ngram/dz_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "dz",
-  "unique_ngrams": 19053,
-  "total_ngrams": 4380562
+  "unique_ngrams": 28498,
+  "total_ngrams": 2898160
 }

models/subword_ngram/dz_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:72af1322dc5108a5a783c65271a08d008eb8788ea196291554e7bb961046ce3d
-size 858312
+oid sha256:dc939eb8f6b49a4d85499e546bee0c3aa02484d5639731734c57e7e77ccc3406
+size 1535380

models/subword_ngram/dz_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "dz",
-  "unique_ngrams": 68338,
-  "total_ngrams": 4379449
+  "unique_ngrams": 106273,
+  "total_ngrams": 2897122
 }

models/subword_ngram/dz_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89c0a9d344f772c56895fcfa4f96ca8389e2d1d65d242db9969fdf57da31004e
+size 2925710

models/subword_ngram/dz_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "dz",
+  "unique_ngrams": 194726,
+  "total_ngrams": 2896084
+}

models/tokenizer/dz_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6e4b153b5c498e9676f1d0be1af974800ada3ef90b84b25fb029c769bb4f2c39
-size 723437
+oid sha256:2f90c7e3cf5a68d178f10a0beb7ddb97feaa78af6989d232df8e51de111001a4
+size 731782

models/tokenizer/dz_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3535359e078e90bde0054e2b711f0dd81cef07816c72d21f67b1611058d511e4
-size 1262637
+oid sha256:ea0364e6b2d160e778e60d5e1c73e3e9b08f8793e92690aafe427a87f8eb980a
+size 1365323

models/tokenizer/dz_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b0639d3f1b896a46c0dddd000ad31618a562b86efd1e742d9ffaaee9eb96d8ca
-size 2493435
+oid sha256:402c9983197ff52d444fc136259bb6b1f0cfb5b38449cd46aef1f96b7701f99e
+size 2390465

models/tokenizer/dz_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a1b86d110a86dab6e1e1c6f0a86b9076034e30c13e2bccd44bf4df01c9c6ed44
-size 466064
+oid sha256:c1416416e068b7dd36e23224aed70c1131a475dc4b0464d18851552856625ace
+size 450802

models/vocabulary/dz_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:62eb74b27face901c271eb032243b46b21201b90fd59225353cbe861179c11f6
-size 68375
+oid sha256:14e984f867b546d329e8a05a2bde8d13e79ac78e9afc51f2e97dc5f7abdcc723
+size 113898

models/vocabulary/dz_vocabulary_metadata.json

@@ -1,15 +1,17 @@
 {
   "language": "dz",
-  "vocabulary_size": 4041,
+  "vocabulary_size": 6761,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.0058438083699328585,
+    "type_token_ratio": 0.013374135154743156,
     "coverage": {
-      "top_100": 0.907915257374594,
-      "top_1000": 0.9895912632305182,
-      "top_5000": 0.9975216652363403
+      "top_100": 0.4871301735883779,
+      "top_1000": 0.9174028542105356,
+      "top_5000": 0.9902080052377329,
+      "top_10000": 0.9976852671066834
     },
-    "hapax_count": 4641,
-    "hapax_ratio": 0.5345542501727713,
-    "total_documents": 1113
+    "hapax_count": 5332,
+    "hapax_ratio": 0.4409162325312164,
+    "total_documents": 1038
   }
 }

models/word_markov/dz_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:61a9184a1b8ab3551fe0bc2b20b0017902678f748e4675c3e748aea5859a0d5c
-size 312387
+oid sha256:76e337706f6a7fb91b004453fee9ac14da9e139b1614cc7f2e3089bc6b4ea3fc
+size 876930

models/word_markov/dz_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "dz",
-  "unique_contexts": 8836,
-  "total_transitions": 3510455
+  "unique_contexts": 12061,
+  "total_transitions": 903170
 }

models/word_markov/dz_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:71526c2583406425808542d0de6b23cbb666ed4109f31c4941ffa339ebc9ddcd
-size 865992
+oid sha256:62fa7abf255f3a8a538ee56225f2684bcea4c021a186d768c55d5ca85930ca27
+size 4355103

models/word_markov/dz_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "dz",
-  "unique_contexts": 32129,
-  "total_transitions": 3509342
+  "unique_contexts": 170162,
+  "total_transitions": 902132
 }