UEA-Prodem

This commit is contained in:
2026-06-10 12:38:42 -03:00
parent 3f33154e16
commit c41625e542
9352 changed files with 1128292 additions and 14752 deletions
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Natural Intelligence
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+142
View File
@@ -0,0 +1,142 @@
# anynum
Normalize Unicode decimal digits and minus signs to ASCII.
Converts digits from any script — Devanagari, Arabic-Indic, Thai, Bengali, Fullwidth, and [50+ others](#supported-scripts) — to their ASCII equivalents (`0``9`). Also normalizes Unicode minus variants (``, ``, `﹣`) to ASCII `-`.
Pairs naturally with [`strnum`](https://github.com/nicolo-ribaudo/strnum) — use `anynum` to normalize first, then `strnum` to detect the numeric type.
```js
import anynum from 'anynum';
anynum('१२.३४') // → '12.34' (Devanagari)
anynum('٣٫١٤') // → '3.14' (Arabic-Indic)
anynum('−४२') // → '-42' (Unicode minus + Devanagari)
anynum('-99.') // → '-99.5' (Fullwidth minus + Fullwidth digits)
anynum('hello') // → 'hello' (no digits — zero allocation)
anynum('100') // → '100' (already ASCII — zero allocation)
```
---
## Install
```bash
npm install anynum
```
---
## Usage
```js
// ESM
import anynum from 'anynum';
import { anynum } from 'anynum';
```
### API
```ts
anynum(str: string): string
```
- Accepts a `string`, returns a `string`.
- Non-string values are returned as-is (no throw).
- Non-digit characters pass through unchanged.
- If no conversion is needed, the **original string is returned** (zero allocation).
---
## What gets converted
### Decimal digits
Any Unicode character in category `Nd` (decimal digit) is mapped to its ASCII equivalent. This covers all positional decimal digit scripts — every script whose digits represent `0``9` by position.
```js
anynum('๑๒๓') // Thai → '123'
anynum('੧੨੩') // Gurmukhi → '123'
anynum('᠑᠒᠓') // Mongolian → '123'
anynum('𝟏𝟐𝟑') // Math Bold → '123'
```
### Unicode minus variants
Three Unicode characters are normalized to ASCII `-` (`U+002D`):
| Code point | Character | Name |
|---|---|---|
| `U+2212` | `` | MINUS SIGN (mathematical) |
| `U+FF0D` | `` | FULLWIDTH HYPHEN-MINUS |
| `U+FE63` | `﹣` | SMALL HYPHEN-MINUS |
Dashes used for punctuation — EN DASH (``), EM DASH (`—`), HYPHEN (``) — are intentionally **not** converted.
```js
anynum('42') // U+2212 MINUS SIGN → '-42'
anynum('42') // U+FF0D FULLWIDTH → '-42'
anynum('42') // U+2013 EN DASH → '42' (unchanged)
```
---
## Use with strnum
`anynum` and `strnum` compose cleanly:
```js
import anynum from 'anynum';
import strnum from 'strnum';
strnum(anynum('१२.३४')) // → 12.34 (number, float)
strnum(anynum('−४२')) // → '-42' (string; strnum handles sign detection)
strnum(anynum('hello')) // → 'hello'
```
---
## Supported scripts
50+ decimal digit scripts from Unicode `Nd` category, including:
| Script | Zero | Sample |
|---|---|---|
| Devanagari (Hindi/Marathi/Nepali) | `U+0966` | `०१२३४५६७८९` |
| Arabic-Indic | `U+0660` | `٠١٢٣٤٥٦٧٨٩` |
| Extended Arabic-Indic (Urdu/Persian) | `U+06F0` | `۰۱۲۳۴۵۶۷۸۹` |
| Bengali | `U+09E6` | `০১২৩৪৫৬৭৮৯` |
| Gurmukhi | `U+0A66` | `੦੧੨੩੪੫੬੭੮੯` |
| Gujarati | `U+0AE6` | `૦૧૨૩૪૫૬૭૮૯` |
| Odia | `U+0B66` | `୦୧୨୩୪୫୬୭୮୯` |
| Tamil | `U+0BE6` | `௦௧௨௩௪௫௬௭௮௯` |
| Telugu | `U+0C66` | `౦౧౨౩౪౫౬౭౮౯` |
| Kannada | `U+0CE6` | `೦೧೨೩೪೫೬೭೮೯` |
| Malayalam | `U+0D66` | `൦൧൨൩൪൫൬൭൮൯` |
| Thai | `U+0E50` | `๐๑๒๓๔๕๖๗๘๙` |
| Lao | `U+0ED0` | `໐໑໒໓໔໕໖໗໘໙` |
| Tibetan | `U+0F20` | `༠༡༢༣༤༥༦༧༨༩` |
| Myanmar | `U+1040` | `၀၁၂၃၄၅၆၇၈၉` |
| Khmer | `U+17E0` | `០១២៣៤៥៦៧៨៩` |
| Mongolian | `U+1810` | `᠐᠑᠒᠓᠔᠕᠖᠗᠘᠙` |
| Fullwidth (CJK context) | `U+FF10` | `0123456789` |
| Mathematical Bold | `U+1D7CE` | `𝟎𝟏𝟐𝟑𝟒𝟓𝟔𝟕𝟖𝟗` |
| Adlam | `U+1E950` | `𞥐𞥑𞥒𞥓𞥔𞥕𞥖𞥗𞥘𞥙` |
| … and 30+ more | | |
---
## What it does NOT convert
- **Kanji/Chinese numeral words** (`三`, `百`, `万`) — these are ideographic numerals, not decimal digits. Each language has its own positional system requiring separate parsing logic.
- **Roman numerals** (`Ⅳ`, `Ⅻ`) — not decimal digits.
- **Punctuation dashes** (`` EN, `—` EM, `` HYPHEN) — not numeric signs.
- **Decimal separators** — commas, periods, Arabic decimal comma (`٫`) are passed through as-is. Separator normalization is the caller's responsibility.
---
## License
MIT
+135
View File
@@ -0,0 +1,135 @@
'use strict';
import { TABLE, TABLE_OFFSET, HIGH_MAP, NOT_DIGIT } from './digitTable.js';
const CHAR_0 = 48; // '0'.charCodeAt(0)
const CHAR_9 = 57; // '9'.charCodeAt(0)
const CHAR_MINUS = 45; // '-'.charCodeAt(0)
// Unicode minus/hyphen variants worth normalizing to ASCII '-' in numeric context:
// U+2212 MINUS SIGN (mathematically correct minus)
// U+FF0D FULLWIDTH HYPHEN-MINUS (Japanese fullwidth context)
// U+FE63 SMALL HYPHEN-MINUS ﹣ (small form variant)
//
// NOT normalized (deliberate):
// U+2013 EN DASH (punctuation, not a numeric sign)
// U+2014 EM DASH — (punctuation)
// U+2010 HYPHEN (typographic hyphen)
//
// Rationale: only characters a human or locale formatter would plausibly use
// as a numeric minus sign are normalized. Dashes used for punctuation are left
// alone to avoid mangling non-numeric strings.
const MINUS_SET = new Set([0x2212, 0xFF0D, 0xFE63]);
/**
* Normalize all Unicode decimal digit characters in a string to ASCII (0-9),
* and normalize Unicode minus variants to ASCII '-' (U+002D).
*
* Non-digit, non-minus characters are passed through unchanged.
*
* Performance design:
* - Fast path: if the string has no convertible characters, return it unchanged
* (zero allocation).
* - BMP digits (0x0660..0xFFFF excl. surrogates): flat Uint8Array lookup (O(1)).
* - Supplementary plane digits (> 0xFFFF, encoded as surrogate pairs): Map lookup.
* - Minus variants: checked inline with a small fixed Set.
*
* @param {string} str
* @returns {string}
*/
function anynum(str) {
if (typeof str !== 'string') return str;
const len = str.length;
if (len === 0) return str;
// Scan for first character needing conversion.
// If none found, return original string (zero allocation).
let firstHit = -1;
for (let i = 0; i < len; i++) {
const cc = str.charCodeAt(i);
// ASCII digit or ASCII minus — already normalized, skip fast
if ((cc >= CHAR_0 && cc <= CHAR_9) || cc === CHAR_MINUS) continue;
// Below first unicode digit script — check minus variants only
if (cc < TABLE_OFFSET) {
if (MINUS_SET.has(cc)) { firstHit = i; break; }
continue;
}
// Surrogate pairs live in BMP range 0xD800-0xDFFF — check before TABLE
if (cc >= 0xD800 && cc <= 0xDBFF) {
if (i + 1 < len) {
const low = str.charCodeAt(i + 1);
if (low >= 0xDC00 && low <= 0xDFFF) {
const cp = 0x10000 + ((cc - 0xD800) << 10) + (low - 0xDC00);
if (HIGH_MAP.has(cp)) { firstHit = i; break; }
}
}
continue;
}
// BMP non-surrogate: flat table lookup; also check minus variants in this range
if (TABLE[cc - TABLE_OFFSET] !== NOT_DIGIT || MINUS_SET.has(cc)) {
firstHit = i;
break;
}
}
// Nothing to replace — return original, zero allocation
if (firstHit === -1) return str;
// Build result: copy unchanged prefix, then convert from firstHit onward
const chars = [];
if (firstHit > 0) chars.push(str.slice(0, firstHit));
for (let i = firstHit; i < len; i++) {
const cc = str.charCodeAt(i);
// ASCII digit or ASCII minus — pass through
if ((cc >= CHAR_0 && cc <= CHAR_9) || cc === CHAR_MINUS) {
chars.push(str[i]);
continue;
}
// Below TABLE_OFFSET — check minus variants, else pass through
if (cc < TABLE_OFFSET) {
chars.push(MINUS_SET.has(cc) ? '-' : str[i]);
continue;
}
// Surrogate pairs
if (cc >= 0xD800 && cc <= 0xDBFF) {
if (i + 1 < len) {
const low = str.charCodeAt(i + 1);
if (low >= 0xDC00 && low <= 0xDFFF) {
const cp = 0x10000 + ((cc - 0xD800) << 10) + (low - 0xDC00);
const d = HIGH_MAP.get(cp);
if (d !== undefined) {
chars.push(String.fromCharCode(d + 48));
i++; // consume low surrogate
continue;
}
}
}
chars.push(str[i]);
continue;
}
// BMP non-surrogate: flat table lookup + minus variants
if (MINUS_SET.has(cc)) {
chars.push('-');
continue;
}
const d = TABLE[cc - TABLE_OFFSET];
chars.push(d !== NOT_DIGIT ? String.fromCharCode(d + 48) : str[i]);
}
return chars.join('');
}
export { anynum };
export default anynum;
+116
View File
@@ -0,0 +1,116 @@
/**
* Flat lookup table: maps Unicode code point → ASCII digit (0-9).
* Only decimal digit characters (Unicode category Nd) are included.
*
* Strategy: Int32Array of size (maxCodePoint - minCodePoint + 1).
* Value 0xFF means "not a digit". Value 0-9 is the ASCII digit value.
* This gives O(1) lookup with no branching, no bisect, no loop.
*
* Memory: range is 0x0660 to 0x1FBF0 → ~129,936 entries × 1 byte = ~127 KB.
* Acceptable for a one-time init; lookup is a single array index.
*/
// All known Unicode Nd (decimal digit) script zero code points.
// Each script has exactly 10 consecutive digits: zero+0 .. zero+9.
const SCRIPT_ZEROS = [
// Basic Latin (ASCII) — included for completeness / pass-through
0x0030, // 0-9
// Arabic scripts
0x0660, // Arabic-Indic ٠١٢٣٤٥٦٧٨٩
0x06F0, // Extended Arabic-Indic (Urdu/Persian/Sindhi) ۰۱۲۳
// Indic scripts
0x0966, // Devanagari ०१२३४५६७८९
0x09E6, // Bengali ০১২৩৪৫৬৭৮৯
0x0A66, // Gurmukhi ੦੧੨੩੪੫੬੭੮੯
0x0AE6, // Gujarati ૦૧૨૩૪૫૬૭૮૯
0x0B66, // Odia ୦୧୨୩୪୫୬୭୮୯
0x0BE6, // Tamil ௦௧௨௩௪௫௬௭௮௯
0x0C66, // Telugu ౦౧౨౩౪౫౬౭౮౯
0x0CE6, // Kannada ೦೧೨೩೪೫೬೭೮೯
0x0D66, // Malayalam ൦൧൨൩൪൫൬൭൮൯
0x0DE6, // Sinhala Archaic ෦෧෨෩෪෫෬෭෮෯
// Southeast Asian scripts
0x0E50, // Thai ๐๑๒๓๔๕๖๗๘๙
0x0ED0, // Lao ໐໑໒໓໔໕໖໗໘໙
0x0F20, // Tibetan ༠༡༢༣༤༥༦༧༨༩
0x1040, // Myanmar ၀၁၂၃၄၅၆၇၈၉
0x1090, // Myanmar Shan ႐႑႒႓႔႕႖႗႘႙
0x17E0, // Khmer ០១២៣៤៥៦៧៨៩
0x1810, // Mongolian ᠐᠑᠒᠓᠔᠕᠖᠗᠘᠙
0x1946, // Limbu ᥆᥇᥈᥉᥊᥋᥌᥍᥎᥏
0x19D0, // New Tai Lue ᧐᧑᧒᧓᧔᧕᧖᧗᧘᧙
0x1A80, // Tai Tham Hora ᪀᪁᪂᪃᪄᪅᪆᪇᪈᪉
0x1A90, // Tai Tham Tham ᪐᪑᪒᪓᪔᪕᪖᪗᪘᪙
0x1B50, // Balinese ᭐᭑᭒᭓᭔᭕᭖᭗᭘᭙
0x1BB0, // Sundanese ᮰᮱᮲᮳᮴᮵᮶᮷᮸᮹
0x1C40, // Lepcha ᱀᱁᱂᱃᱄᱅᱆᱇᱈᱉
0x1C50, // Ol Chiki ᱐᱑᱒᱓᱔᱕᱖᱗᱘᱙
// Fullwidth (CJK context)
0xFF10, // Fullwidth 0123456789
// Mathematical digit variants (Unicode math block)
0x1D7CE, // Mathematical Bold
0x1D7D8, // Mathematical Double-Struck
0x1D7E2, // Mathematical Sans-Serif
0x1D7EC, // Mathematical Sans-Serif Bold
0x1D7F6, // Mathematical Monospace
// Other scripts
0x104A0, // Osmanya 𐒠𐒡𐒢𐒣𐒤𐒥𐒦𐒧𐒨𐒩
0x10D30, // Hanifi Rohingya 𐴰𐴱𐴲𐴳𐴴𐴵𐴶𐴷𐴸𐴹
0x11066, // Brahmi 𑁦𑁧𑁨𑁩𑁪𑁫𑁬𑁭𑁮𑁯
0x110F0, // Sora Sompeng 𑃰𑃱𑃲𑃳𑃴𑃵𑃶𑃷𑃸𑃹
0x11136, // Chakma 𑄶𑄷𑄸𑄹𑄺𑄻𑄼𑄽𑄾𑄿
0x111D0, // Sharada 𑇐𑇑𑇒𑇓𑇔𑇕𑇖𑇗𑇘𑇙
0x112F0, // Khudawadi 𑋰𑋱𑋲𑋳𑋴𑋵𑋶𑋷𑋸𑋹
0x11450, // Newa 𑑐𑑑𑑒𑑓𑑔𑑕𑑖𑑗𑑘𑑙
0x114D0, // Tirhuta 𑓐𑓑𑓒𑓓𑓔𑓕𑓖𑓗𑓘𑓙
0x11650, // Modi 𑙐𑙑𑙒𑙓𑙔𑙕𑙖𑙗𑙘𑙙
0x116C0, // Takri 𑛀𑛁𑛂𑛃𑛄𑛅𑛆𑛇𑛈𑛉
0x11730, // Ahom 𑜰𑜱𑜲𑜳𑜴𑜵𑜶𑜷𑜸𑜹
0x118E0, // Warang Citi 𑣠𑣡𑣢𑣣𑣤𑣥𑣦𑣧𑣨𑣩
0x11950, // Dives Akuru 𑥐𑥑𑥒𑥓𑥔𑥕𑥖𑥗𑥘𑥙
0x11BF0, // Khitan Small Script 𑯰𑯱𑯲𑯳𑯴𑯵𑯶𑯷𑯸𑯹
0x11C50, // Bhaiksuki 𑱐𑱑𑱒𑱓𑱔𑱕𑱖𑱗𑱘𑱙
0x11D50, // Masaram Gondi 𑵐𑵑𑵒𑵓𑵔𑵕𑵖𑵗𑵘𑵙
0x11DA0, // Gunjala Gondi 𑶠𑶡𑶢𑶣𑶤𑶥𑶦𑶧𑶨𑶩
0x11F50, // Kawi 𑽐𑽑𑽒𑽓𑽔𑽕𑽖𑽗𑽘𑽙
0x16A60, // Mro 𖩠𖩡𖩢𖩣𖩤𖩥𖩦𖩧𖩨𖩩
0x16AC0, // Tangsa 𖫀𖫁𖫂𖫃𖫄𖫅𖫆𖫇𖫈𖫉
0x16B50, // Pahawh Hmong 𖭐𖭑𖭒𖭓𖭔𖭕𖭖𖭗𖭘𖭙
0x1E140, // Nyiakeng Puachue Hmong 𞅀𞅁𞅂𞅃𞅄𞅅𞅆𞅇𞅈𞅉
0x1E2F0, // Wancho 𞋰𞋱𞋲𞋳𞋴𞋵𞋶𞋷𞋸𞋹
0x1E4F0, // Nag Mundari 𞓰𞓱𞓲𞓳𞓴𞓵𞓶𞓷𞓸𞓹
0x1E950, // Adlam 𞥐𞥑𞥒𞥓𞥔𞥕𞥖𞥗𞥘𞥙
0x1FBF0, // Segmented digit symbols 🯰🯱🯲🯳🯴🯵🯶🯷🯸🯹
];
// Build a sparse Map for scripts above 0xFFFF (surrogate-pair range).
// These can't go into a flat Uint8Array indexed by code point efficiently.
const NOT_DIGIT = 0xFF;
const HIGH_MAP = new Map(); // codePoint → digit value (0-9)
const LOW_MAX = 0xFFFF;
const LOW_MIN = 0x0660; // first non-ASCII digit script
// Flat Uint8Array covering 0x0660 .. 0xFFFF
const TABLE_OFFSET = LOW_MIN;
const TABLE_SIZE = LOW_MAX - LOW_MIN + 1;
const TABLE = new Uint8Array(TABLE_SIZE).fill(NOT_DIGIT);
for (const zero of SCRIPT_ZEROS) {
for (let d = 0; d < 10; d++) {
const cp = zero + d;
if (cp <= LOW_MAX) {
TABLE[cp - TABLE_OFFSET] = d;
} else {
HIGH_MAP.set(cp, d);
}
}
}
export { TABLE, TABLE_OFFSET, HIGH_MAP, NOT_DIGIT };
+42
View File
@@ -0,0 +1,42 @@
{
"name": "anynum",
"version": "1.0.0",
"description": "Normalize all Unicode decimal digits (Devanagari, Arabic, Thai, etc.) to ASCII numerals. Zero dependencies, performance-first.",
"type": "module",
"main": "./anynum.js",
"files": [
"anynum.js",
"digitTable.js",
"README.md"
],
"scripts": {
"test": "jasmine tests/anynum.test.js",
"bench": "node test/bench.js"
},
"keywords": [
"unicode",
"digits",
"normalize",
"devanagari",
"arabic",
"hindi",
"japanese",
"indic",
"asian",
"math",
"numerals",
"strnum"
],
"repository": {
"type": "git",
"url": "https://github.com/NaturalIntelligence/anynum"
},
"author": "Amit Gupta (https://solothought.work/)",
"license": "MIT",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/NaturalIntelligence"
}
]
}