Library / System / Codecs / UTFCodec
UTFCodec
Unicode Transformation Format codec + scalar validity.
Methods 12
METHOD encodeToUTF32(REFERENCE char8[] source) RETURNS | char32[]reads
DFA-BASED BULK DECODE — UTF-8 code units -> UTF-32 code points Decode a full UTF-8 buffer (char8[] code units) into a char32[] of code points using the four LOOKUP TABLES above (utf_utils- style single-DFA decoder). Pipe-XOR: SUCCESS yields the codepoints buffer; FAILURE on the first malformed lead unit, truncated sequence, bad continuation unit, or DFA-detected illegal form (overlong, surrogate, beyond U+10FFFF — all baked into the transitions[] mapping).
Input / output are both bare value-array storage shorthand — no String / ByteBuffer / DynamicString / Array[T] dependency. UTFCodec sits below the String family in the kernel dependency order.
Algorithm:
1. firstUnitTable[u] -> (firstOctet bits, nextState).
ASCII units (0x00..0x7F) return nextState=BEGIN with
cp == u, so the ASCII path is one table read + one write.
2. For multi-unit leads, nextState moves into a CONTINUEn /
PARTIAL_SEQUENCE state. The continuation loop reads
successive code units, accumulates 6 bits per unit into
cp, and advances state via
transitions[currentState + octetCategory[contUnit]].
Loop exits when state returns to BEGIN/END (= 0; code
point complete) or ERROR (malformed).
3. Result built via the value-array HWM-write proxy —
result[outCount] = cp grows the buffer when outCount
equals result.length.
METHOD appendUTF8(MUTABLE REFERENCE binary[] dst, REFERENCE char32[] src, int64 off, int64 n) RETURNS voidreads
UTF-8 ENCODE — the one encoder. Every UTF-8 encoder in the kernel and its
modules ends here: appendUTF8 for a window of code points, appendScalarUTF8
for one. String windows arrive through TextEncoding (this namespace orders
before String and cannot name it). Both append to a binary[], the buffer every
caller writes into. PRECONDITION: every code point is a Unicode scalar, as a
String's are by construction; encodeToUTF8 checks a raw array before calling.
The widths follow RFC 3629:
< 0x80 → 1 byte (0xxxxxxx)
< 0x800 → 2 bytes (110xxxxx 10xxxxxx)
< 0x10000 → 3 bytes (1110xxxx 10xxxxxx 10xxxxxx)
else → 4 bytes (11110xxx 10xxxxxx 10xxxxxx 10xxxxxx)
METHOD fillUTF8(MUTABLE REFERENCE binary[] dst, REFERENCE char32[] src, int64 off, int64 n) RETURNS STATUSreads
The same UTF-8 as appendUTF8, through simdutf, for a caller that starts from
an EMPTY buffer — dst is filled from index 0, never appended to. The length is
measured first so dst is sized exactly. FAILURE on a non-empty dst, a window
out of range, or a code point that is not a Unicode scalar.
METHOD appendScalarUTF8(MUTABLE REFERENCE binary[] dst, char32 c) RETURNS voidreads
One code point's UTF-8 appended to dst (see appendUTF8).
METHOD encodeToUTF8(REFERENCE char32[] source) RETURNS | char8[]reads
BULK TRANSCODE — char32[] → char8[] (codepoints → UTF-8 bytes)
Encode an array of UTF-32 codepoints into a UTF-8 byte stream.
Pipe-XOR: SUCCESS yields the byte buffer; FAILURE on the first
codepoint that's beyond U+10FFFF or in the UTF-16 surrogate
block. The array is checked first, then encoded by appendUTF8.
METHOD decodeUTF8(REFERENCE binary[] src, int64 off, int64 n) RETURNS | char32[]reads
UTF-8 DECODE — a window of bytes as code points, through simdutf. One call decodes and validates (overlong, surrogate, out-of-range and truncated sequences all fail); every code point is then held to the Tier-1 text rule, so the result can be adopted as a String's storage. FAILURE on either.
METHOD encodeToUTF16(REFERENCE char32[] source) RETURNS | char16[]reads
BULK TRANSCODE — char32[] → char16[] (codepoints → UTF-16 units) Encode an array of UTF-32 codepoints into a UTF-16 code unit stream. BMP codepoints (< U+10000) emit one char16. Supplementary plane codepoints (U+10000..U+10FFFF) emit a surrogate pair — high in [0xD800, 0xDBFF], low in [0xDC00, 0xDFFF]. Pipe-XOR: FAILURE on a codepoint that's beyond U+10FFFF or in the surrogate block (raw surrogate codepoints are not legal scalar values).
METHOD encodeToUTF32(REFERENCE char16[] source) RETURNS | char32[]reads
BULK TRANSCODE — char16[] → char32[] (UTF-16 units → codepoints) Decode a UTF-16 code unit stream into UTF-32 codepoints, pairing surrogate halves into supplementary-plane codepoints. Pipe-XOR: FAILURE on an unpaired surrogate (low surrogate without a preceding high, or high surrogate without a following low).
METHOD encodeToUTF8(REFERENCE char16[] source) RETURNS | char8[]reads
BULK TRANSCODE — char16[] → char8[] (UTF-16 → UTF-8)
Composition: decode UTF-16 → codepoints → encode codepoints as UTF-8. The chain propagates STATUS from either step.
METHOD encodeToUTF16(REFERENCE char8[] source) RETURNS | char16[]reads
BULK TRANSCODE — char8[] → char16[] (UTF-8 → UTF-16)
Composition: decode UTF-8 → codepoints → encode codepoints as UTF-16. The chain propagates STATUS from either step.
METHOD isValid(char32 c) RETURNS booleanreads
VALIDITY PREDICATES
TRUE when c is a legal Unicode scalar value: in range
U+0000..U+10FFFF and not a UTF-16 surrogate half.
METHOD isStringSafeCodePoint(char32 c) RETURNS booleanreads
TRUE when c is admissible as text — a legal scalar value that
is also not a noncharacter and not a junk control byte. This is
the Tier-1 text rule: DynamicString.append(char) rejects
every code point that fails it, so no text value can carry
non-text content.
Allowed control bytes are the text whitespace set only — tab (U+0009), LF (U+000A), VT (U+000B), FF (U+000C), CR (U+000D), and NEL (U+0085); every other C0/C1 control and U+007F (DEL) is rejected. VT/FF/NEL are kept because they are line separators (see splitLines).