Normalises free-text a user typed before it is handed to a model provider and
persisted. Used by the chatbot turn.
Deliberately not an HTML sanitiser, and it must not become one again. This
text has no HTML sink: user turns render as plain text, assistant turns go
through react-markdown without rehype-raw, and React escapes both on the
way out. Escaping here bought nothing and corrupted the payload instead —
I'm building a client's dashboard reached OpenAI, and was stored in
ChatMessage.content, as I'm building a client's dashboard, and
stripping <…> swallowed ordinary prose like if x < 5 and y > 3. Escaping
belongs at the sink; if a new surface ever renders this content as HTML, that
surface escapes it.
What survives is the work that is actually this layer's job: a length ceiling,
and removal of characters that are never legitimate in typed text — C0/C1
controls (NUL in particular, which Postgres rejects outright in text) plus
the bidi-override and zero-width codepoints used to hide instructions from a
human reader while the model still sees them. ZWNJ/ZWJ (U+200C/U+200D) are
kept: Persian and Arabic text and emoji sequences need them.
Normalises free-text a user typed before it is handed to a model provider and persisted. Used by the chatbot turn.
Deliberately not an HTML sanitiser, and it must not become one again. This text has no HTML sink: user turns render as plain text, assistant turns go through
react-markdownwithoutrehype-raw, and React escapes both on the way out. Escaping here bought nothing and corrupted the payload instead —I'm building a client's dashboardreached OpenAI, and was stored inChatMessage.content, asI'm building a client's dashboard, and stripping<…>swallowed ordinary prose likeif x < 5 and y > 3. Escaping belongs at the sink; if a new surface ever renders this content as HTML, that surface escapes it.What survives is the work that is actually this layer's job: a length ceiling, and removal of characters that are never legitimate in typed text — C0/C1 controls (NUL in particular, which Postgres rejects outright in
text) plus the bidi-override and zero-width codepoints used to hide instructions from a human reader while the model still sees them. ZWNJ/ZWJ (U+200C/U+200D) are kept: Persian and Arabic text and emoji sequences need them.