Moderation quirks

Messages are born invisible

Every message starts life invisible to the other person. An instant filter checks it first and blocks a short, fixed list of phrases on the spot: a message it stops is written hidden and is never queued for AI moderation. Everything else starts as pending. Not "sent, then retracted if bad" — it does not exist for the other person until moderation returns a verdict. There is no window, however small, where an unreviewed message is visible.

This applies to everything that crosses the wire: human messages, images, and even the AI companion's own messages. Nothing is exempt from its own pipeline.

When moderation breaks, delivery stops

If the moderation pipeline is unreachable — model down, network gone, anything — messages fail closed: they block instead of slipping through unreviewed.

That is a deliberate trade-off. Blocking traffic during an outage makes the product look broken, and we would rather look broken than deliver a message the check could not clear.

But an outage never counts against you

Here is the counterweight: a message blocked because our infrastructure failed records nothing against the sender. No strike, no history, no suspicion score. Punishment requires an actual verdict about actual content.

The two rules only make sense together. Fail-closed without this rule would mean an outage silently accumulates penalties against innocent people — which would violate the founding rule of the whole system: safety features protect users, they never punish legitimate ones.

An empty answer is a wrong answer

The models that review messages reason before they answer, and reasoning consumes output budget. Early on, a too-small budget meant the model sometimes ran out of tokens before emitting its verdict — and an empty verdict parsed as "no objection". A resource limit had quietly become an approval.

Today the verdict must arrive complete and well-formed. Truncated, malformed, or empty output is treated exactly like an infrastructure failure: the message blocks. Silence is not consent.

The model only speaks in buckets

Moderation scores don't arrive as smooth decimals — in practice the models snap to a coarse grid of a few values. So decision thresholds are never placed on a bucket, where model jitter flips outcomes coin-toss style. They go in the middle of the empty gaps between buckets, where a score can't land.

It's the kind of thing you only learn by staring at real score distributions, and it changed how every threshold in the system is chosen.

How we will publish the false-positive rate

A moderation system's honest failure mode is not only the harmful message it misses — it is the ordinary message it wrongly withholds. On a platform whose founding rule is that safety features protect people rather than punish them, a wrongly blocked message is a top-severity bug. So the number most worth publishing is the one that measures our own mistakes.

There is no number here yet, because there is no traffic yet. Publishing a rate before anyone has sent a message would be inventing one. Here is exactly how it will be produced.

From real traffic — the reversal rate. Every sanction can be appealed, and an appeal is adjudicated automatically, by the same published policy — never by a person reading your chat. The reversal rate is the number of appeals upheld divided by the number of sanctions issued in the same window: the rate at which we restricted someone we should not have. Appeals nobody files are not counted as agreement, so the reversal rate is always reported next to the share of sanctions that were appealed at all — a low reversal rate means much less when almost nobody appealed.

Note what that does not cover. Most withheld messages carry no sanction at all: a block is a message not delivered, and an appeal today attaches to a restriction on your account, not to an individual message. So the reversal rate measures the punitive edge of the system, which is the part that can actually cost you something. Extending it to every withheld message needs a link from an appeal back to the message that triggered it, which does not exist yet; when it does, the message-level rate is published here beside this one.

From the eval set — the block rate and the false-positive rate. We keep a fixed set of messages labelled by hand: some clearly harmful, some deliberately borderline, and — the important half — ordinary, rude, flirty or blunt chat that has to go through untouched. Running the live classifier over that set gives two figures: how much of the harmful half it withholds, and how much of the benign half it withholds by mistake. Because the same set is re-run whenever a model or a threshold changes, those two figures are comparable over time in a way that live traffic alone is not.

When, and where. Both land on this page, each carrying the date it was measured, how big the sample was, and which model and thresholds were in force at the time. A figure without those four things is not a measurement, and we won't print one. Until the first measurement exists, this section says so instead.

What we won't publish. Anything drawn from a sample too small to mean anything, and no single blended score. The two measurements answer different questions, and an average of them answers neither.