How AI moderation works here
No mystery, no magic. Here is the actual order of operations between you pressing send and the other person seeing anything.
Before delivery, not after
Here is the whole pipeline, in order. You write a message and press send. Before it goes anywhere, an AI moderation model reads it and returns a verdict. Only if that verdict comes back clear does the message get delivered to the other person. If it does not, the message stops right there.
The load-bearing word is before. On most platforms the sequence is deliver, then maybe review; here it is review, then maybe deliver. A blocked message never reaches the other person at all — it is not delivered and then retracted, it simply never arrives. That one reordering is most of what separates this from an ordinary chat box.
Fail closed
Systems fail. The only interesting question is which way they fail when they do.
When our moderation system has an outage — the model is unreachable, a request times out, something upstream breaks — messages hold rather than skip the check. They do not sail through unmoderated on the theory that a failure shouldn't be allowed to slow anyone down. We would rather a message be delayed than delivered unchecked, because safety infrastructure that fails open is not safety infrastructure at all. It is the one place we are genuinely happy to be occasionally annoying.
Protect, never punish
A moderation system that punishes people for its own mistakes is worse than no system at all, so ours is built the other way around. A message blocked because of an infrastructure failure never counts against the sender — a timeout is not a strike. Sanctions require a genuine pattern of abuse, not a single tripped wire, because one bad reading should never brand a person as a problem.
And on the occasions we do get it wrong, that is appealable inside the app. Appeals are adjudicated automatically, and false calls get reversed. There is no punishment queue to grovel through and no waiting on someone's mood — the same kind of system that made the call is held to undoing it when the call was wrong.
What it looks for
The check is not vibes. It looks for specific categories: harassment aimed at the person you are talking to, sexual pressure and unsolicited sexual content, attempts to harvest your personal information, spam and scams, and encouragement of self-harm.
That last category has a deliberate seam running through it. Encouraging someone to hurt themselves is blocked. But a person disclosing their own struggle is not treated as an offender — they are met with crisis resources, never a sanction, because showing support and not punishing someone are, here, the same decision. The full category list, with the kind of message each one catches, is on the what gets blocked page.
The limits, honestly
AI moderation is not perfect, and we are not going to tell you it is. It can be too cautious — an edgy-but-benign joke sometimes reads as worse than it is and gets held. It can also miss things, especially novel phrasings that don't resemble anything it has seen before. Both of those failure modes are real, and both of them exist here.
That is exactly why reporting still matters even with pre-delivery moderation in place: if something slips through, reporting it feeds the system and sharpens the next call. We would rather be honest about the ceiling than sell you a guarantee we cannot keep. The AI limits section of the safety page goes further into where the edges are.