Adding AI Content Moderation to a Live Social Platform

When I built Aroundu Social — a location-based social platform where users post, comment, and message in real time — I learned a lesson that still shapes how I think about AI engineering today:

The model is the easy part. Making it survive contact with real users is the actual work.

The problem

Any platform with user-generated content and location-based anonymity will attract spam, harassment, and worse. Waiting for users to report posts before a human reviews them means the damage is already done — the post was seen, reported by dozens of people, and your community is already asking why nobody did anything.

The goal was simple to state and hard to build: flag inappropriate content automatically, before most users ever see it, while keeping humans in charge of the final decision.

The architecture

The system ended up with three layers:

1. Automated flagging — every post and comment passes through Azure AI's content moderation API before it goes fully live. Problematic content gets flagged for review rather than silently published.
2. Human review — a moderator dashboard where flagged content queues up. Moderators can delete posts, ban users for custom durations, and resolve reports manually.
3. Oversight of the overseers — a super-admin view tracking moderator activity, response times, and even idle time, because an unmonitored moderation team has its own failure modes.

A simplified sketch of the ingestion flow looks like this:

// Simplified version of the pre-publish check
async function moderatePost(post: IncomingPost): Promise<Verdict> {
  const result = await contentModerator.scanText(post.body);

  if (result.isAdult || result.isRacy) {
    return { status: "FLAGGED", reason: result.category };
  }

  return { status: "PUBLISHED" };
}

That's maybe ten lines. The other few hundred lines of real work were everything around it:

• Deciding what happens between "published" and "removed" — the flagged state needs its own queue, its own UI, and its own rules
• Handling API failures gracefully (when the moderation service is down, do you fail open or closed?)
• Making report buttons on every post and comment feed into the same pipeline, so human signals and machine signals land in one place
• Building profiles with full post/comment history so moderators can judge context, not just isolated messages

What I'd tell someone starting this

Fail closed for high-risk content, fail open for ambiguous cases. If the classifier is certain something is bad, hold it back even at some risk of a false positive. If it's unsure, publish but queue for review. Treating confidence as a spectrum instead of a boolean changed the whole feel of the system.

Ban durations should be a dial, not a switch. Temporary bans with configurable lengths handled most abuse. Permanent bans were rare, deliberate, and always backed by a human decision.

Log everything the moderators do. Tracking working hours, actions taken, and idle screens wasn't paranoia — it surfaced exactly when coverage gaps appeared, which is when platforms get hurt.

Why this matters beyond one app

This project is why I roll my eyes at demos. A moderation model demoed in a notebook takes five minutes. The production system — verdicts, queues, appeals, admin analytics, failure handling — took real engineering. That gap between working and shipped is exactly where AI engineering lives, and it's where I've decided to build my career.