DEV Community

Build an AI Moderation Classifier in Python

Two-Stage Pipeline

The app uses a two-stage pipeline:

  • Embeddings pre-filter against a known-bad blocklist.
  • LLM moderation judgment when there is no strong blocklist match.

The Flow

Before moderating content, build the blocklist index:

curl -X POST http://localhost:5000/blocklist/index

That embeds the bundled sample_blocklist.json entries through:

POST /v2/ai/openai/embeddings

Then you can classify a single piece of content:

curl -X POST http://localhost:5000/moderate \
  -H "Content-Type: application/json" \
  -d '{ "content": "Great product, really enjoyed using it.", "source": "review", "author_id": "user-123" }'

The app returns structured fields:

{
  "category": "safe",
  "confidence": 1.0,
  "flags": [],
  "blocklist_match": false,
  "recommended_action": "allow",
  "reason": "Benign positive product review with no policy violations."
}

If content matches the blocklist strongly enough, the app skips the LLM and returns the matched category directly. For anything else, it calls:

POST /v2/ai/chat/completions

The default models are:

  • AI_MODEL=moonshotai/Kimi-K2.6
  • EMBEDDING_MODEL=thenlper/gte-large

Endpoints

The example includes:

  • POST /blocklist/index
  • POST /moderate
  • POST /moderate/batch
  • POST /blocklist
  • GET /blocklist
  • GET /moderations
  • GET /moderations/<id>
  • GET /stats
  • GET /health

Batch Moderation

Batch moderation supports up to 20 items:

curl -X POST http://localhost:5000/moderate/batch \
  -H "Content-Type: application/json" \
  -d '{ "items": [ {"content": "Great product!", "source": "review"}, {"content": "Buy cheap followers now!", "source": "message"} ] }'

Why This Shape Is Useful

The important idea is separating obvious moderation cases from nuanced ones. Known-bad patterns can be handled with embeddings similarity. Ambiguous content can go to an LLM for classification and reasoning. The response is structured enough to plug into product logic: allow, flag, remove, escalate.

For production, I would add persistence, audit logs, human review queues, rate limiting, and a feedback loop so reviewers can tune the blocklist and thresholds over time.

Resources

Comments

No comments yet. Start the discussion.