Building an NLP Pipeline That Categorizes Thousands of Deals a Day

Every deals website has the same dirty secret: deal data is a mess. The same pair of headphones is a "Headphone", a "Wireless Audio Device", and an "Electronics > Audio" depending on which retailer's API you ask. Multiply that by thousands of products across Amazon, Best Buy, Walmart, eBay, and Woot — arriving daily — and you have the core engineering problem behind Wisedealing.

The pipeline

The backend I built does four things continuously:

1. Fetch — serverless functions pull product and deal data from multiple retailers on a daily cycle
2. Standardize — every record gets normalized into one internal shape: consistent fields, consistent formats
3. Categorize — a custom Node NLP model assigns each product to a category automatically
4. Store & refresh — everything lands in the database so users always see current offers

The standardization step sounds boring and is secretly the most important one. Downstream of it, categorization, search, and filtering all become tractable. Without it, you're writing special cases forever.

Teaching a model your categories

Off-the-shelf classifiers don't know your taxonomy. The trick that made this work: train a small classifier on your own labeled examples, then let it run over the firehose.

// Simplified sketch of the categorization step
const classifier = new NodeNLPCategorizer({
  labels: ["Electronics", "Home & Kitchen", "Fashion", "Gaming", "..."],
});

function categorize(product) {
  const text = `${product.title} ${product.brand} ${product.description}`;
  const prediction = classifier.predict(text);

  // Low-confidence predictions go to manual review,
  // not straight onto the site with a wrong label.
  return prediction.confidence > 0.6
    ? prediction.label
    : queueForReview(product);
}

Two design decisions carried most of the value:

• Confidence thresholds instead of blind trust. Predictions below a threshold went to review rather than live with a wrong category. A mislabeled product doesn't just look broken — it breaks search and filters for everyone who trusted that category.
• Category text as signal. Concatenating title, brand, and description gave the classifier far more to work with than titles alone.

What the frontend gets to be

With a clean, categorized dataset underneath, the frontend (Next.js + Tailwind CSS) becomes genuinely simple: browse by category, brand, or store, search across everything — including multisearch, which queries multiple retailer sites in real time so users can compare prices side by side.

That's the part of full-stack AI work I enjoy most: when the data pipeline is right, the user-facing features feel almost free. When it's wrong, no amount of UI polish saves you.

Lessons

• Standardize before anything else — every downstream system inherits that decision
• Automation should have an escape hatch: low-confidence predictions routed to humans beat confidently wrong labels
• Daily refresh cycles mean designing for staleness: what happens when a deal dies overnight?

This project taught me that "AI-powered" products are won in the plumbing. The model is one component among many — and usually not the hardest one to get right.