ProductCompat — AI Enrichment & Search Queue — Documentation

## 1. How it works

Each product that enters the AI Search Queue goes through a five-stage pipeline:

1 · Snapshot existing data→ 2 · Query ALL sources→ 3 · AI extraction & validation→ 4 · Candidates (review-first)→ 5 · You approve / reject

- Sources: unlike the classic pipeline (first source that answers wins), the queue collects results from every configured source — Icecat, SearXNG, Google CSE, SerpApi, official/trusted sites — plus their raw text snippets.
- AI pass: the gathered text is sent to your AI provider (via AiCore) which extracts part numbers, device models, barcodes, corrections for wrong data, a clean description, brand, specs and image URLs.
- Hallucination guards: the AI may only report values that appear verbatim in the sources; barcodes must pass the GTIN check-digit; AI-only findings are capped at 60% confidence (corroborated ones get boosted). Anything that fails validation is silently dropped.
- Review-first: structured data lands as needs-review rows; rich data (description, name, brand, specs, images) lands as pending candidates. Nothing touches the product until you click Apply / Approve.

Zero duplicated AI config. ProductCompat has no API keys of its own — every AI call goes through the AiCore provider chain you configure once in AI Core → Settings, and every call is logged to the central AI usage dashboard (module `ProductCompat`, feature `aggressive_enrich`).

## 2. Data sources explained

These are the entries you see in Product Compatibility → Settings. They are data sources the pipeline reads from — each one is optional, and the queue uses every source that is configured:

Source | What it is | Cost / limits
--- | --- | ---
Icecat | A worldwide product catalog (icecat.biz). Looked up by brand + part number or barcode. The most reliable source: returns canonical MPNs, GTIN barcodes, product titles — and now also descriptions and official product images. Best for products that already have a part number. | Free with the open account (`openIcecat-live`); your own account (`tpsegy`) raises the rate limits.
SearXNG | A self-hosted meta search engine (it queries Google/Bing/DuckDuckGo… anonymously from your own server). Used to find part numbers and source text for products that have no part number yet. You host it yourself (e.g. Docker on `localhost:8080`). | Free, unlimited — but the instance must have `format=json` enabled in its `settings.yml`.
Google CSE | Google Programmable Search. Same job as SearXNG (web search snippets) using Google's official API. Needs an API key + search engine id (cx). | Free tier: 100 queries/day, then paid.
SerpApi | A commercial search-results API — a paid, higher-quality alternative to the two above. | Paid subscription.
Official / trusted sites | Direct scraping of manufacturer sites and trusted aggregators (configured via URL templates in the module config). Off unless templates are configured. | Free.
AI (AiCore) | Not a search source — the brain that reads everything the sources returned, extracts structured data, spots wrong values and writes the description. Configured in AI Core → Settings, not here. The "AI enrichment" section in ProductCompat settings only controls how the queue behaves (schedule, batch size, auto-queue…). | Whatever your AI provider costs (Groq has a generous free tier).

Rule of thumb: Icecat gives the cleanest data when a part number exists. The search engines (SearXNG / Google CSE / SerpApi) matter for products without part numbers. The AI turns all of that raw text into structured, validated suggestions. You don't need all of them — Icecat + one search engine + AI is a solid setup.

Product Compatibility → Settings: SearXNG / Google / SerpApi source settings (left) and the new AI enrichment section (right) with the green AiCore status indicator.

## 3. AI provider & failover chain (AI Core → Settings)

The AI engine is shared by the whole system (WhatsApp auto-reply, VetAI, ProductCompat…). Setup is a two-step flow — configure services once, then just pick them in the chain. The main form holds only behavior (business type, on/off, temperature, budget, persona, prompts) plus a read-only Active primary line — credentials live exclusively in the AI services box. Copies upgraded from older versions get their manually-configured provider auto-converted into a service card, linked as Default.

AI Settings main form after simplification: no credential fields — just behavior settings and the "Active primary" status line (★ Groq).

### Step 1 — AI services (configure & test each service once)

- The box starts empty — use Add provider to add only the services you need: Groq, Claude, DeepSeek, z.ai, Bynara, naraya.ai, Custom. Endpoint and suggested model come prefilled; you only paste the API key (for Bynara / naraya.ai also their endpoint URL).
- Each panel has its own Save service, Test Connection and Remove buttons; a green Configured pill shows which services are ready.
- Make default: each configured card has a radio selector — the chosen service becomes the Primary the engine calls first. The first service you configure becomes the default automatically (unless a working manual primary already exists). Unconfigured cards show the selector disabled until saved with a key.
- Editing a service later automatically updates the default and every chain slot that uses it.

AI Core → Settings → AI services: Groq / Claude / DeepSeek panels with prefilled endpoints — configure and connection-test each service once.

### Step 2 — Failover chain (selection only)

- The Primary card is read-only — it shows whichever service carries the Default radio (or "manual configuration" if you configured the main form directly).
- Each fallback slot (fallback → fallback2 → fallback3) is just a dropdown of your configured services — no credentials re-entered, one Save chain button.
- If the primary call fails for any reason (outage, rate limit, timeout), the engine automatically retries the chain in order. The user never sees the failure. Persona, voice and prompt settings are never touched by the chain.

Failover chain: pick a configured service per slot — "none / disabled" skips a slot.

Voice transcription settings appear on this page only for clinical business types (veterinary / medical clinic) — they configure the dictation buttons on consultation screens and are hidden for retail and other verticals.
Monthly token budget: if a profile has a budget and it is exhausted, AI calls stop (they do not fail over — that would burn the backup's budget too). Queue entries stay queued and resume automatically after you raise the budget or the month resets.

## 4. The AI Search Queue

The queue is the list of products waiting for a full data check. Products enter it four ways:

Reason | How
--- | ---
new | Automatically when a product is created (can be disabled in settings).
edited | When you tick the "Re-check" checkbox while editing a product.
manual | From the queue screen — search a product and "Add to queue".
backfill | The "Backfill" button queues existing products that are missing part numbers, a description or an image (capped, default 500).

Product Compatibility → AI Search Queue: add products, backfill, filter by status, retry failed / skip entries.

### Running the queue

Three ways — pick what suits you:

1. Control Panel (live progress): Product Compatibility → Control Panel → AI Search Queue → Run queue now. Processes in small chunks with a progress bar; you can stop anytime.
2. Scheduled (recommended): enable it in Settings → AI enrichment. Requires the server cron to call `php artisan schedule:run` every minute (already the case on your servers).
3. Command line: `php artisan productcompat:process-queue --limit=10`

Control Panel: the classic pipeline cards plus the new AI Search Queue card with queued/failed counters.

## 5. Product create / edit hooks

- Creating a product — it is queued automatically and silently (a small note appears under the compatibility block). Disable via Auto-queue new products on create in settings.
- Editing a product — the compatibility block shows an unchecked box: "Re-check this product with the AI search queue after saving", next to a badge with the product's current queue status. Tick it only when you want a re-check (edited products re-enter with higher priority).

Product edit form: the re-check checkbox with the green "AI queue status: Done" badge, rendered inside the Product Compatibility block.

## 6. Review queue & applying data

Everything the pipeline finds waits for your decision in Product Compatibility → Review Queue. Products with AI findings show an N AI badge:

Review Queue: SKU0001 carries a "3 AI" badge — three AI data suggestions pending.

Opening the product card shows the AI data suggestions section:

Per product: image thumbnails with Download & attach, description Current-vs-Suggested comparison, name/brand suggestions — each with its source and confidence badge.

Suggestion type | What Apply does
--- | ---
Image | Download & attach downloads the image server-side (type & size checked) and sets it as the product image. If the product already has one, a "Replace existing" checkbox appears.
Description | Replaces the product description with the suggested text.
Name / Brand | Updates the product name / links the brand (created if it doesn't exist).
Specs | Appends a formatted specifications list to the description.
Correction flag | The AI found an existing value the sources contradict. Remove deletes the wrong value; Keep dismisses the flag. Values you verified manually are never flagged.

Rejected suggestions are remembered — re-running the queue will not resurrect them.

## 7. Change report (audit trail)

Every pipeline run and every applied suggestion is recorded in Product Compatibility → Change Report. AI activity is labeled AI check (queue run: what was found) and AI applied (what you approved: before & after values), so any change can be traced back or manually reverted.

Change Report: the AI check row for SKU0001 — snapshot of the old data alongside the run result (3 candidates, top source Icecat).

## 8. Settings reference (Product Compatibility → Settings → AI enrichment)

Setting | Meaning | Default
--- | --- | ---
Enable scheduled AI search queue | Runs the queue automatically on the schedule below (needs server cron). | Off
Queue frequency | Every 15 / 30 minutes, hourly or daily. | Hourly
Batch size per run | How many products each scheduled run processes. Each product ≈ one AI call + a few HTTP lookups; start small. | 10
Max attempts per product | After this many failures the entry is marked failed (retry from the queue screen). | 3
Auto-queue new products on create | The silent enqueue on product creation. | On
Run extra search variants | Adds 1–2 extra search queries per product (specs / compatible models) for richer AI source text. Costs extra search-source quota. | On
Max downloaded image size (KB) | Size cap for "Download & attach". | 2048
Use AI in the classic enrichment pipeline | Also adds the AI as a last-resort source in the classic Enrich buttons (not just the queue). | Off

## 9. Console commands & cron

    # Process the AI search queue manually
    php artisan productcompat:process-queue --limit=10 --sleep=2

    # Restrict to one business
    php artisan productcompat:process-queue --business=1 --limit=5

    # Classic commands (unchanged)
    php artisan productcompat:enrich --missing --limit=50
    php artisan productcompat:parse-custom-fields --dry-run

The scheduled run requires the standard Laravel cron entry (once per server):

    * * * * * php /path/to/saas-one/artisan schedule:run >> /dev/null 2>&1

### Deploying to another copy

1. Pull the branch / copy the two module folders (`Modules/AiCore`, `Modules/ProductCompat`).
2. Visit `/product-compat/update` then `/ai-core/update` as an admin — runs migrations, registers the new permission, bumps versions.
3. Grant the AI search queue permission to the roles that need it, configure AI Settings, done.

## 10. Permissions (Role management)

Permission | Grants
--- | ---
`productcompat.view` | See compatibility data.
`productcompat.edit` | Edit part numbers / compatibility; product create/edit hook fires for these users.
`productcompat.review` | Review queue: approve / reject / apply AI suggestions (incl. image download).
`productcompat.queue` new | View & manage the AI Search Queue screen.
`aicore.manage` | Configure AI providers, keys & failover chain.

## 11. Troubleshooting

Symptom | Cause & fix
--- | ---
Queue entries stay queued, job message says "AI monthly token budget reached" | The AI provider's monthly budget is exhausted. Raise it in AI Core → Settings (or wait for the month to reset) — the queue resumes where it stopped; no entries are lost.
Enrichment log shows `SearXNG HTTP 404 (is format=json enabled…)` | Your SearXNG instance blocks the JSON API. In its `settings.yml` add `json` to `search: formats:` and restart it. The pipeline works without it — you just lose that source.
Job stops with "Provider limit reached" | Icecat / Google CSE daily quota hit. Wait for the reset or switch the Icecat account in settings; the queue can be re-run any time.
AI section missing in review / settings shows yellow warning | AiCore isn't configured for this business. Open AI Core → Settings, set the Default provider + key, press Test.
New products aren't auto-queued | The creating user needs the `productcompat.edit` permission (the hook rides on the compatibility form block). Products created by other users can be picked up with the Backfill button.
Image download rejected | By design: only http(s) URLs to public hosts, image content types (jpeg/png/webp/gif) and files under the size cap are accepted.