yelgabo/job-fraud · system report · 6 November 2026
How a WorkBC job posting becomes a published Low, Medium or High rating. This report covers which steps are plain code, which are single model calls and which are agents, and the rules that stop any model or agent from publishing a claim on its own.
job-fraud collects WorkBC postings and rates each one on whether it looks like a genuine attempt to hire locally, as opposed to an advert placed to show that no Canadian applied. The rating is published next to a real employer's name, and the whole design follows from that. Models and agents gather evidence. Code turns the evidence into the number.
There are three phases, and they share one Postgres database without calling each other. A scraper collects postings and marks them pending. A judge evaluates pending postings. A read-only Next.js site shows the judged ones. The judge has two front ends. The API judge is one Node process that calls Claude through the Anthropic API. The keyless judge is a Claude Code session that sends helper agents to research employers on the web. Both hand their evidence to the same composer, so the same evidence gets the same score whichever path gathered it. The evidence itself can differ. The API judge's verifier sees an 800-character description excerpt where keyless helpers see 1,800, and the API path records verbatim search blocks where keyless helpers report their own searches.
judge-keyless:apply on the keyless
path.These counts come from a read-only query against the production database on 6 November 2026.
The scheduler was scraping but not judging. The newest scrape was from
2 November 2026 at 12:24 UTC, the Monday cron run, while the newest judgment was from 27 October, and
730 postings were waiting. The scheduler service turned out to have no ANTHROPIC_API_KEY,
and its script only judges when that key is set. Later on 6 November the backlog was drained with the
keyless judge (729 postings scored, one left pending) and the key was added to the scheduler, so
the Monday run now judges up to 1,000 postings after scraping.
Most published ratings predate the composer. Of the 23,021 judged postings, 17,033
have scoringVersion null, which means a model chose the number before the composer existed
(or the impersonation path wrote it). Another 1,662 are version 1 and 4,326 are version 2. So the rule
that no model chooses the number covers about a quarter of what the site shows today. Moving the rest
onto the composer needs a fresh Jev call per posting, because those rows store no judgments for
npm run recompose to reuse.
Employer verdicts have mixed origins as well. 445 came from Haiku 5.5 alone, 50 from Haiku followed by Opus, and 295 from Opus alone (the re-check script). Another 365 were presumed from an own-ATS match, 963 come from the web check as it ran before the tiers, and 2,426 employers carry neither marker, either because they were never checked or because their verdict predates both fields.
The scraper is deterministic. It reads WorkBC's JSON search and detail APIs, runs regex checks over
each description (free email domain, mail-in résumé, crypto payment, banking details or ID before an
offer, a fee to apply, WhatsApp or Telegram only), buckets the NOC occupation code into a job category,
and classifies the apply link's host against a registry of applicant tracking systems such as Workday
and Greenhouse. New postings arrive with scoredAt null, and null is what pending means everywhere
else in the system. Re-scraping a posting that is already judged updates its fields and leaves it
judged.
The site only shows judged rows. Public pages are cached for 600 seconds, so a judge run is visible
within ten minutes, or immediately when REVALIDATE_TOKEN is set and the run can call
POST /api/revalidate. There is no staging database. A local judge run writes to
production and is live once the cache turns over. The site's only write path is the token-gated
/audit mirror, where the owner can leave a note or ask for a posting to be judged again.
One predicate picks the path: is ANTHROPIC_API_KEY non-empty in .env? If
it is, npm run judge does everything in one process. If it is not, the keyless judge runs
inside a Claude Code session and uses that session's agents for web research. A second predicate,
TYPESAFE_API_KEY, decides whether Jev answers the text questions. Without it the regex
flags stand in for Jev's route judgment and the row stores judgments = null.
| API judge | Keyless judge | |
|---|---|---|
| Command | npm run judge | judge-keyless:fetch, then helper agents, then judge-keyless:apply |
| Who researches employers | Claude through the API, with the server-side web_search tool | Claude Code helper agents with the WebSearch tool |
| Escalation to Opus | Done in code by verifyEmployerTiered | Done by the orchestrating session; apply rejects any verdict that needs escalation and is not marked as Opus-confirmed |
| Brand-impersonation check | Opus 5.5 API call on each posting whose apply link names a different company | Opus helper agents, one answer per employer and tenant pair; apply holds a posting that needs the check until a result clears it (since 6 November 2026) |
Search trail on /audit | Verbatim API search blocks | The helper's own list of queries and result URLs, labelled agent-reported |
| Search limit | None per run; $10 per 1,000 searches | About 200 per session, refilling about 100 an hour |
| Prompt lives in | lib/ai/verify-employer-web.ts | The "Agent prompt" section of .claude/skills/judge-postings/SKILL.md |
| Used for | Bulk and scheduled runs | Machines without the key, and small high-scrutiny sets |
What the two share matters more than what differs. Both decide each employer with
verificationAction, both enforce the same escalation rule, and both score through
buildVerdict in lib/scoring/verdict.ts. The grouping differs in one detail.
The keyless fetch groups through planByEmployer, which sends a posting with no employer
to the web check on its own, while judge.ts groups inline and gives such a posting no
employer check at all.
A failed employer check holds its postings back. When a check fails for any
reason other than billing (a refusal, a malformed answer after the retry, a failed Opus re-check),
judge.ts logs it and stage 2 skips that employer's postings, which stay pending. Until
6 November 2026 stage 2 scored them anyway on whatever verdict was cached. A placeholder key still
causes trouble. It counts as set, every Anthropic call returns 401, and while postings that needed a
check stay pending, postings whose employer has a cached or presumed verdict are scored without the
impersonation check. On a keyless machine, leave the ANTHROPIC_API_KEY line in
.env commented out.
This is the multi-agent part of the system. One Claude Code session owns the whole run. It runs the
fetch CLI, spawns helper agents, writes their output to files, and runs the apply CLI. The repo's
update-postings skill wraps it, and judge-postings holds the procedure and the
agent prompt. The skill insists that one session own the run rather than handing batches to separate
top-level sessions, because a single owner can restart the run and its progress stays visible.
The fetch step writes one item per employer that needs a web check, 15 employers to a batch file by
default. The item carries the employer's representative posting, chosen as the first posting that
gives a mailing address when any does (pickRepresentative). The mailing-address type is the
heaviest signal the check produces, and the verifier can only judge an address it has been shown.
{
"workbcId": "49588691",
"title": "...", "employer": "...", "location": "...", "salary": "...", "postedAt": "...",
"atsProvider": "workday", "externalApplyOk": null,
"flags": [{ "flag": "mail_physical_resume", "evidence": "..." }],
"descriptionExcerpt": "first 1800 characters of the description",
"employerPostings": ["49588691", "49588702"]
}
employerPostings is for the apply step, which scores every posting in that list with the
employer's verdict. The helper does not need it.
A JSON array with one verdict per item. The verdict describes the employer. It has no score field.
{
"workbcId": "49588691",
"web": {
"websiteUrl": "https://acme.com",
"websiteReachable": "yes", // yes | no | unknown
"businessMatch": "match", // match | mismatch | uncertain
"locationMatch": "match", // match | mismatch | uncertain
"hasJobsListing": "yes", // yes | no | unknown
"applicationAddressType": "business",
"confidence": 0.85,
"summary": "at most 400 characters, shown on the site",
"verifiedBy": "haiku-5.5" // set by the orchestrator
}
}
applicationAddressType is one of business, residential,
home_business, po_box, virtual, none or
uncertain. home_business needs two sources: the company's own website and at
least one independent listing must both give the home as the business address, and otherwise the
answer is residential.
workbcId and web.uncertain
would be cached on the employer and reused by every later run, while a missing verdict only keeps
the postings pending.mismatch because its industry differs from the role. A retailer
hiring a developer is a match. Mismatch is reserved for fake, shell, parked or impersonating
entities.judge-keyless:apply is the only place a keyless score is computed. For each verdict, in
order:
VerdictSchema. A partial or malformed web object fails and
the employer's postings stay pending.tierViolation. A verdict that needs escalation must be
marked opus-5.5 or haiku-5.5>opus-5.5; anything else is rejected. On
the API path code escalates, and here this check is what catches a skipped escalation step.web on the employer once, then ask Jev and compose a score for each pending
posting in that employer's employerPostings. A Jev error stops that employer's
remaining postings, which stay pending, though the employer verdict is already stored.Postings in cached.json get one more check. The apply step re-decides each of those
employers against all of its current pending postings. If a new posting has arrived since the fetch,
for example one with a mailing address the cached verdict never saw, that employer now needs a web
check and its postings stay pending for the next fetch. The orchestrator runs apply with
--dry-run first, which validates everything and reports tier violations without writing,
and then loops fetch and apply until fetch reports zero pending.
Both paths check employers with Haiku 5.5 first. Opus 5.5 checks again, and its verdict replaces
Haiku's, when needsEscalation in lib/ai/verify-employer-web.ts returns true:
| Haiku's verdict | Why it gets a second look |
|---|---|
businessMatch: mismatch | Adds 25 points. In the model evaluation Haiku found nothing on a real Vancouver payments company and called it a mismatch, which scored 85, a High rating on a real employer. |
locationMatch: mismatch | Adds 12 points. |
Address residential, po_box or virtual | On every posting that mails applications, adds 40 points and floors the score at 70, which is High. |
Address home_business | This label lowers a rating compared with residential, so the cheap model may never give it alone. |
Address uncertain when the posting gives a mailing address | On 1,000 postings Haiku left a house address uncertain where Opus found the residence. |
The split rests on docs/verifier-model-eval.md. On the 21 hand-labelled postings, Haiku
followed by Opus on alarming verdicts agreed with the labels 20 times out of 21, the same as Opus alone,
and escalated 2 of the 21. The estimated cost was about $0.003 per posting against about $0.026 for
Opus on everything, roughly an eighth. Opus also gets up to 10 searches per check instead of 5, because
at 5 searches 39 of the 584 verdicts stored on 27 October said the search had been cut off.
A cached employer verdict is reused unless one of four things is true. The run is a
--rejudge (an API-judge flag); the cached verdict was presumed from an own-ATS match and a new posting does
not apply through that tenant; the cached address type is none but a pending posting gives
a mailing address; or the cached verdict breaks the tier rule because it was written before the tiers
existed. When the re-check script ran over the first 73 such pending-side verdicts, Opus overturned 47
of them.
The API judge is agentic in a narrower sense than the keyless path. Each employer check is one
Messages API request that carries two tools: the server-side web_search tool, capped at 5
uses for Haiku and 10 for Opus, and record_web_verification, a client tool whose input
schema is the verdict. The model chooses its own search queries and runs them on Anthropic's side. If
the server-side loop pauses with pause_turn, the code resends the turn, at most three
times. The answer counts only if it arrives as a record_web_verification call that passes
zod. Otherwise the request is retried once and then counted as failed. Every search block, including
its encrypted result content, goes into EmployerWebSearchLog so the /audit
pages can show what the model consulted.
Stage 1 checks four employers at a time and stage 2 scores eight postings at a time, both through
p-limit, and the SDK retries rate-limit and overload errors up to five times. Two failure
rules keep partial runs safe. An out-of-credit error, which arrives as a 400 and is never retried, sets
a flag that turns every queued task into a no-op, and the run throws after the current stage with the
completed rows saved. A Jev failure leaves that one posting pending, because composing it without
judgments would store a different kind of row under the same scoringVersion.
The weekly job-fraud-scheduler Railway cron service runs this path. On Mondays at 12:17
UTC it runs both scrape passes and then npm run judge -- --limit $JUDGE_LIMIT (1,000 by
default) if the service has an Anthropic key. It reaches Postgres over Railway's private network.
Before scoring each posting, both judges compare the applicant tracking system tenant in the
apply link with the employer the posting names. On the keyless path the fetch writes one
impersonation-NNN.json item per employer and tenant pair, an Opus helper agent answers
it, and apply writes the answer through the same function the API judge uses.
Below, the API judge's version. If a posting says it is from one company but its
Workday link belongs to another, Opus 5.5 investigates with web search and classifies the relationship
as same, affiliate, impersonation or uncertain. The
prompt tells it to choose impersonation only when it is confident the two companies are unrelated, so
a subsidiary applying through its parent's hiring system is cleared. Results are cached per employer
and tenant pair for the rest of the run.
same or affiliate: the posting is scored normally.uncertain: the posting gets an apply_host_mismatch_review flag and is
scored normally.impersonation: the posting is re-attributed to the real company and written
directly as score 90 (High), with reasoning that starts "Brand impersonation:".This is the one place the system writes a fixed number without the composer. The write sets
scoringVersion to null, and every re-scoring script (backfill-ats,
recheck-cached-verdicts, recompose, reverify-mail) skips rows
whose reasoning starts with that prefix. Before 6 November the write left scoringVersion
unchanged and recompose did not check the prefix, so a re-judged composed row could have
lost its 90; neither of the two impersonation rows in the database was affected. Opus runs this check because a smaller model
labelled a real construction company's division as an impersonation of its parent group. It moved from
Opus 4.8 to Opus 5.5 on 27 October without a side-by-side comparison, and no labelled impersonation
cases exist yet. Both October runs logged zero impersonations.
composeScore in lib/scoring/compose.ts is a pure function: the same inputs
always give the same output, with no clock, network or model. It works in this order:
routePlausibleForEmployer is below 0.57. Without them, it is disowned when the posting
has a free-email or messaging-app-only flag. A disowned route withholds the brand credits (business
match, location match, jobs page, business mailing address, no mailing address) while the penalties
on the same fields still apply. Confirming that a national chain exists says nothing about a
posting that collects replies at a Gmail address.askBeforeHire, worth up to 25 points and nothing
at or below probability 0.5.askBeforeHire above 0.8.The published reasoning is generated from the composed result by explain.ts, so the
prose can never disagree with the number beside it. Before the composer a model wrote that prose
without knowing the final score.
Each row below was computed by running composeScore on the stated inputs.
| Posting | Signals that fire | Score | Band |
|---|---|---|---|
| Small local pub, verified, applications to a Gmail address, route fit 0.77 | baseline 20, free email +20, business match -15, location match -5, applies online -4 | 16 | Low |
| National chain, verified, applications to a Gmail address, route fit 0.10 | baseline 20, free email +20; every brand credit withheld because route fit is under 0.57 | 40 | Medium |
| Software role, résumés mailed to a house, route fit 0.30 | baseline 20, private address +40, mail-in for a software role +20; floor 70 not needed | 80 | High |
| Web-verified employer, software role applying through its own Workday | baseline 20, known ATS -25, business match -15, jobs page -7, location match -5, applies online -4 | 0 | Low |
The first two rows are the case the route gate exists for. The regex sees the same free-email flag on both, and Jev's route judgment is what separates a neighbourhood pub on Gmail from a national chain on Gmail.
From lib/scoring/weights.ts. Every number except apply_address_home_business
(the owner's call on 27 October) was transcribed from the midpoints of the old prompt's prose ranges,
and none has been fitted. A reweight is a diff to this file followed by
npm run recompose, with no model calls.
| Signal | Points | Fires when |
|---|---|---|
apply_address_private | +40 | The posting mails applications to a residential, P.O. box or mail-forwarding address (floor 70) |
apply_address_home_business | +25 | It mails to a home the verified employer publishes as its address (no floor) |
business_mismatch | +25 | The web check found a fake, shell, parked or impersonating entity |
crypto_payment, banking_info_upfront | +25 | Regex on the description |
generic_email_domain | +20 | Applications go to a free mailbox (+5 more if the website is unreachable) |
fee_to_apply, id_upfront, whatsapp_telegram_only | +20 | Regex on the description |
mail_resume_software_role | +20 | A software or IT role asks for mailed résumés |
address_not_geocoded | +20 | A stated address did not resolve, or matched with confidence under 0.5 |
address_city_mismatch | +15 | The address resolves to a different city |
location_mismatch, website_unreachable | +12 | Web check or site probe |
pre_hire_ask (Jev) | up to +25 | Probability of a pre-offer money or ID request above 0.5 (floor 70 above 0.8) |
apply_address_business, location_match, address_city_match | -5 | Credits, the first two withheld on a disowned route |
apply_address_none | -4 | The posting asks for no mailed materials (withheld on a disowned route) |
jobs_listing | -7 | The employer's site has a careers page (withheld on a disowned route) |
business_match | -15 | The web check found a real company (withheld on a disowned route) |
ats_known_provider | -25 | The apply link goes to a recognised tracking system |
Jev answers five questions per posting and only one of them is weighted.
routePlausibleForEmployer drives the route gate. employerIsOrganisation,
brokerRouting and brandProminence are stored on the row and not weighted,
because a question earns a weight only after npm run measure-questions shows it separates
the bands. Brand prominence is the instructive case. It reads about the same on low and medium postings
because the medium band is full of big brands, so weighting it directly would penalise size.
These are the properties a reviewer should check are still true after any change to the judge.
scripts/judge.ts,
scripts/judge-keyless-apply.ts). The owner's /audit actions can still write
during a run.lib/scoring/compose.ts produces every composed score.tierViolation), though that marker is a string the orchestrating session sets
itself.uncertain and unknown add
nothing, null checks match no weight-table condition, and a helper omits what it could not
search..env files connect as
jobfraud_local, which can read and write rows but cannot change the schema or
truncate tables. Schema changes reach production only through a deploy.EmployerWebSearchLog keeps the API judge's raw search blocks and, since 6 November
2026, the keyless helpers' reported queries and result URLs, append-only by convention. Presumed
verdicts involve no search and write none.There is no public correction route, by design. The README says a rating is an automated screening
signal that can be wrong. What exists is an owner-only queue on /audit/<token>
pages, the app's only write path. The owner can add a ReviewNote, or file a
JudgeRequest of one of two kinds. A rerun clears the posting's
scoredAt at once, so the next judge pass re-scores it, and the API judge marks the request
resolved when that happens. A deep request keeps the current score and queues the posting
for a single-agent investigation with the owner's note passed in as context.
The posting stays on the site with its old score while the agent works. When the agent's
verdict names the flagged posting, judge-keyless:apply re-scores it even though it is
judged and stamps the request resolved. Before 6 November apply only scored pending postings, so a
deep verdict updated the employer but never the posting the owner had flagged.
The update-postings skill runs a full refresh in this order:
deep request to one fraud-analyst agent with the
owner's note and apply its verdict, which re-scores the posting and stamps
resolvedAt.ANTHROPIC_API_KEY predicate.docs/self-improving-loop.md describes where this is meant to go: drift reports that
file their own review requests, review decisions that become labels, and weight proposals measured
against those labels in CI. Two steps stay human on purpose, labelling and merging rubric changes,
because every merge changes published claims about named employers.
Both rows are API-judge runs, taken from the summaries the judge printed (files under
logs/). Cost is the web-check estimate at list price and excludes Jev.
| Run | Postings | Employers web-checked | Presumed (own ATS) | Escalated to Opus | Searches | Web-check cost | Wall time | Bands L / M / H |
|---|---|---|---|---|---|---|---|---|
| 26 Oct 2026 | 1,000 | 271 | 13 | 37 | 846 | $16.09 | 842 s | 708 / 250 / 42 |
| 27 Oct 2026, backlog | 4,243 | 219 | 103 | 16 | 585 | $9.14 | 833 s | 4,240 / 3 / 0 |
The backlog run shows why grouping by employer matters. It scored 4,243 postings with 219 web checks, because most of those postings belonged to employers with a reusable or presumed verdict. Jev's text questions cost about $2 for a whole corpus, so employer research is where the money goes.
brokerRouting answer
looks promising on 8 postings and is stored on every composed row, but it carries no weight until a
drawn sample of about 60 is labelled.| File | Role |
|---|---|
scripts/judge.ts | The API judge: both stages, the billing abort, rerun-request resolution |
scripts/judge-keyless-fetch.ts | Keyless step 1: read pending, group by employer, write batch files and cached.json |
scripts/judge-keyless-apply.ts | Keyless step 3: the single writer, schema and tier gates, scoring |
.claude/skills/judge-postings/SKILL.md | Keyless procedure, escalation step, search budget, the agent prompt |
.claude/skills/update-postings/SKILL.md | The full refresh: review queue, two scrape passes, path choice, waves |
lib/signals/verification-plan.ts | verificationAction, planByEmployer, the presumed verdict |
lib/ai/verify-employer-web.ts | Employer check prompt and tools, needsEscalation, tierViolation, verifyEmployerTiered |
lib/ai/verifier-models.ts | Model profiles, search caps and list prices |
lib/ai/check-impersonation.ts, resolve-impersonation.ts | The Opus impersonation check and its re-attribution write |
lib/ai/jev-judgments.ts | The five Jev questions |
lib/scoring/verdict.ts | buildVerdict, the per-posting pipeline both paths call |
lib/scoring/compose.ts, weights.ts, explain.ts | The composer, the weight table, the published prose |
lib/shared/risk-band.ts | Band cut-offs at 30 and 70 |
scripts/scheduled-refresh.sh | The weekly Railway cron: two scrapes, then the judge if keyed |
app/audit/[token]/j/[id]/actions.ts | Owner review actions: notes, rerun and deep requests |