Data Quality & AI Limitations

Row ids are now stable, so updated_since works properly and a cursor can be resumed

The pipeline used to delete and re-insert every row of a document on each run, so every id changed daily and every row looked freshly modified. It now updates rows in place. Two consequences for you, both improvements. FIRST, updated_since is now sharp: updated_at moves only when a row's content actually changed, where previously it moved on every row every day and so returned effectively the whole table. If you tried incremental sync before and abandoned it, try again. SECOND, a stored cursor can now be resumed. Ids are only ever issued upward, so a row that IS re-created moves to the end of the ordering: a resumed cursor may hand you an event you have already seen, but it cannot skip one. The previous documentation said a cursor was valid only within a single pass and could both skip and repeat - the skip half no longer applies. De-duplicate on fold_id. A few pct of rows still get new ids each run, being the ones whose labels our correcting passes rewrite, so updated_since remains the better tool for incremental sync.

/api/v1/catalysts now deduplicates by default; new fold_id is the stable key

BEHAVIOUR CHANGE: the endpoint previously returned every extracted row and asked you to collapse duplicates yourself. It now applies the same fold the calendar, ICS feed, weekly digest and alerts use, so a response matches finscry.com exactly. Roughly 13 pct of rows were alternative labels for an event already present (664 of 4,938 measured on the day of the change). Pass fold=false for the previous behaviour - every raw row, which will NOT match the calendar and is intended for auditing extraction rather than display. Each event now carries fold_id and folded_count. KEY ON fold_id, not event_id: event_id identifies a row within one document, and the row chosen to represent an event changes whenever a newer announcement arrives with a better label, so event_id moves for an event that did not change. fold_id is keyed on the event identity and survives both a relabel and a change of representative. Responses also carry a top-level 'folded' flag.

Milestone roles replace bare instrument labels; event identity clarified

MAJOR: catalyst_type values changed for existing events. Labels that named only the INSTRUMENT now name the MILESTONE - 'SPP Closes' was stored as 'SPP' and is now 'Offer Closing Date'. New values include Offer Closing Date, Offer Opening Date, Allotment Date, Settlement Date, Trading Commencement, Ex-Date, Offer Results Announcement, Offer Document Lodgement, Completion Date and Director Nominations Close. A label naming both a meeting and an instrument now resolves to the meeting. Because event_id derives from catalyst_type, these corrections MOVE event_id for the affected rows - key on (symbol, catalyst_date, fold_group) if you need identity across syncs. Deduplication no longer considers category: the same event arriving from two documents categorised differently used to return as two rows.

Superseded timetables retired; dividend ex/record/payment split restored

When a later announcement republishes an offer or scheme timetable, the dates it replaces are now retired and no longer returned. Separately, Ex-Dividend Date and Dividend Payment Date were being collapsed to a bare Dividend at ingest; the distinction is restored and recovered on existing rows.

Halt resumptions classified by shape; scheme fallback narrowed

Trading-halt resumptions are classified structurally rather than by an enumerated phrase list. The scheme fallback now requires corporate-action context before accepting a headline, and excludes tenement schedule rows from its date window - both were producing dates from documents that were not scheme timetables.

Four label families folded: Loan Maturity, Trading Commencement, Quotation, Despatch

31 label spellings across 218 rows now fold to four canonical groups. Trading Commencement is anchored on commence/commencement so it cannot absorb trading STOPPING. Quotation deliberately uses two groups, because the family runs both directions and one group would merge a listing with a delisting.

Scheme label families collapsed; inferred-year flag added

Three families of scheme labels collapsed to canonical types. A new year_inferred flag marks rows whose YEAR came from document context rather than the text - ASX timetables routinely write a bare day-month, and this makes the inference visible rather than silent.

Scheme documents routed through AI extraction; headline filter widened

Scheme booklets and related documents now reach the model, with merge rules for the labels they produce. Additional allow terms added to the headline filter, so announcement types previously skipped are now processed.

Prospectus documents routed through AI extraction

Prospectuses previously bypassed the model and were handled by a regex fallback. They now go through AI extraction with the fallback retained as a backstop. Effect: more complete offer timetables, and more specific labels on prospectus-derived rows.

Stricter evidence for inferred dates and reporting-period ends

A reporting-period phrase now only produces a catalyst when the date IS the period end, matched against the first date after the phrase - so 'year ended 30 June 2026 on 26 August 2026' keeps 26 August. Date inference and source-snippet evidence tightened generally, reducing dates asserted without supporting text.

Label and date-parsing corrections across four classes

The webinar fallback no longer matches a bare call (it was catching company names such as Callisto). Month-first date parsing is guarded against clock times. Acronyms and apostrophes survive title-casing (ASIC no longer becomes Asic). Court steps are matched before the generic scheme rule, so Second Court Date is no longer absorbed into the scheme meeting label.

Option expiry labels unified; escrow and fold-group consistency fixes

Option expiry was arriving under several spellings and folding inconsistently. Escrow document handling was made consistent, and capital-raising fold groups corrected. Effect: fewer duplicate rows for one event.

Halts and dividends promoted to primary extraction methods; one source of truth for categories

Halt and dividend events are produced deterministically rather than by the AI pass, so they no longer depend on model output. The category taxonomy was consolidated to a single definition shared by every surface, removing drift between the extractor and the API.

Date revision tracking, trading halts, and the price-sensitive flag

Catalysts whose date changes across successive announcements now record the previous date, surfaced as previous_date. Trading halts became extractable events. The ASX price-sensitive marker is carried through from the announcement to the event.

Date ranges resolve to the FIRST date, not the last

Where a document expressed a period (the offer will be open from 3 July to 24 July), the extractor was taking the END of the range as the catalyst date. It now takes the first. Effect: offer-open and similar dates moved earlier, to the date actually being announced.

Failed AI extractions no longer write empty calendar entries

A document whose extraction failed used to write an empty entry, which then looked like a successfully-processed document with no dates. Those entries are no longer written, so a failure is visible as a failure and can be retried.

Headline filters consolidated; NTA substring bug fixed; AGM/EGM allowlist added

Multiple overlapping headline filters were merged into one. A substring bug on NTA was matching headlines it should not have. AGM, EGM and trading-update headlines were added to the allowlist, so announcement types previously skipped entirely are now processed.

Catalyst taxonomy expanded

The set of event types the extractor recognises was broadened beyond the initial results and meeting categories, and the AI prompt updated to match. Effect: event classes that previously produced no catalyst now appear.

Scheme documents: retry logic and pre-extraction

Scheme booklets are long and their timetables sit deep in the document. Added a pre-extraction step to locate the relevant section before the model sees it, plus retry logic for documents that return nothing on the first pass.

Initial extraction rules - AI primary pass with webinar, scheme and capital-raising fallbacks

Extraction pipeline uses llama3.3:70b via Ollama. Passes: ai_primary (high-signal excerpt), ai_focus (targeted retry on specific date context), scheme_fallback (regex on scheme/takeover timetables), webinar_fallback (regex on webinar date lines), capital_raising_fallback (regex on offer timetables in large prospectus documents). Quality scores 0-100 are computed from confidence, source snippet richness, and extraction method.