enrichment: Haiku flat details + image gallery on expand
apply service
- POST /internal/fetch-listing: headless Playwright fetch of a listing URL,
returns {html, image_urls[], final_url}. Uses the same browser
fingerprint/profile as the apply run so bot guards don't kick in
web service
- New enrichment pipeline (web/enrichment.py):
/internal/flats → upsert → kick() enrichment in a background thread
1. POST /internal/fetch-listing on apply
2. llm.extract_flat_details(html, url) — Haiku tool-use call returns
structured JSON (address, rooms, rent, description, pros/cons, etc.)
3. Download each image directly to /data/flats/<slug>/NN.<ext>
4. Persist enrichment_json + image_count + enrichment_status on the flat
- llm.py: minimal Anthropic /v1/messages wrapper, no SDK
- DB migration v5 adds enrichment_json/_status/_updated_at + image_count
- Admin "Altbestand anreichern" button (POST /actions/enrich-all) queues
backfill for all pending/failed rows; runs in a detached task
- GET /partials/wohnung/<id> renders _wohnung_detail.html
- GET /flat-images/<slug>/<n> serves the downloaded image
UI
- Chevron on each list row toggles an inline detail pane (HTMX fetch on
first open, hx-preserve keeps it open across the 3–30 s polls)
- CSS .flat-gallery normalises image tiles to a 4/3 aspect with object-fit:
cover so different source sizes align cleanly
- "analysiert…" / "?" chips on the list reflect enrichment_status
Config
- ANTHROPIC_API_KEY + ANTHROPIC_MODEL wired into docker-compose's web
service (default model: claude-haiku-4-5-20251001)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
2609d3504a
commit
eb66284172
11 changed files with 688 additions and 44 deletions
32
web/db.py
32
web/db.py
|
|
@ -195,6 +195,13 @@ MIGRATIONS: list[str] = [
|
|||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_rejections_user ON flat_rejections(user_id);
|
||||
""",
|
||||
# 0005: LLM enrichment — extracted details + downloaded image count per flat
|
||||
"""
|
||||
ALTER TABLE flats ADD COLUMN enrichment_json TEXT;
|
||||
ALTER TABLE flats ADD COLUMN enrichment_status TEXT NOT NULL DEFAULT 'pending';
|
||||
ALTER TABLE flats ADD COLUMN enrichment_updated_at TEXT;
|
||||
ALTER TABLE flats ADD COLUMN image_count INTEGER NOT NULL DEFAULT 0;
|
||||
""",
|
||||
]
|
||||
|
||||
|
||||
|
|
@ -447,6 +454,31 @@ def get_flat(flat_id: str) -> Optional[sqlite3.Row]:
|
|||
return _conn.execute("SELECT * FROM flats WHERE id = ?", (flat_id,)).fetchone()
|
||||
|
||||
|
||||
def set_flat_enrichment(flat_id: str, status: str,
|
||||
enrichment: Optional[dict] = None,
|
||||
image_count: int = 0) -> None:
|
||||
with _lock:
|
||||
_conn.execute(
|
||||
"""UPDATE flats SET enrichment_status = ?,
|
||||
enrichment_json = ?,
|
||||
enrichment_updated_at = ?,
|
||||
image_count = ?
|
||||
WHERE id = ?""",
|
||||
(status,
|
||||
json.dumps(enrichment) if enrichment is not None else None,
|
||||
now_iso(), image_count, flat_id),
|
||||
)
|
||||
|
||||
|
||||
def flats_needing_enrichment(limit: int = 100) -> list[sqlite3.Row]:
|
||||
return list(_conn.execute(
|
||||
"""SELECT id, link FROM flats
|
||||
WHERE enrichment_status IN ('pending', 'failed')
|
||||
ORDER BY discovered_at DESC LIMIT ?""",
|
||||
(limit,),
|
||||
).fetchall())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Applications
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue