บันทึกการสร้าง · Build notes

ภาพรวม · Overview

motdang.net is a Thai-first directory of Chiang Mai and Chiang Rai. The place pages, shelf pages, front page and the index behind the public API are static files written by one Python program, build.py, and uploaded to an R2 bucket (Cloudflare's object storage). A Cloudflare Worker (a script that runs at Cloudflare's edge) reads the bucket and answers each request. Reader submissions go to a second Worker backed by KV (a key-value store). The search at /find reads a D1 database (Cloudflare's hosted SQLite).

build.py uses the standard library and its own modules, 98 *_layer.py files on 2026-10-11, one per feature area. Pillow is imported inside the functions that draw pictures. The share-card renderers also drive headless Chrome and run before the build.

ข้อมูลและที่มา · Data and provenance

A record is one object in data/canonical/cm.json or cr.json: id, province, cat and sub (the shelf), name, nameTh, nameEn, lat, lng, geoPrecision, address, phone, website, hours, attrs, sources, confidence, updatedAt. The importers rewrite these files whole. Hand-kept field truth sits in data/curated/ and wins over a crawled record with the same source reference. Researched facts go in curated/enrich.json, which a crawl does not overwrite.

sources is a list. An entry has a type, a ref, usually a via and a fetch date, and where it applies a licence and a release. confidence takes six values: third-party, crawled, official, curated, field, register. geoPrecision takes four: exact (77,085 records on 2026-10-11), approx (6,938), needs-pin (4,849), block (375). A place page shows its provenance as a border: dashed for a crawl, solid for a mapper's tag, bold for a person who stood there.

The table counts source entries across all 89,247 records on 2026-10-11. A record carries several, so the rows do not sum to the record count.

Beside the records: weather, PM2.5, river gauges (Royal Irrigation Department), fire hotspots (NASA FIRMS), showtimes, lottery results, exchange rates, events from iCal and JSON feeds, festival dates, iNaturalist sightings and Wikimedia Commons photographs. Each is an importer in importers/ that writes a snapshot to disk and keeps the previous snapshot when its source does not answer. The build makes no network request. importers/ is where the network is used.

A crawl is not a verification. The freshness signal needs a human touch, an owner's claim or a field source, so one bulk import cannot raise it on every record. A dead official site is named and dated, and the Internet Archive copy is linked in place of the dead link; check_links.py runs each morning.

Two crawl details. An Overpass timeout answers HTTP 200 with an empty elements list and the error in remark; with a snapshot-first crawl, caching that leaves a shelf empty for good, so fetch() raises OverpassRemark. A province is an area: provinces are clipped with area["ISO3166-2"="TH-50"], since a rectangle around Chiang Rai returns places in Laos and Myanmar.

รอบเดิน · The walks

Two scheduled jobs on the Mac (launchd). The morning walk runs at 07:09 and gathers the daily sources: weather, air, river gauges, fire hotspots, showtimes, lottery, finance, events, festival dates and link health. It also pulls reader submissions (claims, toilet reports, suggestions) and runs a watchdog that reports a source that came home empty. It writes to disk. The standing walk runs every 20 minutes and publishes whatever is on disk.

A standing-walk round runs these steps in order and stops at the first that says wait:

1. cache/walk-rest exists: rest. An empty file rests until removed; an ISO timestamp rests until then.
2. A shared walk lock, and any cache/build.lock held by a live process: stand down.
3. Source files touched in the last 10 minutes: stand down. A tree nobody has written to for ten minutes is between edits.
4. Sync claims, toilet reports and suggestions. Import, unless a crawl is out. Rebuild the streets, the temple rounds and the opening-hours lamps.
5. The same inputs failed the publish gate last round: stand down. cache/gate-fail.fp holds their hash.
6. Change gate: walk_fingerprint.py hashes what a build reads, drops the generated stamps the writers put inside their output, and compares with cache/last-published.json. No change and the last publish under 20 hours old: stand down.
7. Refresh weather, air and river gauges after that decision. Stamp BUILD_DATE.
8. Hand off to the build server, or build locally when cache/build-on-server is absent.

The change gate exists because the first evening of the standing walk republished a byte-identical site four times in ninety minutes, the weather having been refetched at the top of each round. The quiet window replaced a clean-tree rule that let the morning walk publish on 2 mornings of 14. The walk does not commit; cache/standing-walk.jsonl records each round.

Ledger (cache/standing-walk.jsonl), 2026-08-18 to 2026-10-11, counted at 14:24 Bangkok: 1,638 rounds. From 2026-09-20: 230 stood down, 150 handed to the server, 50 published from the Mac; failures were 64 in build.py, 47 in the import, 21 in the deploy, 17 at the publish gate and 2 in the hand-off.

เซิร์ฟเวอร์สร้าง · Build server

Since 2026-10-05 the standing walk hands each round to motdang-build, a server with two shared cores and 3.8 GB of RAM (importers/server_walk.sh). Removing cache/build-on-server builds on the Mac again.

Laptop side, importers/server_link.sh. rsync sends the source tree to a staging folder under a lock, leaving out docs/, docs.next/, docs.prev/, the build lock, the rest file and one private input folder. Every transfer has a 300-second idle timeout: on 2026-10-10 a stalled copy blocked a round and held a lock the next build needed. Then it writes a hand-off id and the UTC time the copy began. With cache/build-from-commit present it also sends the code of HEAD as a tar, so a half-edited file in the working tree is not built (on since 2026-10-10 19:36).

Server side, importers/server_walk.sh. A systemd timer starts it every five minutes. It takes the lock without waiting, looks for a hand-off id it has not tried, honours the same rest file, copies staging into the build tree under a second lock, overlays the committed code, and runs importers/publish_steps.sh: the same build, gates and deploy the Mac ran. One verdict line per attempt goes to walk.jsonl: published, failed, stood-down.

A hand-off is built once. From 2026-10-05 to 10-11 a failed hand-off was built a second time with the code of the last publish; that second build published 0 times in 14 and was removed. A layer that fails puts back its files from the live tree, and a page or site that fails a gate is held at its live version.

Settings: MD_JOBS=1 (one worker process on 3.8 GB), MD_BUILD_TIMEOUT=360m (a full build was 170 minutes on one Mac core on 2026-09-10; the limit was 180 minutes on the server until 2026-10-10), MD_PROFILE set so build_sampler.py records where time goes every two seconds.

การสร้างและ docs.next · The build and the docs.next swap

build.py writes the site into docs.next/ and renames it over docs/ at the end (_swap_into_place()). Two renames, each atomic on APFS; docs.prev/ is deleted after the swap. A build killed halfway leaves the previous whole docs/ where the publisher looks. This came after 2026-09-08, when a half-written docs/ was published (see Incidents).

The build holds a second copy of the site beside the first until the swap. On 2026-10-05 docs/ was 10.4 GB in 271,090 files, and a build died with Errno 28 after the disk reached 213 MB free. check_room() now measures the last docs/ with du and refuses to start when free space is under that size plus 5 GB. The swap itself needs about 8 GB spare for the minute both trees exist.

cache/build.lock holds the builder's pid. A second build refuses and names the holder; a lock whose process is gone is announced and taken. A build into a different output folder takes no lock, and the tests read MD_DOCS, which is how they run while another build holds docs/.

Same inputs should give the same bytes. PYTHONHASHSEED=0 is set in the publish steps, because an event with no uid took its calendar UID from a salted hash(). On 2026-10-10 a round three hours after the last changed 79,500 files, and 1,000 sampled place pages differed only in the weather and air line. That line is now filled in by md.js from data/nownear.json.

The sitemap dates each page by a fingerprint of its content: 97,113 pages changed in the 2026-10-10 build and 1,572 kept an earlier date.

ด่านตรวจ · Gates

importers/publish_steps.sh is sourced by both walks, so the Mac and the server pass the same gates. A hard gate stops the publish and writes failed to the ledger. An advisory gate prints a line and the publish goes on.

157 test files sit in tests/. test_search.py runs the shipped JavaScript matcher under node over the page's own pipeline. test_ground.py reads one column down a rendered map frame and expects it to cross the moat's water exactly twice.

การเผยแพร่ · Publish: R2 and the Worker

The site once deployed through Cloudflare Pages from a git host. Pages caps a project at 20,000 files and docs/ was 27,577 at the move. R2 has no object cap, and the 116 MB basemap archive lives in the same bucket.

publish/deploy.py wraps rclone sync, which makes the bucket match docs/ and deletes what docs/ lacks. It runs as a dry run unless given --yes, and takes the build lock so it cannot sync a tree a build is writing. publish/safety.py carries the checks that came out of the 2026-09-08 outage: a required-path set, per-subtree floors, a deletion preview built by listing both sides, a ceiling of 2,000 live objects deleted in one run, a list of prefixes never pruned (vendor/, glyphs/, api/), and a smoke test that asks the live origin a list of questions after the upload. --only and --files publish part of the tree. A live file uploaded after the hand-off began is kept, since the build could not know it.

publish/worker.js serves the bucket. .html is served directly and the extensionless path resolves to the same object, so the common path has no redirect; some folder indexes still redirect. Range requests pass through to R2, which is all a PMTiles archive needs, and tiles carry max-age=31536000, immutable. Every response carries Access-Control-Allow-Origin: *. /api/v1/* is answered from a baked index.json, checked against the bucket by an etag request every 60 seconds (OpenAPI 3.1.0, 8 paths on 2026-10-10).

The Worker writes an Analytics Engine row for a page request: referrer host, the landing path with the query dropped, and a category derived from the User-Agent. Blob position is the schema, and test_edge_count.mjs fails when the order drifts from the collector it shares a dataset with.

The claims Worker (worker/worker.js, KV) takes claims, toilet reports, thumbs and notes (/feedback), suggestions (/suggest), listing forms and tester reports. Rate limits per address per hour: 12 suggestions, 120 feedback events, 12 notes. A suggestion queue and the notes have no public GET. Importers read them with an authenticated call into _incoming/, which is gitignored.

Backups are git bundle files of every repo, written by importers/offsite_backup.py to a separate bucket, the last three kept per repo. A backup written into the site bucket is deleted by the next deploy, which is how the first one was lost.

แผนที่พื้นฐาน · Map basemap

Maps are MapLibre GL JS v5.24.0 over PMTiles (v3), a single-file tile archive read with HTTP range requests, with both the library and the archive served from the site's own origin. The archives came from a Protomaps planet build dated 2026-08-11 through pmtiles extract.

map_shell.py is the one module that configures a map. Each map is drawn first as a Python-made SVG, which is what prints, what shows with scripting off and what shows while tiles arrive; MapLibre mounts over it in the same box. data/basemap.json with no url switches the tile layer off and leaves the SVG. The OpenStreetMap credit is configured visible on every map surface, as the ODbL asks.

map_ground.py reads the same archive in stdlib Python (PMTiles v3 and MVT) to paint ground under share cards, plan GIFs and posters that Pillow draws. It returns geometry in longitude and latitude for the caller's own projection, draws roads in metres with a pixel floor, and treats polygon holes as holes, so the moat stays a ring. The credit is drawn into the pixels because a GIF travels without its page.

ค้นหา · Search

The search box matches on the device. assets/searchcore.js is generated from a sibling search-core repo, which also holds a Python twin of the same rules; a parity script runs a few thousand queries through both. Rules: normalization (Thai marks are whitelisted, since vowels and tone marks are Unicode category Mn); bounded edit distance with no slack on very short tokens; a phonetic key for Thai homophone spellings and for Latin romanization variants; dictionary segmentation of Thai chunks, with no character n-grams, applied to the documents too; a thesaurus mined from the site's own shelf vocabularies and a Thai root-word dictionary (4,113 groups in data/search_thesaurus.json on 2026-10-11); and intent lifting, so "cheap dentist open now" becomes a noun and three constraints. Terms loosen by steps, and documents matching only some terms show only when none matches all.

The index is shipped to the browser. On 2026-09-08 it was 25,582 rows and 1,563,679 bytes gzipped; indexsplit_layer.py trimmed coordinates and moved some fields to a second file aligned by position, 1,238,000 bytes. After the Overture fold the catalogue is 89,082 rows, which the module's note puts well over four megabytes before an answer; that figure has not been re-measured here. The public API at /api/v1/places answered in about 120 ms and the search box did not call it, per the module's 2026-09-08 note; moving to it trades offline-first for size, and that trade is open.

/find is a separate search over every page of every site on the roster: SQLite FTS5 in D1, loaded by fleetsearch/load.py from a crawl of each site's build. A reload takes 6 to 7 minutes, during which D1 does not answer queries and /find returns errors in bursts.

เหตุขัดข้อง · Incidents

ค้างอยู่ · Open items

• The /find reload crawls a local docs/. Builds now run on the server.
• D1 coordinates: 3,705 of an intended 76,847 rows carried a pin on 2026-09-21, after a D1 import returned 502 and locked the database for 25 minutes. Distance works where a row has one.
• The index shipped to the browser grows with the catalogue. The API is the alternative, and it costs offline-first.