5 states, 2 working filters: scraping US childcare license registries

Five states, one query language, and an "active licenses only" checkbox that only actually filters two of them. That's the trap in scraping US childcare-license open-data registries: Socrata SODA makes every state's API look identical, but "active" is defined - or not defined at all - differently in every dataset. Quick answer New York, Connecticut, Colorado, Delaware, and Texas all publish their childcare-facility registries through Socrata, and all five accept the same $where query syntax. But only NY and CT ship a server-side status filter this Actor can apply. Colorado and Delaware have no status column in the dataset at all - there's nothing to filter on. Texas does have a status column (operation_status ), it's just not wired into the active-only filter, so toggling activeOnly doesn't touch Texas rows either way. Treating "active only" as a global switch that behaves the same everywhere will silently hand you closed and revoked facilities in three of the five states while you believe you filtered them out. STATE_CONFIGS: dict[str, StateConfig] = { "NY": StateConfig(..., col_status="facility_status", active_where="facility_status='Active'"), "CT": StateConfig(..., col_status="status", active_where="status='ACTIVE'"), "CO": StateConfig(..., col_status=None), # no status column to filter on "DE": StateConfig(..., col_status=None), # no status column to filter on "TX": StateConfig(..., col_status="operation_status"), # status exists, filter isn't wired } Why does "active only" do nothing in three states? Because the filter is applied per-state, not globally, and only two states have both a status column and a configured $where fragment for it: async def _fetch_page(...): params = {"$limit": str(page_limit), "$offset": str(offset), "$order": config.order_key} if active_only and config.active_where: params["$where"] = config.active_where return await _get_with_retry(session, config, params, offset) active_only and config.active_where is the whole guard. If a state's config carries no active_where , the toggle is a no-op for that state - by design, because forcing a fake filter on data that doesn't support one is worse than being honest that it isn't filterable. Texas's registry does return a status value per row (so you can filter it downstream yourself); Colorado and Delaware's registries don't expose status at all. Why is "capacity" typed as int | str | None instead of just int ? Because licensed capacity isn't reliably numeric across five independently-maintained state datasets. A generic Socrata consumer that assumes every "capacity" column parses cleanly to int will crash the run the first time a state ships something like "N/A" or a range string in that field. The parser tries the conversion and falls back to the raw string rather than failing the row: def _parse_capacity(raw: dict[str, Any], col: str | None) -> int | str | None: if col is None: return None val = raw.get(col) if val is None: return None try: return int(val) except (ValueError, TypeError): return str(val) That one try/except is the difference between a run that finishes with 5,000 clean rows and one that dies on row 4,812 because Delaware's capacity column had a value nobody anticipated. Why does only Texas have email and website columns? Because Texas's operation registry happens to publish email_address and website_address fields - the other four states' schemas simply don't collect that data in the open dataset. The row model carries email and website as nullable fields specifically because they're populated for one state out of five and None everywhere else; there's no cross-state equivalent to fall back to. Why is New York's street address built from two separate fields? Because NY's dataset splits it into street_number and street_name instead of shipping one address column, unlike every other supported state. Address assembly is state-specific for exactly the same reason licensee-name assembly is state-specific in our cosmetology-registry Actor: each state government designed its own schema independently, decades apart, with no shared spec. What happens when a facility row has no usable name? It gets dropped rather than emitted as a blank row. Every state's registry has a facility-name column, but a handful of records across five independently-maintained datasets inevitably ship a null or empty value - a placeholder entry, a data-entry gap, a record mid-correction. Rather than push a row your CRM can't use, the builder checks for a usable name first and returns None if there isn't one: def _build_row(raw: dict[str, Any], config: StateConfig, scraped_at: str) -> ResultRow | None: facility_name = _get(raw, config.col_facility_name) if not facility_name: return None ... One bad record in a 50,000-row Colorado page doesn't fail the run either - pagination keeps walking $offset forward on 408 / 429 / 503 , and a page that comes back short simply ends that state's scrape cleanly rather than retrying forever. Is scraping state childcare registries legal? These are official state open-data portals, published for public and commercial reuse with no login wall. We still treat every endpoint as a target that can throttle: curl-cffi impersonates real Chrome, Firefox, and Safari TLS sessions, 408 / 429 / 503 responses get retried with exponential backoff up to 5 attempts honoring Retry-After , and residential proxy rotation is available for higher-volume runs. FAQ Which states does this cover? New York, Connecticut, Colorado, Delaware, and Texas. Does activeOnly work the same everywhere? No - it only filters NY and CT server-side. CO and DE have no status column at all; Texas has a status value in every row but no active-only filter applied to it, so you'd filter on the status field yourself downstream if you need Texas-only active facilities. Why would a row be missing a licensee name? Some states record the operator/governing body under a different field than the facility name, and a handful of records simply don't populate it in the source registry - the Actor doesn't fabricate a value. What does one dataset row represent? One licensed childcare facility from one state's registry, normalized to a single schema regardless of source column names. Packaged and ready to run: Daycare & Childcare License Leads - pick your states, get one normalized row per facility: name, licensee, license number, type, capacity, address, phone, status, and (Texas only) email and website. $0.20 warm-up plus $0.004 per result row (about $4 for 1,000 leads). We run the gauntlet so your lead list lands clean. 😈 Top comments (0)