Synthetic scouting labels: records and field-disjoint splits

Overview

# Synthetic scouting labels: records and field-disjoint splits

## Goal, files and requirements

Test record exchange, explicit boolean parsing and grouped splitting. Download `scouting-observations.csv`, `.tsv`, `.json`, `data-dictionary-v2.json` and `license.txt` to one folder. Save the script as `check_scouting.py` and run `python3 check_scouting.py` using only the Python 3 standard library. There are 24 invented rows, no photos, real labels, pretrained model or benchmark.

## Field meaning and procedure

`observation_id` is unique; `field_id` joins the [sandbox registry](/hub/agridigi/sandbox-field-boundaries). Crop is `demo-crop`; stage is `unspecified`. Per-field index i=0..11 creates yellowing at i%3==0, visible damage at i%4==0 and uncertain quality at i%5==0; timestamps are six hours apart from the arbitrary 2026-01-01 UTC origin. `review_required` is the disclosed toy OR rule, not validated triage. CSV/TSV 'false' must parse as False, not as a truthy nonempty string.
Compare whole records by ID, check the toy rule and exact counts, then assert one field per split. Field A's 12 rows are train; field B's 12 rows are test.

```python
import csv, json
from collections import Counter, defaultdict
from pathlib import Path

root = Path(".")
boolean_fields = ("leaf_yellowing", "visible_leaf_damage", "review_required", "is_synthetic")
def boolean(value):
    assert value in ("true", "false"), value
    return value == "true"
def load_table(filename, delimiter):
    with (root / filename).open(newline="", encoding="utf-8") as f:
        return [{**r, **{k: boolean(r[k]) for k in boolean_fields}}
                for r in csv.DictReader(f, delimiter=delimiter)]
def keyed(rows):
    result = {r["observation_id"]: r for r in rows}
    assert len(rows) == len(result) == 24
    return result

rows = load_table("scouting-observations.csv", ",")
tsv = load_table("scouting-observations.tsv", "\t")
other = json.loads((root / "scouting-observations.json").read_text(encoding="utf-8"))
assert keyed(rows) == keyed(tsv) == keyed(other)
groups = defaultdict(set)
for r in rows:
    groups[r["field_id"]].add(r["split"])
    assert r["is_synthetic"] is True
    assert r["crop"] == "demo-crop" and r["growth_stage"] == "unspecified"
    assert r["observation_quality"] in ("clear", "uncertain")
    assert r["review_required"] == (r["leaf_yellowing"] or r["visible_leaf_damage"] or r["observation_quality"] == "uncertain")
assert dict(groups) == {"sandbox-field-a": {"train"}, "sandbox-field-b": {"test"}}
assert Counter(r["split"] for r in rows) == {"train": 12, "test": 12}
assert sum(r["review_required"] for r in rows) == 16
assert sum(r["leaf_yellowing"] for r in rows) == 8
assert sum(r["visible_leaf_damage"] for r in rows) == 6
assert sum(r["observation_quality"] == "uncertain" for r in rows) == 6
train_fields = {r["field_id"] for r in rows if r["split"] == "train"}
test_fields = {r["field_id"] for r in rows if r["split"] == "test"}
assert train_fields.isdisjoint(test_fields)
assert json.loads((root / "data-dictionary-v2.json").read_text())["row_count"] == 24
print("rows=24; train=12; test=12; review=16; yellowing=8; damage=6; uncertain=6; overlap=0")
```

## Expected output and worked example

`rows=24; train=12; test=12; review=16; yellowing=8; damage=6; uncertain=6; overlap=0`
`scout-a-02` has false symptom flags, clear quality and `review_required=false`; treating the text 'false' as true breaks equivalence. `scout-a-01` exercises the review path. Both belong only to train. Two invented groups do not establish independence or real-world generalization.

## Next action, privacy, provenance and license

Try the cross-dataset join and [14-day scouting plan](/hub/agridigi/field-scouting-14-day). For any model experiment, exclude `review_required` from inputs; consider field IDs and symptom-rule inputs as leakage/shortcut risks. Real farm/time/population splits require a separately justified design. AgriDigi generated the records deterministically; the dictionary states every rule. No personal data, photographs or diagnoses are supplied. Original fixture/text is [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/): credit AgriDigi, link source/license and mark changes. No accuracy, treatment or yield claim can follow.

Dataset

Rows: 24

Columns: observation_id, field_id, timestamp_utc, crop, growth_stage, leaf_yellowing, visible_leaf_damage, observation_quality, review_required, split, is_synthetic

• license.txt · text/plain · 558 bytes
• scouting-observations.csv · text/csv · 2769 bytes
• scouting-observations.tsv · text/tab-separated-values · 2769 bytes
• scouting-observations.json · application/json · 8745 bytes
• data-dictionary-v2.json · application/json · 12878 bytes
• readme-v2-en.txt · text/plain · 4279 bytes
• readme-v2-tr.txt · text/plain · 4429 bytes
• readme-v2-de.txt · text/plain · 4103 bytes
• readme-v2-es.txt · text/plain · 4327 bytes
• readme-v2-fr.txt · text/plain · 4347 bytes
• readme-v2-it.txt · text/plain · 4157 bytes
• readme-v2-az.txt · text/plain · 3996 bytes
• readme-v2-ru.txt · text/plain · 4884 bytes
• readme-v2-sr.txt · text/plain · 4010 bytes
• readme-v2-ar.txt · text/plain · 4749 bytes
• validate-v2.txt · text/plain · 2028 bytes
• review-queue-v2.csv · text/csv · 2279 bytes
• split-audit-v2.csv · text/csv · 153 bytes
• expected-checks-v2.json · application/json · 267 bytes