# Phase 1 — Dataset Reorganization

**Blocked by**: Nothing
**Blocks**: Phase 2 (training)
**Est. time**: 2-3 days
**Machine**: Strix Halo (fast I/O, 128GB RAM for in-memory processing)

## Objective

Reorganize the flat directory structure (`data/dataset/plant-disease-name/`) into a proper hierarchical layout (`data/organized/species/disease/`) with train/val splits and metadata files.

## Current State

- `data/dataset/` — 11,499 flat directories, each named `{plant}-{disease}`
- Files mixed: `.jpg`, `.jpeg`, `.png`, `.webp` per directory
- Total: ~1.47M images, 64-244 images per class (well-balanced)
- **Total size: ~450 GB**
- **SSD available**: 8TB NVMe (7,300 MB/s read, 6,300 MB/s write — PCIe 5.0)

## Deliverables

```
data/
├── organized/
│   ├── train/                          # 85% of images
│   │   ├── {species_1}/
│   │   │   ├── healthy/
│   │   │   ├── {disease_a}/
│   │   │   └── {disease_b}/
│   │   ├── {species_2}/
│   │   └── ...
│   ├── val/                            # 15% of images
│   │   └── ... (mirrors train structure)
│   ├── species_index.json              # Maps species → [disease IDs]
│   ├── class_hierarchy.json            # Full mapping + metadata
│   └── dataset_stats.json              # Counts per class, splits
```

## Steps

### 1.1 Parse directory names → (species, disease) pairs

**Problem**: Directory names like `acorn-squash-powdery-mildew` use an inconsistent separator (hyphen). Need to reliably split plant name from disease name.

**Approach**: Use `src/data/diseases.json` as ground truth. Try matching each directory name against known disease ID suffixes (sorted longest-first). The remainder is the plant name.

**Fallback**: For unmatched dirs, build a plant suffix list from `src/data/plants.json` and try prefix matching. Log any truly unmatched dirs for manual review.

**Script**: `scripts/organize-dataset.py`

```python
# Pseudocode for the matching algorithm:
disease_ids = sorted([d["id"] for d in diseases], key=len, reverse=True)
plant_names = [p["id"] for p in plants]  # or extract from dir prefixes

for dir_name in dataset_dirs:
    matched_disease = next(d for d in disease_ids if dir_name.endswith(d))
    plant = dir_name[:-(len(matched_disease)+1)]  # +1 for hyphen
    hierarchy[plant].append(matched_disease)
```

### 1.2 Split into train/val (85/15)

Use stratified splitting per class to preserve class distribution.

- For each disease-plant class, randomly assign 85% to train, 15% to val
- Copy files (or symlink) to new directory structure
- Verify no data leakage (same image in both splits)

### 1.3 Build metadata files

```json
// species_index.json
{
  "tomato": ["healthy", "early-blight", "late-blight", "bacterial-spot", ...],
  "acorn-squash": ["healthy", "powdery-mildew", "downy-mildew", ...],
  ...
}

// dataset_stats.json
{
  "total_images": 1465818,
  "total_species": 320,
  "total_classes": 11499,
  "images_per_class": { "min": 64, "max": 244, "mean": 127 },
  "train_images": 1245945,
  "val_images": 219873,
  "species_disease_counts": {
    "tomato": { "early-blight": 156, "late-blight": 142, ... }
  }
}
```

### 1.4 Data quality checks

### 1.4 Image normalization & compression (before splitting)

**450GB is unnecessarily large for 224px training.** Many source images are high-resolution (e.g., 4000×3000 from phone cameras), but the model only sees 224×224 crops. Resizing to a reasonable max dimension BEFORE training saves massive I/O and enables faster epochs.

**Strategy**: Resize all images to **max dimension of 512px** (preserving aspect ratio), convert to **JPEG quality 90**.

| Approach                        | Est. Size     | Pros                                               | Cons                               |
| ------------------------------- | ------------- | -------------------------------------------------- | ---------------------------------- |
| **Keep originals**              | 450 GB        | No quality loss                                    | Slow loading, huge storage         |
| **Resize 1024px max, JPEG 90**  | ~120 GB       | Good for future higher-res models                  | Still somewhat large               |
| **Resize 512px max, JPEG 90 ✓** | **~60-80 GB** | **Fast loading, enough detail for 224px training** | Can't go back to full res          |
| **Resize 256px max, JPEG 95**   | ~30 GB        | Fastest loading                                    | Too small if retrain at higher res |

**Recommendation**: Resize to 512px max, JPEG q90. This:

- Reduces storage from 450GB → ~70GB (fits in RAM for caching)
- Preserves enough detail for 224×224 RandomResizedCrop augmentation
- JPEG is hardware-accelerated (libjpeg-turbo) — fastest decode path
- Single format (no more .png/.webp mixed loading)

```python
# resize_and_convert.py
from PIL import Image
import os
from joblib import Parallel, delayed

MAX_SIZE = 512
QUALITY = 90

def process_image(src_path, dst_path):
    img = Image.open(src_path)
    # Resize so max dimension = MAX_SIZE, preserving aspect ratio
    w, h = img.size
    if max(w, h) > MAX_SIZE:
        ratio = MAX_SIZE / max(w, h)
        img = img.resize((int(w * ratio), int(h * ratio)), Image.LANCZOS)
    # Convert to RGB (handles RGBA PNGs)
    if img.mode != 'RGB':
        img = img.convert('RGB')
    # Save as JPEG
    os.makedirs(os.path.dirname(dst_path), exist_ok=True)
    img.save(dst_path, 'JPEG', quality=QUALITY, optimize=True)

# Run in parallel (Strix Halo has many cores)
Parallel(n_jobs=16)(
    delayed(process_image)(src, dst)
    for src, dst in image_pairs
)
```

**Time estimate on Strix Halo**: ~2-3 hours to resize + convert 1.47M images with 16 parallel workers. Each image takes ~5-10ms with PIL+LANCZOS.

### 1.5 Data quality checks

- **Label noise**: Run confidence learning (CleanLab) on a sample to estimate mislabel rate. Web-scraped datasets typically have 8-15% label noise.
- **Duplicate detection**: Check for near-duplicate images (perceptual hashing + Hamming distance) within each class.
- **Format consistency**: Ensure all images decode successfully; remove corrupted files.
- **Background bias**: Verify that no single background dominates a class (subset and eyeball a random grid per class).

## Edge Cases & Gotchas

- **Multi-word plant names**: "acorn-squash", "fiddle-leaf-fig", "chili-pepper" — the disease suffix must match the end of the string, not a substring in the plant name. Sorting disease IDs by length (longest first) handles this.
- **Disease-less "healthy" dirs**: Need to ensure "healthy" is in the disease list as a valid class (index 0 in current model). Some dirs may be `{plant}-healthy`.
- **Cross-platform path length**: Some species+disease combos produce long paths. Use relative symlinks or shorten names if needed on Windows.
- **Original files preserved**: The existing `data/dataset/` structure stays untouched; `data/organized/` is a copy.

## Verification

- [ ] `data/organized/train/` has same total image count as original (minus val split)
- [ ] Every class has at least 50 training images
- [ ] `species_index.json` covers all 11,499 classes
- [ ] No files in both train/ and val/ (no overlap)
- [ ] All images readable (no corrupted files)
- [ ] Train/val split ratios consistent across all classes (±2%)