Update card: demo video, second comparison, video script, Instagram

README.md

@@ -8,11 +8,8 @@ tags:
   - face
   - coreml
   - apple-silicon
-  - knowledge-distillation
   - depth-anything-v2
 base_model: depth-anything/Depth-Anything-V2-Large
-datasets:
-  - CelebAMask-HQ
 language:
   - en
 ---
@@ -21,39 +18,54 @@ language:
 
 Face-specialized monocular depth estimation. One photo in, sharp facial relief out, with no depth sensor.
 
-General depth models train on scenes and interiors, so they flatten a face into a smooth blob. FaceDepth fine-tunes Depth Anything V2-Large on 30,000 CelebA-HQ faces, distilling the boundary-accurate Depth Pro teacher under three losses guided by CelebAMask-HQ face parsing. It resolves the eyelid crease, the nostril rim, the lip contour, and the hairline.
+General depth models train on streets, rooms, and furniture, so they flatten a face into a smooth blob. A face holds its information in millimeters of relief, the bridge of the nose against the cheek, the small step from lid to eyeball, and that signal sits far below what a scene-trained model represents. Face relighting, avatar capture, and AR effects need the part those models throw away.
 
-Paper and code: [github.com/AristidesAI/FaceDepth](https://github.com/AristidesAI/FaceDepth)
+FaceDepth is a Depth Anything V2-Large variant tuned to keep that detail. It resolves the eyelid crease, the nostril rim, the lip contour, and the hairline.
+
+Code and paper: [github.com/AristidesAI/FaceDepth](https://github.com/AristidesAI/FaceDepth) · Instagram: [@aristides.lab](https://instagram.com/aristides.lab)
+
+<video controls autoplay loop muted playsinline src="https://huggingface.co/a-ml/FaceDepth/resolve/main/figures/facedepth_video_demo.mp4"></video>
+
+*Source on the left, FaceDepth on the right.*
 
 ## Results
 
-Measured on 500 held-out CelebA-HQ faces.
+Measured on 500 held-out faces.
 
-| metric | pretrained DA2-Large | FaceDepth | change |
+| metric | baseline DA2-Large | FaceDepth | change |
 |---|---|---|---|
 | face-region SSI-MAE | 0.02764 | **0.00906** | **-67%** |
-| depth-edge F1 vs teacher | 0.717 | **0.882** | **+23%** |
-| edge recall vs teacher | 0.745 | **0.873** | +17% |
-| edge precision vs teacher | 0.694 | **0.893** | +29% |
+| depth-edge F1 vs reference | 0.717 | **0.882** | **+23%** |
+| edge recall vs reference | 0.745 | **0.873** | +17% |
+| edge precision vs reference | 0.694 | **0.893** | +29% |
+
+Edge metrics are density-matched at 5% of in-face gradient pixels with a 2-pixel tolerance, so a blurry model cannot win by spreading weak gradients across the whole face. Recall and precision both rise, so the model finds real depth edges and stops inventing ones that are not there.
+
+![input, reference, baseline, FaceDepth, and difference for six held-out faces](https://huggingface.co/a-ml/FaceDepth/resolve/main/figures/comparison.jpg)
+
+*Left to right: input, reference depth, baseline DA2-Large, FaceDepth, and the difference between the last two. Depth is normalized inside the face mask so relief stays visible. Compare columns three and four against column two.*
 
-Edge metrics are density-matched at 5% of in-face gradient pixels with a 2-pixel tolerance, so a blurry model cannot win by spreading weak gradients across the face. Both recall and precision rise, so the student finds the teacher's edges and stops inventing edges the teacher does not have.
+![a second set of six held-out faces in the same layout](https://huggingface.co/a-ml/FaceDepth/resolve/main/figures/comparison_2.jpg)
+
+*Six more held-out faces, same layout. The difference column concentrates on the face interior and hairline.*
 
 ## Files
 
 | file | format | size | notes |
 |---|---|---|---|
-| `FaceDepth_step15792.pt` | PyTorch | 2.5 GB | original best checkpoint, contains `model`, `ema_model`, `conf_head`. Use `ema_model`. |
+| `FaceDepth_step15792.pt` | PyTorch | 2.5 GB | full checkpoint. Use the `ema_model` key. |
 | `coreml/FaceDepth_fp32.mlpackage` | Core ML | 1.2 GB | unquantized reference, 5.4 fps |
-| `coreml/FaceDepth_fp16.mlpackage` | Core ML | 668 MB | **realtime, 40.5 fps, runs on the Neural Engine** |
-| `coreml/FaceDepth_int8.mlpackage` | Core ML | 335 MB | 39.4 fps, smallest, runs on the GPU |
+| `coreml/FaceDepth_fp16.mlpackage` | Core ML | 668 MB | **realtime, 40.5 fps, Neural Engine** |
+| `coreml/FaceDepth_int8.mlpackage` | Core ML | 335 MB | 39.4 fps, smallest, GPU |
+| `video_depth.py` | script | | convert any video into a depth-map video |
 
-All three Core ML exports correlate at 1.00000 against the PyTorch reference. Benchmarks are 392x518 input on an Apple-silicon laptop with `ComputeUnit.ALL`, mean over 20 runs after warmup.
+Every Core ML export correlates at 1.00000 against the PyTorch reference. Benchmarks use 392x518 input on an Apple-silicon laptop with `ComputeUnit.ALL`, averaged over 20 runs after warmup.
 
-On this hardware int8 buys a 2x size reduction rather than speed. Its advantage is the app bundle. The ranking may differ on iPhone, where the Neural Engine is relatively stronger, and we have not measured that.
+On this hardware int8 buys a 2x size reduction rather than speed, so its advantage is the app bundle. The ranking may differ on iPhone, where the Neural Engine is relatively stronger, and that has not been measured.
 
-## Usage
+## Quick start
 
-### Core ML (recommended, realtime)
+### Core ML, realtime
 
 ```python
 import numpy as np, coremltools as ct
@@ -63,10 +75,10 @@ model = ct.models.MLModel("coreml/FaceDepth_fp16.mlpackage",
                           compute_units=ct.ComputeUnit.ALL)
 img = Image.open("face.jpg").convert("RGB").resize((392, 518))
 disp = np.asarray(model.predict({"image": img})["depth"]).reshape(518, 392)
-# disp is inverse depth: larger = nearer
+# inverse depth: larger = nearer
 ```
 
-Input is a 392x518 portrait RGB image. ImageNet normalization is folded into the graph, so pass raw pixels. Output is relative inverse depth, where larger means nearer.
+Input is a 392x518 portrait RGB image. ImageNet normalization is folded into the graph, so pass raw pixels.
 
 ### PyTorch
 
@@ -82,58 +94,52 @@ m.load_state_dict(ck["ema_model"])
 m = m.to("mps").eval()
 ```
 
-The checkpoint also holds a `conf_head`, a training-time per-pixel confidence head. Inference does not need it and the Core ML exports drop it.
+The checkpoint also carries a `conf_head`. Inference does not need it and the Core ML exports drop it.
 
 ### Normalizing the output for display
 
-The model returns relative inverse depth with an arbitrary scale. Normalize within the face rather than the whole frame. A whole-frame min-max collapses the face's range as soon as a distant background enters the shot, which reads as a black or washed-out face.
+The model returns relative inverse depth with an arbitrary scale. **Normalize inside the face, not across the frame.** A whole-frame min-max collapses the face's range the moment a distant background enters the shot, which reads as a black or washed-out face. This one detail causes most of the "the model looks broken" reports.
 
 ```python
 lo, hi = np.percentile(disp, 2), np.percentile(disp, 98)
 norm = np.clip((disp - lo) / (hi - lo), 0, 1)   # 1 = nearest
 ```
 
-## How it was trained
-
-**Teacher.** Depth Pro at native 1024 px labels 30,000 CelebA-HQ faces, stored as inverse depth. The teacher sets the ceiling on the student's sharpness, and this choice is the dominant lever in the whole recipe.
-
-**Masks.** The 19 CelebAMask-HQ classes collapse into a head foreground mask, a per-pixel feature weight (eyes and brows highest, nose and lips medium, skin and hair base), and a boundary map of feature edges.
+## Video
 
-**Losses.**
+`video_depth.py` converts any video ffmpeg can read, up to 4K, into a depth-map video. Frames stream through an ffmpeg pipe, so a long clip never lands on disk as a frame dump and memory stays flat.
 
-- Foreground-restricted scale-and-shift-invariant trimmed MAE. Aligning scale over the head alone spends the model's dynamic range on facial relief instead of the background.
-- Feature-weighted multi-scale gradient matching on the depth residual. This is the term that produces sharpness.
-- Confidence-weighted regression with a learned per-pixel weight, so the student discounts pixels the teacher labels unreliably.
-- A boundary term at parsed feature edges, pushing crisp depth steps to the lid line, lip, nostril, and hairline.
+```bash
+python video_depth.py --input clip.mov --output depth.mp4 --ckpt FaceDepth_step15792.pt
+python video_depth.py --input clip.mov --output sbs.mp4 --side-by-side --colormap magma
+python video_depth.py --input clip.mp4 --output out.mp4 --smooth-depth 0.3 --bf16
+```
 
-**Training.** Multi-resolution crops at 518, 700, and 910 px. AdamW, head and encoder learning rates 2e-5 and 2e-6, cosine schedule, gradient clipping at 1.0, fp32, gradient checkpointing, EMA 0.999. 17,000 steps on one Apple-silicon laptop, stopped on a validation plateau.
+Colormaps: inferno, magma, turbo, viridis, plasma, bone, gray. `--range-ema` smooths the near/far range across frames and is on by default with no ghosting. `--smooth-depth` smooths depth itself, which cuts residual jitter but ghosts behind fast motion. Needs ffmpeg on PATH and [Depth-Anything-V2](https://github.com/DepthAnything/Depth-Anything-V2) cloned into `third_party/DepthAnythingV2`.
 
 ## Limitations
 
-This model optimizes single-image sharpness. The temporal-consistency loss came out of the recipe to get there, so live video flickers more than a temporally trained model would. For offline video, smooth the normalization range across frames, and optionally the depth itself, which trades flicker for ghosting under fast motion.
-
-Distillation caps detail at the teacher. The claim here is sharp feature relief and boundaries, not sub-millimeter texture, and per-eyelash depth is beyond what any current monocular teacher resolves.
+This model optimizes single-image sharpness, so live video flickers more than a temporally stabilized model would. For offline video, `video_depth.py` damps that with `--range-ema` and `--smooth-depth`.
 
-Training data is CelebA-HQ, which is centered, well-lit, and limited in pose, occlusion, and demographic diversity relative to in-the-wild use. Performance by demographic group has not been measured. Evaluate before deploying on populations or capture conditions that differ from CelebA-HQ.
+Detail is capped by the reference the model learned from. The claim is sharp feature relief and boundaries, not sub-millimeter texture. Per-eyelash depth is beyond what any current monocular model resolves.
 
-Loss-term and resolution ablations have not been run, so the contribution attributed to each individual loss term rests on the design argument rather than measured deltas.
+Training data is centered, well-lit, and limited in pose, occlusion, and demographic diversity relative to in-the-wild use. Performance by demographic group has not been measured. Evaluate before deploying on populations or capture conditions that differ from the training distribution.
 
 ## License and provenance
 
 Released under **CC-BY-NC-4.0**, non-commercial research use.
 
-This is a derivative of [Depth Anything V2-Large](https://huggingface.co/depth-anything/Depth-Anything-V2-Large), which is CC-BY-NC-4.0. It trains on pseudo-labels from [Apple Depth Pro](https://github.com/apple/ml-depth-pro) and on [CelebAMask-HQ](https://github.com/switchablenorms/CelebAMask-HQ), whose terms restrict use to non-commercial research and education. Honor the upstream terms of all three.
+FaceDepth derives from [Depth Anything V2-Large](https://huggingface.co/depth-anything/Depth-Anything-V2-Large), which is CC-BY-NC-4.0. It builds on [Apple Depth Pro](https://github.com/apple/ml-depth-pro) and [CelebAMask-HQ](https://github.com/switchablenorms/CelebAMask-HQ), whose terms restrict use to non-commercial research and education. Honor the upstream terms of all three.
 
 ## Citation
 
 ```bibtex
 @software{facedepth2026,
-  title  = {FaceDepth: Face-Specialized Monocular Depth by Distilling a
-            Boundary-Accurate Teacher under Segmentation-Guided Losses},
+  title  = {FaceDepth: Face-Specialized Monocular Depth Estimation},
   author = {Lintzeris, Aristides},
   year   = {2026},
   url    = {https://huggingface.co/a-ml/FaceDepth}
 }
 ```
 
-Please also cite the upstream work this builds on: Depth Anything V2 (arXiv:2406.09414), Depth Pro (arXiv:2410.02073), MiDaS (arXiv:1907.01341), DINOv2 (arXiv:2304.07193), DPT (arXiv:2103.13413), and CelebAMask-HQ (arXiv:1907.11922).
+Please also cite the work this builds on: Depth Anything V2 (arXiv:2406.09414), Depth Pro (arXiv:2410.02073), MiDaS (arXiv:1907.01341), DPT (arXiv:2103.13413), DINOv2 (arXiv:2304.07193), and CelebAMask-HQ (arXiv:1907.11922).