On-Device Computer Vision in React Native: Auto-Aligning Progress Photos with MediaPipe and Expo

A person can take thirty photos of the same face and still produce a timelapse that looks like camera shake. On Monday they stand closer to the mirror. On Tuesday they tilt their phone. By Friday they frame their eyes near the top of the image. For fitness, skincare, and haircut timelines, viewers need to see the change in the person without chasing their face around the screen. In Phases, the useful visual reference is the eye line: keep both eyes at the same output coordinates across the sequence. This article develops the React Native and Expo architecture for that workflow: detect landmarks on the device, compute a two-point transform, retain originals in local storage, and render a reel without a rendering server. Implementation scope: the public Phases site currently describes a browser app that users can add to their home screen. It also describes consent-based analytics. The native Expo architecture and zero-telemetry policy below describe the reference design for this article, rather than a verified description of that deployed browser build. The project link is github.com/Ishannaik/phases; the implementation examples below stand on their own. Separate the photograph from its alignment A capture guide helps the user repeat their pose, but it cannot guarantee matching camera distance or roll. I would keep the original photo and store its alignment metadata beside it. That lets the user change the reel's framing without reshooting or accumulating JPEG compression from repeated edits. flowchart TD A[Camera capture] --> B[Normalize orientation and mirroring] B --> C[On-device facial landmark detection] C --> D{Usable eye anchors?} D -->|Yes| E[Compute affine matrix] D -->|No| F[Ask user to retake or mark anchors] B --> G[Original photo in app filesystem] E --> H[SQLite metadata and transform] G --> I[Local Canvas or WebGL composite] H --> I I --> J[On-device video encoder] J --> K[Timelapse reel on device] G -. Optional user-enabled backup .-> L[Cloud backup] The engineer should treat capture, detection, and rendering as separate steps. After a successful detection, the renderer needs the photo URI and six transform coefficients. It does not need another inference pass each time the user plays the timeline. For a native app identified as io.phases.app , use its app-private filesystem for photos and SQLite for timestamps, dimensions, eye coordinates, and alignment settings. io.phases.app is an application identifier, not a portable filesystem path. Android and iOS choose the sandbox's physical location. With Expo, expo-file-system provides document storage, and expo-sqlite provides a persistent database. Store large image files in the filesystem rather than putting base64 photo strings into database rows. Choose eye anchors your detector can supply With MediaPipe, you can derive eye anchors from a face mesh. The older Face Mesh API offers iris refinement through refine_landmarks ; the current Face Landmarker task provides a denser landmark result. Google documents 478 landmarks for its current model bundle. Google's canonical face model UV visualization. This repository asset illustrates landmark topology, not a Phases screenshot or a detector result from this article. ML Kit can provide eye landmarks through its face detection API, but an eye landmark is not equivalent to a measured pupil center. If you choose ML Kit, name those anchors eye centers. If you need iris-center estimates, choose a detector configuration that supplies iris landmarks and document your index mapping against that model version. Even an iris center gives an estimate of the visible iris, not a clinical measurement of pupil position. For framing a selfie sequence, the distinction matters because eye movement can introduce jitter. A user looking left and then right can change the anchors without moving their head. A consistent gaze and capture guide help. Eye-contour centers offer another anchor convention when gaze changes dominate the result. I would run detection on each captured still image, with one intended face. A low-resolution preview can guide capture, but I would calculate the saved transform from the final image. Preview crops and final camera output often use different coordinate systems. Before measuring the eye vector: - Decode the photograph into a consistent upright orientation. - Apply the chosen mirror policy to both pixels and landmarks. - Convert detector coordinates into source-image pixels. - Preserve consistent eye identities across photographs. For a detector that returns normalized coordinates, multiply x by image width and y by image height. Measuring an angle in normalized coordinates distorts it whenever width and height differ. Two anchors determine rotation, scale, and translation Let the source eye anchors be: $$ p_1=(u_1,v_1),\qquad p_2=(u_2,v_2). $$ Choose fixed normalized output anchors: $$ q_1=(x_1,y_1),\qquad q_2=(x_2,y_2). $$ For horizontal eyes, choose $y_1=y_2$. In an output with width $W$ and height $H$, convert those targets to pixels: $$ Q_1=(Wx_1,Hy_1),\qquad Q_2=(Wx_2,Hy_2). $$ For example, on a 1080 by 1920 portrait canvas, (0.38, 0.35) and (0.62, 0.35) place the eyes at (410.4, 672) and (669.6, 672) . Their separation is 259.2 pixels. These are illustrative framing choices, not measured Phases defaults. SOURCE PHOTO OUTPUT CANVAS p2 * Q1 ============= Q2 / fixed eye axis / d p1 * tilted eye axis horizontal eye axis variable separation fixed separation translate + rotate + uniform scale Define the source and target eye vectors: $$ d=p_2-p_1,\qquad D=Q_2-Q_1. $$ Then calculate: $$ s=\frac{\lVert D\rVert}{\lVert d\rVert},\qquad \theta=\operatorname{atan2}(D_y,D_x)-\operatorname{atan2}(d_y,d_x). $$ Use the source midpoint $m=(p_1+p_2)/2$ and target midpoint $M=(Q_1+Q_2)/2$. For any source pixel $p$: $$ T(p)=sR(\theta)(p-m)+M, $$ where: $$ R(\theta)= \begin{bmatrix} \cos\theta & -\sin\theta\ \sin\theta & \cos\theta \end{bmatrix}. $$ You can write this as a homogeneous affine matrix: $$ \begin{bmatrix} x'\y'\1 \end{bmatrix} \begin{bmatrix} a&c&e\b&d&f\0&0&1 \end{bmatrix} \begin{bmatrix} x\y\1 \end{bmatrix}. $$ Two point pairs determine this similarity transform, which uses uniform scale, rotation, and translation. They do not determine an arbitrary six-parameter affine transform with shear and independent axis scales. A similarity transform belongs to the affine family and preserves the face's proportions. For a horizontal target eye line, $\theta$ cancels the source eye angle. The coordinate system uses positive y downward, as Canvas does; calculate both angles in that same system. Calculate the Canvas coefficients in TypeScript The following reference implementation accepts source anchors in pixels and target anchors in normalized output coordinates. It returns the six values in the order that Canvas expects for setTransform(a, b, c, d, e, f) . type Point = Readonly ; type Affine = [number, number, number, number, number, number]; function eyeAlignment( source1: Point, source2: Point, target1: Point, target2: Point, width: number, height: number, ): Affine { const values = [ source1.x, source1.y, source2.x, source2.y, target1.x, target1.y, target2.x, target2.y, width, height, ]; if (!values.every(Number.isFinite) || width value 1)) { throw new Error("Target anchors must lie within normalized output bounds"); } const sx = source2.x - source1.x; const sy = source2.y - source1.y; const tx = (target2.x - target1.x) * width; const ty = (target2.y - target1.y) * height; const sourceDistance = Math.hypot(sx, sy); const targetDistance = Math.hypot(tx, ty); if (sourceDistance < 1 || targetDistance < 1) { throw new Error("Eye anchors must be distinct"); } const scale = targetDistance / sourceDistance; const angle = Math.atan2(ty, tx) - Math.atan2(sy, sx); const a = scale * Math.cos(angle); const b = scale * Math.sin(angle); const c = -b; const d = a; const mx = (source1.x + source2.x) / 2; const my = (source1.y + source2.y) / 2; const qx = (target1.x + target2.x) * width / 2; const qy = (target1.y + target2.y) * height / 2; const e = qx - a * mx - c * my; const f = qy - b * mx - d * my; return [a, b, c, d, e, f]; } function drawAlignedFrame( ctx: CanvasRenderingContext2D, image: CanvasImageSource, imageWidth: number, imageHeight: number, matrix: Affine, ): void { ctx.save(); try { ctx.setTransform(1, 0, 0, 1, 0, 0); ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height); ctx.fillStyle = "#000"; ctx.fillRect(0, 0, ctx.canvas.width, ctx.canvas.height); ctx.setTransform(...matrix); ctx.drawImage(image, 0, 0, imageWidth, imageHeight); } finally { ctx.restore(); } } Use imageWidth and imageHeight from the same oriented image coordinate system as the source anchors. The code assumes the caller owns the canvas, with no inherited clipping region. The one-pixel check prevents division by a degenerate eye vector. Production acceptance needs a stronger, image-relative criterion: a two-pixel eye separation on a large image can still produce an unusable scale. Check confidence, face size, and acceptable scale before saving alignment metadata. To check the matrix mathematically, apply it to each source anchor. You should recover the corresponding target pixel coordinate: au1 + cv1 + e = Wx1 bu1 + dv1 + f = Hy1 au2 + cv2 + e = Wx2 bu2 + dv2 + f = Hy2 These equations provide useful unit-test assertions with a floating-point tolerance. They describe invariants, not test results from a running Phases build. React Native needs a rendering boundary React Native does not expose a browser DOM Canvas as a built-in drawing surface. The CanvasRenderingContext2D example belongs in a browser or a WebView. In a native renderer, adapt the same coefficients to that renderer's matrix convention. You can use a WebView-hosted Canvas for a shared web rendering path, or use a native GPU drawing surface. With WebGL, upload a photo texture and apply the transform while drawing a quad. Account for the renderer's origin, te