PawSpective: See the World Closer to How Your Dog Sees It

This is a submission for Weekend Challenge: Dog Days Edition. Have you ever bought a bright red toy that looked impossible to miss only to watch your dog struggle to find it on green grass? Humans and dogs do not experience color in the same way. That inspired me to build PawSpective, an application that combines a canine-vision approximation, AI-assisted scene understanding, deterministic visibility calculations, and playful narrated videos. The central idea is one connected experience: Capture a scene → explore a canine-vision approximation → discover which objects remain visible → turn the moment into a fictional story. I also wanted PawSpective to be entertaining without pretending it can read a dog’s mind. It does not claim to determine exact canine vision, gaze, thoughts, feelings, smell, or intent. What I Built PawSpective helps dog owners explore how objects in an everyday scene may appear under a canine-inspired color transformation. A user can: - Create a lightweight dog profile. - Open the Live Dog Lens. - compare Human View with a canine-vision approximation. - Record or upload a short video. - Let Google Gemini identify visible objects and scene events. - Review, rename, or remove incorrect AI detections. - Calculate the relative dog-visible contrast of an object. - Explore possible attention cues through the Curiosity Map. - Compare alternative toy colors in the Toy Color Lab. - Generate and download a fictional narrated Story Reel. Live Dog Lens The Live Dog Lens uses a browser-based WebGL transformation to provide an immediate Human/Dog Vision comparison. The transformation preserves more blue and yellow differentiation while reducing red and green differentiation. A comparison slider makes the effect easy to understand, and a dog-height guide helps users frame the scene from a lower point of view. This is deliberately described as a canine-vision approximation, not an exact reconstruction of what a particular dog sees. AI scene analysis After a user uploads a 5-15 second video, PawSpective sends a normalized, silent version of the clip to Google Gemini. Gemini returns structured scene evidence, including: - Visible object labels - Approximate bounding boxes - Timestamps - Object categories - Confidence values - Visible evidence - Coarse motion levels The response must pass a strict Pydantic and JSON Schema contract before the application accepts it. Users can then remove incorrect events, rename objects, and select which object should be analyzed. These corrections become the source of truth for visibility calculations and Story Reel generation. Visibility Lab The Visibility Lab measures how strongly a selected object differs from its nearby background. The backend: - Seeks to the event timestamp. - Samples pixels inside the object’s corrected bounding box. - Samples a surrounding background region. - Applies the same canine color transformation used by the frontend. - Converts colors into CIE Lab. - Measures foreground/background color and luminance separation. - Produces a relative dog-visible contrast score. The result is presented as a product score and qualitative visibility band-not as a scientific probability. For example: Dog-visible contrast: High The blue toy remains visually distinct from the surrounding grass after the canine-vision transformation. Curiosity Map The Curiosity Map highlights objects that may visually stand out in the scene. It combines: - AI-inferred motion - Measured dog-visible contrast - Apparent object size - A small optional profile-relevance bonus Every result explains why it appeared. PawSpective calls these possible attention cues, because the application does not track a dog’s gaze or know what the dog is actually paying attention to. Toy Color Lab Toy Color Lab compares six screen colors against the measured background around a selected object. It keeps the surrounding background unchanged, simulates alternative colors for the object, applies the canine transformation, and ranks the colors by approximate dog-visible contrast. The preview is illustrative-it does not claim that a physical toy will look exactly the same or guarantee a dog’s response-but it makes the red-versus-blue-on-grass problem immediately understandable. Story Reel The final step turns the reviewed scene into an 8-10 second vertical video. Google Gemini uses only the user-reviewed scene timeline to generate grounded story and animation direction. ElevenLabs creates the fictional dog narration, and FFmpeg combines the video, narration, captions, overlays, music, watermark, and disclaimer into a downloadable 9:16 MP4. The result remains clearly labeled Just for fun. Demo The application is available at: 👉 https://pawspective.onrender.com/ Demo note: [My Gemini API daily usage limit ran out, so I couldn’t publish the complete demo.] The interface and deployed application can still be explored through the link above. The complete demonstration will be added to the repository once the provider usage limit is available again. Code The complete source code is available on GitHub: kernelKain / pawSpective See the world closer to how your dog sees it. PawSpective combines canine-vision simulation, AI-powered object visibility analysis, curiosity mapping, and playful narrated story reels. PawSpective See the world closer to how your dog sees it. PawSpective combines a canine-vision approximation, reviewed AI scene analysis, deterministic visibility scoring, color comparison, curiosity mapping, and downloadable fictional Story Reels. What it does - Opens a live Human/Dog Vision comparison or accepts a 5-15 second video. - Detects visible objects with Gemini and lets the user correct the result. - Measures foreground/background contrast with OpenCV and CIE Lab color. - Shows possible attention cues in a timestamp-aligned Curiosity Map. - Compares six screen colors in Toy Color Lab. - Creates an 8-10 second animated dog-height POV reel with fictional narration. - Supports a SHA-256-bound controlled demo for offline rehearsals. PawSpective labels deterministic calculations as Research-grounded model interpretation as AI-inferred, and fictional output as Just for fun. It does not claim exact canine vision, gaze, thoughts, emotions, smell, intent, or behavioral diagnosis. See the AI disclosure for data flow, limitations, and provider details. Repository: https://github.com/kernelKain/pawSpective The repository also contains the product contract, AI disclosure, deployment guide, exported schemas, automated tests, Docker configuration, and release checklist. How I Built It PawSpective uses a Next.js frontend and a Python media-processing backend. Frontend The frontend uses: - Next.js 16 - React 19 - TypeScript - WebGL for the live color transformation - Canvas and video APIs for previews and overlays - Vitest and Testing Library - Playwright for end-to-end smoke testing The browser handles the live Dog Lens, comparison slider, dog profile, recording and upload flow, object-correction interface, visibility results, Toy Color Lab, Curiosity Map, job polling, and Story Reel download. Backend The backend uses: - FastAPI - Pydantic - Google Gemini - OpenCV - NumPy - ElevenLabs - FFmpeg and FFprobe - SQLite FastAPI validates uploaded media, normalizes videos, coordinates AI requests, performs deterministic calculations, and manages background Story Reel jobs. SQLite stores job metadata and progress, while generated media is stored on a writable backend volume. Story rendering runs through a bounded background worker so the frontend can submit a job, poll its progress, and download the completed reel. Google Gemini Gemini is used for three related tasks: Scene analysis Gemini examines the short clip and returns visible objects, approximate bounding boxes, timestamps, motion levels, and supporting visual evidence.Grounded story generation Gemini receives the reviewed event timeline instead of being allowed to invent details directly from the raw video.Animation direction Depending on the configured model and access, Gemini Omni or Veo can create a canine-vision-inspired artistic edit using the source clip or reference frames. One of my most important engineering decisions was requiring structured model output. Gemini responses are validated against strict schemas, and malformed coordinates, unsupported fields, impossible timestamps, or ungrounded results are rejected. ElevenLabs ElevenLabs generates the complete fictional dog narration for the Story Reel. Only the finalized narration text is sent to ElevenLabs-the original video is not. The selected voice is intentionally described as a fictional dog voice, rather than an attempt to recreate a real animal’s voice or internal thoughts. The application also supports narration failure handling and cached demo output so a temporary voice-service problem does not break the entire user journey. Deterministic computer vision I did not want an AI confidence value to become a fake scientific visibility score. Instead, the Visibility Lab and Toy Color Lab use deterministic OpenCV and NumPy calculations. They sample pixels from the video, apply the documented color transformation, convert the colors to CIE Lab, and measure relative foreground/background contrast. This creates a clear boundary between what the model inferred and what the application calculated. Video composition FFmpeg performs the final media work: - Video normalization - Portrait 9:16 composition - Captions - Narration synchronization - Curiosity overlays - Visibility result cards - Music - Watermarking - Final MP4 encoding Using a fixed template made the rendering pipeline more reliable and kept the project focused on one polished experience instead of becoming a general-purpose video editor. Reliability and responsible AI PawSpective separates its output into three categories: - Research-grounded: deterministic transformations and calculations - AI-inferred: visible-object and motion interpretation from