Serverless Multimodal Vector Search on Apache Iceberg via Google Apps Script

Turn Google Drive into an AI-Powered Lakehouse Vector Engine across Converted PDFs, Binary Images, and Text without Specialized Vector Databases. Hero Infographic: Unified Multimodal Lakehouse Vector Search via Apache Iceberg & Google Apps Script. Consolidates Google Docs, Sheets, Slides, Forms, binary diagrams, and web-fetched assets into an open Parquet table, driven by Gemini embeddings and BigQuery serverless pushdown for sub-second semantic retrieval and live Google Drive discovery. Structural Analysis of the Hero Infographic: The hero infographic visualizes the end-to-end paradigm shift enabling unified multimodal storage and sub-second semantic discovery across three synchronized operational zones: - Left Zone (Diverse Multimodal Ingestion): Ingests heterogeneous corporate knowledge spanning Google Drive assets (Docs with automated PDF normalization, Sheets data/PDFs, Slides presentations, and Forms intake/response structures), binary media (PNG/JPEG schematics and diagrams), and web-crawled/HTTP-downloaded external files into a unified ingestion pipeline. - Center Zone (Serverless Processing & Unified Iceberg Lakehouse): The IcebergApp.js engine running purely within Google Apps Script extracts payloads and routes textual representations to the Gemini Embedding API ( text-embedding-004 ) to synthesize 768-dimensional normalized float vectors (ARRAY ). These are committed alongside raw binary streams (data BYTES ), metadata, and authentic Google Drive identifiers (file_id STRING ) into a Single Apache Iceberg Table hosted on Google Cloud Storage. - Right Zone (Serverless Query & Live File Discovery): When an operator issues a natural language query, BigQuery executes distributed COSINE_DISTANCE pushdown directly across the open Parquet files on Cloud Storage. Within 0.3 to 0.5 seconds, ranked results (with similarity metrics exceeding0.87 ) are returned, enabling instant 1-click browser navigation back to the live collaborative document or triggering programmatic Workspace automation. Abstract Enterprise knowledge is notoriously fragmented across structured tabular records, raw text notes, binary diagrams, and rich documents stored in Google Drive. Traditional Retrieval-Augmented Generation (RAG) and semantic search architectures address this challenge by deploying dedicated vector database clusters (e.g., Pinecone, Milvus, Qdrant) or specialized Reverse ETL pipelines. However, these systems introduce steep subscription costs, operational complexity, and vendor lock-in, while isolating embeddings from core analytical storage. This article introduces a 100% serverless, zero-maintenance Multimodal Lakehouse Vector Search architecture built on IcebergApp, Google Apps Script (GAS), and BigQuery. Leveraging Apache Iceberg's native support for binary data (BYTES ), this system ingests heterogeneous assets-including live Google Docs automatically converted into PDF binary blobs, raw image diagrams, and direct plain text records-into an open Parquet table on Google Cloud Storage. Crucially, by persisting the authentic Google Drive file_id alongside the binary PDF snapshot, vector search results immediately map back to the live collaborative document (https://docs.google.com/document/d/${file_id}/edit ), transforming Apache Iceberg into an enterprise-grade semantic file search engine for Google Workspace. Vector embeddings are generated on-the-fly using the Gemini API (text-embedding-004 ) and queried using distributed vector distance metrics (COSINE_DISTANCE ) with sub-second latency. Furthermore, to enable developers to safely evaluate the pipeline within their own Google Cloud environment, the article includes an autonomous test suite equipped with an automated self-cleaning mechanism that guarantees zero-residue cleanup of temporary test assets, empowering Workspace developers to evaluate and deploy enterprise-grade multimodal search engines directly from Google Apps Script. 1. Introduction: The Fragmentation of Multimodal Knowledge In modern enterprise operations, knowledge does not reside neatly in relational database rows. It lives in diverse multimodal formats scattered across Google Workspace and cloud storage: - Rich Documents: Standard operating procedures, technical manuals, and architectural specifications in Google Docs; financial models and analytical KPIs in Google Sheets; pitch decks and visual proposals in Google Slides; and questionnaire forms and intake responses in Google Forms. - Visual Artifacts: System architecture schematics, flowcharts, and infographics saved as PNG/JPEG images. - Direct Operational Text: Real-time event notes, error logs, and transactional summaries written directly in code or spreadsheets. To visually capture this operational divide and contrast conventional, costly multi-tier architectures with our unified serverless design, Figure 1 outlines the structural evolution from isolated enterprise silos to an integrated Apache Iceberg lakehouse. Figure 1: Fragmented Enterprise Data Silos & Traditional RAG vs. IcebergApp Unified Multimodal Serverless Lakehouse. Contrasts the high-cost, fragmented traditional RAG pipeline against the unified, zero-egress Apache Iceberg architecture powered by Google Apps Script and BigQuery on Google Cloud. Detailed Architectural Analysis of Figure 1: As clearly mapped in the comparative tiers of Figure 1, enterprise multimodal knowledge management undergoes a fundamental structural paradigm shift: - Top Section (Fragmented Enterprise Data Silos): In everyday operations, assets are created and stored in isolated systems: Google Docs, Sheets, Slides, and Forms inside Google Drive; PNG/JPEG diagrams residing in Cloud Storage or Drive; and operational event notes residing as raw text strings. Because these assets use incompatible storage primitives, traditional analytical systems cannot ingest or index them collectively. - Middle Section (Traditional Complex RAG Stack - The Problem): As depicted by the red-bordered warning flow in Figure 1, conventional architectures attempt to bridge these silos by introducing third-party Python microservices and dedicated vector databases (e.g., Pinecone, Milvus, Qdrant). This approach introduces three fatal operational penalties: - Exorbitant Subscription Tax: Dedicated vector database clusters impose recurring fixed commitments between $500 and $3,000 per month, regardless of actual query volume. - Severe Egress Vulnerabilities: Sensitive enterprise documents and intellectual property must leave the secure Google Cloud Virtual Private Cloud (VPC) and traverse external public networks to reach third-party SaaS platforms. - Dual-Storage Synchronization Drift: Storing vector embeddings in a standalone vector database while keeping binary PDFs and images in separate object storage invariably leads to broken links, synchronization lags, and metadata drift. - Bottom Section (IcebergApp Unified Serverless Lakehouse - The Solution): Indicated by the glowing green and cyber-cyan borders in Figure 1, our architecture replaces this complex multi-tier stack with a clean, 100% serverless pipeline: - Google Apps Script (V8 Engine): Directly interacts with Google Workspace APIs via DriveApp and native services, exporting Docs, Sheets, and Slides into standardized PDF binary blobs on-the-fly (and extracting Forms structure viaFormApp or linked response sheets) without intermediary servers. - Apache Iceberg on Google Cloud Storage: Eliminates storage fragmentation by persisting Text metadata, Binary BYTES (PDF, PNG), and 768-dimensional float embedding vectors into a single, open Apache Parquet table. - BigQuery Vector Accelerator: Leverages distributed COSINE_DISTANCE pushdown directly across open Parquet files, executing sub-second semantic retrieval with 100% data sovereignty entirely within the customer's GCP perimeter. - Google Apps Script (V8 Engine): Directly interacts with Google Workspace APIs via 1.1 The Pitfalls of Traditional Vector Architectures To perform semantic search across these mixed assets, engineering teams typically deploy complex multi-tier pipelines: - Dedicated Vector Database Tax: Specialized vector databases (Pinecone, Weaviate, Qdrant) cost hundreds to thousands of dollars per month, requiring dedicated network infrastructure, VPC peering, and credential rotation. - Metadata Synchronization Drift: When document content changes on Google Drive, keeping the vector database synchronized with external document stores requires complex webhook choreography and distributed transaction handling. - Binary Data Isolation: Most vector databases store only vector embeddings and minimal text metadata, forcing systems to maintain secondary blob storage links that frequently break or drift out of sync. 1.2 The Breakthrough: Apache Iceberg as a Multimodal Vector Store Apache Iceberg fundamentally redefines this paradigm: - Binary Data Support: Iceberg tables are not restricted to scalar primitives; they natively store arbitrary binary payloads ( BYTES ) and complex arrays (ARRAY ). A single open Parquet table can hold document metadata, raw PDF binaries, image bytes, and high-dimensional vector embeddings side by side. - Zero Vendor Lock-In: Assets reside in open Parquet files on Google Cloud Storage, directly queryable by BigQuery, Apache Spark, Trino, or DuckDB. - Serverless Vector Compute: BigQuery provides distributed vector distance functions ( COSINE_DISTANCE ,EUCLIDEAN_DISTANCE ) and vector indexes over Iceberg tables, eliminating the need for standalone vector database servers. 1.3 From Vector Search to Semantic File Discovery: The Power of GAS & file_id One of the most compelling enterprise advantages of implementing this pipeline in Google Apps Script is the direct bridge between AI vector search and live Google Drive file discovery: - In conventional RAG pipelines, vector search engines return isolated text chunks, leaving users stranded with no direct path to the original source document. - In cont