Loading Academy Hub...
Build end-to-end generative AI and vector pipelines on GCP: BigQuery ML text embedding generation (`ML.GENERATE_TEXT_EMBEDDING`), Vertex AI Vector Search (formerly Matching Engine), and scaNN algorithms.
[!NOTE] Google Cloud converges big data and AI by bringing machine learning directly into SQL. Using BigQuery ML (BQML), engineers generate LLM vector embeddings over billions of rows directly with SQL queries, indexing them into Vertex AI Vector Search for sub-millisecond nearest neighbor search.
Instead of exporting data to Python VMs, BigQuery calls Vertex AI foundation models directly in-database using remote models:
SELECT * FROM ML.GENERATE_TEXT_EMBEDDING(
MODEL `finance.text_embedding_model`,
TABLE `finance.customer_feedback`,
STRUCT(TRUE AS flatten_json_output)
);
BigQuery ML + ML.GENERATE_EMBEDDING): Instead of exporting 50M rows out of BigQuery to external GPU VMs, call remote Vertex AI models (text-embedding-005, gemini-2.5-flash) natively inside SQL via BigQuery ML Remote Models + VECTOR_SEARCH with IVF / Tree-AH indexes kept fresh inside Colossus.BigQuery-Backed): Feature Store v2 reads features straight from governed BigQuery tables/views and syncs online serving endpoints (Bigtable or Optimized Vector Search) without managing legacy duplicated feature ingestion code, guaranteeing zero training-serving skew.Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
-- 1. Register Vertex AI text-embedding model in BigQuery
CREATE OR REPLACE MODEL `enterprise-lakehouse-prod.finance.bq_text_embedder`
REMOTE WITH CONNECTION `us.vertex_ai_connection`
OPTIONS (ENDPOINT = 'text-embedding-004');
-- 2. Generate embeddings for raw complaint logs directly via SQL
CREATE OR REPLACE TABLE `enterprise-lakehouse-prod.finance.vector_indexed_complaints` AS
SELECT
complaint_id,
customer_id,
content,
ml_generate_text_embedding_result AS text_vector
FROM ML.GENERATE_TEXT_EMBEDDING(
MODEL `enterprise-lakehouse-prod.finance.bq_text_embedder`,
TABLE `enterprise-lakehouse-prod.finance.raw_complaints`,
STRUCT(TRUE AS flatten_json_output)
);google.api_core.exceptions.ResourceExhausted: 429 Resource has been exhausted (e.g. check quota).