Stop copying your data for Vertex AI pipelines

80% of data scientists spend their time cleaning data, but 90% of data engineers spend their time babysitting ETL pipelines that move the exact same data from a Lakehouse into a training-specific bucket.

We are literally burning cloud credits to store the same Parquet files in two different places because we’re too lazy to build a connector. If you are still running a "data export" step before your Vertex AI pipeline, you are paying a "laziness tax" that kills both your storage budget and your data lineage.

Why I chose this topic: In my time handling HIPAA-regulated datasets, I’ve seen teams lose six figures annually in egress costs and storage bloat simply because they didn't trust their Lakehouse to serve a model directly. I’m writing this because I’m tired of seeing "copy-to-GCS" steps in production pipelines that have no business being there.

We all use BigQuery or Dataproc daily, but most of us treat them like glorified file cabinets rather than compute engines. We treat the Lakehouse as a passive sink. That’s a mistake. The real power of the modern stack is that the feature engineering layer should be an extension of your query engine, not a separate shuttle service.

How it actually works

The goal is to stop moving data and start querying it in place using the Vertex AI SDK for Python. Instead of a pipeline that pulls data into a staging bucket, you should be using BigQuery-backed feature sets or direct integration with your Lakehouse (Databricks or BigQuery) using Vertex AI Feature Store.

Here is the mechanics of how we do this without a single gsutil cp command.

You define your training input as a BigQuery URI rather than a GCS path. When you initialize your CustomJob or PipelineJob, you point the input_data_config directly to your source table.

from google.cloud import aiplatform

job = aiplatform.PipelineJob(
    display_name="feature-eng-pipeline",
    template_path="pipeline.json",
    parameter_values={
        "bq_input_uri": "bq://my-project.feature_dataset.user_features_v1",
        "model_output_uri": "gs://my-bucket/models/"
    }
)

Inside the pipeline component, you shouldn't be reading CSVs. You should be using the google-cloud-bigquery client to create a temporary view or a materialized table that represents your training slice. If your feature engineering involves complex window functions or time-series joins, run those as a BigQueryToDataset operation.

The magic happens when you use BigQuery as the direct source for your Vertex AI TrainingPipeline. By setting the training_task_inputs to point at a BQ table, you allow Vertex to manage the read operation. If you’re using BigQuery ML (BQML), you aren't even leaving the warehouse. You’re shipping the compute to the data, which is the only way to scale in a financial services environment where data gravity is a real, tangible problem.

Photo by A Chosen Soul on Unsplash

The tradeoffs nobody mentions

Let’s be honest: avoiding data duplication isn't free.

First, you lose the "snapshot" guarantee of a flat file. If your source table in BigQuery changes while your training job is running, you get non-deterministic results. You are essentially training on a moving target. To fix this, you have to implement proper partition pruning or point-in-time snapshots in your DDL. If you don't use FOR SYSTEM_TIME AS OF in your BigQuery queries, you will eventually have a production model that fails to reproduce because the underlying data drifted during the training run.

Second, the BigQuery read API has throughput limits. If you have a massive dataset (hundreds of terabytes) and you try to pull it into a custom container running on Vertex AI, you will hit the 100MB/s per project limit for the BigQuery Storage Read API. You’ll be scratching your head wondering why your pipeline is taking five hours to initialize when the query itself took ten seconds. You’ll need to explicitly manage your concurrency settings.

Third, debugging becomes a headache. When you have an error in a CSV on GCS, you can just cat the file. When you have an error in a BQ-backed pipeline, you’re dealing with IAM permissions, service account scopes, and project-level quotas. If your Vertex AI service account doesn't have bigquery.jobs.create and bigquery.datasets.get on the source dataset, the job will fail silently or hang in a state of perpetual "pending."

Photo by Growtika on Unsplash

When to reach for it (and when not to)

Reach for in-place feature engineering when your data is already in BigQuery and you are running iterative experiments. If you are doing fast-paced model iteration, you cannot afford to wait for a 20-minute ETL job to shuffle files to GCS every time you want to add a feature.

Do NOT reach for it if you are dealing with unstructured data. If your Vertex AI pipeline is processing images, audio, or raw binary blobs, keep those in GCS. The overhead of storing binary data in a database as BLOBs is a disaster for performance and cost. Keep your structured features in the Lakehouse and your raw unstructured data in GCS, then use a BigQuery join or an index file to link them during the training loop.

Also, avoid this pattern if your team lacks strong SQL skills. If your data scientists are "Python-native" and refuse to touch SQL, you will end up with broken, unoptimized queries that scan the entire table instead of using partitions. You will bankrupt your cloud account in a week. If you can’t write a partition-aware query, stick to your GCS file copies.

Conclusion

The "data duplication" tax is an artifact of a time when compute and storage were tightly coupled. Today, with BigQuery and Vertex AI, they are logically separated but physically close. By treating your Lakehouse as the primary source of truth for your training pipelines, you reduce your attack surface for data drift, lower your storage costs, and simplify your lineage.

Stop treating your cloud storage like a temporary trash bin for ETL jobs. Start treating your Lakehouse like the compute engine it was designed to be. It’s harder to set up, and the IAM policies will make you pull your hair out for an afternoon, but the result is a clean, scalable pipeline that doesn't duplicate the world every time you want to train a model.

Tags: gcp, vertexai, data, engineering

Cover photo by Tyler on Unsplash.