How to mix and match BigQuery pricing models on your workloads to optimize costs.
When it comes to BigQuery pricing, there’s no “one-size-fits-all” approach.
Since you can utilize different BigQuery pricing models across projects, you should split your workloads across dedicated projects vs. grouping them all in one project.
Placing different workload types in different projects — and using different BigQuery pricing models in those projects — is a great way to reduce your BigQuery costs because it makes it easier to match the right pricing plan for each workload.
For example:
➡️ ETL and ELT workloads should be in a separate project from your R&D workloads
➡️ Spiky, short-running workloads should go in a project that utilizes on-demand pricing
If most of your workloads are in one main project, you might worry that breaking up your project will be too complicated.
Sure there will be configuration changes to consider — you might have to change your macros if you’re using dbt, for example.
But consider the consequences of not splitting your BigQuery workloads into different projects.
Imagine that a company with its daily data pipeline jobs in the same project as ad-hoc, exploratory data analysis jobs. If the project was using a BigQuery Edition, they’d be overspending on the ad-hoc queries and vice-versa if the project were utilizing on-demand pricing.
Catch the full episode on transitioning from BigQuery flat-rate pricing and optimization in the world of BigQuery Editions here:
YouTube: • BigQuery Editions and transitioning from e...
Spotify: https://open.spotify.com/episode/5imh...
Apple Podcasts: https://podcasts.apple.com/il/podcast...
#bigquery #googlecloud