BigQuery
SQL database offered by GCP, used as a Data Warehouse for data analysis and various other use cases.
Use cases :
- logs
- analytics
- ETL (Extract Transform Load)
- Machine Learning
- Data Integration
- Cost Management
Using BigQuery as a versatile, do-everything service is one approach, but with more maturity or a better initial understanding of the need, other dedicated services can be preferred.
Other options
- Analytics :
- Cloud DataFlow
- Cloud Pub-Sub
- ETL :
- Cloud DataFlow
- Cloud Composer
- Machine Learning :
- Cloud AI Platform (formerly ML Engine)
- Cloud AutoML
- Data integration :
- Cloud Data Fusion
- BigQuery Data Transfer Service
BigTable
NoSQL database, used internally for Gmail, Maps, Analytics…
Built for large datasets, commonly used for AI workloads (take this with a grain of salt, not a Data Scientist). Technical depth aside, same applies here : the indexing details are beyond the scope of this note.
As with most managed databases, the classic audit angle is to check the permissions granted on access to the DB and evaluate the impact.
Cloud Storage
Object-typed storage service (images, videos, files), referred to as BLOB storage (Binary Large-object).
Files are immutable. There’s no “edit”, only overwrite. Versioning can be enabled to preserve previous versions and define a maximum number of retained iterations.
IAM defines access at the bucket level for Cloud Storage. If finer-grained control within the bucket is needed, ACLs come into play.
Different storage classes exist depending on access frequency :
Worth checking during audits, can hold sensitive information if a bucket is misconfigured and listing is possible. Sensitive data exposure to check for.
ACL
ACL is not the same as IAM.
- Used at the bucket level, ACLs allow filtering access on individual objects within a single bucket, which IAM does not natively support (IAM grants permissions at the bucket level generally, though it technically can go finer, just not in an optimal way).
- Requires the bucket to be interoperable with Amazon S3 or to use the XML API exclusively.
Fairly specific use case. If none of the above criteria apply, use IAM instead.
To check
During a configuration audit, check for existing ACLs, potential misconfigured permissions.
Firestore
NoSQL database that scales horizontally. Worth comparing NoSQL vs SQL trade-offs before choosing storage. Not much else to detail here, usage-wise it’s worth comparing against BigTable (also NoSQL) which is oriented toward large datasets and heavy processing.
Spanner
A Google Cloud SQL-based database. You essentially instantiate the DB and the rest works like a classic relational database. The main advantage is scalability once again, and as for technical depth, not a Data Scientist, the indexing choices in Spanner are a rabbit hole.
For background reading (even though it’s technically insufficient on its own) : https://en.wikipedia.org/wiki/Spanner_(database)
From an audit standpoint there isn’t much to say specifically, the same logic applies : check permissions, and remember at its core, it’s still a database.
Cloud SQL
(placeholder, to be expanded)