We use this product extensively to analyse users who play our games. We record billions of rows of data per day and can query it effectively/efficiently with Big Query. It is a pretty awesome product and getting better all the time.
That's interesting, you are the only one in this thread who is using it.
Could you tell a bit more about for what kind of analytics you are using it ? And why you use BigQuery and not some other analytics solution (Mixpanel, Kontagent, Flurry etc.) for these scenarios ?
At streak.com we use BigQuery to analyze our logs. All of our application logs are shipped over to BigQuery and we run sql commands over this large data set. Its been amazingly helpful.
We open sourced two things that helped us do this:
Tangential to the discussion, but I am currently working on a system that will handle some 100 million semi-structured rows. Despite all the buzz about "big data", hadoop and nosql etc. etc. there is literally no system that can give me a way to reliably store AND search through this data on multiple columns in near real-time (assuming I can throw a reasonable no. of nodes at it).
I looked at BigQuery, but they don't support updates. It's just dump and analyze.
I needed to do something like this and I ended up settling on a custom workflow connected through zeromq dumping data into into sharded sqlite databases. Single process handled writing to each database and lots of reads were mapped out and then reduced (also via zmq). This handled around 14 billion rows when I left that job. If you want to talk some more about it, my gmail is the same as my username.
Yes, it's the closest to my current needs. But it's unlikely I will be able to use it currently as it's still pretty nascent and I don't feel comfortable just yet. Anyhow - I am looking to run some benchmarks on it next week.
One company I know of literally ships hard drives-it's faster and more reliable than transferring the data over the internet. I'm not sure if that's what they do with BigQuery.
To get data into BigQuery you need to get it into Google Cloud Storage first and then import it from there. To get it into cloud storage you can use rest api's or command line utils. See https://developers.google.com/storage/
You need to read the contract/terms of service to determine what you are granting them with regards to your data. You might find that they want to keep an option open that you can't accept. You might also find that you are okay with what they propse. This is one of those times when you actually need to read the fine print.
You can do similar feats with Amazon Elastic MapReduce + Karmasphere for reasonable cost except that Amazon S3 has the annoying 5GB max file size limit, which is a pain to workaround, where Google Cloud Storage has 5TB max file size.
Isn't Amazon S3 now running with a 5 terabyte limit now[1]? But still, shipping drives is far more sane than cross the t'interwebs for things of that size.
