This morning in Paris we delivered our 7th Google Cloud Platform Next experience of 2015. We started Next because we wanted to bring Google Cloud Platform to you so you can learn about and try the tools to build your next great project. Thanks to Next we've been able to engage with more than 23,000 developers around the world. Google Cloud Platform Next has brought us closer to you and we're looking forward to announcing our 2016 tour on our website soon.

We also want to share some new products which we're announcing today at Next in Paris:

Screen Shot 2015-10-12 at 9.41.18 AM.pngCloud Datalab: The Perfect Tool For Data Exploration
Google Cloud Datalab, available today in Beta is a web-based interactive developer tool that allows you to get insights from your raw data and explore, share, and publish reports in a fast, simple and cost-effective way. Cloud Datalab combines the power of Google BigQuery and Google Cloud Storage with familiar data science ecosystems built around IPython, removing the need for complex integration between products.

Cloud Datalab is an interactive tool that enables you to write code to:
  1. Explore, transform, visualize and process data in Google Cloud Platform.
  2. Combine code from multiple languages seamlessly: Python, SQL and JavaScript (BigQuery UDF).
  3. Build and test data pipeline for deployment to BigQuery.
  4. Create, tune, and deploy Machine Learning models.

Cloud Datalab removes common barriers of getting started. Instead it provides a ready-to-use, fully setup, secure, multi-user environment integrated with source control for developers and data scientists. With Cloud Datalab, you can focus on your data analysis tasks immediately.

Please visit Cloud Datalab to learn more and get started!

Google Cloud Shell - a Google Curated Admin Machine at your fingertips!
Our customers want to be able to easily manage their infrastructure and applications when running on Google Cloud Platform no matter where they are.  Building on the ability to SSH from the browser we released for Google Compute Engine VM instances last year, today we’re launching Google Cloud Shell Beta, which extends from just VMs to all aspects of the platform.

Google Cloud Shell provides you with command-line access to computing resources hosted on Google Cloud Platform and is now available in the Google Developers Console. We deliver Cloud Shell via a VM hosted and managed by Google, so you don’t have to worry about managing it alongside the rest of your production infrastructure.  To ensure that you can always pick up where you left off if you disconnect or move to a different machine, we’ve also set aside sufficient storage space for you to store your Cloud Shell sessions.

Launching Cloud Shell is easy. Just open the Google Developers Console and click the Cloud Shell icon in the top right from any page of the Console.

Screenshot 2015-10-11 12.55.10.png

We have made using Google Cloud Platform easier and we’re excited to see how you’ll use this new feature to be even more productive. Google Cloud Shell is available to all customers free while in beta until early 2016. Learn more about Cloud Shell here.

-Posted by Greg DeMichellie, Director Product Management

Today’s guest post comes from Kevin Baillie, co-founder and visual effects supervisor of Atomic Fiction, a visual effects company in Oakland, CA.

Five years ago, we founded the visual effects company Atomic Fiction with the goal of “creating the most believable fiction imaginable” for film and television. We had lofty aspirations despite the fact that the effects industry had entered a dark period, suffering a distressing rash of closures and bankruptcies. But we saw signs of hope: more effects work in movies was being done than ever before, and the entertainment industry’s longstanding technology model was clearly undergoing transition. Instead of building a giant “render farm” to perform the data processing required to create photo-realistic images for film, our idea was to build Atomic Fiction with the cloud as a core component of our business model. At the time, an overwhelming majority of our most experienced colleagues echoed the following advice to us: “It’ll never work. Don’t do it.”

With some ingenuity combined with bull-headed naïveté, our team encountered then overcame one hurdle after another. We were the first and only visual effects company to complete work on A-List projects including “Transformers,” “Flight,” “Star Trek,” “Cosmos,” and “Game of Thrones” entirely using cloud-based tools. But our biggest challenge was still to come.

Robert Zemeckis, director of “Back to the Future,” “Who Framed Roger Rabbit?,” and “Forrest Gump,” witnessed the benefits of our cloud-based workflow and tasked Atomic Fiction with crafting the iconic plane crash sequence in “Flight” (Denzel Washington). In 2013, on the heels of that project’s success, he approached us and said, “I want you guys to do “The Walk” for me. We’ve got an incredibly tight budget, but you’re going to need to recreate the Twin Towers, and they have to look absolutely real. And you need to build 1974 New York from scratch. Oh, and it’s all going to be featured in about 30 minutes’ worth of the movie.” Our hearts jumped into our throats. With our faith in the future of the cloud, we said, “Sure, Bob!”

To understand why the technical demands on “The Walk” were so intense, it’s important to illustrate the rendering process. In the beginning of a shot’s life, an artist works with a lightweight “wireframe”
representation of a scene, which looks something like this:

Its relative simplicity makes the scene quick to interact with, so that a team of artists can work with programs like Maya and Katana to create “recipes” for how cameras move, how surfaces are textured, and how lighting falls on objects. Once that recipe is defined, it needs to be put into action. This process of calculating how light is received by and reflects off of millions of individual surfaces is called rendering. Here’s the final result:

A single iteration of one frame, which amounted to 1/24th of a single second of “The Walk,” had 83GB of input dependencies and took 7 hours to render on a 16-core instance. Now imagine 1,000 instances all being asked to do other similarly intensive parts of the film, all at the same moment in time. Each second of screen time would require 5,000 processor hours to realize. Given the nature of the deadlines, our teams needed the ability to spike to 15,000 cores simultaneously on-demand just to stay on schedule. And because we don’t realize profits until the end of the project, we needed to spend as little as possible to get up and running.

We knew that in order to achieve the necessary combination of scale and economics, we’d have to tap heavily into the cloud. Since no existing cloud rendering solution could address our needs at that scale, we decided to develop our own software atop Google Cloud Platform. Google’s efficiency, availability of resources, and per-minute billing formed the back-end of a product we call Conductor. Our artists used Conductor to manage the entire cloud rendering workflow from end-to end.

By the time Atomic Fiction had finished “The Walk,” our team had created effects sequences that Deadline Hollywood deemed the “best I have seen all year” and USA today crowned “a visual spectacle.”

The stats associated with “The Walk’s” production constitute the largest use of cloud computing in filmmaking to date. Here’s the final score card:

  • 9.1 million core hours in total were used to render “The Walk.”
  • Movie projects are heavily back-loaded, and “The Walk” was no exception. 2 million core hours were rendered in the final month alone.
  • Peak scale of over 15,000 simultaneous cores was achieved rendering I/O intensive tasks.
  • Average artist productivity increased 20%.
  • Savings vs. traditional infrastructure was in excess of 50%.
  • Because Atomic Fiction had Conductor at its disposal, director Zemeckis was able to realize 4 additional minutes of the movie’s climactic scene.

When looking at “The Walk’s” Conductor processor hours graphed out monthly, it becomes obvious how varied processing needs are over the course of a project.

Having just screened the final cut of the film, Robert Zemeckis called.“I think this is the most spectacular effects movie I’ve ever made. I can’t imagine making a movie in any other way.” Coming from the director who has pushed the art of visual effects perhaps more than any other contemporary director, our team feels it’s one of the highest honors to receive.

Take a look at the video below to see some of our work in action and learn how Google Cloud Platform helped bring it all to life.

Visit to see our recent work. Learn how you can render using Conductor like Atomic Fiction did on “The Walk” at and explore Media Solutions by Google Cloud Platform to see how other companies are tapping Cloud Platform for their media workflows.

-Posted by Kevin Baillie, co-founder and visual effects supervisor, Atomic Fiction

Ever since Google engineers started the work on cGroups, the enabling technology for containers in the Linux Kernel, we’ve led innovation in container technology. In addition to leading development of Kubernetes, the open source container orchestration engine, everything at Google, from Gmail to Maps to YouTube, runs in a container. Each week we launch over 2 billion containers, allowing us to dynamically flex to increase resource utilization and efficiency of our infrastructure.

While containers have been in production for years at web-scale companies, they’ve only recently been popularized and made accessible to all developers. In these early days of container adoption, I hear a few common questions: why do containers matter, how safe are they and how should I use them?

For this month’s Cloud Platform newsletter, I sat down to discuss these issues with some leaders from the container software industry: Alex Polvi, CEO of CoreOS, Kit Colbert, VP & CTO, Cloud-Native Apps at VMware and Craig McLuckie, Product Manager with Google Cloud Platform.

For future content delivered straight to your inbox, subscribe to our newsletter here.

Take a look at a few highlights from our talk below.

Why does anyone care about container technology?

Craig McLuckie: Container technologies have solved a really hard problem around the hermetic sealing and deployment of an application. As we look to the future, every enterprise...needs access to the capabilities that have previously been the provenance of internet giants alone.

What are some myths that we need to bust about containers?

Alex Polvi: [That they are inherently less secure]. What you were doing before containers...was just running a bunch of different applications on one machine. ...If any of those got compromised in any way, the attacker could go get the rest of the server. ...At least with containers you’re putting up some boundaries and walls between every single application.

What needs to happen in the industry that isn’t happening now for containers to take hold?

Kit Colbert: Most of the customers we talk to... [today] are the cutting edge ones...and what you’ll see happening over time is more out-of-the-box solutions. ...Things that just work and are able to let IT teams take it and run.

Learn more about the topics discussed:

-Posted by Cornelius Willis, Global Lead, Product Marketing

Performance is a feature. For many Google applications it is the feature that makes everything else possible—instant text and voice search, directions, translations, and more. The platforms and infrastructure teams at Google are always on the leading edge of development and deployment of new performance best practices. These best practices, in turn, benefit your applications running on Google Cloud Platform.

One great example of Cloud Platform leveraging technology improvements in the Google infrastructure is support for SPDY and HTTP/2. The Google Chrome team began experimenting with SPDY back in 2009 and Google servers were the first to support it. SPDY proved to deliver significant performance benefits and helped shape the development of the HTTP/2 protocol. Now, with HTTP/2 finalized (RFC 7540, RFC 7541) and rapidly gaining client adoption, Google servers are, once again, leading the way by being among the first to offer full HTTP/2 support.

HTTP/2 on Google Cloud Platform
We’re not breaking any news here, because Google Cloud Platform support for SPDY has been enabled for many years, and support for its successor (HTTP/2) was enabled earlier this year. Many performance-conscious developers are already leveraging the performance benefits of the HTTP/2 protocol in their applications, and so should you! Here’s what you need to know:

  1. Your application must run over HTTPS. Support for the HTTP/2 protocol is automatically negotiated as part of the TLS handshake via ALPN.
  2. The server terminating the secure HTTPS connection must support ALPN and HTTP/2 protocols. If either side does not support the new HTTP/2 protocol then they fallback to HTTP/1.1.

If your application is already running over HTTPS, and the secure session is terminated by one of Google servers, then—good news—you’re already HTTP/2 enabled! Let’s take a peek under the hood to see how this works.

HTTP-2 Flowchart - Remove Server Push (v3).png

Google Cloud Storage
Consider a scenario in which you upload a file into Google Cloud Storage and get an HTTPS link for the resource, which you then reference or share with others. When the resource is fetched, the client and Google server negotiate the TLS session and HTTP/2 protocol support. If the client supports HTTP/2 then the Google servers automatically selects it; otherwise the Google servers fallback to HTTP/1.1. In short, just upload a file and provide an HTTPS link to it and your job is done.

Google App Engine
Both in its original incarnation and Managed VM configurations, Google App Engine servers are responsible for negotiating and terminating the TLS tunnel. As long as your application is being served over HTTPS, the Google servers automatically negotiate HTTP/2 on your behalf.

Each HTTP/2 stream is then translated into an HTTP/1.1 request and routed to your application. As a result, no modifications are required within your application to enable HTTP/2. Simply direct your traffic through HTTPS and all capable visitors will be automatically served over HTTP/2.

Google Compute Engine and Google Container Engine
If you’re running a virtual machine, or an entire cluster, via Google Compute Engine or Google Container Engine, then you have multiple options for where and how the inbound connections are routed:

  1. You can expose the virtual instance directly, in which case you must set up and manage the necessary infrastructure to terminate the TLS tunnel, negotiate HTTP/2, and process the HTTP/2 session—see the HTTP/2 wiki for a list of open source servers.
  2. You can use a network load balancer to distribute inbound TCP connections to one of multiple servers, which then have to terminate the TLS tunnel and negotiate HTTP/2—the same as with exposing the virtual instance directly.
  3. You can use an HTTPS load balancer, which terminates the TLS tunnel and negotiate HTTP/2 on your behalf. The load balancer translates inbound HTTP/2 streams into HTTP/1.1 requests and forwards them to one of your servers—same as Google App Engine. This is the quickest and easiest way to enable HTTP/2 for your Compute/Container-powered application.

There shouldn’t be any surprises with any of the above. To enable HTTP/2, you need to enable HTTPS, and then either let one of the Google servers terminate it and do all the heavy lifting on your behalf, or route the connection to one of your HTTP/2 capable servers.

Note: There is much more to be said about the benefits and optimization strategies for HTTP/2it’s a big and important upgradebut that’s outside of scope of this post. If you’re curious, however, check out the free chapters on HTTP/2 and HTTP/2 optimization strategies in High Performance Browser Networking (O’Reilly).

A QUIC peek into the future

The Google Chrome and platform teams continue to innovate by experimenting with QUIC, a UDP-based transport for HTTP/2, which enables faster 0-RTT secure handshakes, improved congestion control, loss recovery, and other mechanisms that promise to further improve performance:

The data shows that 75% percent of connections can take advantage of QUIC’s zero-round-trip feature. QUIC outshines TCP under poor network conditions, shaving a full second off the Google Search page load time for the slowest 1% of connections. These benefits are even more apparent for video services like YouTube. Users report 30% fewer rebuffers when watching videos over QUIC. This means less time spent staring at the spinner and more time watching videos.
                                              - from A QUIC update on Google’s experimental transport

And speaking of being on the leading edge of performance—Many applications powered by App Engine are already speaking QUIC to capable Chrome clients, and we’re hard at work to bring QUIC to all other products as well. As with HTTP/2, the Google servers terminate the UDP flow and translate the QUIC protocol to HTTP/1.1 requests, enabling a transparent performance upgrade to millions of existing applications running on Google Cloud Platform.

Tip: To quickly and easily see what protocol is negotiated, install the HTTP/2 indicator extension for Google Chrome. A blue bolt indicates HTTP/2; a red bolt indicates QUIC.

- Posted by Mark Mandel, Google Cloud Platform Developer Advocate and Ilya Grigorik, Internet Plumber.

Even the most careful developer will make the occasional mistake, and when it comes to security, mistakes can be disastrous. One of our goals is to make it easier for you to develop secure web applications and to find and fix issues early in the development lifecycle.

Today, we are pleased to announce the general availability of Cloud Security Scanner, a tool which enables App Engine developers to proactively test their applications for many common web application security vulnerabilities. For example, it can detect issues like cross-site scripting (XSS), Mixed Content, and Flash Injection or alert you to the usage of insecure Javascript libraries.

The tool is easy to setup and use, and is well suited for the modern, complex, Javascript-heavy applications which App Engine enables you to build and deliver.

Cloud Security Scanner is available free of charge for Google Cloud Platform customers, so please visit to get started.

We’d also like to thank all of the beta testers who have provided great feedback to the product team over the past couple of months. We really appreciate the support.

- Posted by Matthew O’Connor, Product Manager

Cloud native technologies like Kubernetes help you compose scalable services out of a sea of small logical units. In our last post, we introduced Vitess (an open-source project that powers YouTube's main database) as a way of turning MySQL into a scalable Kubernetes application. Our goal was to make scaling your persistent datastore in Kubernetes as simple as scaling stateless app servers - just run a single command to launch more pods. We've made a lot of progress since then (pushing over 2,500 new commits) and we're nearing the first stable version of the new, cloud native Vitess.

Vitess 2.0
In preparation for the stable release, we've begun to publish alpha builds of Vitess v2.0.0. Some highlights of what's new since our earlier post include:

  • Using the final Kubernetes 1.0 API.
  • Official Vitess client libraries in Java, Python, PHP, and Go.
    • Java and Go clients use the new HTTP/2-based gRPC framework.
  • Can now run on top of MySQL 5.6, in addition to MariaDB 10.0.
  • New administrative dashboard built on AngularJS.
  • Built-in backup/restore, designed to plug into blob stores like Google Cloud Storage.
  • GTID-based reparenting for reversible, routine failovers.
  • Simpler schema changes.

We've also been hard at work adding lots more documentation. In particular, the rest of this post will explore one of our new walkthroughs that demonstrates transparent resharding of a live database - that is, changing the number of shards without any code changes or noticeable downtime for the application.

Vitess Sharding
Sharding is bitter medicine, as S. Alex Smith wrote. It complicates your application logic and multiplies your database administration workload. But sharding is especially important when running MySQL in a cloud environment, since a single node can only become so big. Vitess takes care of shard routing logic, so the data-access layer in your application stays simple. It also automates per-shard administrative tasks, helping a small team manage a large fleet.

The preferred sharding strategy in Vitess is what we call range-based shards. You can think of
the shards as being like the buckets of a hash table. We decide which bucket to place a record in based solely on its key, so we don't need a separate table that keeps track of which bucket each key is in.

To make it easy to change the number of buckets, we use consistent hashing. That means instead of using a hash function that maps each key to a bucket number, we use a function that maps each key to a randomly distributed (but consistent) value in a very large set - such as the set of all 8-byte sequences. Then we assign each bucket a range of these values, which we call keyspace IDs.

Transparent Resharding
If you want to follow along with the new resharding walkthrough, you'll need to first bring up the cluster as described in the unsharded guide. Both guides use the same sample app, which is a Guestbook that supports multiple, numbered pages.

In the sample app code, you'll see a get_keyspace_id() function that transforms a given page number to the set of all 8-byte sequences, establishing the mapping we need for consistent hashing. In the unsharded case, these values are stored but not used. When we introduce sharding, page numbers will be evenly distributed (on average) across all the shards we create, allowing the app to scale to support arbitrary amounts of pages.

Before resharding, you'll see a single custom shard named "0" in the Vitess dashboard. This is what an unsharded keyspace looks like.

As you begin the resharding walkthrough, you'll bring up two new shards for the same keyspace. During resharding, the new shards will run alongside the old one, but they'll remain idle (Vitess will not route any app traffic to them) until you're ready to migrate. In the dashboard, you'll see all three shards, but only shard "0" is currently active.

Next, you'll run a few Vitess commands to copy the schema and data from the original shard. The key to live migration is that once the initial snapshot copy is done, Vitess will automatically begin replicating fresh updates on the original shard to the new shards. We call this filtered replication, since it distributes DMLs only to the shards to which they apply. Vitess also includes tools that compare the original and copied data sets, row-by-row, to verify data integrity.

Once you've verified the copy, and filtered replication has caught up to real-time updates, you can run the migrate command, which tells Vitess to atomically shift app traffic from the old shards to the new ones. It does this by disabling writes on the old masters, waiting for the new masters to receive the last events over filtered replication, and then enabling writes on the new masters. Since the process is automated, this typically only causes about a second of write unavailability.

Now you can tear down the old shard, and verify that only the new ones show up in the dashboard.

Note that we never had to tell the app that we were changing from one shard to two. The resharding process was completely transparent to the app, since Vitess automatically reroutes queries on-the-fly as the migration progresses.

At YouTube, we've used Vitess to transparently reshard (both horizontally and vertically) nearly all of our MySQL databases within the last year alone, and we have still more on the horizon as we continue to grow. See the full walkthrough instructions if you want to try it out for yourself.

Scaling Benchmarks
The promise of sharding is that it allows you to scale write throughput linearly by adding more shards, since each shard is actually a separate database. The challenge in achieving that separation while still presenting a simple, unified view to the application is to avoid introducing bottlenecks. To demonstrate this scaling in the cloud, we've integrated the Vitess client with a driver for the Yahoo! Cloud Serving Benchmark (YCSB).

Below you can see preliminary results for scaling write throughput by adding more shards in Vitess running on Google Container Engine. For this benchmark, we pointed YCSB at the load balancer for our Vitess cluster and told it to send a lot of INSERT statements. Vitess took care of routing statements to the various shards.
The max throughput (QPS) for a given number of shards is the point at which round-trip write latency became degraded, which we define as >15ms on average or >50ms for the worst 1% of queries (99th percentile).

We also ran YCSB's "read mostly" workload (95% reads, 5% writes) to show how Vitess can scale read traffic by adding replicas. The max throughput here is the point at which round-trip read latency became degraded, which we define as >5ms on average or >20ms for the worst 1% of queries.
There's still a lot of room to improve the benchmarks (for example, by tuning the performance of MySQL itself). However, these preliminary results show that the returns don't diminish as you scale. And since you're scaling horizontally, you're not limited by the size of a single machine.

With the new cloud native version of Vitess moving towards a stable launch, we invite you to give it a try and let us know what else you'd like to see in the final release. You can reach us either on our discussion forum, or by filing an issue on GitHub. If you'd like to be notified of any updates on Vitess, you can subscribe to our low-frequency announcement list.

- Posted By Anthony Yeh, Software Engineer, YouTube