Google I/O 2012 – Gaming in the Cloud

Google I/O 2012 – Gaming in the Cloud


FRED SAUER: OK, welcome
everyone. It’s a great third day of
an excellent conference. Hopefully you’ve had a really
good time so far. And we’re in the home stretch. I think there’s one more block
of sessions after this. I’m excited to show you a couple
of demos today and walk through some code. Let’s get started. I want to give you a little
bit of background– how I came into web games
and the stuff I work on. So back in 2007– this is before I
joined Google– my son and I wanted
to build a game. And so we sat down. He did some artwork. He was five at the time. And browsers– well, they weren’t that good. Chrome wasn’t even out. I probably got, I think,
initially 10, 12 frames per second on this game. I just used CSS to move
sprites around. So let me show you what
the game looks like. It’s been preserved
in history. [VIDEO GAME SHOOTING
SOUND EFFECTS] It has some sound effects. Actually, let’s turn
it up a bit. Now, this is a quality of, I’ll
say, state-of-the-art web game circa 2007. Lots of fun, but I think
we can do a little bit better than that. Now, last year at the keynote,
you saw Chrome Angry Birds, an HTML5 game using WebGL. Very fast, amazing, much better
than, I would say, that 2007 game that I worked on. And I’m not going to bore you
with playing a lot of this game, but I just want to show
you a little bit of what’s going on here. [VIDEO GAME SOUND EFFECTS] Now, pretty good, huh? Pretty good? Yeah, OK. Now, if you played that game
last year at I/O, you probably realize that the game
has changed a lot. It has a lot of new levels. It has episodes. There’s more functionality. Your progress is now
saved in the cloud. That’s possible in a web game. That wasn’t possible back
in the day when we did shrink-wrap games. If you bought a box of
software, you took it home with you. That’s pretty much the version
of the software you had for the rest of your life. Our phones don’t operate
that way anymore. The phones you got the other
day probably have already received an OTA and are in a
newer version than when you unpackaged them just the
day before yesterday. So with Chrome Angry Birds,
we released a framework called PlayN. And that’s not really
what I’m going to be talking about today. But I wanted to stick this slide
up there and show you that the community
has stepped up. There’s other games out
there that are fun. | like this one that
came out earlier. Just a nice little classic. And I’m going to die pretty
quickly because I’m no good at this game. Yeah, no. So watching me play games for
the rest of the hour is probably not what we
want to be doing. So as a game developer, you
really need to know a lot of things about backends
and frontends. You need to worry about
scalability and authentication, making your
game social, monetization, persisting player data. Oh, and you have to make the
game playable and fun, and have good graphics, and
do all that stuff. So let me show you the sample
game that we built this year. It’s not quite the quality
of some of the other games out there. But we wanted to give you a
showcase game, something that you could open up
the source code. And you could look at it, and
you can copy bits and pieces. And you could get started, kind
of like your first kit for building the game. So I’m just going to launch
that game here. We integrate with– we have authentication. Let’s see. I’m not actually logged in, but
I think I can take care of being logged in. Very secure password. All right. I’ve got a list of friends. Now, this is a multiplayer
game. I’m not sure if anyone’s
on at the moment. Let’s see what happens. At least I’ll get in. It looks like it’s just
me right now. There’s a light team
and a dark team. And I can walk around. I can shoot things. We have sound effects. And the opposite team, they
would appear right here. And then we would go after
each other and shoot each other up. The pictures that you see
here are the party– the six of the willing, those
people that, 120 days ago, decided that we’d build
a web game together. Most of these folks were working
on the frontend. A couple of us were working
on some backend stuff. I take no credit for
all the beauty. That’s the graphics. I’m a backend guy. Now, on day one, there
was a talk by Colt all about the frontend. If you missed that,
catch it online. This talk is the complementary
talk, which is about backend pieces. Now, when we set out to build
this game, called Grits, we had to choose a development
stack. And there’s a lot of
things that go into a development stack. But ultimately, one of the big
choices you’re making is the language that you’re
going to choose. If you’re using PlayN, you’re
using Java, and you’re compiling to multiple
platforms. And we thought we’d do
something different. We wanted something that
was very approachable. We wanted to build
an HTML5 game. We wanted to use pure
JavaScript. We wanted to reuse
the JavaScript source code on the server. And so when you think about your
next project, think about all these things that
are influencing your choice of a platform. And you may pick something
that’s really relevant. Or you may pick some random
thing like the hot new thing that you want to learn
and work on. We decided to go for developer
approachability. Now, my background is
server-side infrastructure. I work on App Engine. And for this project, we used
a combination of App Engine and Compute Engine. The parts that we could put on
App Engine so we didn’t have to write a lot of code,
we put there. The node.js server that we
wanted to kind of run the physics on and share code with
the clients, we put on Compute Engine inside of VM. So let’s take a closer look
at that architecture. You saw the web game, which
is running in a browser. I’m later going to show you an
Android controller app that connects to the same
game as well. That’s pretty cool. Now, on App Engine, we do the
static content serving, so we just host up the basic site. We have what are called
frontend instances. This is the instances where
our application code runs. And they reply to requests from
the browser, like find me a game that I can drop into. And then App Engine responds
back and says, the game is available at this IP address
on this port. Go for it. And then we have a special type
of server on App Engine called the backend. This is a stateful server
that’s a little bit more long-running. And what its job to do is
to keep track of all the different games that are
currently ongoing. And actually, I think
I can pull up the current state right now. There’s a debug URL. And here you see a big
JSON response. Right down at the bottom here–
let me increase the font size a bit– you see players_waiting, none. So there’s no players
currently waiting to get into a game. If I go to the very
top, there’s a list of game servers. And they all have randomly
generated unique strings to identify them. And then each game server has
some state associated with it. In this case, this game
server here has a number of running games. There’s a game, 9ZXL,
going on right now. It’s very exciting. There’s four people in there. Two of them logged in, and
two of them anonymously, identified by the bots. And we have another
game here, mA32. Anyone in that game? No, must be the room
next door. They have quite a few
players going on. So this is just the state that’s
currently represented in this App Engine backend. And any time a player comes
in, they kind of ask this backend for some information. Where can I play? Where’s an open slot? App Engine assigns a slot,
keeps track of the information. So all that’s going on there. And we’ll talk about that
a little bit more. And then on Compute Engine, we
have the actual games running. This is where the logic and the
physics replay happens. And then both our web browser
and later our Android client, they use WebSockets to connect
to these node.js instances. And then the Compute Engine and
App Engine instances, they have to communicate
with each other. They have to keep
track of state. They kind of share the global
state of the system. And so they do a lot
of JSON message passing back and forth. And the parts that I’m going to
talk to you about most are this Android controller and then
most of the App Engine backend stuff. Now, when we set out to choose
a wire protocol, we had two top-level options
in front of us. We could use someone
else’s plumbing. There’s some good
options there. There’s App Engine Endpoints,
which you heard about at this conference. There’s Google Plugin for
Eclipse that provides some synchronization options
and does a lot of the hard work for you. You should use those. Like all engineers, though, we
thought we could just write our own plumbing. Because why use someone
else’s? So we started out with
some basic options. We could do XML– kind of wordy, very chatty,
a lot of bytes. We could do JSON. And we could roll our
own protocol– nice and compact, a little
hard to decode, but probably efficient. And if you compare those, you
see that, for the high bandwidth stuff, we should
probably choose a protocol that’s much smaller
and tighter. And so we ended up rolling our
own protocol that’s fairly compact, but still
human readable. And then for the
server-to-server communication that’s a little bit more low
bandwidth, we chose JSON just because of its ease of use. And XML– no, we didn’t do that one. We could have gone just
a little bit tighter. You see some information here,
four Booleans and a number. We could have actually encoded
that in a single byte. Three is 0011 in binary. And then the four Boolean
flags we could have encoded as bits. This would have been really
hard to debug. It would have been slightly more
efficient, but a little bit overkill for this example. We wanted to be able to look
at messages and understand what’s going on. So for Grits, we chose the
semi-dense, roll your own protocol, and then JSON
for everything else. Now, when we started out, we’re
debugging things, we’re looking at the console, and
we’re looking at JSON that looks like this. And then you’d make a tweak. And then you look
at some JSON. And it’d look just like
that, except it was slightly different. And it was really hard to figure
out what was going on. And we said, this
is not working. If we’re going to use JSON, we
actually need this stuff formatted in a way that
humans can read. We want something that looks
more like this, because then you can actually compare
differences and see what’s going on. So on App Engine, we used the
JSON library, of course. And we did a little something. We have a debug flag. And when we turned that on, we
changed our JSON encoder. And we indent everything
before spaces. And we sort the keys. So that the controller host here
that you see, that first entry, is always the first
thing that shows up. It’s not random depending
on the order of the internal hash. And then we have a couple of
helper functions, tojson method for taking a payload,
turning it into JSON, taking a Python object. And then fromjson to
do the reverse. And it does some exception
handling. And if for some reason it gets
a message back that’s not JSON, it actually prints out
the payload that it got. So we can figure out
what’s going on. In the beginning, a lot
of that was happening. Now, we wanted to make life a
little bit easier as well. We created a JSON handler that
took care of the boilerplate. Every single time we had a
request handler, we needed to inject a JSON payload, do
some work, and then spit out a JSON response. And so you see that here. We have a post handler. It logs the information that’s
coming in, throws an exception if there’s no content. It parses the payload
that’s come in, and it handles response. This is delegated
to a subclass. And then we spit out the right
content type and encode the response and send it back
to the requester. And if there’s any kind of
exception, we print a nice pretty stack trace that you can
actually see on the other end of the connection so
you can debug things. And here’s that handle method
that was being called above. And this is something that you
implement in every single implementation. So the payoff– you’ll just have to trust me. Before, the code looked
like that. After, it looked like that. No, let me show you what
it looks like. So here’s the code without
the helper functions. The blue stuff is really
what we care about. We parse something, and we
spit out a response. Afterwards, there’s a lot
less boilerplate. Everything’s taken care of. And this code [? savings ?] repeats itself throughout
the code. We also turned on the new
PageSpeed service that was announced at this conference. It has a whole bunch
of rewriters. It’s an experimental feature
in App Engine, but it’s something you can
try out today. And by default, it has a number
of rewriters that it turns on that it believes are
safe for almost every single site out there. So like getting rid of
whitespace in CSS and HTML, getting rid of comments, those
things are generally safe and won’t break anything. There’s a few rewriters that
are a little more advanced, like combining JavaScript
files, inlining JavaScript in CSS. And depending on your
application, they may or may not be safe. And so we provide a way to
customize that for you. In your app.yaml, which is your
main configuration file on App Engine, you
add a special block, a pagespeed block. And there, you can explicitly
indicate domains that you want to rewrite, blacklist URLs. And then you can enable
rewriters that aren’t on by default, or you can disable
rewriters that normally are on by default. But if you leave this block
out and you just go to the admin console and you
check the box, we’ll do the default thing. And for most applications,
it’s the right thing. And then the amount of traffic
after a few iterations will get faster. So what happens when you flip
this bit, the first time you hit your website, it will
actually be the same slow, large payload that
you saw before. But behind the scenes, the
PageSpeed service is fetching all the content, rewriting it. And then when you come back
shortly thereafter, it will ship down the rewritten bites,
which are nice and compact and a little bit more efficient. And hopefully, your users’ load
times will go down, and your bandwidth costs will
be affected as well. So one big pain for games that
usually developers delay to the last minute is identity
and access controls. This is always a pain point
of integration. Security is a hard thing. Luckily, we have some existing
tools that we can use in App Engine. There’s a built-in Users API. This is great for sites that
use Google accounts. You get this full-page redirect,
so you don’t have any kind of branding
on the site. And users have to authenticate
themselves every 30 days. So this might be OK for
certain style apps. But for games, it’s probably
not the best experience. It’s fairly easy
to use, though. You just have your
regular handlers. And you add one line of code
that says, Login required. And then App Engine will
take care of the rest. It will create these
interstitial pages, make sure that users cannot get to your
site, or at least to these URLs, until they are logged
in with a Google account. Then, in your main.py, you
use the user’s API. And you can just say
users.get_current_user(). And that object will either be
none if the user’s not logged in, or it will be something. And you can get the name of the
user or the email address. And if they’re not logged
in, you can send them to a login page. And then you pass in the page
that you want the user to come back to after they’ve
logged in. So that’s pretty easy to use. But we decided to use OAuth,
because we get a better user experience. We still have our branding. Our page is up. Our game is up. And then there’s a popup page
with the OAuth flow. So this is where the user can
indicate whether or not they’re willing to
share their email address, share their profile. Whatever the permissions are
that you’re asking for, this is what the dialog
box will show. Now, once the client’s
authenticated, it gets a special access token. And when you have that access
token, and you pass it into App Engine, App Engine can
extract the access token, validate that the access token
is correct, and then give you back the username
for that user. And so it looks very much
like the Users API. Instead of
user.get_current_user, you say, oauth.get_current_user. And then you pass in a scope. And the scope is one of the
permission scopes that you’ve asked for when you created
that token initially. The nice thing about using the
email scope in this case is that the user ID that you’ll
get back is the exact same user ID that the Users API
would have given back. This is useful if you have two
parts to your site, one that is maybe for administrators
or uses the Users API to prevent login. And in another place, you have
maybe mobile devices or you use OAuth for some other
reason to get in. Now you can correlate these two
user objects by lining up the user ID. Then good debugging tip. When you have these access
tokens and you don’t know why they’re working or what they’re
for or how you got them, you can just make
a call to this URL. And Google will tell you where
the access token– who it was issued by, what it’s
for, when it expires. All that good stuff. So it’s a really useful
debugging tool right there. Now, OAuth2 has many
different flows. So I wanted to just walk through
what the flow is like in our application. We have the client, which is the
web browser in this case. We have App Engine hosting the
content and the services. And then we have the Google
Authorization and User Info Services over there
on the right. So what happens is, the client
grabs the initial page. They download the game. And then the user clicks
the login button. We send them to a login
OAuth page. And App Engine redirects them
to this longer URL, which points at the authorization
servers. There, they get this
dialogue box. And the user clicks
Allow Access. The servers generate a
unique access token. And then they send a redirect
to the URL that we specified initially, which is this
other page, logup. And it passes in the access
token and the hash fragment. So we can extract that
on the page. And then we can use that
access token to call this user info URL. And that allows us to get
information about the profile of the user. And then sometime later,
the game calls– when they click Quick
Game, we call a service called findGame. And we pass in the
access token. And that’s how App Engine knows
which user is asking to login to the game. And so App Engine does this
access token verification and extracts the user ID. And we can go on. Now, in Grits, kind of the
middle section, the App Engine section, looks something
like this. We have a Grits service
handler. This is where all those Grits
requests come in, so grits.findGame, for example. And in the post, when we get the
HTTP post coming in, the first thing we do is
determine the user. And that function is
below, right here. And this is a little tricky bit
because, in our game, we allow users to login anonymously
by clicking Quick Game, or click the login button,
go through the OAuth flow, and actually identify
themselves. So we have those two
things going on. And then at the same time, we
want all this to work in production. But we want to be able to do
some easy debugging on the local development server, so
when we’re coding on a laptop. So we really have four different
scenarios going on. We have the Quick Game
and Login both in production and locally. And so the block of code
below here handles all four of those cases. So the first thing we do is
say, if we’re in a local development server, then we
allow the request URL to have a user ID passed in. And we will accept it blindly. We’re in development. We’re on our own machine. We’re not worried about
any kind of security. We just want to test out the
display of names or user IDs or do whatever we want. So in development,
anything goes. If we’re not in development, and
the user ID starts with a special string, bot*, that’s a
string that we’ve whitelisted. And that’s the string
that’s generated by this Quick Game button. If the user ID starts with
that, we allow it. So any user can specify
a name, as long as it starts with bot*. If that’s not the case, then we
do the OAuth look up that you saw earlier. And if we get any kind of OAuth
error because the token was invalid or something like
that, then we throw an exception back to the user. So now we have all four of
those scenarios covered. And we know what the user is, we
have a user ID, and we have a display name. The user ID is the internal key
that we used to save data in the data store. And the display name is what the
users will see above their player on the game. Now, this game was fun. And WebSockets and JSON,
all of that’s fun. But I thought it’d be a little
bit more interesting to get your hands off the keyboard
and use the latest Android toy. So we all have pretty fancy game
controllers, probably in our pockets, if not
in your bags. They have accelerometers. They can do really
cool things. And some of them can even
make phone calls. So what you see here is
a spinning globe. Represented on it the
average gravity by location on the planet. And you see that it
varies a bit. Places where there’s mountains,
there’s actually a little bit more gravity than
in lower-lying areas. Luckily, these variances
are not that great. And they won’t really
affect the next steps in the gameplay. Although players on high
mountainous regions might have to tilt their phone a little
less than we will here at the conference. The average pull of gravity is
about 9.8 meters per second per second. And we’re going to use our
phone’s sensors to measure this gravity, measure the
direction of the gravity, and use that to control the
character in the game. So I created this app called
WASD, W-A-S-D, after the four keys that are used
in this game to control your character. And it’s a simple little app
that looks like this. And let me show you what
that looks like. Now, there’s four
buttons on here. But if I press in the buttons,
you see that nothing happens. I didn’t wire them
up that way. I wired them up to the
accelerometer. So when I tilt the tablet, the
different buttons light up. And so the idea is that you just
rock this tablet around. And that’s how you control
your character. So let’s see what
that looks like. Hopefully, the network gods
will be friendly to us. So I’m going to login
to the game. I’m going to be sure to use the
OAuth flow so that I’m an authenticated user and the App
Engine will be able to marry up the player right here
on the laptop and the player in the game. I’ll do a little split screen. You can look at either screen. So we have the player
dropping in. I’m logged in as me. Here on the tablet,
I have a chooser. There’s two accounts associated
with this device. I’m going to pick my
Gmail account here. You can see a couple
of popup messages. It’s getting an OAuth
access token. It’s looking up a user ID. And it has a connection. It says, yay. Oh, someone else joined us. Network gods are not playing
good with us yet. I’ll try to reconnect, see if
we can make this happen. [? See if ?] [? this character starts in ?]–
there we go. So isn’t that a lot more fun? Except I haven’t implemented– [APPLAUSE] So there’s one problem– there’s
no fire button yet. So it’s not quite fair. All right. So that’s this little app. And I want to show you a little
bit of how that app came together. One of the key things that we’re
doing, of course, is measuring that gravity and
measuring acceleration. This could be because I mount
the phone in a rocket and blasted off to the moon, or
because gravity is pulling. The two both measure the same. The thing that we do is we
implement this sensor event listener interface. And then there’s a special
callback method called onSensorChanged. And inside that method, we
check the current display orientation. So depending on your– you know, when you rotate your
app and the display of the app rotates with the device– the acceleration is always
measured in one direction. But the app can be any
of four directions. And so there is a little bit of
simple math that we have to do to figure out either take the
positive x or the negative y or whatever the
direction is. To simplify things in this app,
I’ve just hard coded the directions so this app always is
landscape in one particular orientation. This Assert statement is here
just to make sure that that’s the case and I’m not
messing things up. So here, you see that math
I was talking about. The sensorX is the second event
value in a negative, and sensorY, the first one
is a positive. If the rotation of the device is
one of the other four, then the minus sign there and the
plus sign, they kind of alternate and switch around. The full source code will show
you what that looks like. In any case, every time we get
a sensor changed, we figure out what the x and y direction
are on the axis of the application display. And then we call this
updateDirections method. And then, in order to make sure
that we’re saving battery life, we have to make
sure that we have start and stop methods. So that we can shut down the
accelerometer and spin it back up when we need to. So we’ll call those methods
in a minute. Now, when we first started
doing this– essentially, the numbers that
you get back is, as you tilt the device, you go from seeing
zero acceleration when the device is completely level to
9.8 meters per second per second when the device is
vertical, and negative 9.8 when the device is
the other way. The same thing happens in the
y direction like that. So if you’re holding the device
level, you see about 0. But the sensor values actually
fluctuate a little bit. They move around. There’s a little bit of
noise in the sensor. And of course, you’re not
holding it entirely steady. So in the first iteration, I
picked a value just by trial and error, and said, OK, well,
at about 2 meters per second per second, the device is
tilted about right. And so if I get a reading that’s
more than 2 in this direction, I move right. And if it’s less than negative
2, then I move this direction. Unfortunately, what you get is
that if you’re very close to that 2 value and you get that
noise, the character will go right, stop, right, stop,
right, stop, and kind of jitter around. And you end up having to push
the sensor a little bit further to make sure that the
noise doesn’t get you back. So to compensate for that, you
can do some low-pass filters and some fancy math. But what I did is something
a little more just straightforward here and
introduced some stickiness. So you see that represented
in this bar here. Roughly between 0 and 2.8
is this blue section. And if the reading I get back is
between 0 and 2.8, I treat that as a key up. In other words, that key is
not currently pressed. And I have to go all the
way to 3.0 before I trigger a key press. And then everything from 3.0
above is key pressed. When I come back down, though,
instead of turning things off at 3, I don’t turn them
off until I reach 2.8. So the trigger is at a different
spot than the release is. And this makes the game
a lot more playable. That way, once I tilt it, as
soon as I get to that 3, then it’s kind of sticky there
until I go back to 2.8. And those numbers are kind of
arbitrary, but those are a little bit of trial and error. Now, I do that in all
four directions. There’s the stickiness
on both x, y, and up, down, left, right. So here’s what that
looks like. We have the updateDirections
method. We pass in the sensor x and y. And here we’re looking at the
threshold to see if the x is greater than 3. That means I’m over the
red line there. In that case, I want to turn
the left button on. Now, if the sensor is less than
that threshold, minus the stickiness factor, then I turn
the left direction off. I repeat this for
the opposite. I repeat it again for the y. And then, at the very end, I
light up the buttons and say, if you’re going left,
do a bright color. And if you’re not, do
a darker color. So then in the main activity– this is the thing displaying the
app, the screen where you see the buttons. That’s the activity– there’s an onCreate method. And when we first get here,
there’s a few bits of information we need. We set a flag to keep the screen
on, because I’m not touching the screen or
interacting with it. I don’t want it to
go to sleep. So I do that flag. And then I get some shared
preferences where I can store which user ID the user wants
to be logged in with. I get the account manager so I
can get identity information and get OAuth keys. I get the sensor manager so
I can get access to the accelerometer. I get the display so I can
figure out the rotation. And then I set the content
view, which is, light up the display. Put the layout on there. And then you can find
the different buttons on the display. So in Android, you lay out
your display either in a WYSIWYG layout or
in XML directly. And then you have unique IDs
for all the elements. And after you display those, you
can go find them and get access references to
those objects. And then we spin up our
sensor listener, which we’ll start up. Now, this is the bit about
saving battery. In the activities onResume and
onPause methods, we spin up and shut down the sensor manager
to save battery power right there. So if another popup comes up,
like a text message, or the user moves over to another app
and then comes back, we want to make sure to shut
down the sensors. Now I’m going to talk a little
bit about how the game is wired initially, and then how
I wired in the controller after that. So we have up to eight
players in the game. And each one of them has a
WebSocket Socket connecting to the game engine on
Compute Engine. And we’re using socket.IO, which
has endpoints, kind of like URL endpoints. And you saw those four character
random games before. So I picked a game
here called abcd. And they’re all connected
to the same game. And each of the players, as
they move, they send up physics and position information to the game engine. So each of the players will say,
my x and y is here, my velocity is this, and by the
way, I just shot off a bullet. And the game engine will compute
what’s going on and then tell all the players about
what actually happened. And then each of the games is
actually predicting its own physics, but listening
to the server. So the server is
authoritative. So each of these players
has their WASD controls on their keyboard. And when they hit one of the
controls, the game logic fires off the necessary events
up to the game engine. And then the game engine fires
off all the events back down to the different players. And this is happening for all
players, all the time. So there’s a lot of traffic
going back and forth. Now, we want to have an Android
controller there. And when I started out doing
this, I was trying to replicate the game logic inside
the Android controller. And that meant I would have
to worry about physics and position and have all of this
rich knowledge about the game. I didn’t want to
do any of that. I’m way too lazy. I wanted to just send keystrokes
for left, right, up, down, and be done with it. I didn’t even want to implement fire, as you noticed. So I decided to create another
WebSocket endpoint and actually create one
for each player. So you see these here. They’re abcd!1, 2, 3, 4. And these are dedicated sockets
that four different controllers can connect to
each of these sockets. Now, here’s the way it works is
the Android device connects to this other socket. And over this channel, it sends
a very simple protocol. It sends, I think, the number 3,
a colon, and then it sends four characters that are either
y or n for the four different directions. So four Booleans. The game server reflects that
back up the normal abcd channel, but in a different
protocol. This is the game protocol which
has a lot of complex messages and is something that
I didn’t want to implement. So this is how I get around
having to implement the protocol is I have the
game server do the translation for me. So I’m on my Android device. And I tilt it, and I send
out a keystroke. And it goes up into the cloud. And the cloud reflects
it back down. It goes down to the browser. And then inside the browser,
rather than integrate into the game, I decided to do something
a lot more hacky, which is just to synthesize
key events and then inject them into the DOM. And that worked beautifully. So I’m going up to the cloud. It reflects down
to the browser. The browser interprets those
four flags as key up and key down events, injects those
into the page. And then the game logic just
takes over and thinks that the player pushed one of the keys. And it then sends information
up to the game engine, which interprets and then sends it
back down to the client. So I’ve been doing an
extra round trip, but it’s a lot easier. So I got away with it. So here’s what the injecting
the keystrokes looks like. In the browser, I have a
little function, wasd. And it gets messages that
look like this. After they’ve been unpacked out
of this dense format, they turn into these little
JavaScript objects. And here I have wasd with
true and false values. And so I just go in a For loop
for all the keys inside of this message. I create an event. If the value is true, I create
a key down event. If it’s false, key up. And then I set the key code,
either W, A, S, or D. And then you dispatch the event. And the game loop
will take over. So that’s my little game
controller hack there. For OAuth, in order to bind this
device to the browser, I’m having the user login with
the same Google account in both places. In both cases, I’m using
OAuth on App Engine to extract the user ID. So this is what it
looks like here. Inside the Android activity,
we get the account manager. Here, I’ve hardcoded it to the
first account, which is the primary account on
the device, 0. And then there’s a built-in
method called getAuthToken. It gives me an OAuth2 token if
I specify an OAuth2 scope. In this case, I’m using
the email scope. So those user IDs
line up nicely. And then in the callback, I can
access the result, grab out the access token. And then I can call
my App Engine API. And I pass in that special HTTP
header, Authorization, Bearer, and then the OAuth2
access token. And App Engine will interpret
that, figure out the user ID, and give that back to me. Now, if the user clicks No
Thanks on this page that pops up, then I’ll actually
get an exception here, operation canceled. And I have to figure
out what to do. I guess I can’t continue
as the app. And I can tell the user,
well, sorry. You can’t use this game
if you don’t give me access to your account. But that is a condition
we need to handle. Now, there’s a couple things
that we didn’t take care of in this game that, if you’re doing
this in a real game, you should worry about. The confused deputy problem
is one of them. And I give you the Wikipedia
link to look this up. But essentially, right now, this
application is requesting a token that proves that
the user was there. But we’re not currently
authenticating that this token actually originated from
my Android app. You could build an Android app,
get one of these tokens. And then you could call
my App Engine service and give me the token. And all I know is that that same
user was behind your app. But I don’t know that they
came from my app. So that’s actually a security
problem here. And it’s called the confused
deputy problem. And there’s some workarounds for
that, but for this demo, we didn’t do it. The other thing you need
to do is that these access tokens expire. So when you look at these
examples, they work great. You go to bed. You wake up in the morning. And then the tokens
stop working. That’s because you need
to expire them and then get new tokens. What a lot of people do is they
just always expire the token before they get a token,
rather than trying to make a connection, and then if it
fails, expire the old one and grab a new one. You can do either way. It’s up to you. So we are using a number of
services on App Engine that we looked at so far. We have the PageSpeed service
for optimizing the content. We have the App Engine
frontends where we run our code. We have the High Replication
Datastore where actually persisting data, player data,
number of wins, number of losses, number of credits
for virtual items– which you can’t buy just yet,
but that’s coming soon. And then we’re using the OAuth
APIs to authenticate users. Now, the High Replication
Datastore– there’s many talks that you
should probably look at if you want to start using it and
fully understand it. A really good one is this one
from Google I/O last year, More 9s Please, talking about
the differences between the Master/Slave Datastore and the
High Replication Datastore, and all its advantages. What we’re using it for is
really simple stuff. We’re storing some global
configuration information. These are OAuth client IDs and
secrets, and then some user information– the wins, losses, and credits. And I mentioned the
virtual items. So the way we do that
is we have a little Python class, db_api. And we’re using NDB, which is
Python’s new database module. It does some fantastic
magic for you. It caches in memory inside
the instance. It caches inside a memcache
before it accesses the datastore. So if you use NDB, it’s
definitely an upgrade from DB. It’s a better API. And it reduces the number of
datastore writes that you have and datastore reads, because
we have this extra caching layer built in. And the use is basically
the same. For this user class, we
have a display name and a create date. And this is all we have to do
to declare the class, just these two lines of
code right here. We also add in number of
wins and number of losses, and then credits. So we have five properties
total storing all the information about the user– sorry, six. And we have these hidden
virtual items that you can’t buy yet. But these are essentially
weapons upgrades and things like that. So as your scores increase,
you get credits. And then with credits, you
can buy virtual items. Now, managing all these
different servers– we have multiple game servers
running, we have App Engine– is quite a bit of work. And I wanted to give you a
little bit of insight of what that looks like, what the JSON
flows look like, and the communication between the
different pieces. So we have, potentially, an
administrator, one of the game creators that’s trying to look
at the game, debug things. We have the actual player. Then we have the App Engine,
which has zero if there’s no traffic, or many backends if
there’s a lot of traffic at that moment. We currently have one App
Engine backend that is coordinating all the
matchmaking. And then on Compute Engine, on
each instance, we have a node.js controller, and then
potentially many games. And all of these components need
to talk to each other and exchange information. So what happens is each time a
node.js instance starts up, in its initial startup script,
it calls home. It calls the App Engine
matchmaker. And it registers itself. And it’s basically telling App
Engine, hey, I’m available. I’m ready to spin up
games for you. Whenever you’re ready,
send me players. Then there’s a player
that comes in. They load up the page. So they go to index.html. They get a bunch of resources. And eventually they
click Quick Game. And they call this Grits method,
this findGame method. And they talk to one of the
frontends, which just passes the information on
to the backend. The backend then goes and
pings all the node.js instances very quickly to find
out if they’re all still alive and healthy. And it picks one at random. And it calls start-game,
requesting it to start up a game. If the instance rejects, then
it just goes on to the next one and calls start-game
there. But assuming it accepts, that
instance will create a brand-new game, empty game
with no players. That information is given all
the way back to the player. And the browser opens a
WebSocket connection directly to the node.js instance using
the IP address in the port provided to it and the name
of the game, which is this four-character random
sequence. So at that point, that
player is in the game, playing the game. Other players follow
this flow. And they’re all communicating
directly to this backend instance. Then as players drop out or the
game finishes, we’ll get this update-game-state message
back to App Engine, giving us the summary of what happened. How many kills? How many wins? How many points did
players do? How many games did they play? And then the game-over URL when
the game actually ends. Separately, the admin might
do some debugging here. We have a couple of callback
hooks like list-games to see how many games are active. We have this debug call, which
is the big JSON block I showed you earlier. And then we can talk to the
individual node backends and get some login information. So here’s that big debug section
you saw earlier. Let’s see list-games,
what it shows us. OK, so this gives us– let me see. There’s one server right here. There’s a game. There’s a second game. And that’s it. And if we look at one of these
games right here, you can see the different information. So this is the control host. This is where the main
instance is running. This is the WebSocket URL where
the game is running. This game supports a
maximum of eight players, minimum of one. And currently, there are three
players inside this game. And it’s been running for
173 seconds right now. So there’s some happy
players right now. Let me get through
this slide again. So that findGame call invokes
this matchmaking service that I’ve been talking about. And so I wanted to walk
through that real briefly with you. We’ve done something
very simple. We’re not doing anything
skills based. We’re just allotting players
to different slots. So we have this queue
of players. And it’s a first come,
first served. As you drop into the queue, you
have the first opportunity to play the game. So a player drops in. We get a couple more. Then we get a few
more players. And now, suddenly, we have a
whole backlog of players. So we call this startGame
method. Start up a game. Minimum of two– so we’re going
to take two players out and shift them into
the first game. We slide everyone up. And we’ll repeat this process. And we’re constantly trying
to fill up new games. So now we have three different
games going, two players in each. And we only have one
player left. Now, if we drop this player into
a new game, they wouldn’t be able to play. Because they would have to
wait for someone else. So what we’re going to do
instead is we’re going to slide them into one of the
remaining slots and have them drop into an existing game. So that’s the entire
matchmaking service right there. So in code, what does
that look like? Well, we do this two-pass flow
that I talked about. If there’s no players waiting,
there’s nothing to do. Otherwise, we do a first pass,
which is we match players based on the minimum number
of players in the game. So that’s two per game. And then we go across until we
can no longer fill up slots. And then we do the second pass,
which is do matchmaking based on the maximum
number of players. So in this phase, we’re filling
everything up until eight players. Now, I’ll skip this
min method. You can read that later. Now, let’s say we want to
scale things up a bit. I said we had one
matcher backend. And that can handle quite
a bit of capacity. But if this game became really,
really popular, we might actually want
more backends. Now, App Engine already takes
care of frontends for us. As traffic comes in, it will
spin up additional instances. And before you know it,
you might have a large number of instances. Backends are a little
bit different. They’re designed for long
running jobs or jobs where you want a specific server that
you can identify. But backends do have this
feature where you can make them be dynamic and behave
like frontends. But at the same time, backends
know which backend number they are, which backend instance
they are. So you can make them
kind of sticky. So we can imagine doing that. So let’s say we have
one backend. And we have all the traffic
going there. We decide we need more. So we would change our
backends.yaml. And we’ll say we’ll have three
instances instead of one. And we specify that they’re
dynamic, which means App Engine will spin up more as
the earlier ones get busy. Now what happens here is that
now, traffic is being routed all over the place. Any time you call a
backend, you get routed to a random backend. And that’s not really what you
want, because we want players that are coming in to always
reach the same backend. And we can do that by specifying
a different URL. We can specify the instance
number, 0, 1, or 2, in front of the backend name, matcher,
in front of the app ID, gritsgame.appspot.com. So what that looks like in the
backend is when one of these random calls come in,
we can actually call backends.get_instance(). That way, we know what
instance we are– say, 2. And then we can create a
new URL based on that. So we get one of those
2.matcher.gritsgame.com. And in our response back to the
client, we can tell them what URL to use next time they
call home so that they contact the same person. And so now we have dynamic
backends that scale up and down with load. But as users connect to
backends, they’ll be told to reconnect to the same backend. So we still get the stickiness
benefit. And then things look
much nicer. They look exactly
the way we want. Now, when you’re building
games, one of things you really need to do is
do load tests. Unless you’re just building
the game for you and three other friends. Hopefully, that’s the weekend
project, and you’re thinking about something else. A lot of people start
out doing load tests essentially like this. They have zero traffic. And then they flip
on the load test. And they want to do 1,000
requests per second, instantaneously. And then they get a bunch
of error messages. And they wonder why. Natural traffic really doesn’t
behave like this. Even if your game were to get
slashdotted, your traffic would not look like this. A graph like this is a
little bit better. Ramp it up over time. Much more ideal is probably
something like this. Now, the basic recipe for
doing load tests goes something like this. Send some steady-state traffic–
about, let’s say, start with five queries
per second. And then you uncover
any bottlenecks. If you have, say, a global
counter that all your requests are connecting to, you’ll
already see issues here. So you fix that up. Everything’s going
beautiful again. And then you wait about
15, maybe 20 minutes. And you look at your dashboard
at all your graphs. You make sure you have
zero quota denials. Your error rates
are really low. All your tasks are keeping up
and not forming a backlog, and no elevated datastore
contention. And if you have any of these
issues, you make adjustments in your app. And then you slowly ramp up to
about twice that traffic. So you go from 5 to 10,
to 20, to 40, to 80. And very quickly, you’ll be in
hundreds or thousands of QPS. Yet through this stepped
process, you can uncover bottlenecks at different
levels of resource requirements in your
application. And it’s a lot easier to
diagnose issues than flipping the switch, going to 1,000
requests per second, and then not being able to see what’s
going on because everything is blowing up in your face. So you uncover the next
bottleneck if you have any. If you’ve done all your homework
and done things correctly, you should be fine. And then you repeat and iterate
until you’ve reached the goal traffic rate. And then you relax and plan
your next feature and plan your next load test. Some other pro tips, especially
if you’re building mobile backends– don’t have all your devices call
home at the same time of day every night. Even though there’s different
time zones in the world, you don’t want 24 of these spikes
across your application. That won’t do very well. And also, make sure that your
retry times are a little bit randomized and that you use
exponential back-off. Otherwise, your app might
be serving right, goes down for a minute. And then you have every single
client out there. You have a million mobile
app users out there. They all wait exactly two
minutes and then call home. And of course, most
of them will fail. So they all sit back
for two minutes. And then they call home. And you get this repeated
Denial of Service attack on yourself. So don’t do that. I’m going to skip over
some of this. You can read this later. So what you saw here is
building JSON APIs, integrating with OAuth, using
matchmaking in backends, taking an Android app. Kind of all the basic building
blocks for backend services. Colt’s talk, which was the
complementary to this one, talked all about the frontend,
how you build atlases and sprite sheets, how you put
together all the JavaScript, how you create the protocol
messages. So really, you should
have pretty good information to get started. Now, I know we’re actually
out of time here. But I wanted to briefly show
you this other idea that didn’t quite come together. So this is almost an
identical app. And here we go. So the next phase– rather than have this controller
wired directly into a game, I thought I could
make a controller that wired into any game. For example, this game
that I heard of that was created in 2007. So we can go to– I built an extension
this morning. And this extension uses
WebSockets to connect to the phone. And let’s see if it
will connect back. Oh, it’s working. So you see all these
messages here. Those are actually coming
down from this device. And you see as I tilt it, we get
different Boolean values. And this one does actually
have a Shoot button. When I hit it, I get a little
asterisk there. Now, the game’s not working yet,
because I didn’t quite finish this. Let me turn off the extension
for a moment. The very last piece of wiring,
that I’m going to put together probably this weekend, is
actually injecting the keystrokes. And this is the code I
showed you before. So you see here– let’s see if I can zoom
in here for you– we have keyCode 32,
which is a space. And this game needs
a space to start. So if I inject this piece of
JavaScript into this page right now, the game starts. So imagine this game controller
wired up into any game with this Chrome
extension. So you could play any game on
the web that has WASD as the keys, or the arrow keys. And you just pick up your
Android device. And you tilt it. And it injects keystrokes
in the page. That is the demo you did
not see here today. Sorry about that. That’s it. Sorry about that. Thank you. [APPLAUSE]

3 Comments on "Google I/O 2012 – Gaming in the Cloud"

Leave a Reply

Your email address will not be published. Required fields are marked *