Rachels Lab Notes

Yes, I’ve been absent from this blog since forever. I can’t help it. There was lots of work to be done at work – we released a tiny little game last week. There was preparation for SIGGRAPH. No, I didn’t give a talk yet, but at least it was the first one where I got a shout out(pg.35). Hey, it’s a start! ;)

And then there’s the biggest issue – I tried to go for a long-form blog, and it’s not working for me. Sometimes, long is good. And sometimes, I just need to throw out a short link. Like today.

If you’ve ever done cross-platform multithreading, you hate it. There’s no unified API, not all operations are supported on all platforms… in short, it sucks. Thankfully, @bjoernknafla to the rescue. He has written amp, a cross-platform threading library, and made the amp source available on github under a BSD license. And, what makes me even happier, he stuck to a C-style API.

So my threading issues are solved. Both in my source code and in terms of resource starvation of this blog. Expect higher throughput!

One of the things that concerns me about the game industry “as-is” is the fact that we take too long to actually produce even playable prototypes. Everything moves at rather slow speeds – you can’t just quickly add a new model, for example.

At least for the prototyping side, there seems to be hope though. A couple of smart guys from the University of Cambridge came up with a way to generate textured models using a simple webcam.

In a nutshell, it allows you to go from here:

to here

without using any artist-time.

If you look closely (or watch the video in the original article) you’ll see that this won’t generate final content for us. The data is still very rough, and you will need an artist to either rebuild it or clean it up. But it will allow us to quickly generate placeholder art that in turn will allow us to iterate on gameplay with a look that is at least closer to the final game than the usual checkerboard boxes.

After my post on the hidden cost of C++, Vince suggested I go write a small game in C, see how I like that.

While that’s an interesting idea – maybe for another day – I’m interested in looking at larger-scale problems than that. And since I’m lately itching to write a renderer again (the last one was 1998), that’s where I started.

The goal of this project is to explore the “pressure points” in writing games in C++¹, to see if in some of these areas, the pain and delay can be alleviated. To make it as painful as possible, it’s of course going to be cross-platform. That means OS X and Windows, unless somebody wants to gift me a devkit for Xenon or PS3…

And even after that first bit of code, just hoisting out the most basic platform abstractions, a few lessons are to be had.

Banned Forever

I hope to keep the preprocessor mostly banned, except in some very limited areas where the libraries abstract actual hardware. It’s cause for a lot of confusion in real-world projects, so let’s try to keep it out. It is interesting to think if you could achieve the same effect without a preprocessor, just with language constructs – and what those would be.

Too Useful to be banned

Some C++ features are too useful to be banned. Writing a render engine in pure C, as Vince suggested, is possible, but tedious. So I guess I have to loosen my stance and allow some features to creep back in.

• namespaces

  Sure, you can prefix every single function, but it’s a messy business. It carries no performance cost to have them, so they’re in.

• limited classes

  Mostly, to get the convenient syntax of member function invocations. On non-virtual members, this carries a predictable cost. (I.e. I can predict what code the compiler will generate without having to look up the class API). Casts and copy constructors are still out, since they can implicitly generate code that I won’t be aware of when examining code.

  So, basically, C-style structs that have member functions and access protection are what’s allowed.

With both of these, I’m curious about their impact on compilation time.

Missing Features

Some features are missing from C++ that would be extremely useful:

• Anonymous functions.

  Yes, the new standard has them – but at a high readability price.

• Headerless compilation

  Header files are a completely pointless waste of time, a remnant from the late 70’s. As a side effect, that would allow mapping platform-specific enums to abstraced enums without having to expose the platform-specific header that contains them.

• Cleaner Syntax

  Braces/semicolons do add a lot of noise. Can I get a whitespace-scoped language, like Python?

Don’t care

There are many issues I don’t care about right now. Consequently, I’ll go “off the shelf” with them.

• math libraries.

  I really don’t want to write the 22nd implementation of a vector class, thank you. I’ve written enough of that. Since they’re all owned by my employers, I’m opting for an open-source one. So far, CML is the chosen one. It is, rather poignantly, making the point that this is a facility that’s sorely missing from C++ for game development.

• Window System Code

  While I really do want to examine the effects cross-platform code carries, some things are too gross to be touched by humans. I started out using GLUT, but its shortcomings meant native window handling. That’s not what really concerns me here, so I’ll be basing things on SDL for now – at least for purposes of handling the windowing system.

• Memory management

  I didn’t need to touch that yet, but if it comes to that, there’s dlmalloc, or Fluid Studios’ memory system (no link, since their atrocious website doesn’t have links – it’s all Flash. But you can find it by starting here), and probably many others. And unless I absolutely have to touch memory management, I don’t want to go there.

To get the formalities out of the way: I work at EA, but this is my private blog. The views and opinions expressed here are mine, and mine alone. (Unless it’s something really stupid, then my cat typed it.)

Also, nothing in here reflects knowledge internal to EA. Not only am I a lowly peon in a large machine who doesn’t get told anything of importance, I certainly wouldn’t share if I was told internal info either. Professional pride and all that.

With that out of the way, let’s look at Playfish first. Many critics claim it was done to appease analysts or, if they are analysts, that it dilutes EAs core business.

While those are valid concerns, they miss a major fact that’s looming for video games: AAA games are getting expensive enough that they just might outgrow the size of their market, core video gamers. And even if they don’t, it is a rather crowded market, and revenue only arrives in bursts every 2-4 years for each title.

What EA is doing here is branching out into a market with a far larger customer base, lower costs, and monthly revenue. And, more importantly, server-based games can’t be pirated. EA just cut out a large threat that is in the process of killing the PC market and even has significant impact on the console market. (After all, Microsoft just killed almost a million accounts on XBox live for hacking)

Furthermore – assuming Playfish indeed delivers on its revenue & profit valuations – EA has traded cash-on-hand against a profitable revenue stream. And while it’s nice to sit on $2.2B cash, a positive cash-flow is a better choice in the long term.

And then there are the layoffs. While I appreciate the sentiment of many online commenters that “EA shafted their employees to buy Playfish”(paraphrased), that’s almost certainly not true. As I said above, EA has plenty of cash to go buy something. There was no need to cut jobs for that acquisition, and I don’t think there’s any relation.

Riccitello simply realizes that the console games market as is has matured and is saturated. There’s only room for so many titles, so it makes sense to focus on those with the best chances to make money.

Yes, gamers are now going to scream about the “evil EA” that just produces sequels – but I believe the sales numbers of all the excellent new IP EA has produced last year clearly tell the story. Vocal gamers profess a love for new IP, but what’s mostly getting bought is the sequel to the established title.

Leaving out Nintendo consoles – because they thrive on mostly first-partly titles – let’s look at the top titles for the year so far:

• Halo 3
• Resident Evil 5
• Killzone 2
• Call of Duty: World at War
• FIFA Soccer 10
• Call of Duty: Modern Warfare 2

Notice a trend there? Every single one is a sequel. Gamers buy what they perceive as “established quality”. And I can’t really blame them – $60 is a lot of money to invest for entertainment.

To be clear: I have NO insight what titles actually have been cancelled. EA remains tight-lipped about that. For all I know, we might have cancelled only sequels and focused on new IP, although I’d consider that unlikely.

And for analysts like Pachter, who now complain that EA “didn’t spot” the underperforming titles last year: It is incredibly hard to judge what a game will be until you’re at least a year or so into execution. Making games is not formulaic. It takes a great idea, an awesome team, and a spark of magic to make a good game happen. You can control the first two, but the last one is out of your control. You’ll see it manifest at some point, but it takes some work before you know it’s there or not.

So, even though I don’t like it on a personal level, I think EA has at least the right ideas. You can certainly debate the merit of the concrete steps taken, but I believe AAA console games will shrink in importance, while online gaming increases – so this seems a decent decision. Lots of things could have done better; we certainly have internal talent capable of building social games. But we missed the boat, and that means there was pressure to buy market share out of the box.

What I do have an issue with is the handling of this, though. If you have to lay off people, let them know. Then people at least know where they stand. There is still no official word on what studios and titles will be affected by those 1,500 layoffs – which means everybody is worried to some extent. It is incredibly hard to do good work if you don’t know you’ll still have a job the next day – and that endangers that spark of magic I mentioned, the one ingredient to a successful game that’s incredibly hard to come by in the first place.

Insomniacs Mike Acton created quite a stir when he claimed yesterday that keeping your game at 60fps doesn’t matter. Unfortunately, his article bases this only on correlation between frame rate and final score. While it’s of great importance to the business end of making games, there’s little reasoning on the technical side why 60fps and 30fps make so little difference.

(Based on what else I read from Mike, I’m sure he’s reasoned about it alright – it’s just not part of this particular article.)

The concept of frame rate matters at all because the update rate of the entire world is tied to the refresh rate – the game progresses “one frame” at a time. That, in turn, is done since the final draw that a world update triggers needs to be tied to the screen refresh to avoid screen tearing)

But “the world” is not a monolithic block – games do many things, and the order in which you do them matters very much. Mick West has an excellent article on what he calls “response lag” – the time it takes for a button press to finally cause a result on screen. And effectively, you measure this lag time by stating how often you need to run “the world” (or the main game loop) until the input causes a visible effect.

That, in turn, means that with the same internal delay, a 30fps game takes twice as long to react to a button press as a 60fps game. Now, if your game is well engineered, that internal delay is “only” 3-4 frames. At 30fps, that’s just below the human reaction time – that’s why most gamers don’t complain about 30fps if it’s a well-engineered game engine.

Things start to get different once the gamer is not reacting, but planning ahead and trying to hit a button at an exact time. Music games, for example. Or FPS players who lead a shot. Here what matters is not so much the lag per se – we can learn to anticipate that. What matters is that that lag is constant, so there are no timing surprises.

So what matters is predictable lag that does not exceed the human reaction time – which is just so achievable with 30fps, and a monolithic step function.

And that’s the core assumption everybody was making for the “60fps or die” argument – your step function is monolithic. It was, for a long time, with a single core synced to the screen refresh. That argument simply isn’t true any more.

We have multiple cores available that can run completely decoupled from each other¹. That means, for example, that I can sample the controller at a much higher rate than I refresh the screen. We might run the physics engine at a time step significantly shorter than the duration of a single frame. In other words, we can run components of the world at different speeds and only have the drawing synced to the screen.

The tricky issue here is that we are sampling – aliasing effects can be introduced along the way. Which is really all that “lag” is – it’s the result of discrete sampling of a continuous world.

And that is the part that surprises me most – while we have a ton of really smart guys working on keeping our lag small, nobody seems to treat it as an aliasing problem, bringing signal theory to bear on it. Am I missing something?

One issue we always face is the fact that we don’t have enough memory. Ever. And even if we did, we like our data structures small and crunchy for performance reasons.

Sure, we pay attention to it all the time – and yet, occasionally, something slips through. So it’s nice that somebody wrote a tool – Cruncher# – to just load your PDB and examine all your structures for unnecessary padding.

And if you desire to look at ELF files (i.e. you’re under Linux, or working on a certain console), there’s pahole. Not only does it point out padding (or ‘holes in your structure’), it’s also friendly enough to tell you how many cache lines your structure will consume.

Let’s hope the two authors inspire each other!

I’d like to be able to regularly write a full article here, but unfortunately time is a scarce resource. (Just participating in the HN discussion on the previous artcle took up a large part of my free time.)

So, instead, here’s “bits and nibbles” – a collection of links that are worth reading.

• Packing Data into fp Render Targets – a quick recap if you need it

• Crytek Presentations – lots of technical details on what the Crysis guys are doing. It’s an older link, but it’s worth adding to your library if you don’t have those presentations yet.

• Chocolux – a GPU raytracer in WebGL. If that was gibberish to you: Run a realtime raytracer in your browser without any plug-ins.

• Twitter Dynamics for Game Developes – If you’re new to twitter (or haven’t even used it yet!), this is an excellent introduction to it, what tools you might want to use, and a short list of interesting people to follow. (Me? I’m @groby.)

• Zynga makes $500K a day – Zynga? You know, the guys who make Mafia Wars on Facebook? Not too shabby. My guess is that it’s about half the revenue Blizzard makes, with much better iteration times, and less capital outlay.

As a game developer, I’m concerned with performance. Yes, we’re living in next-gen land, and there’s a lot of performance – but then somebody comes along and squeezes every last drop out of a platform you develop for, and you better do that too.

As such, I’m occasionally involved in discussions about the merits of using C++. As such, one topic that comes up in every single discussion is the ‘hidden cost’ of using C++. And the reply is inevitably “There’s no such thing! It’s all in the source code!”

Well, yes, it is. But let me explain what I mean by it.

In C, if I have a line of code like this:

I can take a rough guess at the cost of the function by the name of it. The only other part I need to have a mental model of the performance is the overhead involved. And in C, the cost of a function call is pretty fixed. The only ‘surprise’ that can happen is that it’s inline and thus faster than you expected.

Not so in C++. Is it a function call, a member function call, or is it an anonymous constructor? Are b and c implicitly invoking copy constructors for other classes as part of type coercion? Is that a normal assignment, or an assignment operator? Is there a cast operator involved?

And once I have answered those questions, I have to look at all the classes involved. If they have a non-empty destructor, cost is added. Should those destructors be virtual, more cost is added. Virtual cast operators? Add some more.

As the end result, your overhead can grow dramatically. Especially those virtual calls are quite costly. The total runtime of a loop can easily vary by 10x or more based on those parameters.

Of course, they are not really hidden – if I look at the source code, I can easily see them. The real hidden cost is that now, instead of looking at one piece of source – the function itself – I need to look at up to four different classes. Add possible ancestors to find out if a call is virtual.

That is the hidden cost. The mental model for a simple function call became incredibly large and complex, and every function call is potentially as complex. Which makes reasoning about performance a rather hard thing to do.

Worse, it makes profiling harder than necessary. All the type coercions that happen at the API level will show up as separate functions, not attributed to the callee, but the caller.

All that translates ultimately into either worse performance or longer development time. Neither one is something you like to hear about.

… ’cause every good project starts with “Hello World”.

Welcome to my new blog!

What’s the point of this blog? It is, in a way, my lab notebook for everything concerning games development. That’s my profession and my craft, and I have a lot of things that I don’t want to get lost, so I’m keeping lab notes – like every other scientist.

You’ll hear a lot about scaling our game development process, cutting back development effort without impacting the quality, and the occasional other things that cross my mind. There’ll probably be some mumbling about graphics and AI, too, because I just like those two fields.

Anyway – welcome, and I hope you enjoy being here!